[ffmpeg] FindFFMPEG.cmake needs to handle dependencies on libs from the WindowsSDK #38401

petersteneteg · 2024-04-24T19:20:03Z

I also ran into this issue, after some digging I found that FindFFMPEG.cmake will call

find_library(FFMPEG_DEPENDENCY_${lib_name}_${config} NAMES "${lib_name}" REQUIRED
To find a set of required system libraries.
those system libraries comes from the various pc files

If you link dynamically we will find psapi in libavdevice.pc

Libs: "-L${libdir}" -lavdevice
Libs.private: -lpsapi -lole32 -lstrmiids -luuid -loleaut32 -lshlwapi -lgdi32 -lvfw32
note there that the dependency is "private" so it will not be added to the find_library above.
But if we link statically we have

Libs: "-L${libdir}" -lavdevice -lpsapi -lole32 -lstrmiids -luuid -loleaut32 -lshlwapi -lgdi32 -lvfw32
these are now public so they are added to the find_library call above

psapi is a windows system library that lives in for example, (version and so platforms my vary)
C:/Program Files (x86)/Windows Kits/10/Lib/10.0.22621.0/um/x64/psapi.lib

When you run cmake from a developer terminal this path get automatically picked up by cmake (somwhow, not sure exactly which env variable that it will read) and then every thing work.

But if we run the cmake-gui directly we have not such luck and we can't find the needed libs.

Ideally we should be able to give the find_library some hint to look into the "right" WindowsSDK directory. But I have not found a good way to get the "right" path, with out a developer terminal

I have created a solution that searched for the WindowsSDK using some poor heuristics.
And adds that as a PATH hint to the find_library calls mentioned above.

If any one know of a better way to do this? suggestions are welcome.

Changes comply with the maintainer guide.
SHA512s are updated for each updated download.
The "supports" clause reflects platforms that may be fixed by this new version.
Any fixed CI baseline entries are removed from that file.
Any patches that are no longer applied are deleted from the port's directory.
The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
Only one version is added to each modified port's versions file.

dg0yt · 2024-04-25T03:54:31Z

IMO this SDK setup doesn't belong into the domain of a particular port.

Neumann-A · 2024-04-25T05:18:33Z

probably better to add a #pragma comment(lib,"....") in an appropriate header instead

dg0yt · 2024-04-25T07:37:26Z

probably better to add a #pragma comment(lib,"....") in an appropriate header instead

No, this doesn't solve the problem of the library directory.

petersteneteg · 2024-04-26T06:56:34Z

We don't need the find_library call really, The linker will find the libraries it self, we can just pass in the lib files,. Something like this also works:

function(append_dependencies out)
    cmake_parse_arguments(PARSE_ARGV 1 "arg" "DEBUG" "NAMES" "")
    if(${arg_DEBUG})
        set(config DEBUG)
        set(path "${CURRENT_INSTALLED_DIR}/debug/lib/")
    else()
        set(config RELEASE)
        set(path "${CURRENT_INSTALLED_DIR}/lib/")
    endif()

    set(windowsSDK psapi uuid oleaut32 shlwapi gdi32 vfw32 secur32 ws2_32 mfuuid strmiids ole32 user32 bcrypt)
    foreach(lib_name ${arg_NAMES})
        if("${lib_name}" STREQUAL "-pthread")
            list(APPEND ${out} "-pthread")
        elseif("${lib_name}" STREQUAL "-pthreads")
            list(APPEND ${out} "-pthreads")
        elseif("${lib_name}" STREQUAL "gcc")
            list(APPEND ${out} "-lgcc")
        elseif("${lib_name}" STREQUAL "gcc_s")
            list(APPEND ${out} "-lgcc_s")
        elseif("${lib_name}" STREQUAL "stdc++")
            list(APPEND ${out} "-lstdc++")
        elseif("${lib_name}" STREQUAL "atomic")
            list(APPEND ${out} "-latomic")
        elseif(lib_name IN_LIST windowsSDK)
            list(APPEND ${out} "${lib_name}.lib")
        else()
            # first look in ${path} specifically to ensure we find the right release/debug variant
            find_library(FFMPEG_DEPENDENCY_${lib_name}_${config} NAMES "${lib_name}" PATHS "${path}" NO_DEFAULT_PATH)
            # if not found there, must be a system dependency, so look elsewhere
            find_library(FFMPEG_DEPENDENCY_${lib_name}_${config} NAMES "${lib_name}" PATHS #[["${winsdk_path}"]] REQUIRED)
            list(APPEND ${out} "${FFMPEG_DEPENDENCY_${lib_name}_${config}}")
        endif()
    endforeach()
    set("${out}" "${${out}}" PARENT_SCOPE)
endfunction()

My main issue here is that I need to hard code the list of lib from the WindowsSDK, and the capitalization is off.. althoguh that seems to work anyway...

Thoughts?

Neumann-A · 2024-04-26T07:07:40Z

        if("${lib_name}" STREQUAL "-pthread")
            list(APPEND ${out} "-pthread")
        elseif("${lib_name}" STREQUAL "-pthreads")
            list(APPEND ${out} "-pthreads")
        elseif("${lib_name}" STREQUAL "gcc")
            list(APPEND ${out} "-lgcc")
        elseif("${lib_name}" STREQUAL "gcc_s")
            list(APPEND ${out} "-lgcc_s")
        elseif("${lib_name}" STREQUAL "stdc++")
            list(APPEND ${out} "-lstdc++")
        elseif("${lib_name}" STREQUAL "atomic")
            list(APPEND ${out} "-latomic")
        elseif(lib_name IN_LIST windowsSDK)
            list(APPEND ${out} "${lib_name}.lib")
        else()

Looks like you are mixing MinGW and Windows here? The gcc stuff should be implicitly linked except for stdc++ and atomic. If it is not implicitly linked that is your problem

No, this doesn't solve the problem of the library directory.

These paths should be on a env variable which makes the linker lookup work. If it is not your build environment is not correctly configured.

@petersteneteg:

You probably want to configure env variable LIBS for your build environment.

petersteneteg · 2024-04-26T11:33:36Z

@Neumann-A We often run the cmake-gui without any starting it from a msvc developer console. This is what you get if you just start it from the start menu.

When you do that the Windows SDK env variables will not be set. And hence the find_library will fail.

I strongly support this usage, One should definitively not have to start cmake from a developer console.
There should not be a need to configure any build env variables.

The relevant part in the code above for the MSVC build is just. (From the installed FindFFMPEG.cmake,

vcpkg/ports/ffmpeg/FindFFMPEG.cmake.in

Lines 44 to 75 in 90a5b03

    
           function(append_dependencies out) 
        
               cmake_parse_arguments(PARSE_ARGV 1 "arg" "DEBUG" "NAMES" "") 
        
               if(${arg_DEBUG}) 
        
                   set(config DEBUG) 
        
                   set(path "${CURRENT_INSTALLED_DIR}/debug/lib/") 
        
               else() 
        
                   set(config RELEASE) 
        
                   set(path "${CURRENT_INSTALLED_DIR}/lib/") 
        
               endif() 
        
               foreach(lib_name ${arg_NAMES}) 
        
                   if("${lib_name}" STREQUAL "-pthread") 
        
                       list(APPEND ${out} "-pthread") 
        
                   elseif("${lib_name}" STREQUAL "-pthreads") 
        
                       list(APPEND ${out} "-pthreads") 
        
                   elseif("${lib_name}" STREQUAL "gcc") 
        
                       list(APPEND ${out} "-lgcc") 
        
                   elseif("${lib_name}" STREQUAL "gcc_s") 
        
                       list(APPEND ${out} "-lgcc_s") 
        
                   elseif("${lib_name}" STREQUAL "stdc++") 
        
                       list(APPEND ${out} "-lstdc++") 
        
                   elseif("${lib_name}" STREQUAL "atomic") 
        
                       list(APPEND ${out} "-latomic") 
        
                   else() 
        
                       # first look in ${path} specifically to ensure we find the right release/debug variant 
        
                       find_library(FFMPEG_DEPENDENCY_${lib_name}_${config} NAMES "${lib_name}" PATHS "${path}" NO_DEFAULT_PATH) 
        
                       # if not found there, must be a system dependency, so look elsewhere 
        
                       find_library(FFMPEG_DEPENDENCY_${lib_name}_${config} NAMES "${lib_name}" REQUIRED) 
        
                       list(APPEND ${out} "${FFMPEG_DEPENDENCY_${lib_name}_${config}}") 
        
                   endif() 
        
               endforeach() 
        
               set("${out}" "${${out}}" PARENT_SCOPE) 
        
           endfunction()

)

    set(windowsSDK psapi uuid oleaut32 shlwapi gdi32 vfw32 secur32 ws2_32 mfuuid strmiids ole32 user32 bcrypt)
    foreach(lib_name ${arg_NAMES})
        if(lib_name IN_LIST windowsSDK)
            list(APPEND ${out} "${lib_name}.lib")
        else()
            # first look in ${path} specifically to ensure we find the right release/debug variant
            find_library(FFMPEG_DEPENDENCY_${lib_name}_${config} NAMES "${lib_name}" PATHS "${path}" NO_DEFAULT_PATH)
            # if not found there, must be a system dependency, so look elsewhere
            find_library(FFMPEG_DEPENDENCY_${lib_name}_${config} NAMES "${lib_name}" PATHS #[["${winsdk_path}"]] REQUIRED)
            list(APPEND ${out} "${FFMPEG_DEPENDENCY_${lib_name}_${config}}")
        endif()
    endforeach()

The function append_dependencies is used for all platforms... but the exceptions put in there are mostly for non windows I guess

Basically what needs to be done is to bypass the find_library call for the WindowsSDK libs. We don't need to provide absolute paths to them since the MSVC linker has the WindowsSDK path implicitly from my understanding.

Neumann-A · 2024-04-26T11:34:52Z

Which generator do you use?

petersteneteg · 2024-04-26T11:35:33Z

The MSVC one ("Visual Studio 17 2022")

petersteneteg · 2024-04-26T11:40:31Z

The generator should know about the used Windows SDK
From the cmake output:

Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22631.
The CXX compiler identification is MSVC 19.39.33520.0

Neumann-A · 2024-04-26T11:52:58Z

So you are generating VS Solution from that. VS itself knows how to add the correct search paths. In that case #pragma comment(lib,"....") in the correct headers should be enough and generally solve the problem.

petersteneteg · 2024-04-26T12:06:43Z

Added the pragmas would work for getting the linker to link the lib. Although I would rather not have to figure out which of ffmpegs headers to add that to. Also seems quite invasive, to modify the headers to solve this link issue.

And the issue with the find_library is still there. since the current solution in vcpkg it so gather those up from the pc files. so we still need to address that part to avoid having cmake file its configure set.

dg0yt · 2024-04-26T12:29:00Z

If you want find_library to work then add the actual SDK path to the CMAKE_(SYSMEM_)LIBRARY_PATH when configuring your project? Inform your build about your installation. It is not the port's task.
But I think the SDK libs should not be used via find_library.

petersteneteg · 2024-04-26T13:09:22Z

I rather avoid the call to find_library. My main question is who to know which libs that I should have by bass the call
Should I hardcode the list as now. or do something more clever?

dg0yt · 2024-05-24T05:28:39Z

I can't answer the last question, but when it comes to the standard windows libs (-lkernel32), I can share yesterday's findings for vcpkg-ci-curl:x64-windows-static

icu adds windows libs to the pc file.
curl picks them via FindPkgConfig.cmake when it uses libpsl.
pkg_check_modules turns them into absolute paths for TARGET_LINK_LIBRARIES. (Ugly.)
NB: It doesn't need any extra setup to locate the libs in the Windows SDK.
vcpkg_cmake_config_fixup fails to fixup INTERFACE_LINK_LIBRARIES when the value constains (x86).
And then downstream usage of find_package(CURL) is broken for the debug config.

Neumann-A · 2024-05-24T07:13:34Z

icu adds windows libs to the pc file.

If it adds windows libs to the pc file it should do it verbatim, e.g. kernel32.lib, instead of with -l flags, e.g. -lkernel32.

petersteneteg · 2024-05-24T07:37:01Z

FFMpeg does add them verbatim to the pc file: for example libavdecice.pc

prefix=/c/Users/petst55.AD/Projects/vcpkg/packages/ffmpeg_x64-windows-static-md
exec_prefix=${prefix}
libdir=/c/Users/petst55.AD/Projects/vcpkg/packages/ffmpeg_x64-windows-static-md/lib
includedir=/c/Users/petst55.AD/Projects/vcpkg/packages/ffmpeg_x64-windows-static-md/include

Name: libavdevice
Description: FFmpeg device handling library
Version: 60.3.100
Requires: libavfilter >= 9.12.100, libswscale >= 7.5.100, libavformat >= 60.16.100, libavcodec >= 60.31.102, libswresample >= 4.12.100, libavutil >= 58.29.100
Requires.private: 
Conflicts:
Libs: -L${libdir}  -lavdevice psapi.lib ole32.lib strmiids.lib uuid.lib oleaut32.lib shlwapi.lib gdi32.lib vfw32.lib
Libs.private: 
Cflags: -I${includedir}

And x_vcpkg_pkgconfig_get_modules will pull in all of those, which later then gets added the to the FIndFFMPEG.cmake.
Although there are some filtering of libs in the portfile:

x_vcpkg_pkgconfig_get_modules(PREFIX FFMPEG_PKGCONFIG MODULES ${FFMPEG_PKGCONFIG_MODULES} LIBS)

function(append_dependencies_from_libs out)
    cmake_parse_arguments(PARSE_ARGV 1 "arg" "" "LIBS" "")
    string(REGEX REPLACE "[ ]+" ";" contents "${arg_LIBS}")
    list(FILTER contents EXCLUDE REGEX "^-framework$")
    list(FILTER contents EXCLUDE REGEX "^-L.+")
    list(FILTER contents EXCLUDE REGEX "^-libpath:.+")
    list(TRANSFORM contents REPLACE "^-Wl,-framework," "-l")
    list(FILTER contents EXCLUDE REGEX "^-Wl,.+")
    list(TRANSFORM contents REPLACE "^-l" "")
    list(FILTER contents EXCLUDE REGEX "^avutil$")
    list(FILTER contents EXCLUDE REGEX "^avcodec$")
    list(FILTER contents EXCLUDE REGEX "^avdevice$")
    list(FILTER contents EXCLUDE REGEX "^avfilter$")
    list(FILTER contents EXCLUDE REGEX "^avformat$")
    list(FILTER contents EXCLUDE REGEX "^postproc$")
    list(FILTER contents EXCLUDE REGEX "^swresample$")
    list(FILTER contents EXCLUDE REGEX "^swscale$")
    if(VCPKG_TARGET_IS_WINDOWS)
        list(TRANSFORM contents TOLOWER)
    endif()
    if(contents)
        list(APPEND "${out}" "${contents}")
        set("${out}" "${${out}}" PARENT_SCOPE)
    endif()
endfunction()

append_dependencies_from_libs(FFMPEG_DEPENDENCIES_RELEASE LIBS "${FFMPEG_PKGCONFIG_LIBS_RELEASE}")
append_dependencies_from_libs(FFMPEG_DEPENDENCIES_DEBUG   LIBS "${FFMPEG_PKGCONFIG_LIBS_DEBUG}")

There is not any easy way to pick out windows libs here...

dg0yt · 2024-05-24T08:00:33Z

If it adds windows libs to the pc file it should do it verbatim, e.g. kernel32.lib, instead of with -l flags, e.g. -lkernel32.

Why? kernel32.lib wouldn't work for mingw.

Neumann-A · 2024-05-24T08:55:36Z

Why? kernel32.lib wouldn't work for mingw.

Because:
set(CMAKE_C_STANDARD_LIBRARIES_INIT "kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib")

After looking at the pc files is there a specific reason those libs are even listed in the pc file? They should probably be removed if they are intended to be linked by default on windows.

petersteneteg · 2024-05-24T09:12:30Z

seems CMAKE_C_STANDARD_LIBRARIES_INIT covers some of the libs from the WindowsSDK but not all

dg0yt · 2024-05-24T11:06:35Z

Why? kernel32.lib wouldn't work for mingw.

Because: set(CMAKE_C_STANDARD_LIBRARIES_INIT "kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib")

After looking at the pc files is there a specific reason those libs are even listed in the pc file? They should probably be removed if they are intended to be linked by default on windows.

pkg-config is independent of CMake (CMAKE_C_STANDARD_LIBRARIES) and meson (winlibs), so it cannot make many assumptions about what is "linked by default".

pkg-config has a --msvc-syntax option to transform -lkernel32 into the MSVC world.

What I want is a different behavior of pkg_check_modules. Actually many differences.

BillyONeal

This should just add psapi.lib. It's the toolset provider's responsibility to provide a working Windows SDK.

The written change assuming that the Windows SDK is in Program Files is not OK and will fail for many customers who do that deployment themselves to a different location.

petersteneteg · 2024-06-19T09:14:32Z

I've pushed a new version that just bypasses the find_library for the Win SDK libs, avoiding any need of find the winsdk path etc. Similar to how pthread etc is bypassed.

dg0yt · 2024-06-19T20:25:57Z

ports/ffmpeg/FindFFMPEG.cmake.in

+        elseif(lib_name IN_LIST windowsSDKLibs)
+            list(APPEND ${out} "${lib_name}.lib")


This breaks mingw.

(Why does this function transform these libs at all? IIUC the output goes to CMake, so literal atomic, ws2_32 etc. should be fine. And CMake will transform it when generating link commands for a particular generator. This function just needs a single list of values which shall be passed literally instead of going through find_library. And I probably removed the sources of gcc, gcc_s, stdc++ in #39077)

I've updated the PR to just pass the libs through without any modifications

…he WindowsSDK

dg0yt · 2024-06-22T11:03:49Z

ports/ffmpeg/FindFFMPEG.cmake.in

@@ -50,6 +50,7 @@ function(append_dependencies out)
        set(config RELEASE)
        set(path "${CURRENT_INSTALLED_DIR}/lib/")
    endif()
+    set(windowsSDKLibs psapi uuid oleaut32 shlwapi gdi32 vfw32 secur32 ws2_32 mfuuid strmiids ole32 user32 bcrypt)


Suggested change

set(windowsSDKLibs psapi uuid oleaut32 shlwapi gdi32 vfw32 secur32 ws2_32 mfuuid strmiids ole32 user32 bcrypt)

set(windowsSDKLibs psapi uuid oleaut32 shlwapi gdi32 vfw32 secur32 ws2_32 mfuuid strmiids ole32 user32 bcrypt)

cmake_policy(SET CMP0057 NEW)

for IN_LIST.
(Push/pop/save/restore as you like, but IN_LIST should rarely be a problem.)

dg0yt · 2024-07-06T16:48:55Z

Lib list adopted for #39703.

JonLiu1993 assigned MonicaLiu0311 Apr 25, 2024

MonicaLiu0311 added the category:port-bug The issue is with a library, which is something the port should already support label Apr 25, 2024

BillyONeal reviewed Jun 18, 2024

View reviewed changes

petersteneteg force-pushed the ffmpeg-fix branch from 31ffbd1 to 5d6f8d3 Compare June 19, 2024 09:09

dg0yt reviewed Jun 19, 2024

View reviewed changes

petersteneteg added 2 commits June 20, 2024 13:13

[ffmpeg] FindFFMPEG.cmake needs to handle dependencies on libs from t…

b3d0927

…he WindowsSDK

v db

eaa55e3

petersteneteg force-pushed the ffmpeg-fix branch from 5d6f8d3 to eaa55e3 Compare June 20, 2024 11:14

dg0yt reviewed Jun 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ffmpeg] FindFFMPEG.cmake needs to handle dependencies on libs from the WindowsSDK #38401

[ffmpeg] FindFFMPEG.cmake needs to handle dependencies on libs from the WindowsSDK #38401

petersteneteg commented Apr 24, 2024

dg0yt commented Apr 25, 2024

Neumann-A commented Apr 25, 2024

dg0yt commented Apr 25, 2024

petersteneteg commented Apr 26, 2024

Neumann-A commented Apr 26, 2024

petersteneteg commented Apr 26, 2024

Neumann-A commented Apr 26, 2024

petersteneteg commented Apr 26, 2024 •

edited

Loading

petersteneteg commented Apr 26, 2024

Neumann-A commented Apr 26, 2024

petersteneteg commented Apr 26, 2024

dg0yt commented Apr 26, 2024 •

edited

Loading

petersteneteg commented Apr 26, 2024

dg0yt commented May 24, 2024 •

edited

Loading

Neumann-A commented May 24, 2024

petersteneteg commented May 24, 2024

dg0yt commented May 24, 2024

Neumann-A commented May 24, 2024

petersteneteg commented May 24, 2024

dg0yt commented May 24, 2024 •

edited

Loading

BillyONeal left a comment

petersteneteg commented Jun 19, 2024

dg0yt Jun 19, 2024

petersteneteg Jun 20, 2024

dg0yt Jun 22, 2024

dg0yt commented Jul 6, 2024

		elseif(lib_name IN_LIST windowsSDKLibs)
		list(APPEND ${out} "${lib_name}.lib")

	set(windowsSDKLibs psapi uuid oleaut32 shlwapi gdi32 vfw32 secur32 ws2_32 mfuuid strmiids ole32 user32 bcrypt)
	set(windowsSDKLibs psapi uuid oleaut32 shlwapi gdi32 vfw32 secur32 ws2_32 mfuuid strmiids ole32 user32 bcrypt)
	cmake_policy(SET CMP0057 NEW)

[ffmpeg] FindFFMPEG.cmake needs to handle dependencies on libs from the WindowsSDK #38401

Are you sure you want to change the base?

[ffmpeg] FindFFMPEG.cmake needs to handle dependencies on libs from the WindowsSDK #38401

Conversation

petersteneteg commented Apr 24, 2024

dg0yt commented Apr 25, 2024

Neumann-A commented Apr 25, 2024

dg0yt commented Apr 25, 2024

petersteneteg commented Apr 26, 2024

Neumann-A commented Apr 26, 2024

petersteneteg commented Apr 26, 2024

Neumann-A commented Apr 26, 2024

petersteneteg commented Apr 26, 2024 • edited Loading

petersteneteg commented Apr 26, 2024

Neumann-A commented Apr 26, 2024

petersteneteg commented Apr 26, 2024

dg0yt commented Apr 26, 2024 • edited Loading

petersteneteg commented Apr 26, 2024

dg0yt commented May 24, 2024 • edited Loading

Neumann-A commented May 24, 2024

petersteneteg commented May 24, 2024

dg0yt commented May 24, 2024

Neumann-A commented May 24, 2024

petersteneteg commented May 24, 2024

dg0yt commented May 24, 2024 • edited Loading

BillyONeal left a comment

Choose a reason for hiding this comment

petersteneteg commented Jun 19, 2024

dg0yt Jun 19, 2024

Choose a reason for hiding this comment

petersteneteg Jun 20, 2024

Choose a reason for hiding this comment

dg0yt Jun 22, 2024

Choose a reason for hiding this comment

dg0yt commented Jul 6, 2024

petersteneteg commented Apr 26, 2024 •

edited

Loading

dg0yt commented Apr 26, 2024 •

edited

Loading

dg0yt commented May 24, 2024 •

edited

Loading

dg0yt commented May 24, 2024 •

edited

Loading