Modernizing Apothecary #7562
Replies: 22 comments 40 replies
-
|
But I have a question: Why is both TravisCI and Github Actions used? Which is used for what? Based on the files I am seeing, Github Actions is used to build and upload the results, as well as show the badges. Then what is Travis used for? |
Beta Was this translation helpful? Give feedback.
-
|
Travis CI was our old CI before moving to GH Actions. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @HerrNamenlos123 Your plan sounds good. I'd maybe even suggest we create a new repo This wouldn't be a clone of Might make issues/PRs etc better organized? If that makes sense happy for some name suggestions :) |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that is a very good idea. I would suggest My goal was to create many, many different versions of every package, maybe even one for every single gcc version from 4 to 11, and that for every architecture and every version of the package (only one in the beginning, but more if it is updated later, so all of them stay available). I am not yet sure in which way to utilise Docker and Github Actions with its Matrix feature, since there will be so many devops runs. (I am not yet sure if we want one run per package-version-architecture-platform, which would result in hundreds of CI runs, or one run per platform and architecture, that then installs 10 different versions of the same compiler...) Or are we fine with hundreds of parallel CI runs since they are not going to run often and every package is perfectly isolated? |
Beta Was this translation helpful? Give feedback.
-
|
I think the only thing to be mindful of would be avoiding tons of CI runs that aren't needed. I'll create the packages repo and add you. |
Beta Was this translation helpful? Give feedback.
-
|
I already have a perfectly working Docker container that clones and builds GLFW, and archives the library files and exports it. I am working on the matrix that builds the same variation for different compilers. But where should the packages be uploaded to? Github releases in the packages repo or the openFrameworks CI website? Because currently both are kinda used... |
Beta Was this translation helpful? Give feedback.
-
|
I think hosting in the Github Releases of the Packages repo makes the most sense. We are now doing our nightly OF builds by pulling libs from the nightly release of Apothecary - so using the Packages Github Releases file storage would be great. |
Beta Was this translation helpful? Give feedback.
-
|
Is there already a limit on what the supported compilers are? Regarding platforms I will try to get the following combinations to work:
32-bit support is dropped. All of these supported architecture combinations are supposed to be transferred into a visual table, that is then also valid for openFrameworks itself, so this is strictly what OF will support in the future. Did I miss something it should support? Or can more of these be dropped since they're out of date and nobody uses them? (I think especially MSYS, and tvOS are pain points) |
Beta Was this translation helpful? Give feedback.
-
|
I think gcc-7 onwards makes sense. Some notes:
|
Beta Was this translation helpful? Give feedback.
-
|
I will try to reach the list, i will see if there are problems anyways and then I can decide if it's worth it. A note: |
Beta Was this translation helpful? Give feedback.
-
|
Adding a link to the packages repo for anyone interested: |
Beta Was this translation helpful? Give feedback.
-
|
Oh wow this is exciting! @HerrNamenlos123 nice one!! Yes absolutely we need to move the scripts to CMake! As for the CMake Scripts: We are already one step ahead to assist this for other targets and platforms as already re-wrote a lot of apothecary dependencies compile targets to use CMake in the Android PR here: openframeworks/apothecary#231 For Windows the complexities are which target compiler version
Might be good idea both and let the user decide and for future just go 2022 For iOS and macOS, these libraries will need to be compiled into the new required format
Toolchain can be found here for mac/iOS/tvOS etc Apothecary updates have legit been a nightmare, if we can sort out a clear and concise system (via CMake) I'm all for it |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the info, I will take a look at it when i have time to. I am currently working on the build scripts of the packages, trying to get it to work on all systems without major code duplication. Github Actions is good in itself, but the downside is it is weird to write steps and it won't run anywhere else. I for now decided to build the package in a docker container, and start the container using docker-compose. This works amazingly well and with a single command you get an archive file, cross-compiled for any system and always repeatable, regardless of the host. The upside is that you can run this 1:1 on your host or in a github actions container, so you can use the exact same container to build it locally. Also, docker is a lot more comfortable to write. The downside is that Windows containers are not a thing, and the non-standard ones that are, can only be run on a Windows host with Docker Desktop. So we cannot run a Windows Docker container on a Linux host like Github Actions. I however found a docker container that manages to run MSBuild VC++ under Wine32, running Windows programs under Linux. It is a bit of a hack, but it should work since docker containers are so repeatable. The other option would be to do it directly in Github Actions without docker. Github Actions has proper Windows runners with VS installed. The downside there is that the scripts are really awkward to write and you cannot build it anywhere else, it is strictly bound to GH Actions and there is no way to run or even test it locally. I think the best bet is to use MSBuild in a Linux docker container under Wine for now, and (since Microsoft is driving GH Actions so much), hope that either MSBuild soon gets native Linux support, or that Windows containers can be run on a Linux host. Oh and a third option i must test is if you can possibly run a Windows docker container in a windows-latest GH Actions runner. The question is if docker desktop is installed in windows-latest and if yes, if you can switch to Windows Container mode from the command line. If that works, I think this is the way to go now, then you must use Windows for testing locally and switch between Windows and Linux containers depending on what you want to test. Thank you for reading this far :) |
Beta Was this translation helpful? Give feedback.
-
|
To clarify, I am already able to build GLFW for example (something with good CMake support) in a docker container and it works for any compiler that can be installed (gcc, clang, mingw, and so on). And due to it being docker it was just a few lines and it worked in GH Actions too. But then the windows part struck me because I thought there is a docker windows runner just like GH has, but there really isn't. |
Beta Was this translation helpful? Give feedback.
-
|
Another idea: What do you think of providing CMake scripts optionally (the same ones that are used inside the docker container) which allow you to build all your dependencies and your application in one go? This could be implemented by packaging all cmake scripts in a package too, and OF then pulls this package and executes the scripts. There would be a lot of system packages to install (conveniently noted by the CMake errors anyways), but this would be the only way to really create bullet-proof applications. What do you think? |
Beta Was this translation helpful? Give feedback.
-
|
Alright, so there is use for it. While this shouldn't be too hard for modern CMake dependencies, we must CMakeify the ancient dependencies. Depending on the library it can be very easy to pretty difficult. But I think it is the way to go and makes everything a lot easier later on. (I mean writing a CMakeLists and calling it in docker, rather than calling make directly in docker, which would be a step back) We can for sure also do it in a way that there is an error for missing system dependencies, and if a special flag is used all needed dependencies are automatically installed using apt. This is very easy but would need a lot of testing on various distributions since the package names may differ. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah I think all the dependencies should now include CMakeList which will make this easier, and if they don't it is included in the apothecary/forumula/directory currently (if not at source) The only ones that remaining todo were cURL, OpenSSL, GLFW (now done above) Yeah a system of install apothecary build dependencies currently exists in the scripts/ folder (for installing via github actions). This contains the apt-get dependencies and build tools currently used for reference |
Beta Was this translation helpful? Give feedback.
-
|
I am currently using GLFW as an example of the first package and now it finally builds with gcc, clang and msvc, but I can't really get it to work for arm. I am cross-compiling it in an ubuntu container using an arm gcc cross compiler. The issue is that it needs X11 and it cannot use the installed libx11-dev because it is for the wrong architecture. And I can't find a way of getting the arm-x11 version into the x86_64 container. You CAN use Wayland instead or no backend at all, but I don't know if it is of any use then for us. How was this done in the past? |
Beta Was this translation helpful? Give feedback.
-
|
And in which ways were FreeImage and the other libraries patched? Is that still relevant? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah first to support C++11 then later C++17 and then to patch the incredible CVE's included in dependancy dependancy tree's, that some allowed remote execution have fun. Don't worry it's not like Unreal or Unity use them or anything ... hahahaha... I reported to them and they said they have patched it internally... well binaries suggest no they didn't... Anyways I have an updated 3.19 version going with further Nvidia CVE patches as well applied. |
Beta Was this translation helpful? Give feedback.
-
|
I am currently struggling to get MSVC binaries to work (Clang-CL finally works using CMake so I can build CMake libraries into an MSVC binary in a Linux container, the issue is that few of the dependencies we need even have CMake support, and I am currently stuck with OpenSSL. At this point I am re-thinking my re-thinking, thinking if we should even pre-build these libraries in the first place. I already tried to get rid of OpenSSL completely, but libcurl seems to need it. However, libcurl can be compiled statically such that it itself includes OpenSSL. Problem is it also has an arcaic build system. I mean we do need the binaries from somewhere, and one idea that came to my mind: Other package managers already provide all of these binaries, so it would only be smart to consider to re-use them. What would you think of the idea (provided that it is legally allowed) of more or less stealing the binaries from something like vcpkg or Conan? These could be used as a package manager to download the precompiled libraries like if we wanted to use them, but instead of using them we simply copy the files and upload to our repository. This would still keep OF independent of vcpkg or Conan from the user point of view. What do you think of the idea of doing so, and would you do it for all packages or just the MSVC versions, while all non-MSVC versions are compiled normally in our pipeline? |
Beta Was this translation helpful? Give feedback.
-
|
Omg yeah avoid OpenSSL and cURL it's a nightmare fuel. I think we should put ofURLFileLoader as a define/in or out as well for that reason, as well as making lightweight builds if not using said functionality in app + security reasons. If possible can you setup the basic template script for all our current targeted dependancies, I'll throw some time in cutting what we have in that tip apothecary PR, and PR to your packages all the current Apothecary working Cmake commands / scripts to the repo. For example I can setup FreeImage as that is fully working and updated recently in Apothecary PR |
Beta Was this translation helpful? Give feedback.
-
|
Been looking at trying to get iOS simulator and iOS Device builds as separate libs ( see: openframeworks/apothecary#208 ) working for Freetype and what should just be a few small tweaks has turned into a massive headache. I am starting to feel apothecary really needs to be radically changed. Anyway, I am all up for finding better solutions including a cmake only approach :) |
Beta Was this translation helpful? Give feedback.
-
|
Indeed, it needs to radically change. I did not expect to have so many road bumps while writing the new packages system. What I thought would be a week of writing docker images turned into weeks and weeks of just trying around different stuff and nothing works well enough. I wanted to use Docker in the beginning, because it is a lot more comfortable to write than plain GH Actions and would mean you can simply run the container and build said binary, on any system whatsoever, be it your grandmas PC or a GH Actions container. What started making issues was Windows because we need MSVC binaries and Docker only runs Linux containers (when excluding hacks), so you would have to cross-compile or do weird Linux hacks like running Windows programs under Linux using Wine32. It was a lot of pulling my hair out. I at some point even got cross-compiling for MSVC to work using clang-cl, that's when someone mentioned MSVC is still the easy part, MacOS is gonna be the hard one. That is when i realised something: For this reason I scrapped Docker because it only runs Linux containers and expecting to cross-compile for any operating system on Linux hosts is a fever dream and nothing guarantees it should even be possible at all. Thus, everything will be buiit on the native Windows and MacOS runners GH, as much as possible. Minimal cross compilation. |
Beta Was this translation helpful? Give feedback.
-
|
And we really should try and get rid of more dependencies. I did a few Reddit posts and 95% advised to get rid of apothecary completely and simply use a package manager like Conan or vcpkg. I think the OF users could handle that. The biggest push back i see is that many of our dependencies are not even available on Conan because they're so arcaic (Another reason to get rid of them). I couldn't get OpenSSL to build easily because it has an even stranger build system. We do not need it directly, just LibCURL needs it for TLS support. After a lot of trying around I resorted to not switch to cpp-httplib and instead keep LibCURL in combination with MbedTLS for TLS support. Both are now easily compilable using CMake, and they result in a slim binary that is statically linked. I chose it because MbedTLS is an officially supported backend for LibCURL, meaning you must only configure its CMake correctly and the code in OF does not even change, only the backend does. So: OpenSSL is gone. We can live completely without it and still have bundled TLS support, so we also don't need to ship OpenSSL DLLs. I am very happy with this combination. |
Beta Was this translation helpful? Give feedback.
-
|
Oh and one more thing, according to Reddit we should really look into Conan. Conan 2.0 is finally out with tremendous improvements and apparently there are now installers for every OS (you had to install with pip before) and apparently there are even standalone versions where you can call a portable Conan executable without installing anything. If this is true we might be able to embed a Conan executable directly in the repository or download it with CMake and call it. This keeps it hidden from the user but uses an existing packages server, making apothecary completely obsolete. Except for our arcaic dependencies, we could keep apothecary for these until we get rid of them or CMakeify them and propose to get them into Conan. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @HerrNamenlos123 - thanks for the update! Checking out Conan now, from what I can tell it seems like a more formal / developed version of apothecary with good cmake support. I'm definitely down if to give it a whirl. Thanks for your work on this. It is much appreciated. |
Beta Was this translation helpful? Give feedback.
-
|
I am now looking more into Conan. In my tests Python turned out to be a breeze for building the packages, I used it to build MbedTLS and LibCURL with CMake and to install them and to bundle the installed files into an archive. Now, I have used Conan in the past and I once Conanifyed a package that was not provided back then, writing the Conan Python build script which calls CMake in their pipeline. Looking at the build script I wrote in the past weeks, it is scaringly similar to the Conan script back then. That again makes me think again, Conan is significant and at the current stage I think I am doing nothing but reinventing Conan. I have just tried Conan 2.0. They provide exe's and zip files on Github, the exe's are graphical installers, while the zip files are the entire installation directory just zipped, it contains the conan executable, some libraries and an embedded Python installation. This means you can download the zip file, extract it and then call the Conan executable that is found within. So, I will now try to do that from CMake: Download the Conan zip automatically and use it to fetch as many packages as possible. Anything else would at this stage simply reinvent Conan.
|
Beta Was this translation helpful? Give feedback.
-
|
So first, apple now requires a completely new format for libs which will
fix the issues caused by fat lib conflicts.
Re apothecary: Yeah it’s more there are just so many vast charges in time
since the last overhaul with dependencies changing so much over the years.
The android PR makes many of these changes with Cmake first move.
The issues arise with then complexities due to then, latest versions of
said dependencies required to make all platforms then work at the same
time, at the latest dependency versions that are one CMake compatible. This
includes VS
I’ve included the cmake tool chain build file for iOS and macOS in that
branch which is the key for the cross compiler in the same way used for
android (that is packaged with android tools).
Most dependency versions have been patched however to support cmake now
I do think it’s better to cut and dry into a new format
I think it’s best to base any iOS cmake changes to the be based on that
android pr as most of the cmake compatible work is done
…On Wed, 19 Jul 2023 at 7:18 am, Theodore Watson ***@***.***> wrote:
Been looking at trying to get iOS simulator and iOS Device builds as
separate libs ( see: openframeworks/apothecary#208
<openframeworks/apothecary#208> ) working for
Freetype and what should just be a few small tweaks has turned into a
massive headache.
I am starting to feel apothecary really needs to be radically changed.
It is such a mess and doesn't feel sustainable.
Anyway, I am all up for finding better solutions including a cmake only
approach :)
—
Reply to this email directly, view it on GitHub
<#7562 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAGK2HEHD6HXHIQR5B6ZGCLXQ34SXANCNFSM6AAAAAAZPCZQWU>
.
You are receiving this because you commented.Message ID:
<openframeworks/openFrameworks/repo-discussions/7562/comments/6482617@
github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
yeah - agree! @HerrNamenlos123 - this is the PR @danoli3 is working on. I think cmake is used for most of the libs now, just need to fixup the other platforms ( fun ) now that Android is working :) |
Beta Was this translation helpful? Give feedback.
-
|
Currently I am looking into Conan and it truly is amazing, it has changed significantly since i last used it. They are going more and more into the direction that the project is completely unaware it is using Conan at all. That means, there are no more Conan functions used in CMake, there are only Previously Conan generated e.g. Where I am going with this: There have been significant upgrades to many systems, which is amazing, but it also means that OF's approaches need to change radically because they are currently upside-down in many aspects. We still have the project generator however. This means it could be advertised as the main way of using OF, while it would generate example source files, a CMake file, a Conan file, then invoke Conan, then CMake, and then open the IDE. This workflow would ideally be the exact same a professional developer would use, just automated. But now to my points, while ignoring the need for reversing the CMake/Conan/Dependency workflow, I am asking myself if it is even advisable to keep the project generator in the same repository as the OF library itself. What do I mean with that: In an ideal project setup you would choose a project location, have Now, when the project generator generates a project somewhere the user chooses, and then sets the project up so that it clones its own instance of OF, and does not simply copy source files (to keep it easy to update later), it does not make sense that the project generator is even residing between the OF source files. Also, as soon as a project is generated the project itself does not need project generator, that is unnecessary overhead. Also, as the project generator will later only invoke CMake and not do anything specific, it actually has no relation to OF at all. It will simply download stuff and run tools, thus it should be considered as just an external tool for OF, not an essential part of OF. From my current perspective it might be the wisest decision to put project generator into its own repository, and keeping the OF repository clean with JUST its own source files and nothing else. No Conan, no apothecary, no project generator, just OF. When downloading, the user will have two decisions, they can either download the git repository (or a source archive) and do whatever it takes to build it. Or they can download the project generator (which is from another repository but nicely provided), which then downloads the git repo or source files FOR the user, and also runs Conan and CMake: The exact same procedure someone would use to setup a project using the plain git repository. It is just automated. What are your thoughts, do you also see reasons to extract the project generator into its own repository? And what do you think of the Conan 2.0 approach to intercept |
Beta Was this translation helpful? Give feedback.
-
|
By the way, I just checked and Conan provides all dependencies we currently need in OF, except for |
Beta Was this translation helpful? Give feedback.
-
|
In case it's helpful for context, we've always designed OF to be self contained -- that you are encouraged to use relative paths in your projects (you don't have to, but it defaults to /apps) and that you can move the OF folder from computer to computer and your projects still work. This has been super helpful for making sure things break less, and while certain older version of OF become incompatible with modern IDEs and OSs, it's honestly been helpful to have projects live sort of relatively to the version of OF they are using. I will often have multiple versions of OF on my hard drive. That's not to say it has to work like that in the future, but it's been one of the designs of OF -- to avoid having to install system level libraries, to avoid separating OF projects from the library, etc. |
Beta Was this translation helpful? Give feedback.
-
|
I agree with that and this is why I prefer building dependencies as subprojects, as opposed to installing them in the system. What I do not like about the current OF approach is that OF is the root folder and it contains projects that in turn rely on its own parent folders. This is very awkward for me and as said it does not easily allow one project to be continued as a standalone GitHub project. (Pushing the entire OF repo brings its own kind of problems). What I would hope to find in a generated project is that the project itself is at its own root, and OF is in a subfolder. This way every project has its own copy of OF. (Size shouldn't be an issue with Conan's global caching). They will also not break in the future because the OF version used is ideally specified by version or by git tag. You can still have a default folder which contains all projects, but the point is that every single project itself is self-contained, and not THE OF folder with ALL projects is self-contained. I want to move the notion from "This is my openFrameworks installation and somewhere in there might be my project", to "This is my project - and if you are interested, it is made using openFrameworks". (or at least the possibility to do so) |
Beta Was this translation helpful? Give feedback.
-
|
(also, it may be good to keep this conversation specific to modernizing apothecary, vs discussing redesigning how OF works generally -- perhaps branching this conversation might make sense?) |
Beta Was this translation helpful? Give feedback.
-
|
Oh sorry, thanks for pointing it out👍 |
Beta Was this translation helpful? Give feedback.
-
|
Here's another wild idea - what if we switched to Homebrew ( for mac ) and vcpkg ( for windows ) for all our dependencies? I think all the main libs we use are available on both platforms ( assimp, glfw, opencv, etc ) And we could then essentially use a cleaned up apothecary with cmake for the remaining things we need to compile ( like emscripten / android etc ). It would mean a slightly longer initial setup, but it would take away a huge % of our current headaches and we're already doing something similar with Linux. |
Beta Was this translation helpful? Give feedback.
-
|
well well well... as a "comfortable" OF macOS+linux user, I don't see a problem with that (and for GitHub users, the download_libs step is already required on Mac too, and if you're tracking GitHub you 99.9% already have homebrew), but there will be friction for a "new user", or in a Mac (and I presume windows) workshop/lab context. requiring homebrew is not preposterous, but it's another level of technical awareness (need to use the command line, for instance). . there is something very valuable in step1: tell everyone in the room to download some OF release; and step2: pick an example in your desktop environment, hit play in Xcode, and it runs. getting traction in OF (C++) takes time, and at one point an active "decision" to invest energy must be made; until that moment happens in the user's head, everything must be kept as streamlined as possible. so I guess it comes back to "redesigning how OF works generally"... but in the interim, there is perhaps an evaluation of what is required from each library, and can those requirements be met more simply? this is where my question about libcurl leads to in #7656 with (a) if we can use the OS-provided lib, might as well (which I guess is more or less what happens in linux through the "OS-sanctioned" package manager); and if not: (b) if we can use a headers-only lib that gets compiled at project-time, might as well... |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, it definitely is more in terms of setup. The other issue would be packaging universal apps. |
Beta Was this translation helpful? Give feedback.
-
|
more out of curiosity here is the assimp3DModelLoaderExample running purely off of homebrew libs and a few header only libs. 
|
Beta Was this translation helpful? Give feedback.
-
|
can GitHub actions install homebrew? |
Beta Was this translation helpful? Give feedback.
-
|
could be a direct homebrew download_libs from git and a packaging of bin libs from homebrew in the releases... it augments the number of possibilities in the development interval, but I don't think that's a real problem. |
Beta Was this translation helpful? Give feedback.
-
|
apparently yes: https://github.com/marketplace/actions/install-and-cache-homebrew-tools |
Beta Was this translation helpful? Give feedback.
-
|
(also homebrew supports the "latest 3 OS versions" (the meaning of which sort of changed in the past few years). in any case I guess that's OK with general OF use in the sense that the "current version" of OF and C++ is probably going to work down about 3 versions too, but for LTS it will create a hole in dependencies once those 3 versions elapse as the support to get the libraries won't exist anymore). |
Beta Was this translation helpful? Give feedback.
-
|
First of all, I agree that solutions like homebrew and the linux package manager saves a lot of headache, what I am not so fond of is using vcpkg on Windows. I do not particularly like it and hardly know any people or systems using it, and making it a requirement for OF would mean everyone that wants to use OF must first install vcpkg, which is not really desirable. You can correct me on this, but from my perspective, vcpkg has been dying the past decade. But now that I am reading all of this, I am going even further back: How important is that an average OF user even has the source code of OF? I mean just as a theoretical thought, what if OF is built using system libs, but in GH Actions it is compiled into static libs for many architectures (like apothecary now), meaning an average OF user that is not a developer only downloads a prebuilt binary package that only has OF static libraries and those of its dependencies. It would move dependency issues from a non-knowing user to the CI pipeline. OF Developers can still build it locally by serving the dependencies. It would mean we have the freedom of packaging our dependencies how we want and how comfortable it is for the CI/CD pipeline, not how comfortable it is for an average user. Another thing, I did some reddit posts in the past to get some different perspectives. If the dependencies are only needed in CI, then I do not have an issue with vcpkg, I only have an issue with requiring vcpkg from an average user. Another idea was to "steal" packages from Conan or vcpkg and re-upload them in the apothecary, but that is also a very bad idea as it puts a barrier between the library maintainer and the OF user. This is from a security point of view, think of OpenSSL security patches. Realistically, OF dependencies will not be updated even every few years. But it would also be a bad idea in general. Downloading using one package manager and re-uploading as a zip file is a very weird take that goes directly against the idea of having the package manager in the first place. I think we should really be using a package manager like it is meant to be, and not abuse it and put an unnecessary piece of poorly maintained software inbetween. So as a recap, what about the idea of using system package managers (apt, homebrew and vcpkg), while also providing pre-built OF libraries that include JUST the static libraries of OF + dependencies, and their headers? |
Beta Was this translation helpful? Give feedback.
-
|
I think I will start to migrate the CMake branch to where it finds dependencies using find_package and not from a local folder, once that is done finding system libs is trivial |
Beta Was this translation helpful? Give feedback.
-
|
Recently I had some spare time and I took another look at CMake and Conan. I really think Conan is the way to go. This does not mean everyone needs to use Conan. I figured what we could do is create a Conan package called Additionally, I am still a fan of also providing a package with OF already prebuilt, which would then be another simple Conan package, where export the artifacts to a non-Conan package. Some libraries that are not part of Conan, are already built from source as part of OF. I removed ofAppGlutWindow because it is subject to be removed AFAIK. But what about FMOD? It also isn't part of Conan and it's not trivial to build. AFAIK it is also subject to be removed, so what's the state? I don't think I can simply comment it out... |
Beta Was this translation helpful? Give feedback.
-
|
as far as I understand, GLUT is about to be axed, and FMOD has been mostly de-required (there are #ifdefs to take it out), but it's hanging there perhaps for backward-compatibility. I don't think "current" projects should make use of it. so you can brute-force-remove it to validate the approach, and the "proper" final resolution of it's removal can be studied in parallel (perhaps an addon?) for the use of Conan I am in favour as long as the process for "end users" stays as simple as "install an IDE, download OF in a non-admin account, and compile some examples (without additional/external dependencies or 3rd party package managers)". I'm not sure from your proposal if you suggest an intermediary mode where the git-based installs (developers or bleeding edge users) would more or less supersede apothecary with "live" Conan dependencies in the dev workflow? or would that still rely on CI to assemble the required libraries? (I would not mind getting live libs, as I've had a couple of situations stemming from synchronisation between the OF tree and the libs (you tend to forget to download_libs now and then when working with the git source)). |
Beta Was this translation helpful? Give feedback.
-
|
A quick explanation of my current Conan goal: Then, a CMake file can be used to immediately download the correct binaries when configuring OF. This could be achieved by a command-line define for CMake. So TLDR: More advanced users that need better debugging and that can read the docs, call something like Due to using CMake's find_module's, it's also possible to exchange dependencies by custom versions of them. And lastly, OF power users can install Conan and build all depencies locally and build OF by calling This is a course estimate of what the final workflow with Conan would look like. What do you think? |
Beta Was this translation helpful? Give feedback.
-
|
Ah! so let me confirm: OF makes use of Conan not only as a package manager to handle dependencies, but also becomes a Conan package, in 2 flavours: dependencies + OF (which would be a new option; currently OF is always re-compiled by the end user), and only dependencies (but instead of being manual shell scripts, they are automated in CMake). And (thank you for brainstorming here) that means that:
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, that is all correct. A minor correction: The binary release, assembled by Conan in CI, would always be downloaded like now, but as part of the CMake configure step which means it's completely seamless and automatic. It need internet connection on first build though. find_modules are always used, it works such that CMake always looks for libraries using find_modules, that way it could find system libraries. CMake does not know of Conan. And Conan, when called, generates FindModule files for CMake which results that find_modules finds Conan libraries instead of system libraries. And finally, when building the local Conan libraries, most of the time all libraries are built from source. Conan only provides a few precompiled targets and VS2022 is not yet part of it, meaning when you use VS2022 all Conan libraries are built locally, the first time you use them on your machine. Currently this may take up to an hour because huge dependencies like OpenCV, openssl, glibc, etc are used. Just a few insights in the future. I am not entirely sure but there might be something like an Artifactory free version for open source projects. Also, isn't there an OF server hosting the binaries, anyway? The Artifactory self-hosted repository is free and could host the cached Conan packages for build configurations that are not part of Conan. |
Beta Was this translation helpful? Give feedback.
-
|
Just to add in here that the new bleeding Cmake apothecary scripts allow the developer to easily build all dependencies themselves if needed... with a simple run of a bash script with not many system dependencies needed, however install system script included now. I think most developers just don't understand how to use, setup or install apothecary scripts in the past so making that easier is goal. Meaning in the past they would maybe try and run it, it would fail, they become confused and go down another rabbit hole. Lets fix that. If you want to compile a library say OpenCV with a different command than packaged... it should be easy so new system allow for that, not just whole packaged full source binaries you may find in some package managers which will increase app size etc. Downloading latest libraries makes this process a lot quicker as building them however and is preferred and easily accessible from git project If you want to build all dependencies for Android for all platforms, run the .sh from apothecary directory and it will build and deploy all binaries to the libraries of the project with secure hash. Problem with other package built systems is you don't really have control of the bloat or commands and many of the dependencies are custom patched to fix crashes, security problems or to support platforms not available from source. |
Beta Was this translation helpful? Give feedback.
-
|
A quick note, if everything goes as planned with Conan, Apothecary would become completely obsolete. Currently, a few non-Conan packages are built from source, but that is all done on the local side of OF's Conan build script (which is Python). So, the entire OF build and all of its dependencies is in the main OF repository. Apothecary is not needed at all. The only thing where we might think about using Apothecary (or the new This would be a lot of them since we would host one archive for OF+deps, one archive for OF source + deps, and all of that for every compiler, compiler version and OS supported. These are intended to be automatically downloaded by CMake. We might argue if we want these packages in the releases of the main repository where users can find them, or in another hidden repository that only CMake and core devs know about, so new users are not confused by 500 tar files. |
Beta Was this translation helpful? Give feedback.
-
|
And if you really want to exchange dependencies by custom patched versions, I think the best, most permanent solution would be to simply fork the repository, modify the Conan file to point to another package (or a self-created one) and build it using conan. Using the CMake downloaded binary libs and injecting other find_modules might also work, but is more like a hack. We might need to test this to see how plausible it is. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting ideas, and I like your enthusiasm to get what you can see as the biggest hurdle in oF, probably due to getting VS2022 stuff going am I right? I just also know, tested and understood the power of Apothecary many moons ago after package manager built libraries would crash, in production. We can apply and have applied custom fixes, patches while reporting them upstream, however some like FreeImage go un-patched leading to situations like Pegasus re:LibPNG subdepandacy We support 3.19.0 with the Pegasus fix via apothecary for example and you can't find that here. Which means all runtimes built with Conan or Pkg mangers can be hacked into oblivion with remote execution png you can find in the dark. Let's not allow ease of use to blindside power of the potions. Apothecary is bash, multiplatform supported now. No need for powershell or automk systems. Once it's fully operational in unbleedling fashion cmake commands pave to future for many platforms not even envisioned yet by package managers. For example I have the libraries building for visionOS, can't find that on a package manager in the world right now. Custom builds of ANGLE libraries for OpenGL/ES to Metal, I think these package system are okay for nightmares like Boost, SSL even, where the make systems are bananas and they refused cmake scripts and the packages are hardened generally. Package managers just doesnt allow for the customisation and addon ability we are about to spin up. Let us merge all these bleeding changes over the next week and write up tutorials for use of the systems that may seem difficult to understand. Again its literally as easy as the conan commands, I've just spent some time looking into conan after you mentioned and it's pretty cool, however I can see the flaws we already fixed |
Beta Was this translation helpful? Give feedback.
-
|
I see your points. However, I do not see them as issues. Freeimage not fixing a security issue is not Conan's fault, it is FreeImage's. I cannot even find a 3.19 source archive for download on their website. Now, the power of Conan comes from the fact that you can use provided packages IF they work for you if they are well-maintained, and if not, you can roll your own. If you have a library with a good build system that just works, Conan can use it. It can call any buildsystem, and then package the artifacts, you just need to tell it the build command and what files were compiled. So if you have a patched FreeImage version that will compile with a simple command, we can integrate it 1-to-1 into Conan. So Conan can actually replace trivial dependencies and in the case of non-trivial cases, the build scripts can continue to live within Conan. How easy that is depends on the build system. Good, proper CMake support makes it super easy, os-dependent build scripts are harder. But EVERY build system CAN be integrated into a Conan package. Now, even if it seems like it, i am not promoting Conan on purpose or anything, I just think it could be an escape out of the current madness and would be one of the few sane things to do. I am open for alternatives, which the current apothecary approach is not. |
Beta Was this translation helpful? Give feedback.
-
|
A note about By the way, you can browse through all available packages on Conan at https://conan.io/center. Could someone please take a quick look at it and tell if it is possible to switch to another library? |
Beta Was this translation helpful? Give feedback.
-
|
I think this kind of discussion is better, which dependencies can we replace, update and optimise with new ones. With easy cmake commands the power is in our hands not in the what is build on a package manager. We can create it |
Beta Was this translation helpful? Give feedback.
-
|
This is something we must do anyways, without getting rid of old dependencies, Conan and CMake would end up in the current state of what apothecary is now. In fact, CMake is so convoluted currently that it is completely unmaintainable (due to inconsistencies in apothecary). This is what started the discussion about apothecary in the first place, this is why the packages repo was created for me to test alternatives, and this is also what started the discussion if we need apothecary in the first place or using an existing package manager (at least for a few dependencies) would be a better solution. This is how we ended up in this discussion. It was kicked off by apothecary being so inconsistent that 90% of CMake is just fixing inconsistencies. |
Beta Was this translation helpful? Give feedback.
-
|
re SVGtiny, there is no specific requirement but it's what's in the implementation of ofxSVG, which is a builtin addon. It is not a large class, so depending how nanosvg's |
Beta Was this translation helpful? Give feedback.
-
|
Okay as per the QBS discussion, people want to use CMake soon, but apothecary/packages is not yet ready to serve all packages in the new way. I think the current OF CMake should more or less work, but it is a huge mess due to inconsistencies and I don't want to ship it that way. Now, as a temporary fix, we could slightly modify the libs.zip and add a few CMake files into there, that are used to locate and import the binaries. That would significantly reduce complexity and I would be willing to release CMake support under these circumstances. Now, I am not familiar with the old apothecary: Is it feasible to achieve said thing and add a few CMake files into the libs archives, without breaking everything or a lot of manual work? |
Beta Was this translation helpful? Give feedback.
-
Hi there!
I am planning in contributing significantly to modernize OF. I just contributed full CMake-support here: https://forum.openframeworks.cc/t/all-about-cmake/41777
I had many issues while implementing it because many parts of OF have grown significantly over the years that result in a lot of mess and spaghetti code. I love OF for the tremendous amount of features and for what it could be.
I think the apothecary is a central point of mess which could seriously be improved as stated by @ofTheo in the above discussion.
My plan would be to:
This would then allow me to use that for the CMake support, which makes everything a lot easier.
What do you think? Any issues with the plan above, or improvement suggestions?
Disclaimer:
I am not actively using OF currently and I have not been around for long, which is bad because I don't know all the use cases, but also good because I now see many many serious flaws that others simply oversee as they're used to it. I am only trying to help and make OF more modern so more people can approach it and make sure it always simply works (which it never has for me). If I am trying to do something wrong or trying to change something which would violate core ideas of OF, please tell me!
Beta Was this translation helpful? Give feedback.
All reactions