-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prune unused artifacts from non-static builds #59
base: main
Are you sure you want to change the base?
Commits on Nov 8, 2024
-
ioc: prune modules from non-static builds.
Remove EPICS modules that do not have a dynamic library linked to any executables in the target IOC. This shrinks the final image size by removing useless artifacts. We introduce APP_DIRS (application directories) variable to specify paths where relevant binaries are, when no-build target is used, so the proper cleanup can happen in such cases as well. For dynamic-build, it encodes any additional directories besides the base COPY'ed directory to be considered. Cleanup phase must be executed in the build stage (before the final copy), otherwise the COPY layer would still make the image size big. This implies `no-build` target needs to copy from a pruned version of the base image, which is a new stage. The list of modules to be removed is taken from the RELEASE file, so that it contains a valid mapping of all modules that have been installed. EPICS base is assumed to always be needed, even though some binaries in it can certainly be thrown away given the IOC linkage (for instance, PVAccess binaries when the IOC uses only Channel Access). Linkage information is taken from ldd(1), which assumes that inspected binaries are safe to be potentially executed. It is assumed that we are using glibc's ldd, which provides a straightforward way to query all binaries at once. Moreover, we ignore not-found library entries to properly cover binaries already shipped in repositories, such as libopcua.so, which might not contain (properly) defined rpaths. To avoid implementing a function that returns "all EPICS modules whose path is a prefix from the used library paths", we instead reuse `filter_out_paths()` by exploting the fact that it filters *out* everything we care about when given `linked_libs`. Taking its difference to the original set gives us what we need without implementing the variant of `filter_out_paths()`.
Configuration menu - View commit details
-
Copy full SHA for e643d8e - Browse repository at this point
Copy the full SHA e643d8eView commit details -
ioc: remove static libraries from non-static builds.
Static libraries are not used during runtime, and not even compile-time for non-static builds. On the other hand, they take up a large amount of storage: about 312MB for EPICS base and 222MB for modules. Remove them all right before finishing non-static build stages, avoiding their copy to IOC images. This is not needed for static-link targets, since only the cleaned IOC directory is copied, already leaving behind all static libraries. All files with .a are assumed to be static libraries, which should be a good assumption in GNU/Linux. Other static artifacts, such as object files, are correctly removed by the build system clean target and are not handled explictly here for simplification. A lint script for the build system can be implemented in the future if this eventually becomes false.
Configuration menu - View commit details
-
Copy full SHA for cc98b04 - Browse repository at this point
Copy the full SHA cc98b04View commit details -
ioc: prune shared libraries from non-static builds.
Shared libraries may be installed either in /opt or /usr/local/lib but may not be used by the target binaries from the IOC image. Remove them during the prune phase before copying both directories to resulting image to shrink the final image size. Both actual versioned binary and its symbolic links are removed to keep the filesystem consistent. When deciding the set of "used libraries", we assume all libraries inside any APP_DIRS are used. This allows one to specify a dlopen(3)'ed library path in APP_DIRS, and ensure it is kept in the resulting image.
Configuration menu - View commit details
-
Copy full SHA for 63ee161 - Browse repository at this point
Copy the full SHA 63ee161View commit details -
ioc: prune module directories from non-static builds.
Modules may contain several artifacts, including configuration files, graphical interface files and other repository artifacts that do not need to be in the IOC image. Remove them all except the ones containing EPICS database (`.db` and `.template`) or autosave requirement (`.req`) files, besides shared libraries. Binaries directory (`bin`) is also removed, as only $REPONAME and $RUNDIR should contain target executables, which are filtered out from the list.
Configuration menu - View commit details
-
Copy full SHA for 1680f69 - Browse repository at this point
Copy the full SHA 1680f69View commit details -
ioc: prune EPICS base directories from non-static builds.
EPICS base has rather large configuration files for build, and other repository files, which are not needed in the IOC images. Prune them after building the IOCs, shrinking by 40MB the final image size. Prune is performed with the same script as modules, which discards all executables in `bin` (~15MB), as well as Perl scripts. This should be fine considering that `static-link` target also does not preserve EPICS binaries in the resulting image.
Configuration menu - View commit details
-
Copy full SHA for 2ae76d3 - Browse repository at this point
Copy the full SHA 2ae76d3View commit details
Commits on Nov 14, 2024
-
ioc: don't prune pre-selected directories
When pruning modules, some of them might contain special cases where it is harder to automatically detect in the pruning algorithm. While users could workaround this by specifying such special paths under APP_DIRS, this can be handled here by adding extra tooling to be used by modules. Introduce a way for modules to specify extra paths that must be kept, besides those detected by lnls-prune-artifacts. Do this through a .lnls-keep-paths file that specifies all relative paths (directories or files) whose contents must be kept. Paths may be globs, so that they are resolved based on their bash expansion, avoiding hard-coded options when ambiguity can be automatically resolved. These exceptions are considered for all ancestors of a candidate to removal, so that they are not restricted to EPICS modules themselves. This eases implementation because we won't need to detect which parent is a module and should contain .lnls-keep-paths, and instead simply traverse the whole ancestor list looking if any of them defines one.
Configuration menu - View commit details
-
Copy full SHA for 2b46de5 - Browse repository at this point
Copy the full SHA 2b46de5View commit details -
base: don't prune pyDevSup custom python libraries
Some of the dynamic libraries built by pyDevSup are not directly linked into the IOC binary, but rather dlopen(3)'ed by Python interpreter when a "import" statement is reached. To avoid having all users of pyDevSup manually specify which ones to keep, define everything inside python3.<minor-version>/linux-<architecture> to be kept by lnls-prune-artifacts. We use globs here (whose expansion is deferred) to avoid changing the minor version every distro upgrade.
Configuration menu - View commit details
-
Copy full SHA for 0a9c79e - Browse repository at this point
Copy the full SHA 0a9c79eView commit details