Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prune unused artifacts from non-static builds #59

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Commits on Nov 8, 2024

  1. ioc: prune modules from non-static builds.

    Remove EPICS modules that do not have a dynamic library linked to any
    executables in the target IOC. This shrinks the final image size by
    removing useless artifacts. We introduce APP_DIRS (application
    directories) variable to specify paths where relevant binaries are, when
    no-build target is used, so the proper cleanup can happen in such cases
    as well. For dynamic-build, it encodes any additional directories
    besides the base COPY'ed directory to be considered.
    
    Cleanup phase must be executed in the build stage (before the final
    copy), otherwise the COPY layer would still make the image size big.
    This implies `no-build` target needs to copy from a pruned version of
    the base image, which is a new stage.
    
    The list of modules to be removed is taken from the RELEASE file, so
    that it contains a valid mapping of all modules that have been
    installed. EPICS base is assumed to always be needed, even though some
    binaries in it can certainly be thrown away given the IOC linkage (for
    instance, PVAccess binaries when the IOC uses only Channel Access).
    
    Linkage information is taken from ldd(1), which assumes that inspected
    binaries are safe to be potentially executed. It is assumed that we are
    using glibc's ldd, which provides a straightforward way to query all
    binaries at once. Moreover, we ignore not-found library entries to
    properly cover binaries already shipped in repositories, such as
    libopcua.so, which might not contain (properly) defined rpaths.
    
    To avoid implementing a function that returns "all EPICS modules whose
    path is a prefix from the used library paths", we instead reuse
    `filter_out_paths()` by exploting the fact that it filters *out*
    everything we care about when given `linked_libs`. Taking its difference
    to the original set gives us what we need without implementing the
    variant of `filter_out_paths()`.
    henriquesimoes committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    e643d8e View commit details
    Browse the repository at this point in the history
  2. ioc: remove static libraries from non-static builds.

    Static libraries are not used during runtime, and not even compile-time
    for non-static builds. On the other hand, they take up a large amount of
    storage: about 312MB for EPICS base and 222MB for modules. Remove them
    all right before finishing non-static build stages, avoiding their copy
    to IOC images. This is not needed for static-link targets, since only
    the cleaned IOC directory is copied, already leaving behind all static
    libraries.
    
    All files with .a are assumed to be static libraries, which should be a
    good assumption in GNU/Linux. Other static artifacts, such as object
    files, are correctly removed by the build system clean target and are
    not handled explictly here for simplification. A lint script for the
    build system can be implemented in the future if this eventually becomes
    false.
    henriquesimoes committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    cc98b04 View commit details
    Browse the repository at this point in the history
  3. ioc: prune shared libraries from non-static builds.

    Shared libraries may be installed either in /opt or /usr/local/lib but
    may not be used by the target binaries from the IOC image. Remove them
    during the prune phase before copying both directories to resulting
    image to shrink the final image size. Both actual versioned binary and
    its symbolic links are removed to keep the filesystem consistent.
    
    When deciding the set of "used libraries", we assume all libraries
    inside any APP_DIRS are used. This allows one to specify a dlopen(3)'ed
    library path in APP_DIRS, and ensure it is kept in the resulting image.
    henriquesimoes committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    63ee161 View commit details
    Browse the repository at this point in the history
  4. ioc: prune module directories from non-static builds.

    Modules may contain several artifacts, including configuration files,
    graphical interface files and other repository artifacts that do not
    need to be in the IOC image.
    
    Remove them all except the ones containing EPICS database (`.db` and
    `.template`) or autosave requirement (`.req`) files, besides shared
    libraries. Binaries directory (`bin`) is also removed, as only $REPONAME
    and $RUNDIR should contain target executables, which are filtered out
    from the list.
    henriquesimoes committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    1680f69 View commit details
    Browse the repository at this point in the history
  5. ioc: prune EPICS base directories from non-static builds.

    EPICS base has rather large configuration files for build, and other
    repository files, which are not needed in the IOC images. Prune them
    after building the IOCs, shrinking by 40MB the final image size.
    
    Prune is performed with the same script as modules, which discards all
    executables in `bin` (~15MB), as well as Perl scripts. This should be
    fine considering that `static-link` target also does not preserve EPICS
    binaries in the resulting image.
    henriquesimoes committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    2ae76d3 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. ioc: don't prune pre-selected directories

    When pruning modules, some of them might contain special cases where it
    is harder to automatically detect in the pruning algorithm. While users
    could workaround this by specifying such special paths under APP_DIRS,
    this can be handled here by adding extra tooling to be used by modules.
    
    Introduce a way for modules to specify extra paths that must be kept,
    besides those detected by lnls-prune-artifacts. Do this through a
    .lnls-keep-paths file that specifies all relative paths (directories or
    files) whose contents must be kept. Paths may be globs, so that they are
    resolved based on their bash expansion, avoiding hard-coded options when
    ambiguity can be automatically resolved.
    
    These exceptions are considered for all ancestors of a candidate to
    removal, so that they are not restricted to EPICS modules themselves.
    This eases implementation because we won't need to detect which parent
    is a module and should contain .lnls-keep-paths, and instead simply
    traverse the whole ancestor list looking if any of them defines one.
    henriquesimoes committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    2b46de5 View commit details
    Browse the repository at this point in the history
  2. base: don't prune pyDevSup custom python libraries

    Some of the dynamic libraries built by pyDevSup are not directly linked
    into the IOC binary, but rather dlopen(3)'ed by Python interpreter when
    a "import" statement is reached. To avoid having all users of pyDevSup
    manually specify which ones to keep, define everything inside
    python3.<minor-version>/linux-<architecture> to be kept by
    lnls-prune-artifacts.
    
    We use globs here (whose expansion is deferred) to avoid changing the
    minor version every distro upgrade.
    henriquesimoes committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    0a9c79e View commit details
    Browse the repository at this point in the history