Improve balsamic development & verification

[ivadym]

I would like to propose a new way of carrying out the development and validation, speeding up the deployment and making it smoother by:

• Unifying develop and release cache building. We want to have in production what we have tested in stage.
• Allowing the update of the already generated cache. Downloading references in every release seems unnecessary and time consuming.

This will involve changes at three different levels: balsamic init command, balsamic config case command, and containers generation through GitHub actions.

GitHub actions

• Work exclusively with the develop containers, which won't be version tagged. We'll still push to docker hub the release containers to be able to recreate previous production states. However, we won't use them.
• Containers will be tested when they have been changed or when a feature has been pushed to develop.
• Containers will be pushed to docker hub when they have been changed and only from the develop branch.
• Same behaviour for patches/hot fixes as working on develop branch as long as the fix branch follows a specific naming convention: "hotfix/"

balsamic config case & balsamic init

• By default, the cache will continue being associated to the balsamic version. Cache path: /home/proj/<env>/balsamic_cache/<balsamic_version>/
• A new flag (--develop) will be added to the CLI for testing/validation purposes. Cache path: /home/proj/stage/balsamic_cache/develop/. This will allow having both an updated develop cache and a reproducible production environment in stage.

balsamic init

• If the output directory already exists, the balsamic cache will only be updated:
  • All the containers will be pulled again as there is no clear way to know if they have been changed.
  • Only the missing or newly added reference files will be downloaded.

Development process

---

Things to consider:

• The balsamic version inside the balsamic container won't match the release version as the containers will be built before bumping the tags (develop/hotfix branch).
• We can also just fix the balsamic init command, keeping the current structure of github actions and CLI, and instead of generating the cache, only copy it when there have been no changes to the references.

[khurrammaqbool]

It is a nice development centric proposal and indeed will make it easy to deploy/implementation of BALSAMIC in production environment. This is based on assumption that it takes at least one or two additional days to deploy BALSAMIC to production after the validation is complete and final approval has been made for the deployment, which requires building the cache in the production environment after the release with bumped version.

The current model for BALSAMIC up to v11.2.0 is to link the versions of bioinformatics tools used in BALSAMIC with the Method Description and Tools and software in the balsamic documentation. The version of most bioinformatics tools is fetched from the YAML file which is used to build the containers e.g. for varcall_py3 and added to the report for an analysis. We still miss some software versions using this method, but this method covers most of the bioinformatics software used for the analysis.

In other words the bioinformatics tools are linked to a specific BALSAMIC version using the model above. This is to ensure traceability of the bioinformatic tools, along with their versions, to make the analysis/results reproducible. This is the main reason BALSAMIC cache has to be build after making the release, with a bump version, and not before. Decoupling the cache from the BALSAMIC version by manual manipulation of the cache, as proposed above, may make it difficult, or impossible, to trace and link the version of the bioinformatic tools used with a specific version of BALSAMIC. This is because of the flexibility of addition and testing of the bioinformatic tools to be included in the develop branch that we allow in the develop branch for the purpose of development and not for the purpose of analysis, which makes the cache in the develop branch to be flexible and not linked to the documentation mentioned above.

One of the solution, to the arising problem, could be to fetch the version of all the bioinformatic software from within the container and add it to the sample report to enable the reproducibility of the results.

The changes in the development model must also include a plan to ensure that the bioinformatic tools and their versions are traceable and reporting of the tools and versions are implemented to make the analysis and results reproducible.

[ivadym]

#1112 ✅