[IBCDPE-946] Adds Optional Uploading

[BWMac]

Problem:

Every time the agora-data-tools processing pipeline is run, new output data files and GX report files are produced and uploaded to Synapse. This even happens when developers run the pipeline locally for testing. This causes an unnecessarily large number of files to be added to Synapse making it harder than it needs to be to find the output files from a particular run.

Solution:

Add an upload option to the CLI command and make it so that files are not uploaded by default. Additionally, add a platform argument to the CLI command so that the user can indicate whether their run is LOCAL or being executed on GITHUB or NEXTFLOW (Defaults to LOCAL).

The idea now is that when the pipeline is being run locally, it can be run exactly as before:

adt test_config.yaml

This command does not change at all because the default behavior is platform = LOCAL and upload = False

When we want file uploading to occur for testing purposes, we can make the following changes:

adt my_dev_config.yaml --upload

Note that I have provided a different configuration file this time. That is because in this instance I am a developer who needs to test the file uploading capabilities of the pipeline and I am providing a different configuration file with the destination and gx_folder fields pointing to a testing Synapse project that I have so that my test runs do not create new versions of the files in the Agora Synapse project.

Finally, for the Nextflow Tower execution, the command will look like (nf-agora PR):

adt {config} --upload --platform NEXTFLOW

and the GitHub Action command is now (this PR):

adt test_config.yaml --upload --platform GITHUB --token ${{secrets.SYNAPSE_PAT}}

Notes:

• I considered using a destination argument where a developer could change the upload destination on Synapse without changing the configuration file, but this way is simpler to implement and sticks with our current paradigm where the configuration file dictates pipeline behavior.
• Right now, the platform argument is only used to throw a warning if you have platform set to LOCAL and upload is True to remind users to not upload files from a local run unnecessarily. This will be used more extensively in IBCDPE-947.
• Tests are updated/added as needed.
• Documentation is updated as needed.
• Apologies for the large formatting diff on CONTRIBUTING.md. The actual change I made is lines 78-88.

[sonarcloud]

Quality Gate passed

Issues
9 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
9.6% Duplication on New Code

See analysis details on SonarCloud

[jaclynbeck-sage]

Looks good! I'm less good at knowing what to look for in the testing functions so maybe someone else can weigh in on that part of it.

[thomasyu888]

🔥 LGTM! Excellent - just to confirm, the upload parameter just limits to the GX reports? Or is it also the output files?

[BWMac]

@thomasyu888 It prevents both from being uploaded.