[IBCDPE-947] GX Validation Record Keeping

[BWMac]

Problem:

As the agora-data-tools pipeline is run more and more, the versions of output files and GX reports are piling up. It is important that we are able to identify which files were produced during a particular run, and a record-keeping solution is therefore necessary.

Solution:
Design Doc

Implement a method of record keeping within ADT that will upload metadata about a processing run to a Synapse Table in the Agora Project. This way, we will keep a historical record of what files were produced during which ADT runs and we will be able to look up the specific GX report files for those runs.

The majority of logic for this new feature is contained within two new classes:

• DatasetReport: Contains all of the fields needed to populate one row of the Synapse Table. There will be one of these per GX-enabled dataset from each ADT run.
• ADTGXReporter: Contains all of the fields common for all DatasetReports in a run, and also performs the updating of the Synapse table in the end.

Notes:

• To surface the metadata contained within ADT runs, some refactoring needed to be done.
• A new optional parameter run_id has been added to help keep track of specific GH Actions and Nextflow Tower runs. nf-agora PR.
• Added a gx_table parameter to the config files so that testing runs and live runs have their records stored separately.
• The Platform enum is moved to its own module to prevent circular import issues.
• Tests are added and updated as needed.

Future Work:

While working on this feature, a couple of issues came up I have created tickets to track:

• Recording information about ADT input files/configuration files used (ticket).
• Leveraging the GX mostly parameter to create some flexible expectations and report out warnings about how much data has met/not met an expectation (ticket).

[jaclynbeck-sage]

Looks good, nice work!

[thomasyu888]

Nice work here! Just some comments

[sonarcloud]

Quality Gate passed

Issues
47 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
25.4% Duplication on New Code

See analysis details on SonarCloud

[thomasyu888]

🔥 LGTM!