Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

195 implement metatranscriptomics workflow definition in workflow automation framework #246

Merged
merged 53 commits into from
Sep 24, 2024
Merged

195 implement metatranscriptomics workflow definition in workflow automation framework #246

merged 53 commits into from
Sep 24, 2024

Conversation

mbthornton-lbl
Copy link
Contributor

@mbthornton-lbl mbthornton-lbl commented Sep 9, 2024 •
edited
Loading

This PR provides updates to test fixtures and unit tests to enable the MetaTranscriptomics worflow configuration to be run in the workflow automation framework.

Updates to test fixtures:

  • bring existing test fixtures into line with current schema
  • add test fixtures for metaT
  • DRY up and refactor existing test fixtures
  • Add fixtures/db_utils.py for test_db setup / teardown and fixture loading

Updated Tests to cover workflow automation functionality with data fixtures for both MetaG and MetaT

Scheduler tests for both metaG and metaT workflows

  • test_submit
  • test_progress

Load Activities tests for Job creation for MetaG and MetaT workflows

  • test_activities

WorflowJob Model

Workflow Utils

  • Update Workflow model to include Analyte Category

Other

  • move workflows.yaml files to be under nmdc_automation/config to fix path issues
  • bring test_data into tests/

@mbthornton-lbl
Refactor JSON test fixtures for setting up test DB
c1a27ad 
- use the same configs/workflows.py as automation uses
- update test db fixtures to conform better to current schema
- re-named fixture data objects for readability

1 failing test `test_multple_versions`
@mbthornton-lbl
this config file was never used
c2b2e67
@mbthornton-lbl
Update remaining DB fixtures to modern style
0fde295
@mbthornton-lbl
Add fixtures for metatranscriptome sequencing and RQC
66a3495
@mbthornton-lbl
Fix paths and get remaining tests for scheduler working
bef4ef5
@mbthornton-lbl
update mock_progress function to take record from fixture file by idx…
31d0f2e 
…, no always [0]
@mbthornton-lbl
uncomment the rest of test_progress_metagenome
9ccd3b6
@mbthornton-lbl
Dry up test db fixtures
c32f99e
@mbthornton-lbl
refactor test db utilities to fixtures/db_utils
0f1f5fb
@mbthornton-lbl
Add fixture for MetaT genome assembly
9f27c04
@mbthornton-lbl
Refactor to use test_dir and fixtures_dir defined in conftets.py
bb8f7ed
@mbthornton-lbl
move test_data directory to live in tests/
d1a03a2
@mbthornton-lbl
minor change to error message for get_mongo_db()
8424276
@mbthornton-lbl
Update test fixtures to include metatranscriptome data
7b4710e
@mbthornton-lbl
update tests to use test_data and fixtures dirs defined in conftest.py
06f3daf
@mbthornton-lbl
update activity mapper to have type hints and remove hard-coded paths…
766ff86 
… from tests
@mbthornton-lbl
updated nmdc api and related tests to take str or path for site config
5185fbe
@mbthornton-lbl
update metagenome annotation fixture
a3cceab
@mbthornton-lbl
update site config file
0e45aad
@mbthornton-lbl
update .gitignore after moving test_data dir
319e2b9
@mbthornton-lbl
add a db_utils function for updating fixture versions
8abd948
@mbthornton-lbl
update tests to use site_config fixture
4c15813
@mbthornton-lbl
add directory fixtures to conftest
dd0efb7
@mbthornton-lbl
refactor tests to use dirs defined in conftest
7dbb782
@mbthornton-lbl
rename a fixture file
bbbc8f7
@mbthornton-lbl
Merge branch 'main' into 195-implement-metatranscriptomics-workflow
23523ad
@mbthornton-lbl mbthornton-lbl linked an issue Sep 9, 2024 that may be closed by this pull request
Implement MetaTranscriptomics Workflow #195
Closed
@mbthornton-lbl mbthornton-lbl changed the title DRAFT. 195 implement metatranscriptomics workflow definition in workflow automation framework 195 implement metatranscriptomics workflow definition in workflow automation framework Sep 19, 2024
@mbthornton-lbl mbthornton-lbl marked this pull request as ready for review September 19, 2024 18:41
aclum
aclum requested changes Sep 24, 2024
View reviewed changes
Copy link
Contributor

@aclum aclum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments on test_load_workflows.

Also not a blocker but for all test data 'name' should be the file name, it is currently a mix of a description and a file name. Also empty keys should be left out, there are several times in the test data where alternative_identifiers is present but empty.

@mbthornton-lbl
Copy link
Contributor Author

Simplified name values in DataObject - i.e. not file names, are intended to make the fixtures more maintainable

@mbthornton-lbl mbthornton-lbl merged commit c003242 into main Sep 24, 2024
1 check passed
@mbthornton-lbl mbthornton-lbl deleted the 195-implement-metatranscriptomics-workflow branch September 24, 2024 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aclum aclum aclum approved these changes

@kaijli kaijli Awaiting requested review from kaijli

@scanon scanon Awaiting requested review from scanon

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Implement MetaTranscriptomics Workflow
3 participants
@mbthornton-lbl @scanon @aclum