Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

more file submission ops #633

Merged
merged 10 commits into from
Nov 30, 2022
Merged

more file submission ops #633

merged 10 commits into from
Nov 30, 2022

Conversation

blankdots
Copy link
Contributor

@blankdots blankdots commented Oct 31, 2022

Description

Related issues

related to: #148
continuation of #627

Type of change

  • New feature (non-breaking change which adds functionality)

Changes Made

  • check files have the status ready when being read from the submission
  • prevent publish if files have in submission have status added (added but no metadata object) or failed (failed in ingestion, completion, or for any other reason) - in future version we can also check if the submission actually has files
  • missing docs about raise exception in files
  • update status of a file
    • /submission/<submission_id>/files receives a list of files
    [
        {
          "accessionId": 1,
          "version": 1,
          "status": "ready".
          "objectId": {
             "accessionId": 4,
             "schema": "study"
          }
        },
        {
          "accessionId": 2,
          "version": 1,
          "status": "ready"
        },
        {
          "accessionId": 3,
          "version": 1,
          "status": "ready"
        }
      ]
    

and transforms it to mongodb query for each file in update_file_submission e.g.:

{
    "files.$.status": "ready",
    "files.$.version": 1,
    "files.$.objectId": {
      "accessionId": 4,
      "schema": "study"
    }
  • remove file from submission when flagged as deleted, via remove_file_submission which is called when a file is flagged for deletion but can also be called whenever we want to remove all files with a specific accessionId or file path from one or multiple submissions
  • operation to assign metadata object to file, also marking a file as ready, done via /submission/<submission_id>/files as one or multiple files can be marked as ready - will be used also MQ integration.
  • small bug fixes and typo fixes

Testing

  • Needs testing (start an issue or do a follow up PR about it)

Mentions

this is dependent on #642

@blankdots blankdots self-assigned this Oct 31, 2022
@csc-felipe
Copy link
Contributor

Looking good

@blankdots blankdots force-pushed the feature/file-submission-ops branch 3 times, most recently from fad2b5c to c701b90 Compare November 3, 2022 11:10
@blankdots blankdots changed the base branch from develop to feature/extend-problem-json November 3, 2022 11:13
Base automatically changed from feature/extend-problem-json to develop November 3, 2022 12:16
@blankdots blankdots changed the base branch from develop to reafactor/publish-action November 16, 2022 09:15
@blankdots
Copy link
Contributor Author

changed base so that mypy fixes are included

Base automatically changed from reafactor/publish-action to develop November 16, 2022 10:55
@blankdots blankdots marked this pull request as ready for review November 28, 2022 18:51
Copy link
Collaborator

@csc-jm csc-jm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a few things to nitpick but otherwise looks good to me

metadata_backend/database/db_service.py Outdated Show resolved Hide resolved
metadata_backend/database/db_service.py Outdated Show resolved Hide resolved
metadata_backend/api/handlers/submission.py Outdated Show resolved Hide resolved
metadata_backend/api/handlers/submission.py Outdated Show resolved Hide resolved
csc-jm
csc-jm previously approved these changes Nov 29, 2022
- fix comment in submission.py
- the files verified or completed don't need to be submitted again.
blankdots and others added 8 commits November 30, 2022 10:32
- we first check there are files that don't have any failed or added status, that would prevent a submission from being finalized
- remove and append return the document after update
- update specification for `/submission/<submission_id>/files` to update a submission with a list of files. Add the necessary operations for that in db-service and in file operator
- remove file from submission when flag as deleted. Also enable to remove a specific file from submission
- small fixes to description and specification.yml
- add changelog for file operation update and remove
- add delete endpoint to remove a list of files identified by accession id
- add endpoints in specification.yml
Co-authored-by: Joonatan Mäkinen <85632655+csc-jm@users.noreply.github.com>
Copy link
Contributor

@csc-felipe csc-felipe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Need to create an issue for tests.

@blankdots
Copy link
Contributor Author

Need to create an issue for tests.

I thought this was it: #629

@blankdots blankdots merged commit ca69c01 into develop Nov 30, 2022
@blankdots blankdots deleted the feature/file-submission-ops branch November 30, 2022 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants