Integrate with SD-Connect (swift-browser-ui) to receive info about submitted files

[blankdots]

Description

We would need to integrate with https://github.com/neicnordic/sda-pipeline in order to be able to associate metadata to submitted files

We can use https://github.com/neicnordic/sda-orchestration/blob/master/sda_orchestrator/utils/consumer.py to consume files from inbox queue in order to associate them to a submission.

DoD (Definition of Done)

Integration with RabbitMQ message broker to read the messages about submitted files and their IDs as well as dataset IDs

• integration with SD-Connect sharing API to share bucket/objects for submission Add project number to Sharing API  swift-browser-ui#755
• SD-Connect service for data handover to SD-Submit
  • calculate md5/sha256 for entire file (New SD-Connect issue), both encrypted and decrypted checksums
  • re-encrypt when sharing with a key sda-pipeline
  • store information about files in SD-Submit db
  • nice to have: [SDA] support for multiple s3 buckets neicnordic/sensitive-data-archive#378
  • MQ client that sends files' metadata to SD-Submit queue
  • (optional) tagging of Selected Files in SD-Connect to add to the object metadata for filtering them both in SD-Connect (new Issue) and SD-Submit X-Object-Meta-Staged-For-Publishing (and a separate one for buckets)
• integration with sda-pipeline (including the RabbitMQ) Bump github.com/aws/aws-sdk-go from 1.45.6 to 1.45.11 neicnordic/sda-pipeline#622 @blankdots
• new files collection is required Bump github.com/aws/aws-sdk-go from 1.45.11 to 1.45.15 neicnordic/sda-pipeline#623 @csc-felipe
• files operator in sd-submit backend Bump github.com/aws/aws-sdk-go from 1.45.19 to 1.45.24 neicnordic/sda-pipeline#627 @csc-felipe and more file submission ops #633 @blankdots
• Bump github.com/aws/aws-sdk-go from 1.45.24 to 1.45.25 neicnordic/sda-pipeline#629
• MQ consumer
  • requires to process information from multiple vhost and multiple queues, so that we know when a file has been, verified, completed, etc.
  • look into RabbitMQ streams (not what we need) and/or fanout
  • store files information - process message related to file (upload, remove and maybe cancel)
  • publish submission and watch for file statuses
    • process a submission and associated files successfully
    • reflect errors ingesting files in a submission
    • figure out if cancelling should be handled in any way
• integration with Allas backend to list buckets/objects that shared to a project @blankdots
  • integration with allas s3 possible
  • listing projects in the ingest s3 inbox path is not possible - ingest can have different inbox and archive backends, this allows for an inbox to be sftp and the archive s3.
• test sda-pipeline compatibility with Allas - decide if Openstack Swift is needed (should work ™️ ) @blankdots
  • currently not required to adjust for openstack swift
  • only downside are ACLs
• SD-Connect UI
• SD-Submit UI
• nice to have: permanent token instead of short lived AAI token generate permanent token #777

Testing

Integration and unit testing

[blankdots]

the messages we would receive e.g. from SD-Connect to SD-Submit are in the form of:

{
   "operation": "upload",
   "user":"john",
   "project": "csc_project",
   "filepath":"somedir/encrypted.file.gpg",
   "encrypted_checksums": [
      { "type": "md5", "value": "abcdefghijklmnopqrstuvwxyz"},
      { "type": "sha256", "value": "12345678901234567890"}
   ]
}

with operation being upload, remove, rename

for neicnordic/sda-pipeline#295 we will receive file identifiers in the form of

The messages SD-Submit would send to sda-pipeline:
trigger ingestion

{
   "type": "ingest", 
   "user": "user", 
   "project": "csc_project",
   "filepath": "somedir/encrypted.file.c4gh"
   "encrypted_checksums": [
      { "type": "md5", "value": "abcdefghijklmnopqrstuvwxyz"},
      { "type": "sha256", "value": "12345678901234567890"}
   ]
}

assign accession id to file after successful ingestion

{
   "user":"john",
   "filepath":"somedir/encrypted.file.c4gh",
   "accession_id": "EGAF12345678901",
   "decrypted_checksums": [
      { "type": "md5", "value": "abcdefghijklmnopqrstuvwxyz"},
      { "type": "sha256", "value": "12345678901234567890"}
   ]
}

and a message that will connect them to the dataset with:

{
   "type": "mapping",
   "user":"john",
   "dataset_id": "EGAD12345678901",
   "accession_ids": ["EGAF12345678901", "EGAF12345678902"]
}

[blankdots]

we need to create a files database where we keep files uploaded and their ids will go under neicnordic/sda-pipeline#282 groups/projects

or have an API on SD-Connect

[blankdots]

the files that we get from SD-Connect need to be associated as part of an object to be added to the published submission otherwise it does not make sense for those files to be part of a submission

[csc-felipe]

May need to make it part of the workflows, and might need a new schema to be created, and added to workflows neicnordic/sda-pipeline#578.

[blankdots]

To list down how the communication between SD-Connect and SD-Submit will go

1. SD-Connect
   • users uploads file(s)
   • selects set files/buckets for publication in SD-Submit (UI in SD-Connect to reflect that)
   • the files/buckets are shared (read access) with a project SD-Submit knows of
   • the files/buckets are tagged with a specific tag to be able to distinguish them in the SD-Connect UI
   • an API call is made to SD-Submit /files endpoint with information on: project (under the form project_), file path, file name, checksum (and additional information that is deemed necessary)
   • files will need to be re-encrypted with a key that sda-pipeline will know, or at leas the key added to the header
2. SD-Submit
   • in the Files step of a workflow lists the files for a specific project from the files collection in MongoDB and optionally checks the Allas API for those files
   • after metadata has been attached to the files a user would publish the submissions
   • before it is published SD-Submit checks against the Allas API if the files have been removed, changed or checksum differs
   • if all is ok send ingestion message to RabbitMQ, for FEGA communication an Inbox message will need to be sent as well (e.g. at the Files step in the workflow
   • flag the files in the MongoDB that they have been submitted for a project (tbd if files can be selected multiple times to different submissions)
3. sda-pipeline
   • accesses allas to read and split the files

[csc-felipe]

with operation being upload, remove, rename

I'd say we only need upload and remove, or equivalent. Rename would be a delete followed by upload operations. This is in line with the object storage not having a rename feature.

[blankdots]

with operation being upload, remove, rename

I'd say we only need upload and remove, or equivalent. Rename would be a delete followed by upload operations. This is in line with the object storage not having a rename feature.

Our inbox will be SD-Connect and we might not need monitoring these ops and will be skipped, the only use case where inbox messages will be required will be FEGA.