v6.0.4

Important Information

This release is only compatible with Cumulus versions 15.x and 16.x

• Full Change Log available at https://github.com/nasa/cumulus-orca/blob/release-6.0.4/CHANGELOG.md
• Full Change Comparison: v6.0.3...v6.0.4

Fixed

• ORCA-738 Update cumulus-process to v1.2.0 and cumulus-message-adapter-python to v2.1.0. To alleviate potential issues related to timeouts when using CMA calls.

v8.1.0

Important information

🔥 This release is only compatible with Cumulus v17.x.x 🔥

• Full Change Log available at https://github.com/nasa/cumulus-orca/blob/release-8.1.0/CHANGELOG.md
• Full Change Comparison: v6.0.3...v8.1.0

Migration Notes

• If utilizing the copied_to_glacier output property of copy_to_glacier,
  rename to new key copied_to_orca.

• If utilizing the orca_lambda_copy_to_glacier_arn output of Terraform, likely as a means of pulling the lambda into your workflows,
  rename to new key orca_lambda_copy_to_archive_arn

• If utilizing the orca_lambda_request_files_arn output of Terraform, likely as a means of pulling the lambda into your workflows, rename to new key orca_lambda_request_from_archive_arn

• If desired, use the optional recoveryBucketOverride property in extract_filepaths_for_granule input schema to override the default recovery bucket. See example below.

  {
    "input":
      {
        "granules": [
          {
            "granuleId": "MOD09GQ.A0219114.N5aUCG.006.0656338553321",
            "recoveryBucketOverride": "<YOUR_RECOVERY_BUCKET>",
            "files": [
              {
                "key": "MOD09GQ___006/2017/MOD/MOD09GQ.A0219114.N5aUCG.006.  0656338553321.h5",
                "bucket": "cumulus-test-sandbox-protected",
                "fileName": "MOD09GQ.A0219114.N5aUCG.006.0656338553321.h5",
              }
            ]
          }
        ]
    }
  }

• If utilizing the output of the OrcaRecoveryWorkflow, adjust to the simplified output schema. See example below:

  {
      "granules": [
      {
        "granuleId": "integrationGranuleId",
        "keys": [
          {
            "key": "PODAAC/SWOT/ancillary_data_input_forcing_ECCO_V4r4.tar.gz",
            "destBucket": "PREFIX-public"
          }
        ],
        "recoverFiles": [
          {
            "success": true,
            "filename": "ancillary_data_input_forcing_ECCO_V4r4.tar.gz",
            "keyPath": "PODAAC/SWOT/ancillary_data_input_forcing_ECCO_V4r4.tar.  gz",
            "restoreDestination": "PREFIX-public",
            "s3MultipartChunksizeMb": null,
            "statusId": 1,
            "requestTime": "2023-02-10T21:06:13.071287+00:00",
            "lastUpdate": "2023-02-10T21:06:13.071287+00:00"
          }
        ]
      }
    ],
    "asyncOperationId": "770a85f2-f933-4440-90b5-1a8039557538"
  }

• The output format of copy_to_archive lambda and step-function has been simplified. If accessing these resources outside of a Cumulus perspective, instead of accessing output["payload"]["granules"] you now use output["granules"].

• Cumulus is not currently compatible with the changes to copy_to_archive.

  • This section will be updated when a compatible version is created.
  • deployment-with-cumulus.md will also be updated.
  • copy_to_archive_adapter/README.md will also be updated.
  • restore-to-orca.mdx will also be updated.

• Cumulus is not currently compatible with the changes to the Recovery Workflow step-function.

  • This section will be updated when a compatible version is created.
  • deployment-with-cumulus.md will also be updated.
  • orca_recovery_adapter/README.md will also be updated.

• Update the bucket policy for your system-bucket to allow load balancer to post server access logs to the bucket. See the instructions here.

• InternalReconcileReport Phantom and Mismatch reports are now available via GraphQL.

  • API Gateway access is now deprecated, and will be removed in a future update.
  • Use the orca_graphql_load_balancer_dns_name variable to send your queries to GraphQL as json strings in a POST request.

• Users will need to update their orca-user password. The password must have the following requirements otherwise the db_deploy lambda will fail during deployment.

  • one upper case letter
  • one lower case letter
  • one digit
  • one special character
  • minimum length of 12

  Update db_user_password variable in your cumulus-tf/terraform.tfvars file to match the new password requirement and then run terraform. db_deploy lambda will automatically update your new password.

• Changes have been made to SQS message processing that are not backwards compatible. Halt ingest and wait for the PREFIX-orca-status-update-queue.fifo queue to empty before applying update.

  • If the queue is stuck or becomes stuck, it may be necessary to flush the queue and its associated Dead Letter Queue.

• The input format of the ORCA Recovery Workflow step-function has been modified.
  If accessing these resources outside of a Cumulus perspective, go to orca_recover_workflow.asl.json and look at config elements to see the new paths. Additionally, add a collectionId property to each granule passed in.

• collectionId properties have been added to Recovery Jobs and Recovery Granules API.

  • For Recovery Jobs, it is only added to output.
  • For Recovery Granules, it is now required on input and will be returned on output.

• Update the orca.tf file to include aws_region. See example below.

  ## ORCA Module
  ## =============================================================================
  module "orca" {
    source = "https://github.com/nasa/cumulus-orca/releases/download/v6.0.0/cumulus-orca-terraform.zip//modules"
  ## --------------------------
  ## Cumulus Variables
  ## --------------------------
  ## REQUIRED
  aws_region               = var.region
  buckets                  = var.buckets
  lambda_subnet_ids        = var.lambda_subnet_ids
  permissions_boundary_arn = var.permissions_boundary_arn
  prefix                   = var.prefix
  system_bucket            = var.system_bucket
  vpc_id                   = var.vpc_id
  workflow_config          = module.cumulus.workflow_config
  
  ## OPTIONAL
  tags        = local.tags
  
  ## --------------------------
  ## ORCA Variables
  ## --------------------------
  ## REQUIRED
  db_admin_password        = var.db_admin_password
  db_user_password         = var.db_user_password
  db_host_endpoint         = var.db_host_endpoint
  dlq_subscription_email   = var.dlq_subscription_email
  orca_default_bucket      = var.orca_default_bucket
  orca_reports_bucket_name = var.orca_reports_bucket_name
  rds_security_group_id    = var.rds_security_group_id
  s3_access_key            = var.s3_access_key
  s3_secret_key            = var.s3_secret_key
  
  ## OPTIONAL
  db_admin_username                                    = "postgres"
  default_multipart_chunksize_mb                       = 250
  internal_report_queue_message_retention_time_seconds = 432000
  orca_default_recovery_type                           = "Standard"
  orca_default_storage_class                           = "GLACIER"
  orca_delete_old_reconcile_jobs_frequency_cron        = "cron(0 0 ? * SUN *)"
  orca_ingest_lambda_memory_size                       = 2240
  orca_ingest_lambda_timeout                           = 720
  orca_internal_reconciliation_expiration_days         = 30
  orca_recovery_buckets                                = []
  orca_recovery_complete_filter_prefix                 = ""
  orca_recovery_expiration_days                        = 5
  orca_recovery_lambda_memory_size                     = 128
  orca_recovery_lambda_timeout                         = 720
  orca_recovery_retry_limit                            = 3
  orca_recovery_retry_interval                         = 1
  orca_recovery_retry_backoff                          = 2
  s3_inventory_queue_message_retention_time_seconds    = 432000
  s3_report_frequency                                  = "Daily"
  sqs_delay_time_seconds                               = 0
  sqs_maximum_message_size                             = 262144
  staged_recovery_queue_message_retention_time_seconds = 432000
  status_update_queue_message_retention_time_seconds   = 777600
  vpc_endpoint_id                                      = null
  }

v9.0.0-beta

Release v9.0.0-beta

Update teraform to version 1.5.x

v8.0.1

Release v8.0.1

v8.0.0

Release v8.0.0

Added

• ORCA-554, ORCA-561, ORCA-579, ORCA-581
  • GraphQL image, service, and Load Balancer will now be deployed by TF.
  • ORCA-557 Added orca_graphql_load_balancer_dns_name to output variables for GraphQL integration.
  • ORCA-420 Added Internal Reconcile Report Mismatch functionality to GraphQL.
  • ORCA-556 Added Internal Reconcile Report Phantom functionality to GraphQL.
  • ORCA-592 GraphQL logs are json structures, and can thus be queried in CloudWatch.
  • ORCA-622 Added support for integer sizes up to 8 bytes.
• ORCA-597
  • Server access logging is now enabled for graphql application load balancer.
• ORCA-614, ORCA-428 Moved some Internal Reconciliation functionality to GraphQL

Changed

• ORCA-573 Updated ORCA DB user password to now have a stronger password requirement. See migration notes for details.
• ORCA-520 Removed run_cumulus_task function from copy_to_archive to decouple ORCA from Cumulus.
• ORCA-647 Upgraded sqlalchemy from v1.4.11 to v2.0.5.

Migration Notes

• The output format of copy_to_archive lambda and step-function has been simplified. If accessing these resources outside of a Cumulus perspective, instead of accessing output["payload"]["granules"] you now use output["granules"].

• Cumulus is not currently compatible with the changes to copy_to_archive.

  • This section will be updated when a compatible version is created.
  • deployment-with-cumulus.md will also be updated.
  • copy_to_archive_adapter/README.md will also be updated.
  • restore-to-orca.mdx will also be updated.

• The input format of the ORCA Recovery Workflow step-function has been simplified.
  If accessing these resources outside of a Cumulus perspective, go to orca_recover_workflow.asl.json and look at config elements to see the new paths.

• Cumulus is not currently compatible with the changes to the Recovery Workflow step-function.

  • This section will be updated when a compatible version is created.
  • deployment-with-cumulus.md will also be updated.
  • orca_recovery_adapter/README.md will also be updated.

• Update the bucket policy for your system-bucket to allow load balancer to post server access logs to the bucket. See the instructions here.

• InternalReconcileReport Phantom and Mismatch reports are now available via GraphQL.

  • API Gateway access is now deprecated, and will be removed in a future update.
  • Use the orca_graphql_load_balancer_dns_name variable to send your queries to GraphQL as json strings in a POST request.

• Users will need to update their orca-user password. The password must have the following requirements otherwise the db_deploy lambda will fail during deployment.

  • one upper case letter
  • one lower case letter
  • one digit
  • one special character
  • minimum length of 12

  Update db_user_password variable in your cumulus-tf/terraform.tfvars file to match the new password requirement and then run terraform. db_deploy lambda will automatically update your new password.

v6.0.3

Release v6.0.3

Changed

• ORCA-643 Reverted ORCA-437, which introduced IAM authentication for API Gateway endpoints.

v7.0.1

Release v7.0.1

Changed

• ORCA-632 Fixed a bug where excludedFileExtensions was a required property in collection config. Restored default behavior of defaulting to an empty list.

v7.0.0

Release v7.0.0

Changed

• ORCA-336
  • request_from_archive lambda now posts to the new SQS for files that have already been recovered from glacier instead of throwing an error.
  • post_copy_request_to_queue lambda now receives event messages of files recovered from archive from the new archive recovery SQS instead of archive bucket.
• ORCA-522
  • Removed run_cumulus_task function from extract_filepath_for_granule lambda to decouple ORCA from Cumulus.
• ORCA-575
  • Removed run_cumulus_task function from request_from_archive lambda to decouple ORCA from Cumulus.
• ORCA-521
  • Replaced CumulusLogger with AWS powertools logger in all of the lambdas currently present in ORCA.
• ORCA-537
  • Renamed step-function OrcaCopyToGlacierWorkflow to OrcaCopyToArchiveWorkflow.
  • Renamed lambda PREFIX_copy_to_glacier to PREFIX_copy_to_orca. Renamed ORCA repository internal task from copy_to_glacier to copy_to_archive.
    Output of lambda and Terraform updated to match. See Migration Notes below.
• ORCA-540
  • Renamed lambda copy_files_to_archive to copy_from_archive.
  • Output of Terraform updated to match. Unlikely to affect any integrations.
• ORCA-539
  • Renamed lambda request_files to request_from_archive.
  • Output of Terraform updated to match. Unlikely to affect any integrations.
• ORCA-534
  • extract_filepaths_for_granule now raises a descriptive error when no destination bucket (destBucket) is found in fileBucketMaps for a given file.
    Previously was a general JsonSchemaException.
    Now is a ExtractFilePathsError with a description of which file could not be placed.
  • extract_filepaths_for_granule now takes the first match in fileBucketMaps instead of the last.
• ORCA-461
  • Invalid database connection parameters will now be detected earlier and more consistently.
  • Postgres table/user names can now begin with an '_' and contain '$' if your Postgres DB version supports this.
• ORCA-533 RecoveryWorkflow no longer requires the bucket property on files. Was unused by ORCA.

Added

• ORCA-336
  • Added a new standard SQS between archive ORCA bucket and post_copy_request_to_queue lambda so that the bucket now triggers the SQS upon successful object retrieval from glacier.
• ORCA-351
  • Added new optional recoveryBucketOverride property to extract_filepaths_for_granule input schema so that data managers can now specify their own buckets for recovery if desired.
• ORCA-574/580 Added additional logging to the extract_filepaths_for_granule and request_from_archive steps of the recovery workflow to identify when an input granule is entirely excluded, or otherwise has no files to request. Status entries for these granules will display an ERROR status.

Migration Notes

• If utilizing the copied_to_glacier output property of copy_to_glacier,
  rename to new key copied_to_orca.
• If utilizing the orca_lambda_copy_to_glacier_arn output of Terraform, likely as a means of pulling the lambda into your workflows,
  rename to new key orca_lambda_copy_to_archive_arn
• If utilizing the orca_lambda_request_files_arn output of Terraform, likely as a means of pulling the lambda into your workflows,
  rename to new key orca_lambda_request_from_archive_arn
• Use the optional recoveryBucketOverride property in extract_filepaths_for_granule input schema to specify a recovery bucket. See example below.

  {
    "input":
      {
        "granules": [
          {
            "granuleId": "MOD09GQ.A0219114.N5aUCG.006.0656338553321",
            "recoveryBucketOverride": "<YOUR_RECOVERY_BUCKET>",
            "files": [
              {
                "key": "MOD09GQ___006/2017/MOD/MOD09GQ.A0219114.N5aUCG.006.  0656338553321.h5",
                "bucket": "cumulus-test-sandbox-protected",
                "fileName": "MOD09GQ.A0219114.N5aUCG.006.0656338553321.h5",
              }
            ]
          }
        ]
    }
  }

• If utilizing the output of the OrcaRecoveryWorkflow, adjust to the simplified output schema. See example below:

  {
      "granules": [
      {
        "granuleId": "integrationGranuleId",
        "keys": [
          {
            "key": "PODAAC/SWOT/ancillary_data_input_forcing_ECCO_V4r4.tar.gz",
            "destBucket": "PREFIX-public"
          }
        ],
        "recoverFiles": [
          {
            "success": true,
            "filename": "ancillary_data_input_forcing_ECCO_V4r4.tar.gz",
            "keyPath": "PODAAC/SWOT/ancillary_data_input_forcing_ECCO_V4r4.tar.  gz",
            "restoreDestination": "PREFIX-public",
            "s3MultipartChunksizeMb": null,
            "statusId": 1,
            "requestTime": "2023-02-10T21:06:13.071287+00:00",
            "lastUpdate": "2023-02-10T21:06:13.071287+00:00"
          }
        ]
      }
    ],
    "asyncOperationId": "770a85f2-f933-4440-90b5-1a8039557538"
  }

v6.0.2

Release v6.0.2

Changed

• ORCA-570 Fixed an error that could prevent deployment of the database on fresh installations.

v6.0.1

Release v6.0.1

Changed

• ORCA-566 Shortened S3 inventory report name due to length limitation causing errors when a user's naming schema is long.