AWS SQS

DAGs in the ils-middleware Airflow responds to messages posted to AWS SQS service using the SQSSensor.

Current SQS Queues and adding as Airflow Variables

We are storing the root URL for each of the queues as Airflow Variables so that they are available to the calling institutional DAGs.

Sinopia Development Environment:

• Name: cornell-ils URL: https://sqs.us-west-2.amazonaws.com/418214828013/cornell-ils
• Name: stanford-ils URL: https://sqs.us-west-2.amazonaws.com/418214828013/stanford-ils
• Airflow Variable:
  • key: SQS_DEV
  • value: https://sqs.us-west-2.amazonaws.com/418214828013/

Sinopia Stage Environment:

• Name: cornell-ils URL: https://sqs.us-west-2.amazonaws.com/742528315309/cornell-ils
• Name: stanford-ils URL: https://sqs.us-west-2.amazonaws.com/742528315309/stanford-ils
• Airflow Variable:
  • key: SQS_STAGE
  • value: https://sqs.us-west-2.amazonaws.com/742528315309/s

Setting up AWS Airflow Connection

The credentials for connecting to the AWS SQS queue are set as an Airflow Connection. This can be manually set through the UI but we should investigate retrieving theses credentials from our Vault instance. More information on the specific values that can be set are at Amazon Web Services Connection.

Through the UI, the following should be set:

• Conn Type: Amazon Web Services
• Login: AWS access key ID,
• Password: AWS secret access key

In the Extra field, we add a JSON string with the following field:

• region_name: "us-west-2", this shouldn't change

{
  "region_name": "us-west-2" 
}