Skip to content

Commit

Permalink
Add lambda handler for attaching thumbnails to records
Browse files Browse the repository at this point in the history 
Now that Archivematica is generating thumbnails for us, we want to use
those instead of generating our own. This commit adds a lambda handler
that will be subscribed to S3 events fired when Archivematica generates
new .jpg files inside DIPs. The handler checks whether the .jpg is a
thumbnail, and if so creates a CDN URL for it and adds that URL, and the
S3 location of the thumbnail, to the corresponding record in our
database.
  • Loading branch information
Liam Lloyd committed Aug 28, 2024
1 parent 3df4003 commit 748a973
Show file tree
Hide file tree
Showing 23 changed files with 3,118 additions and 1,371 deletions.
2 changes: 2 additions & 0 deletions .github/workflows/test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,5 @@ jobs:
- run: (cd stela/packages/api; docker compose run stela npm run test-ci)
- run: (cd stela; npm run test -w @stela/account_space_updater)
- uses: codecov/codecov-action@v2
- run: (cd stela; npm run test -w @stela/record_thumbnail_attacher)
- uses: codecov/codecov-action@v2
29 changes: 29 additions & 0 deletions Dockerfile.record_thumbnail_attacher
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
FROM public.ecr.aws/lambda/nodejs:18 as builder
WORKDIR /usr/local/apps/stela/

COPY package.json ./
COPY tsconfig.build.json ./
COPY tsconfig.json ./
COPY jest.config.js ./
COPY packages ./packages

RUN npm install -g npm@8.19.3
RUN npm install
RUN npm install -ws
RUN npm run build -ws


FROM public.ecr.aws/lambda/nodejs:18 as final
WORKDIR ${LAMBDA_TASK_ROOT}

COPY --from=builder /usr/local/apps/stela/packages/record_thumbnail_attacher/dist ./packages/record_thumbnail_attacher/dist
COPY --from=builder /usr/local/apps/stela/packages/record_thumbnail_attacher/package.json ./packages/record_thumbnail_attacher/package.json
COPY --from=builder /usr/local/apps/stela/packages/logger/dist ./packages/logger/dist
COPY --from=builder /usr/local/apps/stela/packages/logger/package.json ./packages/logger/package.json
COPY --from=builder /usr/local/apps/stela/package.json ./package.json
COPY --from=builder /usr/local/apps/stela/package-lock.json ./package-lock.json

RUN npm install -g npm@8.19.3
RUN npm install --workspace @stela/record_thumbnail_attacher

CMD ["packages/record_thumbnail_attacher/dist/index.handler"]
82 changes: 59 additions & 23 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,31 +30,34 @@ npm install -ws
Depending on the work being done, some environment variable will not be required for the service to run.
For these, simply fill in any fake value to prevent `require-env-variable` from throwing errors.

| Variable | Default | Notes |
| --------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| ENV | local | Tells stela what environment it's running in |
| DATABASE_URL | postgres://postgres:permanent@database:5432/permanent | Run tests to generate default database |
| PORT | 8080 | Tells stela what port to run on |
| FUSIONAUTH_HOST | <none. needs to be set> | Fusionauth's host URL. Should be different between prod and other envs. |
| FUSIONAUTH_API_KEY | <none. needs to be set> | Find it in Fusionauth admin panel -> settings -> API keys -> the one called "back-end (local)" |
| FUSIONAUTH_TENANT | <none. needs to be set> | Find it in Fusionauth admin panel -> Tenants -> the one called "Local" |
| FUSIONAUTH_BACKEND_APPLICATION_ID | <none. needs to be set> | Find it in Fusionauth admin panel -> Applications -> the one called "back-end (local)" |
| FUSIONAUTH_ADMIN_APPLICATION_ID | <none. needs to be set> | Find it in Fusionauth admin panel -> Applications -> the one called "admin-local" |
| Variable | Default | Notes |
| --------------------------------- | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| ENV | local | Tells stela what environment it's running in |
| DATABASE_URL | postgres://postgres:permanent@database:5432/permanent | Run tests to generate default database |
| PORT | 8080 | Tells stela what port to run on |
| FUSIONAUTH_HOST | <none. needs to be set> | Fusionauth's host URL. Should be different between prod and other envs. |
| FUSIONAUTH_API_KEY | <none. needs to be set> | Find it in Fusionauth admin panel -> settings -> API keys -> the one called "back-end (local)" |
| FUSIONAUTH_TENANT | <none. needs to be set> | Find it in Fusionauth admin panel -> Tenants -> the one called "Local" |
| FUSIONAUTH_BACKEND_APPLICATION_ID | <none. needs to be set> | Find it in Fusionauth admin panel -> Applications -> the one called "back-end (local)" |
| FUSIONAUTH_ADMIN_APPLICATION_ID | <none. needs to be set> | Find it in Fusionauth admin panel -> Applications -> the one called "admin-local" |
| LEGACY_BACKEND_HOST_URL | http://load_balancer:80/api |
| LEGACY_BACKEND_SHARED_SECRET | none | Can be found in `back-end`'s library/base/constants/base.constants.php |
| MAILCHIMP_API_KEY | none | Can be found in `back-end`'s library/base/constants/base.constants.php |
| MAILCHIMP_TRANSACTIONAL_API_KEY | none | Can be found in `back-end`'s library/base/constants/base.constants.php, where it is called `MANDRILL_API_KEY` |
| LEGACY_BACKEND_SHARED_SECRET | none | Can be found in `back-end`'s library/base/constants/base.constants.php |
| MAILCHIMP_API_KEY | none | Can be found in `back-end`'s library/base/constants/base.constants.php |
| MAILCHIMP_TRANSACTIONAL_API_KEY | none | Can be found in `back-end`'s library/base/constants/base.constants.php, where it is called `MANDRILL_API_KEY` |
| MAILCHIMP_DATACENTER | us12 |
| MAILCHIMP_COMMUNITY_LIST_ID | 2736f796db | The default value corresponds to the `dev` list |
| SENTRY_DSN | none | Can be found in Sentry under Projects > stela > Settings > Client Keys (DSN) |
| DEV_NAME | none | Only set in local environments. Should be your given name. all lowercase. Used to create Sentry envs for developers |
| AWS_REGION | us-west-2 | |
| AWS_ACCESS_KEY_ID | none | The same one you use in `devenv` |
| AWS_SECRET_ACCESS_KEY | none | The same one you use in `devenv` |
| LOW_PRIORITY_TOPIC_ARN | test | Doesn't need to be set to a real ARN unless your work touches it specifically |
| MIXPANEL_TOKEN | none | Found in Mixpanel at Settings > Project Settings > Project Token |
| ARCHIVEMATICA_BASE_URL | none | It is the url of the EC2 instance on which archivematica is running |
| ARCHIVEMATICA_API_KEY | none | Found in Bitwarden, not needed unless you're running the cleanup cron |
| MAILCHIMP_COMMUNITY_LIST_ID | 2736f796db | The default value corresponds to the `dev` list |
| SENTRY_DSN | none | Can be found in Sentry under Projects > stela > Settings > Client Keys (DSN) |
| DEV_NAME | none | Only set in local environments. Should be your given name. all lowercase. Used to create Sentry envs for developers |
| AWS_REGION | us-west-2 | |
| AWS_ACCESS_KEY_ID | none | The same one you use in `devenv` |
| AWS_SECRET_ACCESS_KEY | none | The same one you use in `devenv` |
| LOW_PRIORITY_TOPIC_ARN | test | Doesn't need to be set to a real ARN unless your work touches it specifically |
| MIXPANEL_TOKEN | none | Found in Mixpanel at Settings > Project Settings > Project Token |
| ARCHIVEMATICA_BASE_URL | none | It is the url of the EC2 instance on which archivematica is running |
| ARCHIVEMATICA_API_KEY | none | Found in Bitwarden, not needed unless you're running the cleanup cron |
| CLOUDFRONT_URL | none | Can be found as `CDN_URL` in `back-end`'s library/base/constants/base.constants.php. Not required for API server |
| CLOUDFRONT_KEY_PAIR_ID | none | Can be found as `CLOUDFRONT_KEYPAIR` in `back-end`'s library/base/constants/base.constants.php. Not required for API server |
| CLOUDFRONT_PRIVATE_KEY | none | Can be found in `back-end`'s library/static/certs/pk-APKAJP2D34UGZ6IG443Q.pem. Not required for API server |

## Linting

Expand Down Expand Up @@ -113,6 +116,39 @@ Outside a container: Run
npm run start -w @stela/api
```

## Running Lambdas Locally

1. Build the lambda image

```bash
docker build --platform linux/amd64 -t <LAMBDA NAME>:test -f Dockerfile.<LAMBDA NAME> .
```

2. Run the lambda container. Add additional --env arguments as needed. Note that the `CLOUDFRONT_PRIVATE_KEY` required
by the `record_thumbnail_attacher` must include literal newlines; docker does not appear to interpret `\n` correctly.

```bash
docker run --platform linux/amd64 -p 9001:8080 --env DATABASE_URL=postgres://postgres:permanent@database:5432/permanent <LAMBDA NAME>:test
```

3. Find the container name

```bash
docker ps
```

4. Connect to the local env docker network

```bash
docker network connect devenv_default <YOUR CONTAINER NAME>
```

5. Trigger the lambda

```bash
curl "http://localhost:9001/2015-03-31/functions/function/invocations" -d '<YOUR PAYLOAD>'
```

## Deployment

### To `dev`
Expand Down
Loading

0 comments on commit 748a973

Please sign in to comment.