Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partial implementation with Opensearch serverless #471

Merged
merged 79 commits into from
Jul 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
cf97269
add new opensearch client implementation for opensearch serverless (wip)
Mar 15, 2024
1b495bc
add new opensearch client implementation for opensearch serverless (wip)
Mar 15, 2024
db5caea
fix dependency version discrepencies which cause the debug mode to fail
Mar 15, 2024
6a8743f
Merge branch 'opensearchclient' of https://github.com/NASA-PDS/regist…
Mar 15, 2024
39aa95a
one end poit product/lidvid works with the new opensearch client.
Mar 27, 2024
117f2a6
move API response objects in sub-package of model
Mar 28, 2024
f9d59bf
add explicit discipline node list in configuration for multitenant re…
Apr 3, 2024
48ca86a
add support for discipline node index prefix in multitenant setup.
Apr 3, 2024
e017624
wip: add AWS opensearch serverless connection, remove ununsed constant.
Apr 5, 2024
7f385cf
finally works on local opensearch and AWS opensearch serverless
Apr 23, 2024
42be227
complete lidvid resolver, handle errors.
May 1, 2024
05a6881
remove redundant code in controllerAdvice. Add pagination support.
May 1, 2024
a3cad10
add forgotten file
May 2, 2024
5bff508
start q param implementation, simple filter, exception handling
May 14, 2024
03553c1
make the q parmeter work on /products
May 29, 2024
16710d1
add new files for opensearch serverless, ignore locl application.prop…
May 29, 2024
51e48b1
wip: update terraform scripts for MCP
May 29, 2024
4d6a656
terraform work in dev, try to make the API support being in a context…
Jun 6, 2024
d24207e
add README for terraform
Jun 7, 2024
69c8461
add new opensearch client implementation for opensearch serverless (wip)
Mar 15, 2024
f870906
fix dependency version discrepencies which cause the debug mode to fail
Mar 15, 2024
2145d2e
add new opensearch client implementation for opensearch serverless (wip)
Mar 15, 2024
95c2fc9
one end poit product/lidvid works with the new opensearch client.
Mar 27, 2024
0df99f0
move API response objects in sub-package of model
Mar 28, 2024
83f1088
add explicit discipline node list in configuration for multitenant re…
Apr 3, 2024
d96f7a1
add support for discipline node index prefix in multitenant setup.
Apr 3, 2024
feb9830
wip: add AWS opensearch serverless connection, remove ununsed constant.
Apr 5, 2024
3023cd7
finally works on local opensearch and AWS opensearch serverless
Apr 23, 2024
347be1f
complete lidvid resolver, handle errors.
May 1, 2024
6e4d075
remove redundant code in controllerAdvice. Add pagination support.
May 1, 2024
6387f88
add forgotten file
May 2, 2024
825571d
start q param implementation, simple filter, exception handling
May 14, 2024
63e0fb5
make the q parmeter work on /products
May 29, 2024
55b1083
add new files for opensearch serverless, ignore locl application.prop…
May 29, 2024
ef8d095
wip: update terraform scripts for MCP
May 29, 2024
b8af7fc
terraform work in dev, try to make the API support being in a context…
Jun 6, 2024
29aa08f
add README for terraform
Jun 7, 2024
4c5d706
upgrade minor spring framework version as on main
Jun 7, 2024
0a2bfb3
add the load balancer to the terraform script
Jun 27, 2024
e6068fd
re-implement #343
alexdunnjpl Jul 3, 2024
c53e3e0
generalise MissSortWithSearchAfterException -> SortSearchAfterMismatc…
alexdunnjpl Jul 9, 2024
df5bc52
expand sort/searchAfter error checks to catch any invalid case
alexdunnjpl Jul 9, 2024
9143a8a
clarify /products endpoint behavior
alexdunnjpl Jul 10, 2024
d2cebb4
apply onlyLatest() to /products route
alexdunnjpl Jul 10, 2024
0fd8b07
Merge pull request #476 from NASA-PDS/343-reimplementation
alexdunnjpl Jul 11, 2024
e249241
Merge pull request #480 from NASA-PDS/products-route-only-latest
alexdunnjpl Jul 11, 2024
17759eb
Merge pull request #479 from NASA-PDS/products-endpoint-doc-update
alexdunnjpl Jul 11, 2024
d3edf80
Merge pull request #477 from NASA-PDS/sort-search-after-exceptions
alexdunnjpl Jul 11, 2024
323e3b3
convert RegistrySearchRequestBuilder to subclass of org.opensearch.cl…
alexdunnjpl Jul 11, 2024
1155708
reimplement field match methods
alexdunnjpl Jul 11, 2024
9454f1c
rename/document query-string method
alexdunnjpl Jul 11, 2024
225aa59
excise RegistrySearchRequestBuilder copy constructor
alexdunnjpl Jul 11, 2024
f5378ad
excise RegistrySearchRequestBuilder.searchRequestBuilder
alexdunnjpl Jul 11, 2024
19d2a50
replace RegistrySearchRequestBuilder must/mustNot with queryBuilder
alexdunnjpl Jul 11, 2024
36ed047
minor naming tweaks
alexdunnjpl Jul 11, 2024
2fdd259
remove unnecessary getters
alexdunnjpl Jul 11, 2024
f2dffb2
minor re-order
alexdunnjpl Jul 11, 2024
d29c557
fix bug
alexdunnjpl Jul 11, 2024
4427ca7
Merge pull request #489 from NASA-PDS/edunn-develop
alexdunnjpl Jul 12, 2024
77248b7
Update branch-cicd.yaml per new default branch
jordanpadams Jul 16, 2024
482532e
Update unstable-cicd.yaml per new default branch
jordanpadams Jul 16, 2024
9b7efb3
implement ProductClasses.java
alexdunnjpl Jul 18, 2024
dd809bb
implement MiscellaneousBadRequestException
alexdunnjpl Jul 18, 2024
5d0fa0f
implement members and members/members products subroutes
alexdunnjpl Jul 18, 2024
d93984d
implement product_class filter helper methods
alexdunnjpl Jul 19, 2024
3773889
[partial] re-implement ancestry endpoints and related helper functions
alexdunnjpl Jul 18, 2024
bac4a52
fix bug
alexdunnjpl Jul 18, 2024
7c9af2a
implement ProductsController.resolveExtantLidvids()
alexdunnjpl Jul 18, 2024
973ad53
implement ProductsController.resolveLatestLidvid()
alexdunnjpl Jul 18, 2024
c00412b
[partial] implement ancestry-related endpoints and associated helper …
alexdunnjpl Jul 18, 2024
0547f29
implement BadRequestException handling
alexdunnjpl Jul 19, 2024
acc790f
Merge pull request #499 from NASA-PDS/edunn-membership-endpoints
alexdunnjpl Jul 24, 2024
58d3c68
add TODOs for controller common code refactor
alexdunnjpl Jul 24, 2024
924cffc
implement swagger-name conversions in PdsProductClasses
alexdunnjpl Jul 23, 2024
1a36e9a
implement RegistrySearchRequestBuilder.matchProductClass()
alexdunnjpl Jul 24, 2024
1475d1f
implement /classes and /classes/{class} API routes
alexdunnjpl Jul 24, 2024
d2a2895
Remove blob (#513)
tloubrieu-jpl Jul 24, 2024
142ed47
Merge pull request #512 from NASA-PDS/classes-endpoints
jordanpadams Jul 29, 2024
c9408ff
Merge pull request #496 from NASA-PDS/default_branch_update
jordanpadams Jul 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/branch-cicd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ on:
branches:
- '**'
- '!main'
- '!develop'

workflow_dispatch:

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unstable-cicd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ name: 🤪 Unstable integration & delivery
on:
push:
branches:
- main
- develop
workflow_dispatch:
concurrency: roundup

Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ target/
lexer/target
service/target
model/target
application-*.properties

# macOS specific stuff
.DS_Store
14 changes: 7 additions & 7 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -76,14 +76,14 @@ RUN : &&\
#
# External context.

ARG SERVER_PORT=80
ARG SPRING_BOOT_APP_ARGS=" "
ARG JAVA_OPTS="-Xmx6144m -Xms512Mb"

WORKDIR /usr/local/registry-api-service
EXPOSE 80
CMD [ \
"java", "-Xmx6144m", \
"-Xms", "512Mb", \
"-jar", "/usr/local/registry-api-service/registry-api-service.jar", \
"gov.nasa.pds.api.registry.SpringBootMain" \
]
EXPOSE ${SERVER_PORT}
CMD java $JAVA_OPT -jar /usr/local/registry-api-service/registry-api-service.jar \
gov.nasa.pds.api.registry.SpringBootMain --server.port=${SERVER_PORT} ${SPRING_BOOT_APP_ARGS}


# Labels
Expand Down
100 changes: 0 additions & 100 deletions docker/Dockerfile.aws

This file was deleted.

28 changes: 24 additions & 4 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,22 @@

There's one Dockerfile here that can be used to make images for development and production.

👉 **Note:** It's a security risk to include private keys in images. As a result, we've removed the old `Dockerfile.https` and `Dockerfile.https.dev`. We've also removed all the other `Dockerfile.*` files to eliminate confusion, except for `Dockerfile.local`; see below.

## Build the image

To build an image, run:


cd ./service
docker image build --build-arg api_jar=URL --tag [OWNER/]registry-api-service:TAG .


Replace `URL` with the URL (or relative file path) to a `registry-api-service.jar` and `TAG` with the desired version tag. You can add `OWNER/` to tag the image for a specific owner, such as `nasapds/`.

The GitHub Actions configured in this repository automatically make images after each `stable-cicd.yaml` workflow (with a `:X.Y.Z` tag) and `unstable-cicd.yaml` workflow (with a `:latest` tag) and publishes them to the Docker Hub.


## 🧱 Example Builds
## 🧱 Examples

Building a local image:
```console
Expand All @@ -28,9 +32,25 @@ Building an image from a released jar file:
$ docker image build --build-arg api_jar=https://github.com/NASA-PDS/registry-api/releases/download/v1.0.0/registry-api-service-1.0.0.jar --tag nasapds/registry-api-service:1.0.0 --file docker/Dockerfile .
```

## 📍 Dockerfile.aws
## Run the image


For a local deployment simply do:

docker run -t -i -e SERVER_PORT=8082 -p 8082:8082 [OWNER/]registry-api-service:TAG


The same image can be used to run in multiple environments by passing arguments at the service start up, through environment variables. The provided options can override the `application.properties` configuration of the application. The server port needs to define a specific environment variable and match -p option of the docker run.

For example on AWS, with OpenSearch serverless as a back-end:

SPRING_BOOT_APP_ARGS=--openSearch.host=<your opensearch serverless URL> --openSearch.CCSEnabled=true --openSearch.username="" --openSearch.disciplineNodes=atm-delta,en-delta --registry.service.version=1.5.0-SNAPSHOT
SERVER_PORT=80





This Dockerfile is used to make images for AWS deployments of the registry manager. It copies in an AWS-specific application.properties file which has been setup to inform the service to obtain certain properties as environment variable values that have been injected by the Elastic Container Service (ECS) runtime. An example of this is the Opensearch credentials from the Secrets Manager.

## 📍 Dockerfile.local

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ expression : andStatement | orStatement | queryTerm ;
andStatement : queryTerm (AND queryTerm)+ ;
orStatement : queryTerm (OR queryTerm)+ ;
comparison : FIELD operator ( NUMBER | STRINGVAL ) ;
likeComparison : FIELD NOT? LIKE STRINGVAL ;
likeComparison : FIELD LIKE STRINGVAL ;
operator : EQ | NE | GT | GE | LT | LE ;

NOT : 'not' ;
Expand Down
Loading
Loading