OpenSearch indexing and queries #2834

phixMe · 2024-06-07T21:31:52Z

Problem

Our search right now does not enable nested queries on OpenLineage facets, code, linked entities, and ids. We want to
enable our search to be the very best place to absorb and catalog OpenLineage based data.

Opensearch.Demo.mov

Includes

Indexing of documents on OL events
New APIs to retrieve search for jobs and datasets
Search UI updates that can be enabled or fallback to legacy search system.
Local and Helm updates for new OpenSearch containers needed for this. (Including dashboard)

To Follow Up

Update jobs and datasets for all tag related tag actions to include in indexes.
Adding support for more facets and integrations (Trino, Flink...)

Checklist

You've signed-off your work
Your changes are accompanied by tests (if relevant)
Your change contains a small diff and is self-contained
You've updated any relevant documentation (if relevant)
You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
You've included a header in any source code files (if relevant)

netlify · 2024-06-07T21:32:06Z

✅ Deploy Preview for peppy-sprite-186812 canceled.

Name	Link
🔨 Latest commit	`d096138`
🔍 Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/66bfdfaf894d82000853712d

…es-client

api/src/main/java/marquez/MarquezApp.java

wslulciuc · 2024-08-09T22:19:33Z

api/src/main/java/marquez/api/OpenLineageResource.java

@@ -68,6 +68,7 @@ public OpenLineageResource(
  public void create(@Valid @NotNull BaseEvent event, @Suspended final AsyncResponse asyncResponse)
      throws JsonProcessingException, SQLException {
    if (event instanceof LineageEvent) {
+      serviceFactory.getSearchService().indexEvent((LineageEvent) event);


If search.enabled=false, will the index call fail/error?

Handling in the indexEvent method:

public void indexEvent(@Valid @NotNull LineageEvent event) { if (!searchConfig.isEnabled()) { log.debug("Search is disabled, skipping indexing"); return; } UUID runUuid = runUuidFromEvent(event.getRun()); log.debug("Indexing event {}", event); if (event.getInputs() != null) { indexDatasets(event.getInputs(), runUuid, event); } if (event.getOutputs() != null) { indexDatasets(event.getOutputs(), runUuid, event); } indexJob(runUuid, event); }

Ok, when we follow up with an SearchService interface, we'll want to bind search to an engine (psql, opensearch) and can do away with the flag.

wslulciuc · 2024-08-09T22:22:25Z

api/src/main/java/marquez/api/OpenSearchResource.java

+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package marquez.api;


I would move the pkg to marquez.api.v2beta.SearchResource until we promote the API to v2.

wslulciuc · 2024-08-09T22:47:38Z

build.gradle

@@ -64,6 +64,8 @@ subprojects {

    dependencies {
        implementation "org.projectlombok:lombok:${lombokVersion}"
+        implementation 'org.opensearch.client:opensearch-rest-client:2.15.0'
+        implementation 'org.opensearch.client:opensearch-java:2.6.0'


Move to api/build.gradle

codecov · 2024-08-09T23:00:13Z

Codecov Report

Attention: Patch coverage is 22.36025% with 125 lines in your changes missing coverage. Please review.

Project coverage is 83.28%. Comparing base (422fd43) to head (d096138).
Report is 1 commits behind head on main.

Files	Patch %	Lines
...i/src/main/java/marquez/service/SearchService.java	8.40%	106 Missing and 3 partials ⚠️
...c/main/java/marquez/api/v2beta/SearchResource.java	21.05%	15 Missing ⚠️
...src/main/java/marquez/api/OpenLineageResource.java	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2834      +/-   ##
============================================
- Coverage     84.77%   83.28%   -1.49%     
- Complexity     1470     1477       +7     
============================================
  Files           256      259       +3     
  Lines          6626     6785     +159     
  Branches        308      313       +5     
============================================
+ Hits           5617     5651      +34     
- Misses          856      977     +121     
- Partials        153      157       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

# Conflicts: # .env.example # docker-compose.web.yml

wslulciuc · 2024-08-09T23:14:26Z

web/src/types/api.ts

+  run_id: string
+  name: string
+  namespace: string
+  eventType: string


How's eventType used?

wslulciuc

Amazing work, @phixMe! We definitely have some follow up work, but great to see the progress on search 💯 🚀 🥇

Signed-off-by: wslulciuc <willy@datakin.com>

…es-client

Signed-off-by: wslulciuc <willy@datakin.com>

wslulciuc

🔍 ❤️

* Elasticsearch code. * Adding basic responses for elasticsearch. * Saving highlights. * Saving code cleanup. * Adding EsSearch. * Saving partial progress. * Refinements. * Small bug fixes. * Fixing alignment * Migrating es jobs naming to be specific. * Adding boilerplate for dataset es search * Adding datasets. * Adding polish for more data. * Empty state and other small enhancements. * Adding arrow key functionality. * Removing console log * Spotless * Refinements to queries. * Adding debounce. * Fixing alignment issues. * Saving updates for password setting via env config for elasticsearch. * Setting up startup scripts and adding corresponding waits. * Adding logs and more fields for jobs. * Resolving jackson serialization issue. * Small updates for search display. * Adding onClick handlers. * Fixing null cases, adding more search options for datasets. * Handling enter key. * Fixing minor encoding and layout issues for spark related open lineage events. * Additional fixes for text overflow on names and namespaces. * Fixing indexing problem. * Transitioning to opensearch. * Removing elasticsearch references. * Isolation of search code, calling services. * Adding config to support multiple instances. * Spotless * Adding helm files. * Adding in stronger password for search. * Handling debouncing. * Adding "ADVANCED_SEARCH" configurable variable for web. * Fixing some tests. * Moving indexing down a row. * Spotless * Putting back removed code. * Merge spotless resolution. * Skipping over search for db migration tests. * Adding search back to migration * Trying out ci config setting. * Removing search from base config as a whole. * Pushing out header updates. * Review comment on search service init. * Fixing up dependencies in docker to apply migrations. * Code review updates and naming changes. * newline * Updating for beta vs. non beta endpoints in search resource. * Moving search resource to its own place. * Removing prints. * Removing all helm changes for this work stream. * Adding back lock file contents. * Adding header * Adding middleware proxy. * Code review updates. * Moving from outer gradle to api gradle. * Removing extra containers. * Removing extra containers. * Set timeout for seed container to 60s Signed-off-by: wslulciuc <willy@datakin.com> * Fixing `--no-search` and frontend config. * Add check before indexing ol event Signed-off-by: wslulciuc <willy@datakin.com> * Fix db migration CI job Signed-off-by: wslulciuc <willy@datakin.com> --------- Signed-off-by: wslulciuc <willy@datakin.com> Co-authored-by: phix <peter.hicks@astronomer.io> Co-authored-by: wslulciuc <willy@datakin.com> Signed-off-by: Isa Inalcik <isa.inalcik@gmail.com>

phix added 2 commits May 30, 2024 23:46

Elasticsearch code.

f34e016

Adding basic responses for elasticsearch.

70c4ce0

boring-cyborg bot added the api API layer changes label Jun 7, 2024

phix added 3 commits June 11, 2024 13:09

Saving highlights.

51a5fa8

Saving code cleanup.

52606f4

Adding EsSearch.

c489de4

boring-cyborg bot added the web label Jun 11, 2024

phix added 17 commits June 11, 2024 19:42

Saving partial progress.

f05bfb1

Refinements.

b5667f2

Small bug fixes.

e78bafb

Fixing alignment

9f9abfe

Migrating es jobs naming to be specific.

545f7cb

Adding boilerplate for dataset es search

ec4c0c6

Adding datasets.

897f1be

Adding polish for more data.

38c8db6

Empty state and other small enhancements.

6713e1d

Adding arrow key functionality.

bb5ce2d

Removing console log

cffd9cc

Spotless

6f62898

Refinements to queries.

4a5f1aa

Adding debounce.

40bc497

Fixing alignment issues.

19c76c8

Saving updates for password setting via env config for elasticsearch.

c7f5860

Setting up startup scripts and adding corresponding waits.

fd37df9

boring-cyborg bot added the docker label Jun 26, 2024

phix added 4 commits June 27, 2024 14:49

Adding logs and more fields for jobs.

0a4cdcc

Resolving jackson serialization issue.

f9023fb

Small updates for search display.

96ab8f8

Adding onClick handlers.

e828b2b

phixMe and others added 5 commits August 6, 2024 16:26

Adding back lock file contents.

3cb90b2

Merge branch 'main' into feature/es-client

247b79d

Adding header

4158d54

Merge remote-tracking branch 'origin/feature/es-client' into feature/…

e672bf1

…es-client

Adding middleware proxy.

f05bd15

wslulciuc reviewed Aug 9, 2024

View reviewed changes

api/src/main/java/marquez/MarquezApp.java Show resolved Hide resolved

wslulciuc reviewed Aug 9, 2024

View reviewed changes

Code review updates.

61672cd

wslulciuc reviewed Aug 9, 2024

View reviewed changes

Moving from outer gradle to api gradle.

fdeaaa1

Merge branch 'refs/heads/main' into feature/es-client

f04e9b1

# Conflicts: # .env.example # docker-compose.web.yml

wslulciuc reviewed Aug 9, 2024

View reviewed changes

web/src/types/api.ts

run_id: string

name: string

namespace: string

eventType: string

Copy link

Member

wslulciuc Aug 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How's eventType used?

wslulciuc approved these changes Aug 9, 2024

View reviewed changes

phixMe and others added 4 commits August 9, 2024 18:02

Removing extra containers.

dfd60e6

Removing extra containers.

d4649ff

Resolve merge conflicts

ac5102b

Signed-off-by: wslulciuc <willy@datakin.com>

Set timeout for seed container to 60s

b83254c

Signed-off-by: wslulciuc <willy@datakin.com>

wslulciuc self-requested a review August 11, 2024 19:33

phixMe and others added 4 commits August 11, 2024 19:01

Fixing --no-search and frontend config.

73726a7

Merge remote-tracking branch 'origin/feature/es-client' into feature/…

907d96e

…es-client

Add check before indexing ol event

15c6697

Signed-off-by: wslulciuc <willy@datakin.com>

Fix db migration CI job

d096138

Signed-off-by: wslulciuc <willy@datakin.com>

wslulciuc approved these changes Aug 16, 2024

View reviewed changes

phixMe merged commit ead480b into main Aug 16, 2024
15 of 16 checks passed

phixMe deleted the feature/es-client branch August 16, 2024 23:34

wslulciuc mentioned this pull request Aug 23, 2024

Add SearchBackend #2879

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenSearch indexing and queries #2834

OpenSearch indexing and queries #2834

phixMe commented Jun 7, 2024 •

edited

Loading

netlify bot commented Jun 7, 2024 •

edited

Loading

wslulciuc Aug 9, 2024

phixMe Aug 9, 2024

wslulciuc Aug 9, 2024

wslulciuc Aug 9, 2024 •

edited

Loading

wslulciuc Aug 9, 2024

codecov bot commented Aug 9, 2024 •

edited

Loading

wslulciuc Aug 9, 2024

wslulciuc left a comment •

edited

Loading

wslulciuc left a comment

OpenSearch indexing and queries #2834

OpenSearch indexing and queries #2834

Conversation

phixMe commented Jun 7, 2024 • edited Loading

Problem

Includes

To Follow Up

Checklist

netlify bot commented Jun 7, 2024 • edited Loading

✅ Deploy Preview for peppy-sprite-186812 canceled.

wslulciuc Aug 9, 2024

Choose a reason for hiding this comment

phixMe Aug 9, 2024

Choose a reason for hiding this comment

wslulciuc Aug 9, 2024

Choose a reason for hiding this comment

wslulciuc Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

wslulciuc Aug 9, 2024

Choose a reason for hiding this comment

codecov bot commented Aug 9, 2024 • edited Loading

Codecov Report

wslulciuc Aug 9, 2024

Choose a reason for hiding this comment

wslulciuc left a comment • edited Loading

Choose a reason for hiding this comment

wslulciuc left a comment

Choose a reason for hiding this comment

phixMe commented Jun 7, 2024 •

edited

Loading

netlify bot commented Jun 7, 2024 •

edited

Loading

wslulciuc Aug 9, 2024 •

edited

Loading

codecov bot commented Aug 9, 2024 •

edited

Loading

wslulciuc left a comment •

edited

Loading