feat(ingest/dremio): Dremio Source Ingestion #11598

sagar-salvi-apptware · 2024-10-11T11:52:24Z

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_datahub_source_mapping.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py

mayurinehate

Can you please make use of report.warning and/or report.error for cases where user needs to be aware of error details - such as login failure due to missing PAT or incorrect password ?

…ar-salvi-apptware/datahub into feat/dremio-connector-source

metadata-ingestion/docs/sources/dremio/dremio_recipe.yml

datahub-web-react/src/app/ingest/source/builder/sources.json

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_api.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_aspects.py

mayurinehate · 2024-10-25T15:07:33Z

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py

+    dataset_pattern: AllowDenyPattern = Field(
+        default=AllowDenyPattern.allow_all(),
+        description="Regex patterns for schemas to filter",
+    )


Looks like schema_pattern uses TABLE_SCHEMA column which is fully qualified so this is fine.

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py

mayurinehate

Mostly good to get in. A few minor things around tests and a decision pending on profiling.

datahub-web-react/src/app/ingest/source/builder/sources.json

metadata-ingestion/docs/sources/dremio/dremio_recipe.yml

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_aspects.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_entities.py

mayurinehate · 2024-10-28T09:16:15Z

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_profiling.py

+logger = logging.getLogger(__name__)
+
+
+class DremioProfiler:


Would be good to integrate DatahubGEProfiler here instead to make use of other improvements(multithreading, query optimisation, etc) around it. Not sure if this needs to be done right away. cc: @hsheth2 to weigh in.

metadata-ingestion/tests/integration/dremio/dremio_to_file.yml

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_aspects.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py

metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py

feat: Dremio Source Ingestion

38e5946

github-actions bot added ingestion PR or Issue related to the ingestion of metadata docs Issues and Improvements to docs product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment labels Oct 11, 2024

vercel bot deployed to Preview October 11, 2024 12:11 View deployment

sagar-salvi-apptware added 4 commits October 11, 2024 18:48

refactor: improvement to metadata gathering

a5b8c45

fix: Update dremio_entity.py

4a81b19

fix: Update dremio_sql_queries.py

51c2443

refactor: Update dremio_source.py

84227b6

sagar-salvi-apptware changed the title ~~feat: Dremio Source Ingestion~~ feat(ingest/dremio): Dremio Source Ingestion Oct 11, 2024

vercel bot deployed to Preview October 11, 2024 13:42 View deployment

mayurinehate reviewed Oct 17, 2024

View reviewed changes

mayurinehate requested changes Oct 17, 2024

View reviewed changes

vercel bot deployed to Preview October 17, 2024 11:37 View deployment

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from 01ffbcd to bd1ad49 Compare October 17, 2024 16:29

sagar-salvi-apptware added 6 commits October 17, 2024 23:23

feat: Dremio Source Ingestion

e462be6

refactor: improvement to metadata gathering

703a3f8

fix: Update dremio_entity.py

4283cb6

fix: Update dremio_sql_queries.py

f897b9e

refactor: Update dremio_source.py

ac56777

test: add integration test for dremio

3ce57b9

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from c4fe5c5 to 084ce54 Compare October 17, 2024 17:53

vercel bot deployed to Preview October 17, 2024 18:21 View deployment

fix: added minor changes + fix testcase

e1ee817

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from 084ce54 to e1ee817 Compare October 18, 2024 06:32

vercel bot deployed to Preview October 18, 2024 06:54 View deployment

sagar-salvi-apptware and others added 3 commits October 21, 2024 13:41

fix: PR Comments

b23e003

Merge branch 'feat/dremio-connector-source' of https://github.com/sag…

7247ac1

…ar-salvi-apptware/datahub into feat/dremio-connector-source

switch to drill dialect

b9d7b8a

mayurinehate reviewed Oct 25, 2024

View reviewed changes

metadata-ingestion/docs/sources/dremio/dremio_recipe.yml Show resolved Hide resolved

fix: PR comments

9c1e3fb

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from 1a27cb9 to 9c1e3fb Compare October 25, 2024 10:22

vercel bot deployed to Preview October 25, 2024 10:52 View deployment

docs: minor changes

7b938b2

vercel bot deployed to Preview October 25, 2024 11:49 View deployment

build: added dependacy for sql

6914f75

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from 3d2c4fa to 6914f75 Compare October 25, 2024 11:55

vercel bot deployed to Preview October 25, 2024 12:28 View deployment

test: fix ci test

3176e71

vercel bot deployed to Preview October 25, 2024 14:14 View deployment

fix: add minor change in report failure

9c1c556

mayurinehate reviewed Oct 25, 2024

View reviewed changes

fix: statefull ingestion error

6670a98

vercel bot deployed to Preview October 25, 2024 16:03 View deployment

fix: PR Comments and added mysql as a source to test

3e2eae7

vercel bot had a problem deploying to Preview October 27, 2024 10:58 Failure

mayurinehate reviewed Oct 28, 2024

View reviewed changes

sagar-salvi-apptware and others added 3 commits October 28, 2024 17:02

fix: PR Comments

195f78c

test: Updated the test for platform instance

9de8be0

Merge branch 'master' into feat/dremio-connector-source

102b55f

vercel bot deployed to Preview October 28, 2024 12:26 View deployment

mayurinehate reviewed Oct 28, 2024

View reviewed changes

sagar-salvi-apptware added 2 commits October 28, 2024 20:06

fix: minor comments

8efa349

test: fix ci test

82988fd

mayurinehate self-requested a review October 28, 2024 14:58

vercel bot deployed to Preview October 28, 2024 15:12 View deployment

fix: added fixes for dremio cloud apis

7433b8b

sagar-salvi-apptware force-pushed the feat/dremio-connector-source branch from ed91d1f to 7433b8b Compare October 28, 2024 16:35

vercel bot deployed to Preview October 28, 2024 17:13 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ingest/dremio): Dremio Source Ingestion #11598

feat(ingest/dremio): Dremio Source Ingestion #11598

sagar-salvi-apptware commented Oct 11, 2024

mayurinehate left a comment

mayurinehate Oct 25, 2024

mayurinehate left a comment

mayurinehate Oct 28, 2024

		logger = logging.getLogger(__name__)


		class DremioProfiler:

feat(ingest/dremio): Dremio Source Ingestion #11598

Are you sure you want to change the base?

feat(ingest/dremio): Dremio Source Ingestion #11598

Conversation

sagar-salvi-apptware commented Oct 11, 2024

Checklist

mayurinehate left a comment

Choose a reason for hiding this comment

mayurinehate Oct 25, 2024

Choose a reason for hiding this comment

mayurinehate left a comment

Choose a reason for hiding this comment

mayurinehate Oct 28, 2024

Choose a reason for hiding this comment