-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(ingest/dremio): Dremio Source Ingestion #11598
base: master
Are you sure you want to change the base?
feat(ingest/dremio): Dremio Source Ingestion #11598
Conversation
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_datahub_source_mapping.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_datahub_source_mapping.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please make use of report.warning
and/or report.error
for cases where user needs to be aware of error details - such as login failure due to missing PAT or incorrect password ?
01ffbcd
to
bd1ad49
Compare
c4fe5c5
to
084ce54
Compare
084ce54
to
e1ee817
Compare
…ar-salvi-apptware/datahub into feat/dremio-connector-source
1a27cb9
to
9c1e3fb
Compare
3d2c4fa
to
6914f75
Compare
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_aspects.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_aspects.py
Outdated
Show resolved
Hide resolved
dataset_pattern: AllowDenyPattern = Field( | ||
default=AllowDenyPattern.allow_all(), | ||
description="Regex patterns for schemas to filter", | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like schema_pattern uses TABLE_SCHEMA
column which is fully qualified so this is fine.
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mostly good to get in. A few minor things around tests and a decision pending on profiling.
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_aspects.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_entities.py
Outdated
Show resolved
Hide resolved
logger = logging.getLogger(__name__) | ||
|
||
|
||
class DremioProfiler: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be good to integrate DatahubGEProfiler
here instead to make use of other improvements(multithreading, query optimisation, etc) around it. Not sure if this needs to be done right away. cc: @hsheth2 to weigh in.
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_aspects.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_aspects.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_config.py
Outdated
Show resolved
Hide resolved
metadata-ingestion/src/datahub/ingestion/source/dremio/dremio_source.py
Outdated
Show resolved
Hide resolved
ed91d1f
to
7433b8b
Compare
Checklist