Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest/transform): extend ownership transformer to other entities #11700

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

anshbansal
Copy link
Collaborator

@anshbansal anshbansal commented Oct 23, 2024

  • extend pattern cleanup transformer to other entities
  • future suggestion - might be good to auto generate list of entities that have ownership aspects. Will allow us to extend all ownership transformers very easily. Added some common entities for now

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@anshbansal anshbansal marked this pull request as draft October 23, 2024 12:25
@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Oct 23, 2024
Copy link

Hello @anshbansal 😄

Thank you so much for opening a pull request!

Image
You can check out your contributor card and see all your past stats here!

@skrydal skrydal self-requested a review October 24, 2024 10:32
Copy link
Collaborator

@skrydal skrydal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think overall we should refactor our transformers, it's strange that we are having more generic transformer OwnershipTransformer inheriting from less generic one DatasetTransformer - it will of course work, but I think we should consider, not as a part of this PR, to refactor all our transformers and allow for other entities to be transformed (there are other requests related to, for example, tagging other entities than datasets).
We really need at least one test case for the new transformer though.
Also shouldn't we add new transformer to the list of transformers in setup.py?

@anshbansal
Copy link
Collaborator Author

@skrydal There is no new transformer. It's name has not been changed. It is the same transformer so no need to change in setup.py

@skrydal
Copy link
Collaborator

skrydal commented Oct 24, 2024

@skrydal There is no new transformer. It's name has not been changed. It is the same transformer so no need to change in setup.py

You are right, sorry. So the essence of the change here is that AddDatasetOwnership now will be adding ownership to other entities too. Won't it be misleading - both in terms of name, as well as changing the behavior of an existing transformer - users might not expect that. Why don't we create a generalized AddOwnershipTransformer?

Also, I understand that in the test case if we kept the DataJob we would have 2 more aspects in the output - first that of DataJobInfo and second Ownership (for dataJob this time). Wouldn't it exemplify the nature of the change introduced better?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ingestion PR or Issue related to the ingestion of metadata
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants