Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for recursive datatypes #866

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

abhi-glitchhg
Copy link

@abhi-glitchhg abhi-glitchhg commented Oct 28, 2022

since release of mypy 0.981 recursive types are supported; i have just removed the # as per suggestion in the TODO comment and have changed the mypy version in the ci jobs.

This PR is to check if ci is breaking with the changes are not. fixes #640

Let me know if any further changes need to be made
thanks..

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 28, 2022
Copy link
Contributor

@NivekT NivekT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ejguan Any issue form upgrading mypy to a newer version? Aside from fixing the lints?

@abhi-glitchhg
Copy link
Author

Hmm, mypy is raising one error on this line

returning TypeVar should receive at least one argument containing the same
Typevar  [type-var]
Found 1 error in 1 file (checked 76 source files)
        def draw(self) -> T:

def draw(self) -> T:
selected_key = self._rng.choices(self.keys, self.weights)[0]

How can i fix this? can anyone help me out here?

@NivekT
Copy link
Contributor

NivekT commented Nov 7, 2022

Based on this issue, it seems to be warning that a single TypeVar will not do anything. I think you can either remove -> T or suppress the warning.

@NivekT NivekT requested a review from ejguan November 8, 2022 17:23
@facebook-github-bot
Copy link
Contributor

@NivekT has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Copy link
Contributor

@ejguan ejguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution. The PR looks good. However, I am sorry that I have to block this PR for now until PyTorch Core has updated their pinned mypy version from 0.960 to 0.981.

The main problem is that torchdata is still partially in the tree of PyTorch. And PyTorch has strong opinion on the version of mypy. See: https://github.com/pytorch/pytorch/blob/6bb7f4f29f0a36ce410ce53d824f531eaf74c76e/mypy.ini#L5
If we make this PR landed, in order to develop torchdata without mypy complain on my local environment, I have to update mypy to 0.981. However, if I need to work on something in PyTorch Core, the updated mypy will prevent me running static type checking on PyTorch.

IMHO, to land this PR, we accomplish one of the following 2 options:

  • Make TorchData fully out of PyTorch
  • Update PyTorch's mypy requirement

LMK if it makes sense. And, please correct me if anything wrong or we have other options.

@abhi-glitchhg
Copy link
Author

abhi-glitchhg commented Nov 9, 2022

hello @ejguan! Thanks for the feedback.

Make TorchData fully out of PyTorch

Honestly, I do not know the consequences and the amount of work required for this but the second option seems easier than the first.

Anyway, there is no hurry from my side; I just noticed the TODO issue and was aware of the recent mypy version release.

We could try updating the pytorch's mypy version. Are there any existing blockers in doing this that you are aware of?

Would love to hear your thoughts.
Thanks!

@@ -64,7 +64,7 @@ jobs:
run: |
set -eux
STATUS=
if ! mypy --config=mypy.ini; then
if ! mypy --config=mypy.ini --enable-recursive-aliases --install-types --non-interactive; then
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also like to point out that, --install-types --non-interactive will install the missing package stubs hence slower.

Quoting relevant part from the above documentation link :- This is somewhat slower than explicitly installing stubs before running mypy, since it may type check your code twice – the first time to find the missing stubs, and the second time to type check your code properly after mypy has installed the stubs.

I don't think downloading the package stubs would be a bottleneck; but if it is, then we could think of using cached installations. (just a suggestion)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for providing the idea. I think it's fine for now as torchdata is still a small package. We might want to investigate if we have noticed significant overhead.

@ejguan
Copy link
Contributor

ejguan commented Nov 9, 2022

BTW, regarding the recursive type hints, 0.981 is not the best version as it's an experimental feature. 0.990 is the version that officially supports recursive type hints.

@ejguan
Copy link
Contributor

ejguan commented Nov 9, 2022

Let's wait until PyTorch has officially updated mypy

@abhi-glitchhg
Copy link
Author

abhi-glitchhg commented Nov 9, 2022

BTW, regarding the recursive type hints, 0.981 is not the best version as it's an experimental feature. 0.990 is the version that officially supports recursive type hints.

I was just going to mention this.

@abhi-glitchhg abhi-glitchhg marked this pull request as ready for review October 25, 2023 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[TODO] TFRecordLoader - uncomment as soon as mypy supports recursive type
4 participants