Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 2.6 Validations Failures #6077

Open
atalman opened this issue Dec 17, 2024 · 1 comment
Open

Release 2.6 Validations Failures #6077

atalman opened this issue Dec 17, 2024 · 1 comment

Comments

@atalman
Copy link
Contributor

atalman commented Dec 17, 2024

Looks like there are multiple issues revealed with Release 2.6 validation:

Runs:
https://github.com/pytorch/test-infra/actions/runs/12375056109/job/34539017465
https://github.com/pytorch/audio/actions/runs/12359503259/job/34537783507

  1. AArch64 cpu and CUDA failures, except python 3.12 :
import setuptools
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
ModuleNotFoundError: No module named 'setuptools'
@atalman
Copy link
Contributor Author

atalman commented Dec 17, 2024

Resolved Issues:

  1. Rocm 6.2.4 failure to download the binary - RESOLVED
++ eval pip3 install --force-reinstall torch --index-url https://download.pytorch.org/whl/test/rocm6.2.4
+++ pip3 install --force-reinstall torch --index-url https://download.pytorch.org/whl/test/rocm6.2.4
Looking in indexes: https://download.pytorch.org/whl/test/rocm6.2.4
ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
ERROR: No matching distribution found for torch

deployed
#6071
and
#6080

  1. Rocm 6.1 failure gxx ABI 0 not expected - RESOLVED
+++ echo 'gcc ABI 0 not as expected.'
gcc ABI 0 not as expected.

deployed #6080

  1. XPU linux - RESOLVED
+++ echo 'gcc ABI 1 not as expected.'
gcc ABI 1 not as expected.

Should be fixed by: #6078

  1. Windows failures unable to checkout repo
    Caused by [ROCm] CK Flash Attention Backend pytorch#138947 (comment) will probably need to be reverted

  2. Torch compile failure on py3.13:

esting smoke_test_compile for cuda and torch.float16
Traceback (most recent call last):
  File "/pytorch/pytorch/.ci/pytorch/./smoke_test/smoke_test.py", line 385, in <module>
    main()
    ~~~~^^
  File "/pytorch/pytorch/.ci/pytorch/./smoke_test/smoke_test.py", line 379, in main
    smoke_test_cuda(
    ~~~~~~~~~~~~~~~^
        options.package, options.runtime_error_check, options.torch_compile_check
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/pytorch/pytorch/.ci/pytorch/./smoke_test/smoke_test.py", line 186, in smoke_test_cuda
    smoke_test_compile("cuda" if torch.cuda.is_available() else "cpu")
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pytorch/pytorch/.ci/pytorch/./smoke_test/smoke_test.py", line 286, in smoke_test_compile
    x_pt2 = torch.compile(foo)(x)
  File "/opt/conda/envs/conda-env-12375056109/lib/python3.13/site-packages/torch/_dynamo/eval_frame.py", line 573, in _fn
    return fn(*args, **kwargs)
  File "/opt/conda/envs/conda-env-12375056109/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1380, in __call__
    return self._torchdynamo_orig_callable(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        frame, cache_entry, self.hooks, frame_state, skip=1
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/conda/envs/conda-env-12375056109/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1164, in __call__
    result = self._inner_convert(
        frame, cache_entry, hooks, frame_state, skip=skip + 1
    )
  File "/opt/conda/envs/conda-env-12375056109/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 547, in __call__
    return _compile(
        frame.f_code,
    ...<14 lines>...
        skip=skip + 1,
    )
  File "/opt/conda/envs/conda-env-12375056109/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 986, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/opt/conda/envs/conda-env-12375056109/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 715, in compile_inner
    return _compile_inner(code, one_graph, hooks, transform)
  File "/opt/conda/envs/conda-env-12375056109/lib/python3.13/site-packages/torch/_utils_internal.py", line 95, in wrapper_function
    return function(*args, **kwargs)
  File "/opt/conda/envs/conda-env-12375056109/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 750, in _compile_inner
    out_code = transform_code_object(code, transform)
  File "/opt/conda/envs/conda-env-12375056109/lib/python3.13/site-packages/torch/_dynamo/bytecode_transformation.py", line 1361, in transform_code_object
    transformations(instructions, code_options)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Need to rebuild the rc. Triton installation is missing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

1 participant