Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tailing kubernetes logs doesn't put logs on new lines #983

Open
1 of 10 tasks
matthen opened this issue Dec 6, 2024 · 2 comments
Open
1 of 10 tasks

Tailing kubernetes logs doesn't put logs on new lines #983

matthen opened this issue Dec 6, 2024 · 2 comments

Comments

@matthen
Copy link

matthen commented Dec 6, 2024

🐛 Bug

When running torchx log --tail with a kubernetes job, the logs appear all on one line, with no new lines between them.

Module (check all that applies):

  • torchx.spec
  • torchx.component
  • torchx.apps
  • torchx.runtime
  • torchx.cli
  • torchx.schedulers
  • torchx.pipelines
  • torchx.aws
  • torchx.examples
  • other

To Reproduce

Steps to reproduce the behavior:

torchx log --tail kubernetes:${JOB_NAME}

output on terminal looks like:

name/0 2024-12-06T09:57:30.580070417Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Using CUDA driver version 12060name/0 2024-12-06T09:57:30.580072867Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Configuring AWS-specific optionsname/0 2024-12-06T09:57:30.580075167Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Setting provider_filter to efaname/0 2024-12-06T09:57:30.580077497Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1name/0 2024-12-06T09:57:30.580079757Z [0]:torchxgpu-bn14n59m2fr1jd-name-0-0:16:36 [0] NCCL INFO NET/OFI Setting NCCL_NVLSTREE_MAX_CHUNKSIZE to 512KiB

(but running without --tail puts these all on new lines)

Environment

  • torchx version (e.g. 0.1.0rc1): torchx-0.7.0
  • Python version: 3.11.0rc1
  • OS (e.g., Linux): ubuntu
  • How you installed torchx (conda, pip, source, docker): pip
  • Docker image and tag (if using docker): /
  • Git commit (if installed from source): /
  • Execution environment (on-prem, AWS, GCP, Azure etc): local macbook in Terminal app
  • Any other relevant information:
@Sanjay-Ganeshan
Copy link
Contributor

Can you try with a version built from source? I think this might already be addressed

@matthen
Copy link
Author

matthen commented Dec 6, 2024

Thanks for suggestion! I get the same issue after doing pip install -e git+https://github.com/pytorch/torchx.git#egg=torchx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants