-
Notifications
You must be signed in to change notification settings - Fork 619
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CLI]: Wandb agent fails with no log #7444
Comments
Hey @arkadiusz-czerwinski , thanks for writing to W&B support. We'll investigate this on our end and get back to you soon. |
Hey @arkadiusz-czerwinski ,
launch_queue_error_modal.movPlease let me know if this video helps locating the stack trace on your end. |
Hi @arkadiusz-czerwinski , I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved. |
Hi. Thank you for giving me a reminder. In case of our setup, the script starts with wandb init. The error, however, states that the script failed before reaching the wandb init, although highest verbose settings were selected. @anmolmann |
Hi @arkadiusz-czerwinski , could you share your queue config and the script as well? This will help us in reproducing this issue on our end for further investigation. In addition to the above, launch jobs in the run queue do not always show the underlying error message especially when the job fails before init - we made progress here in the last few months as seen in the video shared above, however, I wouldn't be able to guarantee that we will extract the exact right failure message each time. We'll keep working on improvements and investigate the case you brought up, though I don't anticipate us achieving perfection in that regard. |
Fair point. The code will be visible in this repo. gpus: all
label:
- tutorial
volume:
- /mnt/space:/mnt/space The main issue is that the error is not very descriptive, and the agent provides no error even after specifying verbose level. |
Update: the issue seems to be that during creation of a job from git, entry point is required, but is then not passed forward to the wandb, which caused the error, as it had to be defined in Dockerfile.wandb. |
I see, thanks for the context @arkadiusz-czerwinski . I will create a feature request to improve the error catching functionality for launch. |
Thank you for being so cooperative. |
Describe the bug
After setting up wandb launch environment, successfully adding job to queue with
wandb job create
, the run fails, with no feedback other than.: "The submitted job exited successfully but failed to call wandb.init". There is no feedback in the agent terminal.The repo is available here, with the code and Dockerfile.
Additional Files
No response
Environment
WandB version: 0.16.6
OS:
Python version:
Versions of relevant libraries:
Additional Context
No response
The text was updated successfully, but these errors were encountered: