New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KeyError: 'Message' when encountering an error in _send_metrics #4482
Comments
Hi sziem, are you repeatedly seeing this issue? If so can you share some sample code that we can use to replicate this. |
Hi, thanks for your reply. After seeing this about 2-3 times, I wrapped my calls in a try-except and just ignored it, so I'm not sure if this is still an issue, sorry. Also, it's been a while since I looked at it. As I said above, it is a bit hard to create a minimal example for the issue, because of the large time delay until it occurs. Unfortunately, I'm not at liberty to share my code. But the way I've been using log_metrics is like this:
Then there must have been something (maybe a connection error?) that caused send_metrics to fail at some point. |
Seems like an issue with the sdk.
This statement needs to reference "Code" instead of "Message". As that is what the api returns (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-metrics/client/batch_put_metrics.html) It would still error out in the next line:
but the error message would be more helpful. |
Yes I agree. That should be the fix and the correct behavior. |
Describe the bug
When an error occurs while calling
run.log_metric
, it does not show the error message, but a KeyError.To reproduce
It is a bit hard for me to describe this as it occured randomly after working for 42 epochs.
Expected behavior
Get a message of the actual Error cause.
Screenshots or logs
System information
A description of your system. Please provide:
Additional context
The text was updated successfully, but these errors were encountered: