Disable StalledStreamProtection for S3 CopyObject #1141
Comments
mlartz
added
bug
|
PutObject also has this issue. Wasted hours on this how did it get through the tests? Increase priority of this. It occurs over a high latency connection with a large payload. |
|
Offender: Seems related to upload only. |
|
Workaround: I don't like this feature anyway, so ill keep it off and let default timeouts do their job. The consequence of this is that it dramatically increases the chance the upload succeeds but the client still reports a failure. |
|
I'm sorry to hear that this feature hasn't been helpful for you. Timeouts and stalled stream protection both serve different purposes. We chose to enable both by default because we believe that it provides a better user experience for the majority of users. Timeouts and stalled stream protection is generally a benefit. However, you've run into the one case where both may not be helpful: requests that are expected to be very long-running. Timeouts set a timer that may be too short for the request you want to send. Stalled stream protection is better for long-running requests because it's only checking that the transfer is still ongoing, but it will close a connection that sees no activity for the grace period (five seconds by default.) Instead of disabling stalled stream protection, I'd advise you to fiddle with the One more thing: I've written a guide on configuring stalled streams that's available here. Let me know if you'd like me to add anything to it. |
|
I think the best thing to do in this case is to automatically disable stalled stream protection for any API that's modeled without the |
|
I've added this issue to our backlog. |
|
I think the implementation is broken, if it was implemented correctly then it would probably make sense. Looking at This error may occur on any upload that takes longer than the grace period since its not counting those bytes...? |
|
Upload throughput is here. |
|
What about during a long running upstream request and there were no bytes downstream for a while this is occurring? |
|
Again, this really isn't something that should be done at the application layer, do other language AWS SDK's support it? TCP keep alive at the transport layer is more applicable and let application layer do its normal timeout. |
|
This should be P1. It breaks all upload API's that take longer than the grace periods - always. |
aajtodd
added
p1
…lete (#3644) ## Motivation and Context <!--- Why is this change required? What problem does it solve? --> <!--- If it fixes an open issue, please link to the issue here --> * awslabs/aws-sdk-rust#1141 * awslabs/aws-sdk-rust#1146 * awslabs/aws-sdk-rust#1148 ## Description * Disables stalled stream upload protection for requests with an empty/zero length body. * Disables stalled stream upload throughput checking once the request body has been read and handed off to the HTTP layer. ## Testing Additional integration tests added covering empty bodies and completed uploads. Tested SQS issue against latest runtime and can see it works now. The S3 `CopyObject` issue is related to downloads and will need a different solution. ## Checklist <!--- If a checkbox below is not applicable, then please DELETE it rather than leaving it unchecked --> - [x] I have updated `CHANGELOG.next.toml` if I made changes to the smithy-rs codegen or runtime crates - [x] I have updated `CHANGELOG.next.toml` if I made changes to the AWS SDK, generated SDK code, or SDK runtime crates ---- _By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice._ --------- Co-authored-by: Zelda Hessler <zhessler@amazon.com> Co-authored-by: ysaito1001 <awsaito@amazon.com>
|
We just released the runtime crate updates: https://github.com/smithy-lang/smithy-rs/releases/tag/release-2024-05-21 |
|
Let us know whether today's release has improved the behavior of your use case. |
Describe the bug
In running a S3
CopyObjectof a fairly large object (2+ GB), which takes some time (30s+) to complete, the Rust SDK runs into the following ThroughputBelowMinimum error, which gets retried a couple of times (3 total attempts) and always fails:The S3 CopyObject eventually succeeds as I see the copied file in the new location, but the Rust SDK keeps "failing".
I was thinking that this was a timeout, but also see:
In talking to the experts, this is a result of StalledStreamProtection, which doesn't seem valid for potentially long-running "silent" operations like S3's
CopyObject. Could you disable this protection by default for (at least) S3's CopyObject?Expected Behavior
Long-running AWS API calls (like S3 CopyObject) do not get terminated unless associated with a timeout configuration.
Current Behavior
Long-running S3 CopyObject API call fails due to StalledStreamProtection
Reproduction Steps
Execute a long-running API call, such as a S3 CopyObject on a large file
Possible Solution
Disable StalledStreamProtection for potentially long-running (and "silent") AWS API calls.
Additional Information/Context
No response
Version
Environment details (OS name and version, etc.)
Amazon Linux 2, AWS Lambda
Logs
No response
The text was updated successfully, but these errors were encountered: