Failed to persist state to backend when I use S3 as backend!??

[fdervisi]

System Information

Linux distribution

Ubuntu 22.04

Terraform version

v1.7.3

Provider and libvirt versions

v0.7.6

---

Issue Description:

During the execution of a Terraform apply operation aimed at setting up a VyOS VM, we encountered an error related to saving the Terraform state to the S3 backend. The process was unable to persist a state snapshot, leading to a failure in saving the state. This issue occurred immediately after successfully creating the libvirt_domain.vyos_vm resource. Further investigation revealed an EOF error from the provider and a subsequent failure to upload the state to S3 due to the request stream not being seekable. As a result, Terraform suggested using terraform state push errored.tfstate to recover from this error.

Error Details:

• Initial Error: State storage *remote.State declined to persist a state snapshot.
• Provider EOF Error: Received EOF, stopping recv loop; error reading from server: EOF.
• Libvirt Provider Debug Info: Cleaning up connection for URI: qemu+ssh://kvmuser@[IP]/system?sshauth=privkey.
• S3 Upload Attempt: Backend-s3 attempted to upload remote state to customer-terraform-states bucket with path Customer-v1/kvm/vyos/NLD.tfstate.
• S3 Upload Error: Failed to upload state: operation error S3: PutObject, failed to rewind transport stream for retry, request stream is not seekable.
• Recovery Suggestion: To retry writing this state, use the command: terraform state push errored.tfstate.

Steps to Reproduce:

Run Terraform apply for the configuration targeting the creation of a VyOS VM in a KVM environment with S3 as the backend for state storage.
Observe the error during the state saving process to the S3 backend.
Expected Outcome:
Terraform should successfully save the state to the S3 backend without encountering stream-related errors.

Actual Outcome:

Terraform failed to save the state to the S3 backend due to a non-seekable stream error during the PutObject operation.

The funny thing is that sometimes it works sometimes it does not. I always delete the state in S3 to start fresh

Detailed Log Output:

2024-02-14T21:53:34.084Z [DEBUG] State storage *remote.State declined to persist a state snapshot
2024-02-14T21:53:34.085Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-02-14T21:53:34.085Z [INFO]  provider.terraform-provider-libvirt_v0.7.6: 2024/02/14 21:53:34 [DEBUG] cleaning up connection for URI: qemu+ssh://kvmuser@131.153.158.234/system?sshauth=privkey: timestamp=2024-02-14T21:53:34.085Z
libvirt_domain.vyos_vm: Creation complete after 5s [id=5d0618c2-4aaa-4cc3-8ffd-9b58e6e50431]
2024-02-14T21:53:34.236Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.7.6/linux_amd64/terraform-provider-libvirt_v0.7.6 pid=3004
2024-02-14T21:53:34.236Z [DEBUG] provider: plugin exited
2024-02-14T21:53:34.236Z [DEBUG] states/remote: state read serial is: 1; serial is: 1
2024-02-14T21:53:34.236Z [DEBUG] states/remote: state read lineage is: 79cc23e2-4445-09ae-30d5-7b83bccb0af7; lineage is: 79cc23e2-4445-09ae-30d5-7b83bccb0af7
2024-02-14T21:53:34.238Z [INFO]  backend-s3: Uploading remote state: tf_backend.operation=Put tf_backend.req_id=d066bb77-3978-c982-afa7-ca82e6a90fb3 tf_backend.s3.bucket=customer-terraform-states tf_backend.s3.path=Customer-v1/kvm/vyos/NLD.tfstate
2024-02-14T21:53:34.238Z [DEBUG] backend-s3: HTTP Request Sent: aws.region=*** aws.s3.bucket=customer-terraform-states aws.s3.key=Customer-v1/kvm/vyos/NLD.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_aws.signing_region="" tf_backend.operation=Put tf_backend.req_id=d066bb77-3978-c982-afa7-ca82e6a90fb3 tf_backend.s3.bucket=customer-terraform-states tf_backend.s3.path=Customer-v1/kvm/vyos/NLD.tfstate http.request.header.x_amz_sdk_checksum_algorithm=SHA256 http.request.header.x_amz_trailer=x-amz-checksum-sha256 http.request.header.x_amz_server_side_encryption=AES256 http.request.header.amz_sdk_invocation_id=650dea4c-c1cc-4b28-b0ca-2747557a1cf4 http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.7.3 (+https://www.terraform.io) aws-sdk-go-v2/1.24.0 os/linux lang/go#1.21.5 md/GOOS#linux md/GOARCH#amd64 api/s3#1.47.5 ft/s3-transfer" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=AKIA************EE3C/20240214/***/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-server-side-encryption;x-amz-trailer, Signature=*****" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_content_sha256=STREAMING-UNSIGNED-PAYLOAD-TRAILER net.peer.name=customer-terraform-states.s3.***.amazonaws.com http.request.body="[Redacted: 37.8 KB (38,736 bytes), Type: application/json]" http.request.header.accept_encoding=identity http.url=https://customer-terraform-states.s3.***.amazonaws.com/Customer-v1/kvm/vyos/NLD.tfstate?x-id=PutObject http.request_content_length=38736 http.request.header.x_amz_decoded_content_length=38655 http.request.header.x_amz_date=20240214T215334Z http.request.header.content_type=application/json http.request.header.content_encoding=aws-chunked http.method=PUT
╷
│ Error: Failed to save state
│ 
│ Error saving state: failed to upload state: operation error S3: PutObject,
│ failed to rewind transport stream for retry, request stream is not seekable
╵
╷
│ Error: Failed to persist state to backend
│ 
│ The error shown above has prevented Terraform from writing the updated
│ state to the configured backend. To allow for recovery, the state has been
│ written to the file "errored.tfstate" in the current working directory.
│ 
│ Running "terraform apply" again at this point will create a forked state,
│ making it harder to recover.
│ 
│ To retry writing this state, use the following command:
│     terraform state push errored.tfstate
│ 
╵
Error: Process completed with exit code 1.