Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

azurerm_storage_share_file content_md5 incompatible with filemd5() #25704

Open
1 task done
w3bward opened this issue Apr 22, 2024 · 1 comment 路 May be fixed by #25715
Open
1 task done

azurerm_storage_share_file content_md5 incompatible with filemd5() #25704

w3bward opened this issue Apr 22, 2024 · 1 comment 路 May be fixed by #25715

Comments

@w3bward
Copy link

w3bward commented Apr 22, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Community Note

  • Please vote on this issue by adding a 馃憤 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.7.1

AzureRM Provider Version

3.99.0

Affected Resource(s)/Data Source(s)

azurerm_storage_share_file

Terraform Configuration Files

resource "azurerm_storage_share_file" "local_settings_php" {
  name             = "LocalSettings.php"
  storage_share_id = azurerm_storage_share.example.id
  source           = "./files/LocalSettings.php"
  content_type     = "text/plain"
  content_md5      = filemd5("./files/LocalSettings.php")
}

Debug Output/Panic Output

See "Actual Behavior"

Expected Behaviour

Azure Storage, and tools using Azure Storage expect the bytes of an MD5 content hash to be base64 encoded. Terraform's native filemd5() function outputs the hash as a string of hexadecimal characters. Terraform does not have any native tools to support properly converting the byte value of filemd5() to base64.

Considering that setting content_md5 with the filemd5() function appears to be the only Terraform native means of tracking and planning changes to the source file for the azurerm_storage_share_file resource, content_md5 should either properly convert the output of filemd5() or an alternative argument such as content_md5_hex should be added that performs the appropriate conversion internally within the provider.

Actual Behaviour

azurerm_storage_share_file set the ContentMD5 property of the file to the literal string output from filemd5 which causes hash mismatch errors for other clients (e.g. Azure Storage Explorer, AzCopy) trying to download the file.

Terraform has no built in functions or tools that I've been able to find to support calculating an MD5 hash and outputting it as base64, nor any functions that would support converting the byte value of the hex output by filemd5 to base64. Terraform's native base64encode only converts the text value of the hex to base64, which also results in an inconsistent value.

Terraform filemd5() returns the hash encoded as hexadecimal:
SNAG-4-22-2024 9 48 57 AM

On apply, the Azure provider sets the md5 hash using the literal string value of the hexadecimal:
SNAG-4-22-2024 9 53 45 AM

The file fails an integrity check when downloaded via Azure Storage Explorer:
SNAG-4-22-2024 10 00 38 AM

A file uploaded via Azure Storage Explorer has the same MD5 checksum, but the bytes have been encoded as base64. I have confirmed the match when doing a proper hex -> byte -> base64 conversion on the output of filemd5():
SNAG-4-22-2024 10 03 25 AM

Steps to Reproduce

  1. Create a terraform configuration with an azurerm_storage_share_file resource that sets content_md5 using the filemd5 function within terraform.
  2. Apply the configuration.
  3. Attempt to download the file using Azure Storage Explorer, and see the data integrity error in the logs

Important Factoids

No response

References

The most official documentation on MD5 encoding for Azure Storage that I was able to find was in the AzCopy github project, which states:

Note that, in Azure, the Content-MD5 blob property (where the hash is stored) is not just the raw bytes out of the MD5 algorithim. Instead, to produce a valid Azure Content-MD5, you must take the raw bytes returned by the MD5 algorithm, and base64 encode them. AzCopy does this automatically. This paragraph is just for users who want to do their own hash computations and compare to those produced by AzCopy

@rcskosir
Copy link
Contributor

Thank you for taking the time to open this issue. Please subscribe to PR #25715 created by @magodo for this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants