Mandrill: Attachment file name characters garbled

[Thorbenl]

Hey Anymailnauts :)
We have recently introduced sending attachments to our customers. Specifically, receipts as pdfs. We have encountered one small problem regarding Macs Mail Client.. (who would have thought..)
Our filename contains special characters, as the german ö, ä and ü or the danish ø, æ and å.
Gmail correctly shows the filename, without issues, Office365 also does, just not while using the native Mac client. Sadly, a lot of customers use tha client, and we want to try to support it. Hoewever, I am a bit lost into why this happens. I have tried to investigate the email that got sent using mandrill in gmail a bit. And I found this:

--_av-d56RGzgCJLMLL--5b-KtDg--
--_av-GOTuZ3AI5IdoZ8M47HxDCg--
--_av-yPHP6N86pEcJ0_j6AyoVCA
Content-Type: application/pdf
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="Rechnung - Møl"

This is how it looks in the Mac Mail Client, which problably doesnt support base64.

Any tip for me, on how to solve this? Can we set Content-Transfer-Encoding somehow?
Thanks!

[medmunds]

Ah, welcome to the complex and often frustrating world of email charset encodings.

The short version is, you'll probably need to contact MailChimp support. It sounds like they're generating not-really-spec-compliant email messages. And while Gmail overlooks the sloppiness, some other clients don't (or rather, they guess wrong about how to interpret the data Mandrill sent).

Here's the long version…

This is probably the root of your problem:

Content-Disposition: attachment; filename="Rechnung - Møl"

That's a UTF-8 encoded filename. Why would that be a problem? Because there's nothing that says it's using UTF-8 encoding. (And also because you can't reliably transport raw 8-bit data through SMTP, even in 2022.)

Gmail assumes unknown text is in UTF-8 (or maybe has some clever heuristics to identify charset). The Mac mail client probably assumes unknown text is ISO-8859-1, which is why it's displaying the wrong characters.

Here's how RFC 2231 says that attachment header should be sent:

Content-Disposition: attachment; filename*=utf-8''Rechnung%20-%20M%C3%B8l

Notice the *=utf-8'' that specifies the charset for the filename text. (Also notice the "ø" was %-encoded to %C3%B8, to avoid troublesome 8-bit data that might get garbled passing through various SMTP servers and email content scanners on its way to your inbox.)

So this seems like a bug in how Mandrill is composing the outgoing email. I don't see anything in their API docs that you could use to work around it. (Unfortunately, I can't test it myself, due to MailChimp's policies against providing development access for projects like Anymail.) When you contact their support, it may be helpful to eliminate Anymail and Django from the picture. You can use Anymail's DEBUG_API_REQUESTS setting to capture the raw Mandrill API call Anymail makes.

BTW, the other two attachment headers refer to the data that goes into the attachment file itself (but not the attachment's filename). Content-Transfer-Encoding: base64 means the PDF file data is sent base64 encoded, which is the correct thing to do, dates back to the earliest days of email attachments, and so is supported by pretty much every email client.

Finally, if that wasn't enough information for you, I came across a helpful 2017 blog post The mess that is attachment filenames that goes into detail on all of this and more.

[Thorbenl]

First and foremost, thank you so much for the elaborate answer. I could not have asked for more. I appreciate it!

However, I used DEBUG_API_REQUESTS, but the output from Mandrills API is useless in my eyes:

----- Response
HTTP 200 OK
server: nginx/1.4.6 (Ubuntu)
date: Wed, 19 Jan 2022 07:46:11 GMT
content-type: application/json; charset=utf-8
transfer-encoding: chunked
access-control-allow-origin: *
access-control-allow-methods: POST, GET, OPTIONS
access-control-allow-headers: Content-Type
access-control-allow-credentials: false
content-encoding: gzip
vary: Accept-Encoding

[{"email":"turbzcoding@gmail.com","status":"queued","_id":"234234234","reject_reason":null}]

Thanks Mandrill, I guess :D

I am currently in contact in their chat, as we speak. Maybe they can give me some more information into this. I will update this in a few hours, after I have more information!

[medmunds]

Ah, I was thinking you might need the raw API request (not response), which should be in a section starting ===== Anymail API request right above the response. (Sometimes ESP support blames third-party packages without investigating further, so providing the raw API call data is a way to avoid that. Also, it's always possible there's a bug in how Anymail composes the API request, since it's been about six years since we last ran Mandrill live integration tests.)

[medmunds]

I'm not sure I understand what this has to do with SMTP?

Seems like there's some confusion. Anymail uses Mandrill's messages/send HTTP API, not SMTP. (So the encoding of the message for SMTP takes place on Mandrill's side.)

Could this be indeed something in Anymail?

There's always a chance, but I really don't think so.

Here's what Anymail posts to the Mandrill API (I added line breaks for readability):

POST https://mandrillapp.com/api/1.0/messages/send.json
User-Agent: django-anymail/8.5-mandrill python-requests/2.27.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 319

{"key": "[REDACTED]", "message": {
"from_email": "from@example.com",
"to": [{"email": "to@example.com", "name": "", "type": "to"}], 
"subject": "Attachment filename encoding", 
"headers": {"Reply-To": ""}, 
"text": "Test", 
"attachments": [{"type": "application/pdf", "name": "Rechnung - M\u00f8l", "content": "W3Rlc3Rd"}]
}}

The attachment filename is sent as "name": "Rechnung - M\u00f8l", and \u00f8 is a valid JSON Unicode escape sequence for the "ø" character. I can't see anything wrong there, but maybe I'm missing something.

The most likely explanation is that Mandrill's API gateway is not properly using RFC 2231 to construct the attachment header fields as it builds the outgoing message.

[Thorbenl]

Hey @medmunds :)

They just answered me:

Just took a look!
In regards to the formatting, I reviewed your example :

https://d.pr/i/UHKimG

This does appear to be rendering fine in gmail, and lends credence to the fact that the behavior is likely email client specific.

“name”: “Lejekontrakt - M\u00f8llehatten 8240 Risskov.pdf” appears fine, I know you mentioned it’s only certain inboxes that you’re finding can’t handle these characters; we aren’t really going to be able to troubleshoot certain email clients specifically, if you do have access to the types of clients in question, ideally we would need to test in those clients with different encoding.

I was trying to add a different Content-disposition header to the ‘headers’ object in an API request while testing, but it doesn’t look like that changes the headers in the final email. With that being said, I'm positive we’re generating those headers automatically, I just see Content-Disposition: attachment; filename=“Lejekontrakt - Møllehatten 8240 Risskov.pdf” in the headers for the test email I sent myself with your JSON.

But I've elevated the request to our development team, and I'm grateful for your candor and understanding, especially given the circumstances.

Don't hesitate to reach back out should you need anything else whatsoever.

[medmunds]

It sounds like you'll have to wait until their dev team takes a look.

I'm fairly confident the Apple Mail client will properly display non-ASCII attachment filenames for validly encoded email messages. But what Mandrill seems to be doing—just stuffing raw UTF-8 characters into the attachment header, rather than following the spec and using RFC 2231 encoding—is absolutely, positively not valid. (Even if it sometimes works in some other email clients.)

You could try sending the same email using a different ESP. "Works when sent through (e.g.) Mailgun, garbled when sent through Mandrill" might be helpful to escalate the issue.

[Thorbenl]

It sounds like you'll have to wait until their dev team takes a look.

I'm fairly confident the Apple Mail client will properly display non-ASCII attachment filenames for validly encoded email messages. But what Mandrill seems to be doing—just stuffing raw UTF-8 characters into the attachment header, rather than following the spec and using RFC 2231 encoding—is absolutely, positively not valid. (Even if it sometimes works in some other email clients.)

You could try sending the same email using a different ESP. "Works when sent through (e.g.) Mailgun, garbled when sent through Mandrill" might be helpful to escalate the issue.

I appreciate your assistance on this.
Yes, I have the resources and time to escalate this further, so I will be doing that. :)
Sorry for keeping you so much in the loop, when you might not be interested, but it could be for future reference :)

I will update here, when the developers got back to me. Honestly, they have been faster than expected with the replies, so I am positive.

EDIT:
I provided them with the RFC context.

Good deal! I appreciate the additional context. 

So while I can't guarantee any sort of direct contact with our development team, I can provide you with Ticket #xxxx should you want to reach back out and inquire for updates periodically. Definitely just as interested in how that solution would be applied at this point, but thank you for making us aware to begin with! 

Cordially, 
****

I think you can close this for now, and then all open a new discussion, when I have some more info

[medmunds]

Definitely interested in the updates, thanks.

[medmunds]

I signed up for a trial account, and have confirmed this is a bug in Mandrill itself (not Anymail, and not any particular email client). Also, it affects more than just attachment filenames…

Correction: it only affects attachment filenames, not other fields. [An earlier version of this comment incorrectly claimed Mandrill was not using RFC 2047 encoding in the Subject, To, Cc, and Bcc headers. They are. It's only the attachment filename field that is improperly sent as raw UTF-8. My apologies for the confusion.]

---

Bug: Mandrill's /messages/send API does not properly handle non-ASCII characters in attachment filenames. As a result, many email clients will display incorrect garbled characters in their place.

Specifically, Mandrill does not use required RFC 2231 encoding for the filename parameter in an attachment part's Content-Disposition header. Instead, Mandrill appears to place the raw UTF-8 bytes from the API request directly into the header. The RFC 5322 email message format doesn't allow that. Non-ASCII characters cannot appear in header fields; they must use an encoding that specifies the character set.

Faced with the non-standard message Mandrill generates, email clients can interpret the non-ASCII bytes in any way they choose. Gmail appears to guess correctly that they are UTF-8. Apple Mail appears to guess incorrectly that they are ISO-8859-1, resulting in garbled text. Other email clients might strip out the non-ASCII bytes, incorrectly guess other character sets, or (theoretically) even crash.

---

Unless you hear back from Mandrill that they've fixed the problem, I'll probably add a note about this to the "quirks" section in Anymail's Mandrill docs.

A possible workaround is to use Django's SMTP EmailBackend (rather than Anymail) with Mandrill's SMTP integration, which would avoid their buggy /messages/send API. Django uses Python's standard email package under the hood, which does follow the standards for header encoding. (Incidentally, the Mandrill SMTP integration docs specifically call out the need to use RFC 2047 header encoding for things like Subject. Attachment filenames also need to be encoded, though they use a different standard.)

[medmunds]

Sorry, I misinterpreted what I was seeing, and I've edited my comment above to correct it. Mandrill does have a bug with attachment filename character encoding, but Mandrill correctly handles non-ASCII characters in the Subject and recipient header fields.

[Also, note for my own future self: Gmail's "View Original" doesn't actually show the original raw message—it decodes RFC 2231 and RFC 2047 encoded headers for display in the browser. The "Download Original" option on that page does seem to get the actual raw message content without additional transformation.]

[Thorbenl]

@medmunds Sorry for the late reply. Had some time off of work. Mandrill answered me, but there is no news on this, sadly. I think it would be nice to put on a nice message in the docs. :)

I have contacted them once again, and usually get an answer in 7-12 days.. (premium support we pay for, btw :D)

Thanks again for all the work you do on Anymail, and investigating this issue.
I learned a lot about making a bug report and seeing things from different angles, thanks! :)

[medmunds]

I think it would be nice to put on a nice message in the docs.

Done.

Thank you for (noticing and) reporting this, and for pressing the issue with MailChimp support.