Implementation of RFC 6532 sends email without activating SMTPUTF8

[RichardSteele]

Your implementation of RFC 6532 may cause the sending of an email without activating SMTPUTF8 although it is mandatory according to RFC 6532:

messages in this format require the use of the
SMTPUTF8 extension [RFC6531] to be transferred via SMTP

When an SMTP server announces that it is capable of SMTPUTF8, SMTPTransport::send calls

vmime/src/vmime/net/smtp/SMTPTransport.cpp

Line 428 in 182e8f5

ctx.setInternationalizedEmailSupport(m_connection->hasExtension("SMTPUTF8"));

Under certain circumstances, this keeps word::generate from outputting encoded words when it has to auto detect if encoding is neccessary:

vmime/src/vmime/word.cpp

Line 460 in 182e8f5

encodingNeeded = wordEncoder::isEncodingNeeded(ctx, m_buffer, m_charset, m_lang);

For example, if you are in RFC 6532 mode and use a 'simple' subject header with umlauts, wordEncoder::isEncodingNeeded says not to encode:

vmime/src/vmime/wordEncoder.cpp

Lines 245 to 284 in 182e8f5

	bool wordEncoder::isEncodingNeeded(
	const generationContext& ctx,
	const string& buffer,
	const charset& charset,
	const string& lang
	) {
	if (!ctx.getInternationalizedEmailSupport()) {
	// Charset-specific encoding
	encoding recEncoding;
	if (charset.getRecommendedEncoding(recEncoding)) {
	return true;
	}
	// No encoding is needed if the buffer only contains ASCII chars
	if (utility::stringUtils::findFirstNonASCIIchar(buffer.begin(), buffer.end()) != string::npos) {
	return true;
	}
	}
	// Force encoding when there are only ASCII chars, but there is
	// also at least one of '\n' or '\r' (header fields)
	if (buffer.find_first_of("\n\r") != string::npos) {
	return true;
	}
	// If any RFC-2047 sequence is found in the buffer, encode it
	if (buffer.find("=?") != string::npos || buffer.find("?=") != string::npos) {
	return true;
	}
	// If a language is specified, force encoding
	if (!lang.empty()) {
	return true;
	}
	return false;
	}

Eventually, this leads to output of UTF-8 encoded text because of:

vmime/src/vmime/word.cpp

Lines 487 to 500 in 182e8f5

	} else if (!encodingNeeded) {
	string buffer;
	if (ctx.getInternationalizedEmailSupport()) {
	// Convert the buffer to UTF-8
	charset::convert(m_buffer, buffer, m_charset, charsets::UTF_8);
	} else {
	// Leave the buffer as-is
	buffer = m_buffer;
	}

SMTPUTF8 must be activated in this situation. It is, however, activated only if SMTPTransport::sendEnvelope deems it necessary:

vmime/src/vmime/net/smtp/SMTPTransport.cpp

Lines 210 to 223 in 182e8f5

	const bool hasSMTPUTF8 = m_connection->hasExtension("SMTPUTF8");
	bool needSMTPUTF8 = false;
	if (!sender.isEmpty()) {
	needSMTPUTF8 = needSMTPUTF8 || mailboxNeedsUTF8(sender);
	} else {
	needSMTPUTF8 = needSMTPUTF8 || mailboxNeedsUTF8(expeditor);
	}
	for (size_t i = 0 ; i < recipients.getMailboxCount() ; ++i) {
	const mailbox& mbox = *recipients.getMailboxAt(i);
	needSMTPUTF8 = needSMTPUTF8 || mailboxNeedsUTF8(mbox);
	}

Since SMTPTransport::sendEnvelope only considers email addresses and knows nothing of the yet to be generated message, we're in a mess!

[RichardSteele]

Is this ever going to be fixed? We're facing this problem increasingly because of the popularity of Office365 resp. Exchange servers (see also #209).

Our quick fix has been to revert your efforts to detect if SMTPUTF8 is really necessary and instead use it whenever the server supports it (#186). I know that this violates the following recommendation of RFC 6531:

If the SMTPUTF8-aware SMTP client is aware that neither the envelope nor the message being sent requires any of the SMTPUTF8 extension capabilities, it SHOULD NOT supply the SMTPUTF8 parameter with the MAIL command.

A proper solution would probably include some form of pre-pass over all headers (see also section 3.6 of RFC 6531):

The MAIL command parameter SMTPUTF8 asserts that a message is an internationalized message or the message being sent needs the SMTPUTF8 support. There is still a chance that a message being sent via the MAIL command with the SMTPUTF8 parameter is not an internationalized message. An SMTPUTF8-aware SMTP client or server that requires accurate knowledge of whether a message is internationalized needs to parse all message header fields and MIME header fields [RFC2045] in the message body.

But even then VMime doesn't honour section 3.5 of RFC 6531:

An SMTPUTF8-aware SMTP client MUST NOT send an internationalized message to an SMTP server that does not support SMTPUTF8. If the SMTP server does not support this option, then the SMTPUTF8-aware SMTP client has three choices according to Section 3.2 of this specification.

[vincent-richard]

Couldn't we fix this easily by calling ctx.setInternationalizedEmailSupport(true) only if SMTPUTF8 extension is available and we detected it should be used (ie. if needSMTPUTF8 == true)?

[arnt]

I came back to this after several years because someone else has reported a similar problem, which was solved for now by setting their "use smtputf8" variable if both any address in the message has a non-ascii localpart and the next-hop server supports it.

I became curious what the concrete problem was with setting ctx.setInternationalizedEmailSupport(true) based only on whether the addresses require it, ignoring any other header fields. I know it's against the rules in the RFC, but surely that wasn't the problem you observed?