Support for IMAP4rev2 and modern extensions

[nevans]

This is a meta-ticket to track support for IMAP capabilities. I'm also listing capabilities that are already supported by Net::IMAP as well as extensions that are obsolete or unsupported by IMAP servers, so that (when completed) the list will be exhaustive. Later, we can copy the list of supported RFCs and extensions into the rdoc.

This list hasn't been carefully audited yet. Some unchecked items can probably be checked off without any extra effort. As tickets and PRs are created for various features, I'll update this list to include them.

Basic protocol specifications

• RFC3501 (2003): Internet Message Access Protocol - version 4rev1
  • support all basic IMAP4rev1 commands
  • Update parser grammar to match RFC3501 and RFC4466 #50
  • Honoring server-reported and client-enabled capabilities #49
    several extensions require this. e.g. non-synchronizing literals
  • Prevent AUTHENTICATE without matching AUTH=#{mechanism} capability #48
  • UTF-7 should be automatically encoded/decoded #30
    this can be a client option, to preserve backwards compatibility
• RFC4466 (2006): Collected Extensions to IMAP4 ABNF #35
• RFC9051 (2021): IMAP4rev2
  • Appendix E: Changes since RFC3501
    Only the relevant items are listed below. In some cases, all that we need do for Net::IMAP is add (or link to) documentation.
    • 1. Support for 64-bit message and body part sizes.
      Net::IMAP doesn't validate number or number64 size, and thus automatically supports the larger 64-bit sizes
    • 2. Folded in:
      • RFC2177 (1997): IDLE
      • net/imap: support NAMESPACE extension (RFC2342) #5 NAMESPACE
      • RFC3516 (2003): BINARY FETCH extensions (included in IMAP4rev2) #41 FETCH BINARY
      • RFC3691 (2004): UNSELECT (included in IMAP4rev2) #40 UNSELECT
      • RFC4315 (2005): UIDPLUS (included in IMAP4rev2) #36 UIDPLUS
      • RFC4466 (2006): Collected Extensions to IMAP4 ABNF #35 RFC4466 ABNF extensions
      • RFC4731 (2006): ESEARCH (included in IMAP4rev2) #44 ESEARCH
      • RFC4959 (2007): SASL-IR (included in IMAP4rev2) #34 SASL-IR
      • RFC5182 (2008): SEARCHRES (included in IMAP4rev2) #42 SEARCHRES
      • RFC5161 (2008): ENABLE (included in IMAP4rev2) #33 ENABLE
      • RFC5258 (2008): LIST-EXTENDED (mostly included in IMAP4rev2) #43 LIST-EXTENDED
      • RFC5530 (2009): response codes (simply update documentation)
      • RFC5819 (2010): LIST-STATUS (included in IMAP4rev2) #45 LIST-STATUS
      • RFC6154 (2011): SPECIAL-USE (mailbox attributes are in IMAP4rev2) #108 SPECIAL-USE
      • RFC6851 (2013) MOVE
      • RFC7888 (2016): LITERAL- (included in IMAP4rev2) #37 LITERAL-
    • 3. Added STATUS SIZE RFC8438 and STATUS DELETED.
      These are both numeric status attributes and thus were already implicitly supported.
      ⚡ Better Faster Cleaner STATUS parsing #225 added explicit support and documented both.
    • 4. SEARCH command now requires to return the ESEARCH response (SEARCH response is now deprecated)
      RFC4731 (2006): ESEARCH (included in IMAP4rev2) #44
    • 7. Clarified that the COPYUID response code is returned for both MOVE and UID MOVE.
      Support UIDPLUS extension #65
    • 14. For future extensibility, extended ABNF for tagged-ext-simple to allow for bare number64
      RFC4466 (2006): Collected Extensions to IMAP4 ABNF #35
    • 16. Mailbox names and message headers now allow for UTF-8. Support for modified UTF-7 in mailbox names is not required, unless compatibility with IMAP4rev1 is desired.
      RFC6855 (2013): UTF8=ACCEPT (recommended by IMAP4rev2 for backward compatibility) #38
    • 20. IDLE command can now return updates not related to the currently selected mailbox state.
      Not applicable: Net::IMAP does not interpret unsolicited responses, and leaves this up to the library users.
    • 22. Clarified that client implementations MUST ignore response codes that they do not recognize. (Changed from a SHOULD to a MUST.)
    • 23. resp-text ABNF non-terminal was updated to allow for empty text. #110
    • 24. After ENABLE, IMAP4rev2 human-readable response text can include non-ASCII encoded in UTF-8. #109
    • Use modern TLS-related recommendations as per RFC7525, RFC7817, and RFC8314 #118
    • 26. Added warnings about use of ALERT response codes and PREAUTH response.
    • 27. Replaced DIGEST-MD5 SASL mechanism with SCRAM-SHA-256. DIGEST-MD5 was deprecated.
      • SCRAM-SHA-1(-PLUS) + SCRAM-SHA-256(-PLUS) + SCRAM-SHA-512(-PLUS) + SCRAM-SHA3-512(-PLUS) supports #54
  • Appendix F. Other Recommended IMAP Extensions
    • RFC7162 (2014): CONDSTORE (recommended by IMAP4rev2) #106
    • RFC7162 (2014): QRESYNC (recommended by IMAP4rev2) #107
    • RFC8474 (2018): OBJECTID (recommended by IMAP4rev2) #39
• RFC3501's official "updated by" RFCs:
  • RFC4466 (2006): Collected Extensions to IMAP4 ABNF #35
  • RFC4469 (2006): CATENATE
  • RFC4551 (2006): obsoleted by RFC7162 (2014): CONDSTORE
  • RFC5032 (2007): WITHIN
  • RFC5182 (2008): SEARCHRES (included in IMAP4rev2) #42
  • RFC5738 (2010): obsoleted by RFC6855 (2013): UTF-8 support
  • RFC6186 (2011): SRV Records for Locating Email Services
    • updated by: RFC8314 (2018): Cleartext Obsolete: Use TLS for Email
    • updated by: [RFC8553] (2019): DNS AttrLeaf, fixing underscored node names
  • RFC6858 (2013): Simplified Downgrading for I18n Email
    • values simplicity of implementation over fidelity of representation, since implementing a high-fidelity downgrade algorithm... is likely more work than implementing proper UTF-8 support RFC6855.
  • RFC7817 (2016): Updated TLS Server Identity Check Procedure
  • RFC8314 (2018): Cleartext Obsolete: Use TLS for Email
  • RFC8437 (2018): UNAUTHENTICATE
  • RFC8474 (2018): OBJECTID (recommended by IMAP4rev2) #39

IANA registries

Registry	references
Capabilities	RFC3501
Mailbox attributes	RFC3501
SASL Mechanisms	RFC4422
LIST EXTENDED	RFC5258
Response codes	RFC5530
Keywords	RFC5788, RFC8621

Commonly supported extensions

This is not an exhaustive list of all extensions, but an opinionated selection of the most important extensions that Net::IMAP should support. It is partially based on CAPABILITY data for hundreds of thousands of IMAP4 accounts, and partially based on subjective judgement. Additionally, any RFCs we already support, even if only partially, have been promoted to this list. Some extension behaviors are out-of-scope for Net::IMAP, so "support" simply means parsing and adding the necessary documentation for users of net-imap to implement the extension bevavior. Please comment below if you think something should be promoted to this list!

sorted by original RFC year

• RFC2087 (1997): QUOTA
• RFC2177 (1997): IDLE (included in IMAP4rev2)
• RFC2342 (1998): net/imap: support NAMESPACE extension (RFC2342) #5 NAMESPACE (included in IMAP4rev2)
• RFC2971 (2000): net/imap: support IMAP4 ID extension (RFC2971) #4 ID
• RFC3348 (2002): CHILDREN
  • MailboxList predicate methods
• RFC3516 (2003): BINARY
  • RFC3516 (2003): BINARY FETCH extensions (included in IMAP4rev2) #41
  • APPEND command extensions
• RFC3691 (2004): UNSELECT (included in IMAP4rev2) #40
• RFC4314 (2005): ACL
  • commands: GETACL, SETACL
  • response: ACL
  • commands: DELETEACL, LISTRIGHTS, MYRIGHTS
  • response: LISTRIGHTS, MYRIGHTS
• RFC4315 (2005): UIDPLUS (included in IMAP4rev2) #36
  • it is possible to manually extract the raw COPYUID data
• RFC4466 (2006): Collected Extensions to IMAP4 ABNF #35
• RFC4616 (2006): AUTH=PLAIN
• RFC5256 (2008): SORT
• RFC4731 (2006): ESEARCH (included in IMAP4rev2) #44
• RFC4959 (2007): SASL-IR (included in IMAP4rev2) #34
• RFC4978 (2007): COMPRESS=DEFLATE #46
• XOAUTH2 (2007): AUTH=XOAUTH2
• GMAILext (2007): X-GM-EXT-1
  • XLIST deprecated for SPECIAL-USE
  • X-GM-RAW, X-GM-MSGID, X-GM-THRID, X-GM-LABELS
    n.b. GMail's non-standard extensions can almost be obsoleted by a combination of OBJECTID, SORT, THREAD, SEARCH=FUZZY, ESEARCH, ESORT, and AUTH=OAUTHBEARER. But GMail and other large email providers still haven't upgraded to use all of these new standards. Until such time as they are deprecated by GMail (and AUTH=XOAUTH2 is used by many providers), I think it's useful to add these to the standard library instead of installing a separate gem.
• RFC5161 (2008): ENABLE (included in IMAP4rev2) #33
• RFC5182 (2008): SEARCHRES (included in IMAP4rev2) #42
• RFC5957 (2010): SORT=DISPLAY (only requires documentation)
• RFC5258 (2008): LIST-EXTENDED (mostly included in IMAP4rev2) #43
• RFC5530 (2009): Response Codes (IANA registry) (included in IMAP4rev2)
• RFC6855 (2013): UTF8=ACCEPT (recommended by IMAP4rev2 for backward compatibility) #38
• RFC5802 (2010): AUTH=SCRAM-*
  SCRAM-SHA-1(-PLUS) + SCRAM-SHA-256(-PLUS) + SCRAM-SHA-512(-PLUS) + SCRAM-SHA3-512(-PLUS) supports #54
• RFC5819 (2010): LIST-STATUS (included in IMAP4rev2) #45
• RFC6154 (2011): SPECIAL-USE (mailbox attributes are in IMAP4rev2) #108
  • Constants for the mailbox name attributes are added by Add and document flags from RFC9051 #28
• RFC6851 (2013): MOVE (included in IMAP4rev2)
• RFC7162 (2014): CONDSTORE (recommended by IMAP4rev2) #106 (obsoletes: RFC4551 2006)
• RFC7162 (2014): QRESYNC (recommended by IMAP4rev2) #107 (obsoletes: RFC5162 2008)
• RFC7628 (2015): AUTH=OAUTHBEARER #47
• RFC7888 (2016): LITERAL+ (obsoletes: RFC2088 1997)
• RFC7888 (2016): LITERAL- (included in IMAP4rev2) #37
• RFC7889 (2016): APPENDLIMIT
• RFC8437 (2018): UNAUTHENTICATE
• RFC8438 (2018): STATUS=SIZE (included in IMAP4rev2)
  • n.b. Net::IMAP imposes no restrictions on STATUS attributes and can parse any atom number extension in the response, so it automatically supports STATUS mbox (SIZE). It is the user's responsibility to check for the capability.
• RFC8457 (2018): Keyword: $Important
  • n.b. Net::IMAP ignores keyword semantics, leaving that to the user. So "support" simply means that the attributes are parsed correctly.
• RFC8457 (2018): Special-use attribute: \Important
• RFC8474 (2018): OBJECTID (recommended by IMAP4rev2) #39
• RFC8970 (2020): PREVIEW, Message Preview Generation
• RFC9208 (2022): QUOTA (obsoletes RFC2087)
• RFC9394 (2023): PARTIAL, for Paged SEARCH and FETCH

Additional standard recommendations

• RFC2180 - IMAP4 Multi-Accessed Mailbox Practice
• RFC2595 - Using TLS with IMAP, POP3 and ACAP
• RFC2683 (1999): IMAP4 Implementation Recommendations
  • mailbox listing, heirarchy, delimiters, NAMESPACE, etc
  • mailbox "\Noselect", "\Noinferiors", "[TRYCREATE]", "[READ-ONLY]"
  • UIDs and UIDVALIDITY
  • FETCH (any order, extra information, unsolicited)
  • "Clients must present ALERT text clearly to the user."
• RFC4549 (2006): Synchronization Operations for Disconnected IMAP4 Clients
• RFC5550 (2009): The Internet Email to Support Diverse Service Environments (Lemonade) Profile
  • obsoletes: [RFC4550] (2006)
• RFC5598 (2009): Internet Email Architecture (Errata Exist)

Other extensions

Some of these are easy to add, and many of them would be very useful. I placed them down here instead of prioritizing them above mostly due to a quick and unscientific sampling of email accounts and server capabilities.

sorted by estimated current server support

• RFC2221 - LOGIN-REFERRALS
• RFC5032 - WITHIN
• RFC4469 - CATENATE
• RFC5256 - THREAD=ORDEREDSUBJECT
• RFC5267 - ESORT
• RFC5256 - THREAD=REFERENCES
• RFC5267 - CONTEXT=SEARCH
• RFC5267 - CONTEXT=SORT
• RFC5550 - URL-PARTIAL
• RFC5092 - imap:// URL Scheme
• RFC5593 - URL Access Identifier Extension

By my estimation, the following are currently supported by relatively few email addresses. They are sorted by RFC number.

• RFC3502 - MULTIAPPEND
• RFC4467 - URLAUTH
• RFC5255 - I18NLEVEL=1
• RFC5255 - I18NLEVEL=2
• RFC5255 - LANGUAGE
• RFC5257 - ANNOTATE-EXPERIMENT-1
• RFC5464 - METADATA
• RFC5464 - METADATA-SERVER
• RFC5465 - NOTIFY
• RFC5466 - FILTERS
• RFC5524 - URLFETCH
• RFC6154 - CREATE-SPECIAL-USE
• RFC6203 - SEARCH=FUZZY
• RFC6785 - IMAPSIEVE=
• RFC6785 - UTF8=ONLY
• RFC7377 - MULTISEARCH
• RFC8437 - UNAUTHENTICATE
• RFC8440 - LIST-MYRIGHTS
• RFC8508 - REPLACE
• RFC8514 - SAVEDATE

New specifications

Looking at recent standards and others that are still in draft form (as of 2021-10-19):

IETF "extra" WG (Email mailstore and eXtensions To Revise or Amend)

• The JMAPACCESS Extension for IMAP
• IMAP MESSAGELIMIT Extension
• IMAP Paged SEARCH & FETCH Extension
• IMAP Extension for only using and returning UIDs

Deprecations?

We might want to remove obsolete authentication mechanisms from the default set of authenticators. For people who still need them, the code could be preserved in separate files, with explicit requires, e.g:

require "net/imap/sasl/cram-md5_authenticator"
require "net/imap/sasl/digest-md5_authenticator"
require "net/imap/sasl/login_authenticator"

• RFC6331: Moving DIGEST-MD5 to Historic #55
• Warn when using deprecated SASL mechanisms #62
  • LOGIN (only published as a draft. use AUTH PLAIN or LOGIN instead)
  • AUTH=CRAM-MD5
  • AUTH=DIGEST-MD5 - officially obsoleted by RFC6331 (2011)

[nevans]

n.b. IMAP4rev2 (RFC9051) was released on 2021-08-26.

I didn't have as much time earlier this year as I'd hoped, but I'll have some more time to devote to this next week and the week after, so hopefully I can make good progress towards updating our parser for IMAP4rev2. I'd like to see what can be accomplished before RubyConf 2021! :)

[nevans]

n.b. I lightly modified RFC-9051's ABNF to valid ABNF syntax, and I generated railroad diagrams in unicode line-art, using kgt: https://gist.github.com/nevans/ae5a82bb81c1595b06b9e167a0376924

e.g.

response:
    │├─────╭─────────────────────────╮── response-done ─────┤│
           │                         │
           ╰──╮── continue-req ───╭──╯
              │                   │
              ╰── response-data ──╯

response-data:
    │├───── "*" ── SP ──╮── resp-cond-state ──╭── CRLF ─────┤│
                        │                     │
                        ╰─── resp-cond-bye ───╯
                        │                     │
                        ╰─── mailbox-data ────╯
                        │                     │
                        ╰─── message-data ────╯
                        │                     │
                        ╰── capability-data ──╯
                        │                     │
                        ╰──── enable-data ────╯

response-done:
    │├──╮── response-tagged ──╭──┤│
        │                     │
        ╰── response-fatal ───╯

response-fatal:
    │├───── "*" ── SP ── resp-cond-bye ── CRLF ─────┤│

response-tagged:
    │├───── tag ── SP ── resp-cond-state ── CRLF ─────┤│

and so on...

I spent a little bit of time this week going through the collection of response parser patches I've been using, to figure out which ones to convert into PRs and which ones to throw away. I hope to start on the big RFC9051 updates next week.

In most cases, the ABNF in RFC9051 seems like an easier starting point than RFC3501, because it already incorporates many of the updates from various popular extensions. And it also includes simpler extension points for the extensions which aren't included. Likewise, I think it will be simpler to retain backwards compatibility with an RFC9051 parser than to force forward compatibility into an RFC3501 parser.

[nevans]

Updated the checklist with links to #55 and #54. Hopefully I'll starting ticking things off this checklist soon, and not only adding things to it! 😆

[Neustradamus]

To follow :)

[Neustradamus]

Good news, it is official for TLS 1.3 Binding!

• RFC 9266: Channel Bindings for TLS 1.3: https://tools.ietf.org/html/rfc9266

Details:

• tls-unique for TLS =< 1.2
• tls-exporter for TLS = 1.3

---

Thanks to @singpolyma who has done a PR in net-sasl for SCRAM:

• Add support for SCRAM-SHA-* nevans/net-sasl#5

[nevans]

Thanks that's great. I'm (slowly, sporadically) working on merging that PR and hope to finish very soon. But let's discuss SCRAM-* SASL mechanisms in #54, maybe open a new issue (or PR! 😉) for new features like channel binding, and reserve talk on this issue to be about the meta-ticket: the IMAP4rev2 checklist and extensions that are in one of the specific IMAP IANA registries, e.g: IMAP capabilities.

[ssunday]

I'm planning on raising a PR soon™ to incorporate the XOAUTH2 authenticator. 👍🏻