non-blocking API

[glyph]

Twisted has long desired something just like oscrypto, so that users get their platform's trust settings by default. oscrypto provides something like this.

However, many of the APIs in oscrypto are unfortunately hard-coded to do I/O in in a blocking way, as well as in places like constructors which makes it extra tricky to interpose.

It would be neat if oscrypto could expose a buffer-like (as opposed to socket-like) interface for TLS where I could just drop in some encrypted bytes from the wire and get back some authenticated / verified plaintext bytes, and vice versa.

[wbond]

Sorry for not getting back to you sooner. Often times GitHub never notifies me about events on repos I am actively working with. Go figure…

This should be possible. Right now I am working on building out the TLS support for a software package of mine, but I would be happy to try and make it usable by others too.

I've completed the Windows and OS X portions. Both of them handle the encryption and decryption and rely on you actually sending and receiving the data. On Windows you have to deal with all of the logic, which is gross and as I have found, is more prone to logic errors. On OS X you just provide read/write callbacks and the high-level functions SSLHandshake, SSLRead and SSLWrite do most of the dance for you.

I haven't really dealt with async I/O in Python, so I think I would probably come up with a terrible API for deal with this. I imagine you might be able to help in this regard.

For instance, when dealing with a handshake on Windows, there are a couple of places where data is sent and received:

• https://github.com/wbond/oscrypto/blob/master/oscrypto/_win/tls.py#L381
• https://github.com/wbond/oscrypto/blob/master/oscrypto/_win/tls.py#L387
• https://github.com/wbond/oscrypto/blob/master/oscrypto/_win/tls.py#L418

Then there is read():

• https://github.com/wbond/oscrypto/blob/master/oscrypto/_win/tls.py#L574

And write():

• https://github.com/wbond/oscrypto/blob/master/oscrypto/_win/tls.py#L789

How would you envision the flow working here? If we inverted the flow so that you are doing the work of providing the ciphertext and plaintext, then Twisted would need to know a bit more about the SChannel and Secure Transport dances, right?

[glyph]

What I would like is a low-level API very much like SSLHandshake/Read/Write; put bytes in, get calls to an application-provided (where "application" here would be Twisted, or asyncio, or circuits) SSLReadFunc or SSLWriteFunc which can return errSSLWouldBlock.

Although I need to look at it more closely, I think the Windows stuff could all be mapped to this high-level encrypted-bytes-in/plain-bytes-out plain-bytes-in/encrypted-bytes-out API, by examining the out buffers after each relevant call and issuing a callback.

Obviously I'd want something with actual objects rather than C functions that take a "context", but beyond that I would leave the "nice" async API integration up to each specific set of bindings that binds oscrypto to a specific transport abstraction.

Hopefully I'll have more time to look at this later, but does that sound like a reasonable starting point?

[wbond]

Yeah, once I have the OpenSSL backend finished up, then I think I'll have an idea of what sort of context needs to be persisted and how an API can be built to accomplish this.

OS X uses callbacks for I/O, whereas SChannel and OpenSSL (using memory BIOs) returns a result code indicating they need more input, or have output. With the async stuff, would callbacks be feasible, or should it be something more like result codes?

I think the next step is to perhaps write some high-level pseudocode of what the API might looks like and how it would function.

My initial plan is to extract all of the handshake, read and write type functionality into something like a TLSProvider class that the existing TLSSocket would wrap around. Then ideally you could wrap TSLProvider with your own wrappers for Twisted, or anything else.

Hopefully I'll have the OpenSSL implementation mostly done today and then if we have an idea for an API for TLSProvider, I can refactor that code out into a separate class.

[glyph]

I like callbacks better because they make it much clearer when various things need to happen; you provide a callback that does that thing. Pretty much the SecureTransport API is exactly what I want. If you wanted to do an API with return codes, that's not that much worse, but return codes just have to be translated into making the thing identified by the specific return code happen, and it's easier for the caller to screw that up.

[wbond]

Ok, great - I believe callbacks will be easier to implement and simpler to use.

Now, the final bit I want to understand is related to the statement: "Pretty much the SecureTransport API is exactly what I want."

So let's say I build a callback API where your async code will provide the input and consume the output. Is this going to behave like a non blocking socket, where it will return instantly with no data, or will it use some sort or co-routine or thread-like mechanism where the call will not return until data is available?

The final bit is related to select(). Currently that is used with the socket to look to see if additional information is available. Do you envision providing a callback for this, such that the full TLSSocket API would be usable, or do you only want the raw encryption/decryption functionality that you will be making available via a different API?

If this would be easier to hash out on IRC, you can try reaching me on Freenode: wbond.

[wbond]

Initial implementation and testing of TLSSocket and TLSSession are complete. With that work done, the refactoring for this change should be fairly straight forward.

Once you have a chance to answer about the blocking vs non-blocking nature of the callbacks, I can take a stab at making this happen.

[wbond]

This will actually take some more work, possibly quite a bit since the OS TLS libraries deal with fetching revocation information in a blocking manner.

To work around this, there will need to be options in TLSSession that allow disabling CRL and OCSP fetching. I know OS X allows for async trust evaluation, however I am not familiar with OpenSSL to Crypt32. If such async options are not available, we would need to see if there is a way to provide the CRL/OCSP responses. If that is not possible, we'd have to fall back to the pure-python verification library I have been working on.

[glyph]

Which OS TLS library are you referring to? SecureTransport doesn't, OpenSSL doesn't. Does Windows?

[glyph]

I know OS X allows for async trust evaluation, however I am not familiar with OpenSSL to Crypt32.

https://www.openssl.org/docs/manmaster/crypto/OCSP_sendreq_new.html indicates that the relevant OpenSSL API is OCSP_sendreq_nbio at least for OCSP; I'm assuming there's a similar API for CRLs…

[wbond]

Windows does.

[wbond]

But yeah, either way there should be a consistent API in oscrypto for disabling or enabling revocation checks. And then, that would need to be expose to/via the async API.

[wbond]

And yes, the testing I just did confirmed that OpenSSL and Secure Transport by default don't care about revoked certificates.

[wbond]

According to the SecureTransport reference, SecEvaluateTrust can trigger network access. https://developer.apple.com/library/ios/documentation/Security/Reference/certifkeytrustservices/index.html#//apple_ref/c/func/SecTrustEvaluate

Apparently that is not for CRL/OCSP checking, but network-hosted trust roots. So manual validation using SecTrustEvaluateAsync would need to be used to prevent blocking on that.

[wbond]

Discovered that OS X 10.7-10.9 do revocation checks by default.

[glyph]

Meaning that the async API actually blocks on those OS versions?

[wbond]

No, just CRL and OCSP checks do happen during a normal request cycle on 10.7-10.9 even though they do not in 10.10+. Currently I am in the process of working around this issue to provide a consistent API in terms of TLS functionality.

Mostly just documenting findings for future work and reference.

[wbond]

Another note about an issue to deal with when/if this issue gets more time:

On Windows, there are some edge-case bugs that cause a handshake to fail, which require dropping down to TLS 1.1 (versus TLS 1.2) or require retrying the handshake. Thus there would need to be a way to signal to the async network code to create a new socket connection and try again:

• Dropping down to TLS 1.1 is required if the self-signed trust root used has a signature using the MD5 or MD2 hash algorithm. This is fairly rare, I believe I mostly ran into it with Chinese TLS hosts from the Alexa top 1000.
• Retrying a connection (to get a fresh handshake) is required when using TLS 1.2 and the DHE_RSA key exchange is used and the server sends back a public key that is encoded slightly under 1024 bits (usually 127 bytes instead of 128 bytes). This is a reasonable common occurrence.

Both of these issue are currently worked around when creating a tls.TLSSocket() with a hostname and port. tls.TLSSocket.wrap() just throws an exception since it does not originate the socket.

[markfriedman]

for those of us who haven't programmed in 30 years, how can I fix this?
will a new build be created (with Sierra)?)

[wbond]

I don't have a VM of Sierra yet, but there should not need to be a new build until Apple removed some Security.framework or SecureTransport functions.

In terms of making it non-blocking, the biggest issue I see so far is just getting a sense of what the API would look like so the current implementation can be adapted to it.

[wbond]

I think the next step in supporting this would be to add a class tls.TLSDelegate, or similar, that would:

1. Be configured similarly to tls.TLSSocket, minus the actual socket connection
2. Would have a method to feed in raw bytes from the wire (provided by the async library)
3. Would require a callback to be provided that would called when plaintext data is available
4. Would require a callback to be provided that would be called when data needs to be written to the wire

Here is an example API that can hopefully be discussed:

class TLSDelegate(object):
    def __init__(self, received_plaintext_data_callback,
                 encrypted_data_to_send_callback, session=None):
        """
        :param received_plaintext_data_callback:
            A callback that accepts a single parameter, a byte string of
            decrypted plaintext from the TLS connection

        :param encrypted_data_to_send_callback:
            A callback that accepts a single parameter, a byte string of
            encrypted ciphertext/protocol data to be written to a socket/network

        :param session:
            An existing TLSSession object to allow for session reuse, specific
            protocol or manual certificate validation
        """

        _internal_start_handshake()

    def received_tls_data(self, data):
        """
        A method that should be called anytime incoming network data is
        available

        :param data:
            The ciphertext/protocol data from the socket/network/async library
        """

        _read_implementation()

    def write(self, data):
        """
        A method to be called anytime plaintext should be written to the
        conenction

        :param data:
            A byte string of plaintext data to encrypt into TLS and send
        """

        _write_implementation()

    def process(self):
        """
        A method to be called to trigger processing of data provided to the
        write() and received_tls_data() methods
        """

        _process_data_using_tls_library()

[njsmith]

We've also been discussing this in Trio recently too – it would be neat to be able to support oscrypto! The thread also includes some thoughts on possible API abstractions: python-trio/trio#1145

[glyph]

Good to see conversation about this picking up again! Has anything changed in the intervening couple of years? :)

[njsmith]

After doing some more research, it's actually not obvious to me whether the goal should be to wrap SChannel/SecureTransport in a memory-buffer-based API, so maybe we should back up and talk about that.

There are two separate things that a library like Twisted or Trio needs:

• An implementation of the TLS protocol itself: the state machine, record format parser, crypto, etc.
• Certificate validation

Windows and macOS provide platform-native versions of both of these, and they're often lumped together. It would be very convenient if you could have a simple async-friendly API that implements both features together using platform-native tools. But it turns out that this isn't possible: on both Windows and macOS, the certificate validation routines are fundamentally implemented as synchronous blocking operations. (On macOS there's also SecTrustEvaluateAsync, but AFAICT that's just a trivial wrapper that uses a thread to call SecTrustEvaluate. I don't think it unlocks any new functionality.)

Also, I don't think there are many people that want to use the platform-native TLS protocol code, but then skip the platform-native cert validation in favor of some home-grown thing.

Put these two facts together, and my conclusion is that for Twisted/Trio's purposes, any kind of platform-native TLS support is going to have two separate pieces:

• A nice buffer-based TLS state machine that we drive from async-land
• A synchronous cert validation routine that we have to push off into a thread

It's not really possible to hide a thread-based synchronous call inside an buffer-based state machine API. Much better to expose the two pieces separately, and then let projects like Twisted or Trio figure out how to wire the pieces together into their I/O model. (E.g., they both have simple ways to push blocking code out into a thread, but they do it differently.)

Also, these two pieces solve two orthogonal problems:

• Using the platform-native TLS protocol implementation is valuable because it means that any protocol bugs get fixed automatically by OS updates
• Using the platform-native cert validation routines is valuable because it allows us to make more accurate trust decisions and handle local configuration like company-internal CAs

So I think ideally oscrypto should actually have two issues to provide two separate APIs, for the protocol and for the cert validation. And @glyph's original request in the first post in this thread – that oscrypto expose a protocol API, so that he can do better cert validation – doesn't actually make sense.

Both of these APIs would be "nice to have", but of the two of them, the cert validation one seems likely to be both simpler and more immediately useful? It's simpler because it's more-or-less a single blocking function, instead of a complex state-machine with callbacks and all that. And it's more useful because AFAICT cert validation is the part that causes more pain for users, and it's relatively straightforward to keep using pyopenssl for the protocol part while dropping in a call to the platform-native API for validation.

[njsmith]

Oh, and another reason why exposing the platform native TLS state-machine isn't as exciting as you might think: Apple has actually deprecated SecureTransport, so it won't get new features like TLS 1.3, and they have no plans to support memory buffers in its replacement. See python-trio/trio#1165 for more details.

[wbond]

We've also been discussing this in Trio recently too – it would be neat to be able to support oscrypto!

Conceptually this sounds like a good idea. I've heard a little about Trio, and it seems like useful project.

Good to see conversation about this picking up again! Has anything changed in the intervening couple of years? :)

In my case, unfortunately, things have mostly changed in ways that I personally would be unlikely to have any major contributions to such a project. When oscrypto started and I did the bulk of the TLS implementation I had two fewer children, was working for myself and housing was a rental. Due to my current life circumstances, I have very little free time for open source these days.

After doing some more research, it's actually not obvious to me whether the goal should be to wrap SChannel/SecureTransport in a memory-buffer-based API, so maybe we should back up and talk about that.

Based on this and the following comment, it seems to me that while oscrypto having a TLS layer is useful for those working in a blocking environment (or where a thread is useful for IO), that trying to utilize SChannel and SecureTransport in a non-blocking API would:

A. Likely be a significant refactoring or rewrite
B. Is likely impossible
C. Is only so useful since Apple is moving to a new API for TLS 1.3

---

On a higher level note, oscrypto started originally to access legacy ciphers for dealing with PDF encryption and signing. Supporting junk like RC4 and triple DES mostly has to do with loading poorly encrypted files, or dealing with poorly armored private keys. I added on TLS because I had the need to supporting both Python 2.6 and 3.3 with TLS connections, but I needed more access than the ssl module in Python supported, and trying to keep up with recompiling cryptography releases across three operating systems, multiple versions of Python and multiple architectures was a nightmare. Plus cryptography was/is keen on moving quickly and only supporting the latest versions of Python.

In the process of building this all out and trying to keep it working I've had to come up with my own solutions for installing packages to run CI since pip just doesn't work on older Python releases now. Luckily coverage seems to be more keen on supporting older Pythons, especially since it would be effectively impossible to replace coverage, whereas replacing pip wasn't the end of the world.

All of that to say, this project is never really going to be about greenfield development, supporting the latest in high performance IO or even being the correct solution for projects that are deployed in a known environment. This project really is defined by:

• A simple API that should have as few knobs as possible
• Works pretty much anywhere you put it
• Supports lowest-common-denominator crypto that is used by various (older) protocols

If someone is starting a new project and dealing with crypto and isn't shackled to an existing protocol, they should definitely be using NaCl and probably scrypt.

If someone has a single deploy environment and can deal with only "good" crypto which isn't supported by NaCl, cryptography is probably the correct project to use.

If someone needs to work with crypto and support end-user install on some random CentOS 5 box and needs it to just deal with the environment, oscrypto will hopefully be useful.

Ideally I'd love the work I put in to be useful to as many people as possible, but at this point I don't foresee myself making any big changes to the project. Most of my time on the project these days is keeping the stupid CI pipeline working. Maybe the way in which this will be useful is providing some examples of dealing with the warts of the OS crypto libraries. I know work in urllib3 was derived from the Mac TLS implementation here.

Probably the most sane thing to do is say: async sounds great, but I'm not sure this project is the right kind of home for it.

To keep true to this motivations of this project, it probably means not adding features only available on a single OS. If someone needs crypto features not present in oscrypto already, there aren't many left that are supported by Win, Mac and Linux and 95%+ of the install base. DH key agreement may or may not be reasonably possible. I know @wiml did some work here, but I think OpenSSL may not support it until 1.0.2? and Windows probably not unless you are running 7+. These days those are rather long in the tooth and unsupported in various ways, but everything we've got right now works in those environments. If you need more, it probably makes sense to figure out how to deploy cryptography to your end users. That, or use a language/ecosystem that is easier to deploy and support on varied end-user machines.

---

I don't say any of this to discourage anyone, mostly this is just a brain dump and reflection of where I am personally. I know you @njsmith and @glyph are involved in a huge way in the Python world. Hopefully some of this is helpful at some point. Unfortunately I don't think I will end up being able to be so helpful, mostly just since I am very much overcommitted and trying to find ways to get back to a place of balance.