.github/ISSUE_TEMPLATE/1-issue.md

@@ -0,0 +1,16 @@
+---
+name: Issue
+about: Please only raise an issue if you've been advised to do so after discussion. Thanks! 🙏
+---
+
+The starting point for issues should usually be a discussion...
+
+https://github.com/encode/httpcore/discussions
+
+Possible bugs may be raised as a "Potential Issue" discussion, feature requests may be raised as an "Ideas" discussion. We can then determine if the discussion needs to be escalated into an "Issue" or not.
+
+This will help us ensure that the "Issues" list properly reflects ongoing or needed work on the project.
+
+---
+
+- [ ] Initially raised as discussion #...

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,11 @@
+# Ref: https://help.github.com/en/github/building-a-strong-community/configuring-issue-templates-for-your-repository#configuring-the-template-chooser
+blank_issues_enabled: false
+contact_links:
+- name: Discussions
+  url: https://github.com/encode/httpcore/discussions
+  about: >
+    The "Discussions" forum is where you want to start. 💖
+- name: Chat
+  url: https://gitter.im/encode/community
+  about: >
+    Our community chat forum.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,12 @@
+<!-- Thanks for contributing to HTTP Core! 💚
+Given this is a project maintained by volunteers, please read this template to not waste your time, or ours! 😁 -->
+
+# Summary
+
+<!-- Write a small summary about what is happening here. -->
+
+# Checklist
+
+- [ ] I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
+- [ ] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
+- [ ] I've updated the documentation accordingly.

.github/workflows/publish.yml

@@ -17,7 +17,7 @@ jobs:
       - uses: "actions/checkout@v3"
       - uses: "actions/setup-python@v4"
         with:
-          python-version: 3.7
+          python-version: 3.8
       - name: "Install dependencies"
         run: "scripts/install"
       - name: "Build package & docs"

.github/workflows/test-suite.yml

@@ -14,7 +14,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
 
     steps:
       - uses: "actions/checkout@v3"

CHANGELOG.md

@@ -4,12 +4,21 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
-## 0.17.3 (5th July 2023)
+## 0.18.0 (September 8th, 2023)
+
+- Add support for HTTPS proxies. (#745, #786)
+- Drop Python 3.7 support. (#727)
+- Handle `sni_hostname` extension with SOCKS proxy. (#774)
+- Handle HTTP/1.1 half-closed connections gracefully. (#641)
+- Change the type of `Extensions` from `Mapping[Str, Any]` to `MutableMapping[Str, Any]`. (#762)
+
+## 0.17.3 (July 5th, 2023)
 
 - Support async cancellations, ensuring that the connection pool is left in a clean state when cancellations occur. (#726)
 - The networking backend interface has [been added to the public API](https://www.encode.io/httpcore/network-backends). Some classes which were previously private implementation detail are now part of the top-level public API. (#699)
 - Graceful handling of HTTP/2 GoAway frames, with requests being transparently retried on a new connection. (#730)
 - Add exceptions when a synchronous `trace callback` is passed to an asynchronous request or an asynchronous `trace callback` is passed to a synchronous request. (#717)
+- Drop Python 3.7 support. (#727)
 
 ## 0.17.2 (May 23th, 2023)
 

README.md

@@ -25,7 +25,7 @@ Some things HTTP Core does do:
 
 ## Requirements
 
-Python 3.7+
+Python 3.8+
 
 ## Installation
 

docs/index.md

@@ -25,7 +25,7 @@ Some things HTTP Core does do:
 
 ## Requirements
 
-Python 3.7+
+Python 3.8+
 
 ## Installation
 

docs/proxies.md

@@ -51,10 +51,33 @@ proxy = httpcore.HTTPProxy(
 )
 ```
 
-## Proxy SSL and HTTP Versions
+## Proxy SSL
 
-Proxy support currently only allows for HTTP/1.1 connections to the proxy,
-and does not currently support SSL proxy connections, which require HTTPS-in-HTTPS,
+The `httpcore` package also supports HTTPS proxies for http and https destinations.
+
+HTTPS proxies can be used in the same way that HTTP proxies are.
+
+```python
+proxy = httpcore.HTTPProxy(proxy_url="https://127.0.0.1:8080/")
+```
+
+Also, when using HTTPS proxies, you may need to configure the SSL context, which you can do with the `proxy_ssl_context` argument.
+
+```python
+import ssl
+import httpcore
+
+proxy_ssl_context = ssl.create_default_context()
+proxy_ssl_context.check_hostname = False
+
+proxy = httpcore.HTTPProxy('https://127.0.0.1:8080/', proxy_ssl_context=proxy_ssl_context)
+```
+
+It is important to note that the `ssl_context` argument is always used for the remote connection, and the `proxy_ssl_context` argument is always used for the proxy connection.
+
+## HTTP Versions
+
+If you use proxies, keep in mind that the `httpcore` package only supports proxies to HTTP/1.1 servers.
 
 ## SOCKS proxy support
 

httpcore/__init__.py

@@ -130,7 +130,7 @@ def __init__(self, *args, **kwargs):  # type: ignore
     "WriteError",
 ]
 
-__version__ = "0.17.3"
+__version__ = "0.18.0"
 
 
 __locals = locals()

httpcore/_async/connection.py

@@ -21,9 +21,16 @@
 
 
 def exponential_backoff(factor: float) -> Iterator[float]:
+    """
+    Generate a geometric sequence that has a ratio of 2 and starts with 0.
+
+    For example:
+    - `factor = 2`: `0, 2, 4, 8, 16, 32, 64, ...`
+    - `factor = 3`: `0, 3, 6, 12, 24, 48, 96, ...`
+    """
     yield 0
-    for n in itertools.count(2):
-        yield factor * (2 ** (n - 2))
+    for n in itertools.count():
+        yield factor * 2**n
 
 
 class AsyncHTTPConnection(AsyncConnectionInterface):

httpcore/_async/http11.py

@@ -20,6 +20,7 @@
     ConnectionNotAvailable,
     LocalProtocolError,
     RemoteProtocolError,
+    WriteError,
     map_exceptions,
 )
 from .._models import Origin, Request, Response
@@ -84,10 +85,21 @@ async def handle_async_request(self, request: Request) -> Response:
 
         try:
             kwargs = {"request": request}
-            async with Trace("send_request_headers", logger, request, kwargs) as trace:
-                await self._send_request_headers(**kwargs)
-            async with Trace("send_request_body", logger, request, kwargs) as trace:
-                await self._send_request_body(**kwargs)
+            try:
+                async with Trace(
+                    "send_request_headers", logger, request, kwargs
+                ) as trace:
+                    await self._send_request_headers(**kwargs)
+                async with Trace("send_request_body", logger, request, kwargs) as trace:
+                    await self._send_request_body(**kwargs)
+            except WriteError:
+                # If we get a write error while we're writing the request,
+                # then we supress this error and move on to attempting to
+                # read the response. Servers can sometimes close the request
+                # pre-emptively and then respond with a well formed HTTP
+                # error response.
+                pass
+
             async with Trace(
                 "receive_response_headers", logger, request, kwargs
             ) as trace:

httpcore/_async/http_proxy.py

@@ -64,6 +64,7 @@ def __init__(
         proxy_auth: Optional[Tuple[Union[bytes, str], Union[bytes, str]]] = None,
         proxy_headers: Union[HeadersAsMapping, HeadersAsSequence, None] = None,
         ssl_context: Optional[ssl.SSLContext] = None,
+        proxy_ssl_context: Optional[ssl.SSLContext] = None,
         max_connections: Optional[int] = 10,
         max_keepalive_connections: Optional[int] = None,
         keepalive_expiry: Optional[float] = None,
@@ -88,6 +89,7 @@ def __init__(
             ssl_context: An SSL context to use for verifying connections.
                 If not specified, the default `httpcore.default_ssl_context()`
                 will be used.
+            proxy_ssl_context: The same as `ssl_context`, but for a proxy server rather than a remote origin.
             max_connections: The maximum number of concurrent HTTP connections that
                 the pool should allow. Any attempt to send a request on a pool that
                 would exceed this amount will block until a connection is available.
@@ -122,8 +124,17 @@ def __init__(
             uds=uds,
             socket_options=socket_options,
         )
-        self._ssl_context = ssl_context
+
         self._proxy_url = enforce_url(proxy_url, name="proxy_url")
+        if (
+            self._proxy_url.scheme == b"http" and proxy_ssl_context is not None
+        ):  # pragma: no cover
+            raise RuntimeError(
+                "The `proxy_ssl_context` argument is not allowed for the http scheme"
+            )
+
+        self._ssl_context = ssl_context
+        self._proxy_ssl_context = proxy_ssl_context
         self._proxy_headers = enforce_headers(proxy_headers, name="proxy_headers")
         if proxy_auth is not None:
             username = enforce_bytes(proxy_auth[0], name="proxy_auth")
@@ -141,12 +152,14 @@ def create_connection(self, origin: Origin) -> AsyncConnectionInterface:
                 remote_origin=origin,
                 keepalive_expiry=self._keepalive_expiry,
                 network_backend=self._network_backend,
+                proxy_ssl_context=self._proxy_ssl_context,
             )
         return AsyncTunnelHTTPConnection(
             proxy_origin=self._proxy_url.origin,
             proxy_headers=self._proxy_headers,
             remote_origin=origin,
             ssl_context=self._ssl_context,
+            proxy_ssl_context=self._proxy_ssl_context,
             keepalive_expiry=self._keepalive_expiry,
             http1=self._http1,
             http2=self._http2,
@@ -163,12 +176,14 @@ def __init__(
         keepalive_expiry: Optional[float] = None,
         network_backend: Optional[AsyncNetworkBackend] = None,
         socket_options: Optional[Iterable[SOCKET_OPTION]] = None,
+        proxy_ssl_context: Optional[ssl.SSLContext] = None,
     ) -> None:
         self._connection = AsyncHTTPConnection(
             origin=proxy_origin,
             keepalive_expiry=keepalive_expiry,
             network_backend=network_backend,
             socket_options=socket_options,
+            ssl_context=proxy_ssl_context,
         )
         self._proxy_origin = proxy_origin
         self._proxy_headers = enforce_headers(proxy_headers, name="proxy_headers")
@@ -222,6 +237,7 @@ def __init__(
         proxy_origin: Origin,
         remote_origin: Origin,
         ssl_context: Optional[ssl.SSLContext] = None,
+        proxy_ssl_context: Optional[ssl.SSLContext] = None,
         proxy_headers: Optional[Sequence[Tuple[bytes, bytes]]] = None,
         keepalive_expiry: Optional[float] = None,
         http1: bool = True,
@@ -234,10 +250,12 @@ def __init__(
             keepalive_expiry=keepalive_expiry,
             network_backend=network_backend,
             socket_options=socket_options,
+            ssl_context=proxy_ssl_context,
         )
         self._proxy_origin = proxy_origin
         self._remote_origin = remote_origin
         self._ssl_context = ssl_context
+        self._proxy_ssl_context = proxy_ssl_context
         self._proxy_headers = enforce_headers(proxy_headers, name="proxy_headers")
         self._keepalive_expiry = keepalive_expiry
         self._http1 = http1

httpcore/_async/socks_proxy.py

@@ -216,6 +216,7 @@ def __init__(
 
     async def handle_async_request(self, request: Request) -> Response:
         timeouts = request.extensions.get("timeout", {})
+        sni_hostname = request.extensions.get("sni_hostname", None)
         timeout = timeouts.get("connect", None)
 
         async with self._connect_lock:
@@ -258,7 +259,8 @@ async def handle_async_request(self, request: Request) -> Response:
 
                         kwargs = {
                             "ssl_context": ssl_context,
-                            "server_hostname": self._remote_origin.host.decode("ascii"),
+                            "server_hostname": sni_hostname
+                            or self._remote_origin.host.decode("ascii"),
                             "timeout": timeout,
                         }
                         async with Trace("start_tls", logger, request, kwargs) as trace:

httpcore/_backends/sync.py

@@ -2,6 +2,7 @@
 import ssl
 import sys
 import typing
+from functools import partial
 
 from .._exceptions import (
     ConnectError,
@@ -17,6 +18,103 @@
 from .base import SOCKET_OPTION, NetworkBackend, NetworkStream
 
 
+class TLSinTLSStream(NetworkStream):  # pragma: no cover
+    """
+    Because the standard `SSLContext.wrap_socket` method does
+    not work for `SSLSocket` objects, we need this class
+    to implement TLS stream using an underlying `SSLObject`
+    instance in order to support TLS on top of TLS.
+    """
+
+    # Defined in RFC 8449
+    TLS_RECORD_SIZE = 16384
+
+    def __init__(
+        self,
+        sock: socket.socket,
+        ssl_context: ssl.SSLContext,
+        server_hostname: typing.Optional[str] = None,
+        timeout: typing.Optional[float] = None,
+    ):
+        self._sock = sock
+        self._incoming = ssl.MemoryBIO()
+        self._outgoing = ssl.MemoryBIO()
+
+        self.ssl_obj = ssl_context.wrap_bio(
+            incoming=self._incoming,
+            outgoing=self._outgoing,
+            server_hostname=server_hostname,
+        )
+
+        self._sock.settimeout(timeout)
+        self._perform_io(self.ssl_obj.do_handshake)
+
+    def _perform_io(
+        self,
+        func: typing.Callable[..., typing.Any],
+    ) -> typing.Any:
+        ret = None
+
+        while True:
+            errno = None
+            try:
+                ret = func()
+            except (ssl.SSLWantReadError, ssl.SSLWantWriteError) as e:
+                errno = e.errno
+
+            self._sock.sendall(self._outgoing.read())
+
+            if errno == ssl.SSL_ERROR_WANT_READ:
+                buf = self._sock.recv(self.TLS_RECORD_SIZE)
+
+                if buf:
+                    self._incoming.write(buf)
+                else:
+                    self._incoming.write_eof()
+            if errno is None:
+                return ret
+
+    def read(self, max_bytes: int, timeout: typing.Optional[float] = None) -> bytes:
+        exc_map: ExceptionMapping = {socket.timeout: ReadTimeout, OSError: ReadError}
+        with map_exceptions(exc_map):
+            self._sock.settimeout(timeout)
+            return typing.cast(
+                bytes, self._perform_io(partial(self.ssl_obj.read, max_bytes))
+            )
+
+    def write(self, buffer: bytes, timeout: typing.Optional[float] = None) -> None:
+        exc_map: ExceptionMapping = {socket.timeout: WriteTimeout, OSError: WriteError}
+        with map_exceptions(exc_map):
+            self._sock.settimeout(timeout)
+            while buffer:
+                nsent = self._perform_io(partial(self.ssl_obj.write, buffer))
+                buffer = buffer[nsent:]
+
+    def close(self) -> None:
+        self._sock.close()
+
+    def start_tls(
+        self,
+        ssl_context: ssl.SSLContext,
+        server_hostname: typing.Optional[str] = None,
+        timeout: typing.Optional[float] = None,
+    ) -> "NetworkStream":
+        raise NotImplementedError()
+
+    def get_extra_info(self, info: str) -> typing.Any:
+        if info == "ssl_object":
+            return self.ssl_obj
+        if info == "client_addr":
+            return self._sock.getsockname()
+        if info == "server_addr":
+            return self._sock.getpeername()
+        if info == "socket":
+            return self._sock
+        if info == "is_readable":
+            return is_socket_readable(self._sock)
+        return None
+
+
 class SyncStream(NetworkStream):
     def __init__(self, sock: socket.socket) -> None:
         self._sock = sock
@@ -47,16 +145,30 @@ def start_tls(
         server_hostname: typing.Optional[str] = None,
         timeout: typing.Optional[float] = None,
     ) -> NetworkStream:
+        if isinstance(self._sock, ssl.SSLSocket):  # pragma: no cover
+            raise RuntimeError(
+                "Attempted to add a TLS layer on top of the existing "
+                "TLS stream, which is not supported by httpcore package"
+            )
+
         exc_map: ExceptionMapping = {
             socket.timeout: ConnectTimeout,
             OSError: ConnectError,
         }
         with map_exceptions(exc_map):
             try:
-                self._sock.settimeout(timeout)
-                sock = ssl_context.wrap_socket(
-                    self._sock, server_hostname=server_hostname
-                )
+                if isinstance(self._sock, ssl.SSLSocket):  # pragma: no cover
+                    # If the underlying socket has already been upgraded
+                    # to the TLS layer (i.e. is an instance of SSLSocket),
+                    # we need some additional smarts to support TLS-in-TLS.
+                    return TLSinTLSStream(
+                        self._sock, ssl_context, server_hostname, timeout
+                    )
+                else:
+                    self._sock.settimeout(timeout)
+                    sock = ssl_context.wrap_socket(
+                        self._sock, server_hostname=server_hostname
+                    )
             except Exception as exc:  # pragma: nocover
                 self.close()
                 raise exc

httpcore/_models.py

@@ -6,6 +6,7 @@
     Iterator,
     List,
     Mapping,
+    MutableMapping,
     Optional,
     Sequence,
     Tuple,
@@ -20,7 +21,7 @@
 HeadersAsMapping = Mapping[Union[bytes, str], Union[bytes, str]]
 HeaderTypes = Union[HeadersAsSequence, HeadersAsMapping, None]
 
-Extensions = Mapping[str, Any]
+Extensions = MutableMapping[str, Any]
 
 
 def enforce_bytes(value: Union[bytes, str], *, name: str) -> bytes:

httpcore/_sync/connection.py

@@ -21,9 +21,16 @@
 
 
 def exponential_backoff(factor: float) -> Iterator[float]:
+    """
+    Generate a geometric sequence that has a ratio of 2 and starts with 0.
+
+    For example:
+    - `factor = 2`: `0, 2, 4, 8, 16, 32, 64, ...`
+    - `factor = 3`: `0, 3, 6, 12, 24, 48, 96, ...`
+    """
     yield 0
-    for n in itertools.count(2):
-        yield factor * (2 ** (n - 2))
+    for n in itertools.count():
+        yield factor * 2**n
 
 
 class HTTPConnection(ConnectionInterface):

httpcore/_sync/http11.py

@@ -20,6 +20,7 @@
     ConnectionNotAvailable,
     LocalProtocolError,
     RemoteProtocolError,
+    WriteError,
     map_exceptions,
 )
 from .._models import Origin, Request, Response
@@ -84,10 +85,21 @@ def handle_request(self, request: Request) -> Response:
 
         try:
             kwargs = {"request": request}
-            with Trace("send_request_headers", logger, request, kwargs) as trace:
-                self._send_request_headers(**kwargs)
-            with Trace("send_request_body", logger, request, kwargs) as trace:
-                self._send_request_body(**kwargs)
+            try:
+                with Trace(
+                    "send_request_headers", logger, request, kwargs
+                ) as trace:
+                    self._send_request_headers(**kwargs)
+                with Trace("send_request_body", logger, request, kwargs) as trace:
+                    self._send_request_body(**kwargs)
+            except WriteError:
+                # If we get a write error while we're writing the request,
+                # then we supress this error and move on to attempting to
+                # read the response. Servers can sometimes close the request
+                # pre-emptively and then respond with a well formed HTTP
+                # error response.
+                pass
+
             with Trace(
                 "receive_response_headers", logger, request, kwargs
             ) as trace:

httpcore/_sync/http_proxy.py

@@ -64,6 +64,7 @@ def __init__(
         proxy_auth: Optional[Tuple[Union[bytes, str], Union[bytes, str]]] = None,
         proxy_headers: Union[HeadersAsMapping, HeadersAsSequence, None] = None,
         ssl_context: Optional[ssl.SSLContext] = None,
+        proxy_ssl_context: Optional[ssl.SSLContext] = None,
         max_connections: Optional[int] = 10,
         max_keepalive_connections: Optional[int] = None,
         keepalive_expiry: Optional[float] = None,
@@ -88,6 +89,7 @@ def __init__(
             ssl_context: An SSL context to use for verifying connections.
                 If not specified, the default `httpcore.default_ssl_context()`
                 will be used.
+            proxy_ssl_context: The same as `ssl_context`, but for a proxy server rather than a remote origin.
             max_connections: The maximum number of concurrent HTTP connections that
                 the pool should allow. Any attempt to send a request on a pool that
                 would exceed this amount will block until a connection is available.
@@ -122,8 +124,17 @@ def __init__(
             uds=uds,
             socket_options=socket_options,
         )
-        self._ssl_context = ssl_context
+
         self._proxy_url = enforce_url(proxy_url, name="proxy_url")
+        if (
+            self._proxy_url.scheme == b"http" and proxy_ssl_context is not None
+        ):  # pragma: no cover
+            raise RuntimeError(
+                "The `proxy_ssl_context` argument is not allowed for the http scheme"
+            )
+
+        self._ssl_context = ssl_context
+        self._proxy_ssl_context = proxy_ssl_context
         self._proxy_headers = enforce_headers(proxy_headers, name="proxy_headers")
         if proxy_auth is not None:
             username = enforce_bytes(proxy_auth[0], name="proxy_auth")
@@ -141,12 +152,14 @@ def create_connection(self, origin: Origin) -> ConnectionInterface:
                 remote_origin=origin,
                 keepalive_expiry=self._keepalive_expiry,
                 network_backend=self._network_backend,
+                proxy_ssl_context=self._proxy_ssl_context,
             )
         return TunnelHTTPConnection(
             proxy_origin=self._proxy_url.origin,
             proxy_headers=self._proxy_headers,
             remote_origin=origin,
             ssl_context=self._ssl_context,
+            proxy_ssl_context=self._proxy_ssl_context,
             keepalive_expiry=self._keepalive_expiry,
             http1=self._http1,
             http2=self._http2,
@@ -163,12 +176,14 @@ def __init__(
         keepalive_expiry: Optional[float] = None,
         network_backend: Optional[NetworkBackend] = None,
         socket_options: Optional[Iterable[SOCKET_OPTION]] = None,
+        proxy_ssl_context: Optional[ssl.SSLContext] = None,
     ) -> None:
         self._connection = HTTPConnection(
             origin=proxy_origin,
             keepalive_expiry=keepalive_expiry,
             network_backend=network_backend,
             socket_options=socket_options,
+            ssl_context=proxy_ssl_context,
         )
         self._proxy_origin = proxy_origin
         self._proxy_headers = enforce_headers(proxy_headers, name="proxy_headers")
@@ -222,6 +237,7 @@ def __init__(
         proxy_origin: Origin,
         remote_origin: Origin,
         ssl_context: Optional[ssl.SSLContext] = None,
+        proxy_ssl_context: Optional[ssl.SSLContext] = None,
         proxy_headers: Optional[Sequence[Tuple[bytes, bytes]]] = None,
         keepalive_expiry: Optional[float] = None,
         http1: bool = True,
@@ -234,10 +250,12 @@ def __init__(
             keepalive_expiry=keepalive_expiry,
             network_backend=network_backend,
             socket_options=socket_options,
+            ssl_context=proxy_ssl_context,
         )
         self._proxy_origin = proxy_origin
         self._remote_origin = remote_origin
         self._ssl_context = ssl_context
+        self._proxy_ssl_context = proxy_ssl_context
         self._proxy_headers = enforce_headers(proxy_headers, name="proxy_headers")
         self._keepalive_expiry = keepalive_expiry
         self._http1 = http1

httpcore/_sync/socks_proxy.py

@@ -216,6 +216,7 @@ def __init__(
 
     def handle_request(self, request: Request) -> Response:
         timeouts = request.extensions.get("timeout", {})
+        sni_hostname = request.extensions.get("sni_hostname", None)
         timeout = timeouts.get("connect", None)
 
         with self._connect_lock:
@@ -258,7 +259,8 @@ def handle_request(self, request: Request) -> Response:
 
                         kwargs = {
                             "ssl_context": ssl_context,
-                            "server_hostname": self._remote_origin.host.decode("ascii"),
+                            "server_hostname": sni_hostname
+                            or self._remote_origin.host.decode("ascii"),
                             "timeout": timeout,
                         }
                         with Trace("start_tls", logger, request, kwargs) as trace:

pyproject.toml

@@ -0,0 +1,112 @@
+[build-system]
+requires = ["hatchling", "hatch-fancy-pypi-readme"]
+build-backend = "hatchling.build"
+
+[project]
+name = "httpcore"
+dynamic = ["readme", "version"]
+description = "A minimal low-level HTTP client."
+license = "BSD-3-Clause"
+requires-python = ">=3.8"
+authors = [
+    { name = "Tom Christie", email = "tom@tomchristie.com" },
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Web Environment",
+    "Framework :: AsyncIO",
+    "Framework :: Trio",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: BSD License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Topic :: Internet :: WWW/HTTP",
+]
+dependencies = [
+    "anyio>=3.0,<5.0",
+    "certifi",
+    "h11>=0.13,<0.15",
+    "sniffio==1.*",
+]
+
+[project.optional-dependencies]
+http2 = [
+    "h2>=3,<5",
+]
+socks = [
+    "socksio==1.*",
+]
+
+[project.urls]
+Documentation = "https://www.encode.io/httpcore"
+Homepage = "https://www.encode.io/httpcore/"
+Source = "https://github.com/encode/httpcore"
+
+[tool.hatch.version]
+path = "httpcore/__init__.py"
+
+[tool.hatch.build.targets.sdist]
+include = [
+    "/httpcore",
+    "/CHANGELOG.md",
+    "/README.md",
+]
+
+[tool.hatch.metadata.hooks.fancy-pypi-readme]
+content-type = "text/markdown"
+
+[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]]
+path = "README.md"
+
+[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]]
+path = "CHANGELOG.md"
+
+[tool.mypy]
+strict = true
+show_error_codes = true
+
+[[tool.mypy.overrides]]
+module = "tests.*"
+disallow_untyped_defs = false
+check_untyped_defs = true
+
+[[tool.mypy.overrides]]
+module = "h2.*"
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+module = "hpack.*"
+ignore_missing_imports = true
+
+[tool.pytest.ini_options]
+addopts = ["-rxXs", "--strict-config", "--strict-markers"]
+markers = ["copied_from(source, changes=None): mark test as copied from somewhere else, along with a description of changes made to accodomate e.g. our test setup"]
+filterwarnings = ["error"]
+
+[tool.coverage.run]
+omit = [
+    "venv/*", 
+    "httpcore/_sync/*"
+]
+include = ["httpcore/*", "tests/*"]
+
+[tool.ruff]
+exclude = [
+    "httpcore/_sync",
+    "tests/_sync",
+]
+line-length = 120
+select = [
+    "E",
+    "F",
+    "W",
+    "I"
+]
+
+[tool.ruff.isort]
+combine-as-imports = true

requirements.txt

@@ -5,25 +5,22 @@ trio==0.21.0
 
 # Docs
 mkdocs==1.4.2
-mkdocs-autorefs==0.3.1
+mkdocs-autorefs==0.5.0
 mkdocs-material==9.1.15
 mkdocs-material-extensions==1.1.1
 mkdocstrings[python-legacy]==0.22.0
 jinja2==3.1.2
 
 # Packaging
+build==0.10.0
 twine
-wheel
 
 # Tests & Linting
-anyio==3.6.2
-autoflake==1.7.7
-black==23.3.0
-coverage==7.2.7
-flake8==3.9.2  # See: https://github.com/PyCQA/flake8/pull/1438
-isort==5.11.4
-importlib-metadata==4.13.0
-mypy==1.2.0
+anyio==3.7.1
+black==23.7.0
+coverage[toml]==7.3.0
+ruff==0.0.277
+mypy==1.5.1
 trio-typing==0.8.0
 types-certifi==2021.10.8.3
 pytest==7.4.0

scripts/build

@@ -10,6 +10,6 @@ fi
 
 set -x
 
-${PREFIX}python setup.py sdist bdist_wheel
+${PREFIX}python -m build
 ${PREFIX}twine check dist/*
 ${PREFIX}mkdocs build

scripts/check

@@ -8,8 +8,7 @@ export SOURCE_FILES="httpcore tests"
 
 set -x
 
-${PREFIX}isort --check --diff --project=httpcore $SOURCE_FILES
-${PREFIX}black --exclude '/(_sync|sync_tests)/' --check --diff --target-version=py37 $SOURCE_FILES
-${PREFIX}flake8 $SOURCE_FILES
+${PREFIX}ruff check --show-source $SOURCE_FILES
+${PREFIX}black --exclude '/(_sync|sync_tests)/' --check --diff $SOURCE_FILES
 ${PREFIX}mypy $SOURCE_FILES
 scripts/unasync --check

scripts/lint

@@ -8,9 +8,8 @@ export SOURCE_FILES="httpcore tests"
 
 set -x
 
-${PREFIX}autoflake --in-place --recursive --remove-all-unused-imports $SOURCE_FILES
-${PREFIX}isort --project=httpcore $SOURCE_FILES
-${PREFIX}black --target-version=py37 --exclude '/(_sync|sync_tests)/' $SOURCE_FILES
+${PREFIX}ruff --fix $SOURCE_FILES
+${PREFIX}black --exclude '/(_sync|sync_tests)/' $SOURCE_FILES
 
 # Run unasync last because its `--check` mode is not aware of code formatters.
 # (This means sync code isn't prettified, and that's mostly okay.)

tests/_async/test_connection.py

@@ -9,10 +9,13 @@
     SOCKET_OPTION,
     AsyncHTTPConnection,
     AsyncMockBackend,
+    AsyncMockStream,
     AsyncNetworkStream,
     ConnectError,
     ConnectionNotAvailable,
     Origin,
+    RemoteProtocolError,
+    WriteError,
 )
 
 
@@ -83,7 +86,109 @@ async def test_concurrent_requests_not_available_on_http11_connections():
                 await conn.request("GET", "https://example.com/")
 
 
+@pytest.mark.filterwarnings("ignore::pytest.PytestUnraisableExceptionWarning")
 @pytest.mark.anyio
+async def test_write_error_with_response_sent():
+    """
+    If a server half-closes the connection while the client is sending
+    the request, it may still send a response. In this case the client
+    should successfully read and return the response.
+
+    See also the `test_write_error_without_response_sent` test above.
+    """
+
+    class ErrorOnRequestTooLargeStream(AsyncMockStream):
+        def __init__(self, buffer: typing.List[bytes], http2: bool = False) -> None:
+            super().__init__(buffer, http2)
+            self.count = 0
+
+        async def write(
+            self, buffer: bytes, timeout: typing.Optional[float] = None
+        ) -> None:
+            self.count += len(buffer)
+
+            if self.count > 1_000_000:
+                raise WriteError()
+
+    class ErrorOnRequestTooLarge(AsyncMockBackend):
+        async def connect_tcp(
+            self,
+            host: str,
+            port: int,
+            timeout: typing.Optional[float] = None,
+            local_address: typing.Optional[str] = None,
+            socket_options: typing.Optional[typing.Iterable[SOCKET_OPTION]] = None,
+        ) -> AsyncMockStream:
+            return ErrorOnRequestTooLargeStream(list(self._buffer), http2=self._http2)
+
+    origin = Origin(b"https", b"example.com", 443)
+    network_backend = ErrorOnRequestTooLarge(
+        [
+            b"HTTP/1.1 413 Payload Too Large\r\n",
+            b"Content-Type: plain/text\r\n",
+            b"Content-Length: 37\r\n",
+            b"\r\n",
+            b"Request body exceeded 1,000,000 bytes",
+        ]
+    )
+
+    async with AsyncHTTPConnection(
+        origin=origin, network_backend=network_backend, keepalive_expiry=5.0
+    ) as conn:
+        content = b"x" * 10_000_000
+        response = await conn.request("POST", "https://example.com/", content=content)
+        assert response.status == 413
+        assert response.content == b"Request body exceeded 1,000,000 bytes"
+
+
+@pytest.mark.anyio
+@pytest.mark.filterwarnings("ignore::pytest.PytestUnraisableExceptionWarning")
+async def test_write_error_without_response_sent():
+    """
+    If a server fully closes the connection while the client is sending
+    the request, then client should raise an error.
+
+    See also the `test_write_error_with_response_sent` test above.
+    """
+
+    class ErrorOnRequestTooLargeStream(AsyncMockStream):
+        def __init__(self, buffer: typing.List[bytes], http2: bool = False) -> None:
+            super().__init__(buffer, http2)
+            self.count = 0
+
+        async def write(
+            self, buffer: bytes, timeout: typing.Optional[float] = None
+        ) -> None:
+            self.count += len(buffer)
+
+            if self.count > 1_000_000:
+                raise WriteError()
+
+    class ErrorOnRequestTooLarge(AsyncMockBackend):
+        async def connect_tcp(
+            self,
+            host: str,
+            port: int,
+            timeout: typing.Optional[float] = None,
+            local_address: typing.Optional[str] = None,
+            socket_options: typing.Optional[typing.Iterable[SOCKET_OPTION]] = None,
+        ) -> AsyncMockStream:
+            return ErrorOnRequestTooLargeStream(list(self._buffer), http2=self._http2)
+
+    origin = Origin(b"https", b"example.com", 443)
+    network_backend = ErrorOnRequestTooLarge([])
+
+    async with AsyncHTTPConnection(
+        origin=origin, network_backend=network_backend, keepalive_expiry=5.0
+    ) as conn:
+        content = b"x" * 10_000_000
+        with pytest.raises(RemoteProtocolError) as exc_info:
+            await conn.request("POST", "https://example.com/", content=content)
+        assert str(exc_info.value) == "Server disconnected without sending a response."
+
+
+@pytest.mark.anyio
+@pytest.mark.filterwarnings("ignore::pytest.PytestUnraisableExceptionWarning")
 async def test_http2_connection():
     origin = Origin(b"https", b"example.com", 443)
     network_backend = AsyncMockBackend(

tests/_sync/test_connection.py

@@ -9,10 +9,13 @@
     SOCKET_OPTION,
     HTTPConnection,
     MockBackend,
+    MockStream,
     NetworkStream,
     ConnectError,
     ConnectionNotAvailable,
     Origin,
+    RemoteProtocolError,
+    WriteError,
 )
 
 
@@ -83,7 +86,109 @@ def test_concurrent_requests_not_available_on_http11_connections():
                 conn.request("GET", "https://example.com/")
 
 
+@pytest.mark.filterwarnings("ignore::pytest.PytestUnraisableExceptionWarning")
 
+def test_write_error_with_response_sent():
+    """
+    If a server half-closes the connection while the client is sending
+    the request, it may still send a response. In this case the client
+    should successfully read and return the response.
+
+    See also the `test_write_error_without_response_sent` test above.
+    """
+
+    class ErrorOnRequestTooLargeStream(MockStream):
+        def __init__(self, buffer: typing.List[bytes], http2: bool = False) -> None:
+            super().__init__(buffer, http2)
+            self.count = 0
+
+        def write(
+            self, buffer: bytes, timeout: typing.Optional[float] = None
+        ) -> None:
+            self.count += len(buffer)
+
+            if self.count > 1_000_000:
+                raise WriteError()
+
+    class ErrorOnRequestTooLarge(MockBackend):
+        def connect_tcp(
+            self,
+            host: str,
+            port: int,
+            timeout: typing.Optional[float] = None,
+            local_address: typing.Optional[str] = None,
+            socket_options: typing.Optional[typing.Iterable[SOCKET_OPTION]] = None,
+        ) -> MockStream:
+            return ErrorOnRequestTooLargeStream(list(self._buffer), http2=self._http2)
+
+    origin = Origin(b"https", b"example.com", 443)
+    network_backend = ErrorOnRequestTooLarge(
+        [
+            b"HTTP/1.1 413 Payload Too Large\r\n",
+            b"Content-Type: plain/text\r\n",
+            b"Content-Length: 37\r\n",
+            b"\r\n",
+            b"Request body exceeded 1,000,000 bytes",
+        ]
+    )
+
+    with HTTPConnection(
+        origin=origin, network_backend=network_backend, keepalive_expiry=5.0
+    ) as conn:
+        content = b"x" * 10_000_000
+        response = conn.request("POST", "https://example.com/", content=content)
+        assert response.status == 413
+        assert response.content == b"Request body exceeded 1,000,000 bytes"
+
+
+
+@pytest.mark.filterwarnings("ignore::pytest.PytestUnraisableExceptionWarning")
+def test_write_error_without_response_sent():
+    """
+    If a server fully closes the connection while the client is sending
+    the request, then client should raise an error.
+
+    See also the `test_write_error_with_response_sent` test above.
+    """
+
+    class ErrorOnRequestTooLargeStream(MockStream):
+        def __init__(self, buffer: typing.List[bytes], http2: bool = False) -> None:
+            super().__init__(buffer, http2)
+            self.count = 0
+
+        def write(
+            self, buffer: bytes, timeout: typing.Optional[float] = None
+        ) -> None:
+            self.count += len(buffer)
+
+            if self.count > 1_000_000:
+                raise WriteError()
+
+    class ErrorOnRequestTooLarge(MockBackend):
+        def connect_tcp(
+            self,
+            host: str,
+            port: int,
+            timeout: typing.Optional[float] = None,
+            local_address: typing.Optional[str] = None,
+            socket_options: typing.Optional[typing.Iterable[SOCKET_OPTION]] = None,
+        ) -> MockStream:
+            return ErrorOnRequestTooLargeStream(list(self._buffer), http2=self._http2)
+
+    origin = Origin(b"https", b"example.com", 443)
+    network_backend = ErrorOnRequestTooLarge([])
+
+    with HTTPConnection(
+        origin=origin, network_backend=network_backend, keepalive_expiry=5.0
+    ) as conn:
+        content = b"x" * 10_000_000
+        with pytest.raises(RemoteProtocolError) as exc_info:
+            conn.request("POST", "https://example.com/", content=content)
+        assert str(exc_info.value) == "Server disconnected without sending a response."
+
+
+
+@pytest.mark.filterwarnings("ignore::pytest.PytestUnraisableExceptionWarning")
 def test_http2_connection():
     origin = Origin(b"https", b"example.com", 443)
     network_backend = MockBackend(

unasync.py

@@ -2,22 +2,19 @@
 import os
 import re
 import sys
+from pprint import pprint
 
 SUBS = [
     ('from .._backends.auto import AutoBackend', 'from .._backends.sync import SyncBackend'),
     ('import trio as concurrency', 'from tests import concurrency'),
-    ('AsyncByteStream', 'SyncByteStream'),
     ('AsyncIterator', 'Iterator'),
-    ('AutoBackend', 'SyncBackend'),
     ('Async([A-Z][A-Za-z0-9_]*)', r'\2'),
     ('async def', 'def'),
     ('async with', 'with'),
     ('async for', 'for'),
     ('await ', ''),
     ('handle_async_request', 'handle_request'),
     ('aclose', 'close'),
-    ('aclose_func', 'close_func'),
-    ('aiterator', 'iterator'),
     ('aiter_stream', 'iter_stream'),
     ('aread', 'read'),
     ('asynccontextmanager', 'contextmanager'),
@@ -33,10 +30,14 @@
     for regex, repl in SUBS
 ]
 
+USED_SUBS = set()
 
 def unasync_line(line):
-    for regex, repl in COMPILED_SUBS:
+    for index, (regex, repl) in enumerate(COMPILED_SUBS):
+        old_line = line
         line = re.sub(regex, repl, line)
+        if old_line != line:
+            USED_SUBS.add(index)
     return line
 
 
@@ -81,6 +82,13 @@ def main():
     unasync_dir("httpcore/_async", "httpcore/_sync", check_only=check_only)
     unasync_dir("tests/_async", "tests/_sync", check_only=check_only)
 
+    if len(USED_SUBS) != len(SUBS):
+        unused_subs = [SUBS[i] for i in range(len(SUBS)) if i not in USED_SUBS]
+
+        print("These patterns were not used:")
+        pprint(unused_subs)
+        exit(1)   
+
 
 if __name__ == '__main__':
     main()