[release/8.0] Removed unused sessions from SSL_CTX internal cache

[github-actions]

Backport of #101684 to release/8.0-staging

/cc @wfurt @rzikm

Customer Impact

Reported by customer via official support. Small repro available.
Customer sees increased memory usage when establishing large amount of connections to the same host in a quick succession.

Workaround is lowering the cache size via

• System.Net.Security.TlsCacheSize AppCtx switch
• DOTNET_SYSTEM_NET_SECURITY_TLSCACHESIZE environment variable

The mechanism of the (bounded) memory leak is as follows:

• on new TLS connection, we receive TLS session ticket so that future connections can use TLS Resume (for faster handshake)
• There are 2 TLS session caches: internal one in OpenSSL, and managed one in .NET, these caches are supposed to be in sync via callbacks (e.g. OpenSSL may remove an entry internally and notifies managed code via callback that a ticket has been invalidated)
• when many connections are opened in a burst, we receive many TLS session tickets in quick succession, but the implementation keeps only the last one
• Sessions discarded from managed (.NET) TLS session cache were still being kept in the internal OpenSSL cache, until the ticket expired or the internal cache reached a predefined maximum size

The fix is to keep the two caches in sync and remove the dropped TLS session tickets from the internal cache as well.

Regression

Yes, the bug is part of TLS Session resumption feature on Linux, introduced in .NET 7.

Testing

Tested on customer provided repro, verified by tracking OpenSSL allocations in the app.

Risk

Low, the issue is well understood and the change is localized to the feature. Functional tests will verify TLS resumption works.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[wfurt]

LGTM