Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection leaks #32

Open
micsiu opened this issue May 2, 2022 · 0 comments
Open

Connection leaks #32

micsiu opened this issue May 2, 2022 · 0 comments

Comments

@micsiu
Copy link

micsiu commented May 2, 2022

Hi, in the project using r2dbc-mariadb 1.0.3 driver, we face a problem with connection leaks.

It happens only under high load, however it's pretty serious than, as in such cases it leads to leaking couple connections per minute and in the result hitting maximal number of allowed connections on database. When it happens, only way to solve the issue and release those connections is application restart.

After some investigation, I've managed to notice that number of leaked connections matches exactly number of following errors in the logs:

Connection unexpected error
	io.netty.handler.codec.DecoderException: java.lang.NegativeArraySizeException: -5
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
	...
Caused by: java.lang.NegativeArraySizeException: -5
	at org.mariadb.r2dbc.message.server.ColumnDefinitionPacket.decode(ColumnDefinitionPacket.java:129)
	at org.mariadb.r2dbc.client.DecoderState$8.decode(DecoderState.java:114)
	at org.mariadb.r2dbc.client.MariadbPacketDecoder.handleBuffer(MariadbPacketDecoder.java:97)
	at org.mariadb.r2dbc.client.MariadbPacketDecoder.decode(MariadbPacketDecoder.java:79)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
	...

Checking further correlated place in r2dbc-mariadb code:
image
I wonder if condition in line 114 is really correct. It causes that connection is being disposed (line 115) only if channel is already closed.
I did some experiments, forked r2dbc-mariadb project, added extra logging inside that condition and noticed that in our situation, condition is never true, what causes that line 115 (connection dispose) is never invoked. In the same time, compareAndSet in line 109 cause that connection is marked as closed. That from my understanding leaves connection open but marked as closed.

When I applied a test change, removed condition from line 114 (leaving line 115 without extra condition) and included such modified library in our project, problem with connection leaks disappeared for us.

Let me also add, we execute queries using DatabaseClient from spring-r2dbc 5.3.19, which internally handles closing connection, what in general seems to work fine, besides that specific error.

I would be grateful for verification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant