You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, in the project using r2dbc-mariadb 1.0.3 driver, we face a problem with connection leaks.
It happens only under high load, however it's pretty serious than, as in such cases it leads to leaking couple connections per minute and in the result hitting maximal number of allowed connections on database. When it happens, only way to solve the issue and release those connections is application restart.
After some investigation, I've managed to notice that number of leaked connections matches exactly number of following errors in the logs:
Connection unexpected error
io.netty.handler.codec.DecoderException: java.lang.NegativeArraySizeException: -5
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
...
Caused by: java.lang.NegativeArraySizeException: -5
at org.mariadb.r2dbc.message.server.ColumnDefinitionPacket.decode(ColumnDefinitionPacket.java:129)
at org.mariadb.r2dbc.client.DecoderState$8.decode(DecoderState.java:114)
at org.mariadb.r2dbc.client.MariadbPacketDecoder.handleBuffer(MariadbPacketDecoder.java:97)
at org.mariadb.r2dbc.client.MariadbPacketDecoder.decode(MariadbPacketDecoder.java:79)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
...
Checking further correlated place in r2dbc-mariadb code:
I wonder if condition in line 114 is really correct. It causes that connection is being disposed (line 115) only if channel is already closed.
I did some experiments, forked r2dbc-mariadb project, added extra logging inside that condition and noticed that in our situation, condition is never true, what causes that line 115 (connection dispose) is never invoked. In the same time, compareAndSet in line 109 cause that connection is marked as closed. That from my understanding leaves connection open but marked as closed.
When I applied a test change, removed condition from line 114 (leaving line 115 without extra condition) and included such modified library in our project, problem with connection leaks disappeared for us.
Let me also add, we execute queries using DatabaseClient from spring-r2dbc 5.3.19, which internally handles closing connection, what in general seems to work fine, besides that specific error.
I would be grateful for verification.
The text was updated successfully, but these errors were encountered:
Hi, in the project using r2dbc-mariadb 1.0.3 driver, we face a problem with connection leaks.
It happens only under high load, however it's pretty serious than, as in such cases it leads to leaking couple connections per minute and in the result hitting maximal number of allowed connections on database. When it happens, only way to solve the issue and release those connections is application restart.
After some investigation, I've managed to notice that number of leaked connections matches exactly number of following errors in the logs:
Checking further correlated place in r2dbc-mariadb code:
I wonder if condition in line 114 is really correct. It causes that connection is being disposed (line 115) only if channel is already closed.
I did some experiments, forked r2dbc-mariadb project, added extra logging inside that condition and noticed that in our situation, condition is never true, what causes that line 115 (connection dispose) is never invoked. In the same time, compareAndSet in line 109 cause that connection is marked as closed. That from my understanding leaves connection open but marked as closed.
When I applied a test change, removed condition from line 114 (leaving line 115 without extra condition) and included such modified library in our project, problem with connection leaks disappeared for us.
Let me also add, we execute queries using DatabaseClient from spring-r2dbc 5.3.19, which internally handles closing connection, what in general seems to work fine, besides that specific error.
I would be grateful for verification.
The text was updated successfully, but these errors were encountered: