Numerous TCP stack warnings when Tempesta is involved #439
Comments
|
The test bed set up is as follows: A separate local network is set up over these 10G links. The network consists of point-to-point connections between the neighbouring servers. The server with Tempesta (and the patched kernel) is in the middle and has A simple benchmark test is usually as simple as this: Below are the combinations tried and the observed results:
In all of these tests GRO is set on all network interfaces, and I can confirm that GROed SKBs are indeed seen in Tempesta. I certainly appears that the issue is most pronounced when Tempesta transfers SKBs between sockets that are on different 10G interfaces controlled by the same |
|
Tempesta has two types of modifications of SKBs.
In tests, the appearance of kernel warnings from Linux TCP/IP stack is NOT related to these modifications of SKBs. When these modifications are completely commented out, the warnings still occur at about the same rate. |
|
In view of multiple out of order segments and multiple out of order ACKs the Linux TCP/IP stack resets these connections from time to time by sending RST. The connections are closed, and the outstanding requests are deleted. These are the requests without responses, and deletetion of these requests is reflected in Tempesta statistics as shown below: |
|
The issue is also reliably reproduced on the test bed that consist of a host and a VM running on the host:
|
|
I've done with some tests with customized nginx and ab. First of all, I've patched nginx (1.9.15) to enable SO_DEBUG option on it's sockets: diff --git a/src/core/ngx_connection.c b/src/core/ngx_connection.c
index 5a53bac..71823e3 100644
--- a/src/core/ngx_connection.c
+++ b/src/core/ngx_connection.c
@@ -559,6 +559,8 @@ ngx_open_listening_sockets(ngx_cycle_t *cycle)
continue;
}
+ setsockopt(s, SOL_SOCKET, SO_DEBUG, "\x01\x00\x00\x00", sizeof(int));
+
#if (NGX_HAVE_UNIX_DOMAIN)
if (ls[i].sockaddr->sa_family == AF_UNIX) {
diff --git a/src/event/ngx_event_accept.c b/src/event/ngx_event_accept.c
index 1c87a34..63be385 100644
--- a/src/event/ngx_event_accept.c
+++ b/src/event/ngx_event_accept.c
@@ -135,6 +135,8 @@ ngx_event_accept(ngx_event_t *ev)
return;
}
+ setsockopt(s, SOL_SOCKET, SO_DEBUG, "\x01\x00\x00\x00", sizeof(int));
+
#if (NGX_STAT_STUB)
(void) ngx_atomic_fetch_add(ngx_stat_accepted, 1);
#endifNext, I've compiled nginx with the following configure options: And finally, I run nginx and benchmark on the same host: $ cat stress-test.sh
#!/bin/bash
for i in $(eval echo {1..$1}); do
(sleep 1 && siege -b -c 100 http://127.0.0.1:80/index.html &>/dev/null)&
done
$ stress-test.sh 100Next, I've got the following in kernel's log: So, it seems that it's normal for the TCP to have such messages produced under the heavy load and |
|
Need to test whether we have the problem on standard Linux kernel + Nginx on real hardware testbed. |
|
OFO triggers when the kernel meets a packet in the stream which SEQ-number is greater than SEQ expected by the TCP. With At the moment, the following issues were gathered:
In my opinion packet drops causes the OFO's. But that might be not the only reason. More research is needed. |
|
Let's try to reproduce the issue with our 3 machines testbed with standard 3.16 Debian kernel. It has sense to build wrk with SOCK_DEBUG as well as both the Nginx's (upstream and proxy) and check the results. Meantime, we must learn reason of the reordering - which mechanism exactly causes packets reordering (probably it's caused by TSQ on ixgbe adapter itself). |
|
Now, when #645 is fixed, I don't see , then we get the problem, but only on connections establishing, there are no the warnings while the becnhmark is working. Also note work queue overrun messages. Nginx running on Linux 4.1.27-tfw doesn't exhibit the behaviour. |
…te device queues; some debugging
minimizing probability of livelock between new connections establishing and processing packets in already established connections.
|
This is just a performance issue, linked with #488. Many (8192 in my tests) concurrently establishing connections are stuck at
Nginx doesn't exhibit the problem because it doesn't process application layer in softirq. Doing more work in softirq obviously emphasises the problem. We also brutally drop TCP segments in performance issues worsen the problem. Thus, the problem is multi-layer. Firstly, I add performance optimizations by proper system tuning in It worth to mention that kernel compiled with totally enabled SOCK_DBG for all the sockets and without loaded Tempesta FW spits OFO messages. So I move setting of the option under |
…avoid overruns on huge proxied traffic
Tempesta runs on a test bed with two 10G Ethernet adapters. The back end server and the benchmark host are connected to the test bed via 10G links as well. Tempesta sets
SOCK_DBGflag on server sockets. When the benchmark is running, the kernel on the test bed spits numerous warnings like those below. These warnings come in different combinations.The text was updated successfully, but these errors were encountered: