Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kernel BUG under MTU 80 #2059

Open
EvgeniiMekhanik opened this issue Feb 20, 2024 · 4 comments
Open

Kernel BUG under MTU 80 #2059

EvgeniiMekhanik opened this issue Feb 20, 2024 · 4 comments
Assignees
@EvgeniiMekhanik
Labels
bug
Milestone
0.9 - LA

Comments

@EvgeniiMekhanik
Copy link
Contributor

Start test: t_stress.test_stress.H2LoadStressMTU80.test
[ 256.266865] [tempesta fw] Open listen socket on: 0.0.0.0:443
[ 256.282750] [tempesta fw] Tempesta FW is ready
[ 256.923694] ------------[ cut here ]------------
[ 256.924052] kernel BUG at net/ipv4/tcp_input.c:1440!
[ 256.924390] invalid opcode: 0000 [#1] SMP NOPTI
[ 256.924710] CPU: 6 PID: 7478 Comm: h2load Tainted: G OE 5.10.35+ #273
[ 256.925254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
[ 256.925861] RIP: 0010:tcp_shifted_skb+0x32a/0x3b0
[ 256.926213] Code: ff 0f 0b 48 c7 c7 50 c4 35 92 e8 ce a8 16 00 0f 0b e9 9e fd ff ff 4c 89 e7 e8 b2 8f ae ff 49 89 85 20 08 00 00 e9 16 fe ff ff <0f> 0b 48 c7 c7 40 c4 35 92 e8 a5 a8 16 00 48 c7 43 50 00 00 00 00
[ 256.927483] RSP: 0018:ffffa815001fc9a0 EFLAGS: 00010246
[ 256.927829] RAX: 0000000000000010 RBX: ffff8c3e72494000 RCX: 0000000000000004
[ 256.928298] RDX: 0000000000000000 RSI: 000000000f50d071 RDI: ffff8c3fb828d580
[ 256.928879] RBP: ffffa815001fc9d0 R08: 0000000000000001 R09: 0000000000000000
[ 256.929406] R10: 0000000000000000 R11: ffff8c3e72499800 R12: ffff8c3e72499800
[ 256.929931] R13: ffff8c3fb828d580 R14: 0000000000000004 R15: 0000000000000090
[ 256.930427] FS: 00007fe2f2fff640(0000) GS:ffff8c4077d80000(0000) knlGS:0000000000000000
[ 256.930978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 256.931700] CR2: 0000562b5920e870 CR3: 00000001581be000 CR4: 0000000000750ee0
[ 256.932292] PKRU: 55555554
[ 256.932494] Call Trace:
[ 256.932669]
[ 256.932830] tcp_sacktag_walk+0x37d/0x570
[ 256.933139] tcp_sacktag_write_queue+0x6b6/0x1620
[ 256.933477] ? tcp_rack_mark_lost+0x65/0x100
[ 256.933824] ? tfw_h2_find_not_closed_stream+0x1b/0x60 [tempesta_fw]
[ 256.934284] ? bzero_fast+0xe/0x10 [tempesta_lib]
[ 256.934718] tcp_ack+0xd9b/0x1760
[ 256.934967] tcp_rcv_established+0x1a5/0x6b0
[ 256.935265] tcp_v4_do_rcv+0x140/0x200
[ 256.935572] tcp_v4_rcv+0xcd0/0xe20
[ 256.935864] ip_protocol_deliver_rcu+0x44/0x230
[ 256.936175] ip_local_deliver_finish+0x48/0x60
[ 256.936488] ip_local_deliver+0xf8/0x110
[ 256.936760] ? ip_protocol_deliver_rcu+0x230/0x230
[ 256.937092] ip_rcv_finish+0x87/0xa0
[ 256.937346] ip_rcv+0xce/0xe0
[ 256.937557] ? ip_rcv_finish_core.constprop.0+0x470/0x470
[ 256.937925] __netif_receive_skb_one_core+0x86/0xa0
[ 256.938260] __netif_receive_skb+0x18/0x60
[ 256.938557] process_backlog+0x9e/0x170
[ 256.938831] net_rx_action+0x13b/0x430

Was reproduced when I work on 2047. Run http2_general.test_h2_headers.AddBackendLongHeaders.test, http2_general.test_h2_headers.TestIPv6.test_request_with_some_data , then run t_stress.test_stress.H2LoadStressMTU80.test
@EvgeniiMekhanik
Copy link
Contributor Author

ee2d801 Tempesta FW
22b085391310464e2ea26d34723c0b7a3a807aa5 tempesta-test

@krizhanovsky krizhanovsky added this to the 0.8 - Beta milestone Feb 20, 2024
@krizhanovsky
Copy link
Contributor

Probably one more trace for this bug, but maybe another, frequently appearing on the webserver

2024-03-06 17:19:29.258	
[598756.795801] WARNING: CPU: 19 PID: 0 at net/ipv4/tcp_input.c:1958 tcp_sacktag_write_queue+0x71b/0x780
2024-03-06 17:19:29.258	
[598756.795228] ------------[ cut here ]------------
2024-03-06 17:19:29.258	
[598756.794712] ---[ end trace d2a3a97994ee5a52 ]---
2024-03-06 17:19:29.258	
[598756.794327]  secondary_startup_64_no_verify+0xb0/0xbb
2024-03-06 17:19:29.258	
[598756.793906]  cpu_startup_entry+0x14/0x20
2024-03-06 17:19:29.258	
[598756.793569]  do_idle+0x68/0xb0
2024-03-06 17:19:29.258	
[598756.792901]  ? tsc_verify_tsc_adjust+0x2d/0xc0
2024-03-06 17:19:29.258	
[598756.792411]  cpuidle_idle_call+0x133/0x190
2024-03-06 17:19:29.258	
[598756.791681]  default_idle_call+0x2d/0xa0
2024-03-06 17:19:29.258	
[598756.790869] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
2024-03-06 17:19:29.258	
[598756.790194] R10: 000000000002e42e R11: 0000000000000000 R12: 0000000000000000
2024-03-06 17:19:29.258	
[598756.789488] RBP: 0000000000000000 R08: 000000007c5145a0 R09: 0000000000000000
2024-03-06 17:19:29.258	
[598756.788704] RDX: 0000000000957312 RSI: 0000000000000013 RDI: ffff9cef2fcde180
2024-03-06 17:19:29.258	
[598756.788155] RAX: ffff9cef2fceab40 RBX: ffff9cedc023d700 RCX: ffff9cef2fceab40
2024-03-06 17:19:29.258	
[598756.786470] RSP: 0018:ffffad1a40103ef0 EFLAGS: 00000206
2024-03-06 17:19:29.258	
[598756.785864] Code: fb 65 48 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f ae 38 0f ae f0 eb b6 90 e9 07 00 00 00 0f 00 2d 16 c8 47 00 fb f4 <c3> cc 65 8b 15 c9 89 88 59 89 d2 48 8b 05 30 26 67 01 48 03 04 d5
2024-03-06 17:19:29.258	
[598756.785434] RIP: 0010:default_idle+0xe/0x10
2024-03-06 17:19:29.258	
[598756.784850]  asm_sysvec_call_function_single+0x12/0x20
2024-03-06 17:19:29.258	
[598756.784412]  sysvec_call_function_single+0x2e/0x80
2024-03-06 17:19:29.258	
[598756.783912]  irq_exit_rcu+0x83/0xb0
2024-03-06 17:19:29.258	
[598756.783676]  do_softirq_own_stack+0x32/0x40
2024-03-06 17:19:29.258	
[598756.783131]  </IRQ>
2024-03-06 17:19:29.258	
[598756.782817]  asm_call_irq_on_stack+0xf/0x20
2024-03-06 17:19:29.258	
[598756.782431]  __do_softirq+0xc0/0x26e
2024-03-06 17:19:29.258	
[598756.781822]  net_rx_action+0x95/0x180
2024-03-06 17:19:29.258	
[598756.781501]  napi_poll+0x8a/0x1b0
2024-03-06 17:19:29.258	
[598756.780959]  process_backlog+0x83/0x120
2024-03-06 17:19:29.258	
[598756.780459]  __netif_receive_skb_one_core+0x40/0x50
2024-03-06 17:19:29.258	
[598756.779835]  ip_local_deliver_finish+0x3f/0x50
2024-03-06 17:19:29.258	
[598756.779490]  ip_protocol_deliver_rcu+0x14/0x170
2024-03-06 17:19:29.258	
[598756.778815]  ? tcp_v4_early_demux+0xe7/0x140
2024-03-06 17:19:29.258	
[598756.778334]  ? tcp_v4_early_demux+0xa3/0x140
2024-03-06 17:19:29.258	
[598756.778043]  tcp_v4_rcv+0xbb4/0xce0
2024-03-06 17:19:29.258	
[598756.777407]  tcp_v4_do_rcv+0x126/0x1e0
2024-03-06 17:19:29.258	
[598756.776899]  ? tfw_filter_check_ip+0x3a/0xa0 [tempesta_fw]
2024-03-06 17:19:29.258	
[598756.776527]  tcp_rcv_established+0x173/0x620
2024-03-06 17:19:29.258	
[598756.776003]  ? tcp_write_xmit+0x399/0x9c0
2024-03-06 17:19:29.258	
[598756.775603]  tcp_ack+0x4a2/0x810
2024-03-06 17:19:29.258	
[598756.775208]  ? free_unref_page_commit+0x86/0x110
2024-03-06 17:19:29.258	
[598756.774869]  <IRQ>
2024-03-06 17:19:29.257	
[598756.774404] Call Trace:
2024-03-06 17:19:29.257	
[598756.773830] PKRU: 55555554
2024-03-06 17:19:29.257	
[598756.773135] CR2: 00007f7057397f58 CR3: 0000000302364001 CR4: 0000000000770ee0
2024-03-06 17:19:29.257	
[598756.772393] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2024-03-06 17:19:29.257	
[598756.771743] FS:  0000000000000000(0000) GS:ffff9cef2fcc0000(0000) knlGS:0000000000000000
2024-03-06 17:19:29.257	
[598756.770917] R13: ffff9cedcc1daf80 R14: 00000000ef3bb94e R15: 0000000000000003
2024-03-06 17:19:29.257	
[598756.770373] R10: 0000000000000000 R11: 00000000ef3d3232 R12: 0000000000000003
2024-03-06 17:19:29.257	
[598756.769699] RBP: ffff9cedccce7b00 R08: 0000000000000000 R09: ffffad1a40458d50
2024-03-06 17:19:29.257	
[598756.768748] RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff9cedcc1daf80
2024-03-06 17:19:29.257	
[598756.768229] RAX: 0000000000000000 RBX: ffff9cedccce7c84 RCX: 0000000000000030
2024-03-06 17:19:29.257	
[598756.766758] RSP: 0018:ffffad1a40458ca8 EFLAGS: 00010283
2024-03-06 17:19:29.257	
[598756.765962] Code: 68 3b 47 2c 79 55 48 83 c3 08 e9 58 fc ff ff 48 83 c7 08 48 8b 3f 48 85 ff 0f 85 b3 fe ff ff e9 c0 fe ff ff 48 83 c7 10 eb e9 <0f> 0b e9 81 fa ff ff 0f 0b e9 6e fa ff ff 0f 0b e9 58 fa ff ff 0f
2024-03-06 17:19:29.257	
[598756.765299] RIP: 0010:tcp_sacktag_write_queue+0x71b/0x780
2024-03-06 17:19:29.257	
[598756.764441] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
2024-03-06 17:19:29.257	
[598756.761163] CPU: 19 PID: 0 Comm: swapper/19 Tainted: G        W  O      5.10.35-tfw-secperf+ #1
2024-03-06 17:19:29.257	
[598756.759920] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sg serio_raw virtio_balloon virtio_console button loop sch_fq_codel msr drm efi_pstore configfs ip_tables x_tables ext4 crc16 mbcache jbd2 efivars linear md_mod sr_mod cdrom virtio_net virtio_rng net_failover rng_core virtio_blk failover ahci libahci libata crct10dif_pclmul scsi_mod psmouse i2c_i801 lpc_ich i2c_smbus mfd_core virtio_pci virtio_ring virtio

@EvgeniiMekhanik
Copy link
Contributor Author

2024-03-05 11:00:26.002
[489613.530884] WARNING: CPU: 18 PID: 0 at net/ipv4/tcp_input.c:2899 tcp_fastretrans_alert+0x5eb/0xa00

@krizhanovsky krizhanovsky mentioned this issue Apr 25, 2024
Merged
@krizhanovsky
Copy link
Contributor

Both the 2 warnings (#2059 (comment) and #2059 (comment)) are fixed in #2079.

The initial BUG_ON() isn't fixed yet and did not reporduced.

@krizhanovsky krizhanovsky modified the milestones: 0.8 - Beta, 0.9 - LA May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EvgeniiMekhanik EvgeniiMekhanik

Labels
bug
Projects
None yet
Milestone
0.9 - LA
Development

No branches or pull requests
2 participants
@krizhanovsky @EvgeniiMekhanik