Peering full-tables with VyOS leads to flaps/crashes #15786

f0o · 2024-04-18T06:34:37Z

f0o
Apr 18, 2024

Hi,

I'm peering with a VyOS that provides me full BGP tables. For simplicity we're only going to consider IPv6 but this happens on IPv4 as well. It's worth to mention that this is a peering session over vlan subinterface inside a VRF (not netns).

After getting the first full tables update from VyOS I will see FRR disconnect from the peer with error: [HTHRX-GQYGJ][EC 33554454] bgp_process_packet: BGP UPDATE receipt failed for peer: a:b:c:d::6

Full log:

Apr 18 06:23:53 bgpd[470486]: [YZBG0-KRQST][EC 33554503] a:b:c:d::6 unrecognized capability code: 71 - ignored
Apr 18 06:24:05 bgpd[470486]: [GTTPK-RX2GP][EC 33554436] Malformed AS path from 2a0a:3500:0:2::6, length is 20
Apr 18 06:24:05 bgpd[470486]: [P7TRR-4J6XT][EC 33554487] a:b:c:d::6: Attribute AS_PATH, parse error - treating as withdrawal
Apr 18 06:24:05 bgpd[470486]: [WX70K-6XXVF][EC 33554454] a:b:c:d::6 rcvd UPDATE with errors in attr(s)!! Withdrawing route.
Apr 18 06:24:05 bgpd[470486]: [MX1G3-T5RQD][EC 33554438] Martian nexthop 0.0.0.0
Apr 18 06:24:05 bgpd[470486]: [HTHRX-GQYGJ][EC 33554454] bgp_process_packet: BGP UPDATE receipt failed for peer: a:b:c:d::6

After issuing debug bgp allow-martians the session will remain up and all routes (~190k) will be installed with 0 filtered.

Investigating the actual next-hop attribute, since the error mentions martians, shows not a single martian. All next-hops are the correct IPv6LL of the VyOS node and ip -6 neigh shows both GUA and LL as reachable.

I've also thought that maybe a handful of routes might be having martians and FRR is throwing away the entire table so I recorded a pcap file and inspected every single BGP Update and the Next-Hop Attribute was always the correct GUA and LL.

Next I've changed the MAC of the subinterface thinking that it might collide with the IPv6LL of a different subinterface which also has a full-table peer connected to it. After changing the MAC and cycling the interface down/up I verified a new IPv6LL was assigned to me but the error persists.

At this point I'm out of ideas. This feels specific to VyOS as my other sessions (Juniper, Cisco, Arista, FRR, Bird) are all fine with full-tables and tagged interfaces. However the VyOS partner also has fully working sessions to at least Juniper and Arista.

The setup is running on Ubuntu 22.04 LTS with FRRouting 8.1-1ubuntu1.9 (from ubuntu repositories).

Anyone got any ideas what I could try next to debug this?

Answered by f0o

Apr 18, 2024

To summarize:

10.x Solves VyOS Peerings
Breaks blackhole route leak into VRFs; Workaround for that #15786 (reply in thread)
Breaks bgp suppress-fib-pending which leads to prefixes not being advertised ever (even when peer was shut/noshut)
Upgrade adds no neighbor xyz enforce-first-as and in the process breaks other parts of the config (name rt import/export statements missing or incomplete randomly) leading to unusable state

This was a very bumpy upgrade and I hope nobody else has to go through it. Or if you do, rather start with a blank config as you will end up rewriting or correcting a lot anyway.

View full answer

ton31337 · 2024-04-18T06:48:29Z

ton31337
Apr 18, 2024
Collaborator

Please use 9.x or 10.x releases, 8.1 is too old to be supported.

6 replies

ton31337 Apr 18, 2024
Collaborator

Possible to get the backtrace of the crash or even a coredump?

f0o Apr 18, 2024
Author

so 10.x wont work; it's starting off with errors adding blackhole routes to the default vrf:

[WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=317253, pid=2669282327

f0o Apr 18, 2024
Author

so about the Blackholes:

I got this:

vrf IBGP
 ip route 185.243.23.0/24 blackhole 254
 ip route 193.182.61.0/24 blackhole 254
 ipv6 route 2a0e:b106:20::/44 blackhole 254
 ipv6 route 2a0a:3507:1::/48 blackhole 254
exit-vrf

Those are installed correctly into the IBGP VRF:

rt1 ~ # ip r sh vrf IBGP | grep black
blackhole 185.243.23.0/24 proto static metric 20
blackhole 193.182.61.0/24 proto static metric 20

But they're not redistributed into any other VRF anymore. This was working in 8.1 and no config changes have been done.

f0o Apr 18, 2024
Author

Logs about redistribute static failing to redistribute blackhole routes into other VRFs:

Apr 18 07:21:53 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7017818, pid=2669282327
Apr 18 07:21:53 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7017819, pid=2669282327
Apr 18 07:21:53 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7017820, pid=2669282327
Apr 18 07:21:53 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7017821, pid=2669282327
Apr 18 07:21:53 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7017822, pid=2669282327
Apr 18 07:21:53 rt1 zebra[2039]: [VYKYC-709DP] CUST_BODEMS(11:3010):185.243.23.0/24: Route install failed
Apr 18 07:21:53 rt1 zebra[2039]: [VYKYC-709DP] TRANSIT_FOILHAT(16:1011):185.243.23.0/24: Route install failed
Apr 18 07:21:53 rt1 zebra[2039]: [VYKYC-709DP] TRANSIT_FOILHAT(16:1011):193.182.61.0/24: Route install failed
Apr 18 07:21:53 rt1 zebra[2039]: [VYKYC-709DP] TRANSIT_OBE(17:1010):185.243.23.0/24: Route install failed
Apr 18 07:21:53 rt1 zebra[2039]: [VYKYC-709DP] TRANSIT_OBE(17:1010):193.182.61.0/24: Route install failed
Apr 18 07:21:53 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7017827, pid=2669282327
Apr 18 07:21:53 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7017828, pid=2669282327
Apr 18 07:21:53 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7017834, pid=2669282327
Apr 18 07:21:53 rt1 zebra[2039]: [VYKYC-709DP] IX_SONIX(13:2010):185.243.23.0/24: Route install failed
Apr 18 07:21:53 rt1 zebra[2039]: [VYKYC-709DP] IX_SONIX(13:2010):193.182.61.0/24: Route install failed
Apr 18 07:21:53 rt1 zebra[2039]: [VYKYC-709DP] CUST_BODEMS(11:3010):193.182.61.0/24: Route install failed
Apr 18 07:22:24 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7021434, pid=2669282327
Apr 18 07:22:24 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7021435, pid=2669282327
Apr 18 07:22:24 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7021436, pid=2669282327
Apr 18 07:22:24 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7021437, pid=2669282327
Apr 18 07:22:24 rt1 zebra[2039]: [VYKYC-709DP] TRANSIT_FOILHAT(16:1011):2a0a:3507:1::/48: Route install failed
Apr 18 07:22:24 rt1 zebra[2039]: [VYKYC-709DP] TRANSIT_FOILHAT(16:1011):2a0e:b106:20::/44: Route install failed
Apr 18 07:22:24 rt1 zebra[2039]: [VYKYC-709DP] TRANSIT_OBE(17:1010):2a0a:3507:1::/48: Route install failed
Apr 18 07:22:24 rt1 zebra[2039]: [VYKYC-709DP] TRANSIT_OBE(17:1010):2a0e:b106:20::/44: Route install failed
Apr 18 07:22:24 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7021446, pid=2669282327
Apr 18 07:22:24 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7021447, pid=2669282327
Apr 18 07:22:24 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7021454, pid=2669282327
Apr 18 07:22:24 rt1 zebra[2039]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=7021455, pid=2669282327
Apr 18 07:22:24 rt1 zebra[2039]: [VYKYC-709DP] CUST_BODEMS(11:3010):2a0a:3507:1::/48: Route install failed
Apr 18 07:22:24 rt1 zebra[2039]: [VYKYC-709DP] CUST_BODEMS(11:3010):2a0e:b106:20::/44: Route install failed
Apr 18 07:22:24 rt1 zebra[2039]: [VYKYC-709DP] IX_SONIX(13:2010):2a0a:3507:1::/48: Route install failed
Apr 18 07:22:24 rt1 zebra[2039]: [VYKYC-709DP] IX_SONIX(13:2010):2a0e:b106:20::/44: Route install failed

//EDIT:

Changing from blackhole to Null0 has the same error/effect

f0o Apr 18, 2024
Author

OK I got the blackholes to work by:

Add a new VRF (I named it NULL)
Add blackhole routes into NULL VRF (like above IBGP block)
Replace static routes in IBGP block with NULL nexthop-vrf NULL 254 instead of blackhole
It's now redistributed correctly to other VRFs.

f0o · 2024-04-18T07:16:25Z

f0o
Apr 18, 2024
Author

Possible to get the backtrace of the crash or even a coredump?

#15543 you mean like I did here? if not then please be more specific of what you want me to execute/log/upload

0 replies

f0o · 2024-04-18T07:42:09Z

f0o
Apr 18, 2024
Author

Ok so good news: 10.x does solve it

bad news:

10.x breaks VRF leaks for blackholes (cannot be leaked without using yet another helper-vrf as nexthop prior to redistribution)
10.x also breaks different VRF-leak issue which I'm still trying to understand - will update this comment when I get more information.

So 10.x has this oddity:

# do sh bgp vrf TRANSIT_OBE ipv4 route-map TRANSIT-OUT wide
BGP table version is 966738, local router ID is 185.243.23.254, vrf id 17
Default local pref 100, local AS 203038
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network                                      Next Hop                                  Metric LocPrf Weight Path
 *> 185.243.20.0/22                              185.243.23.253@11                                   300      0 200490 i
 *> 185.243.23.0/24                              0.0.0.0@12                                     0    500  32768 i
 *> 193.182.61.0/24                              0.0.0.0@12                                     0    500  32768 i

Displayed 3 routes and 933769 total paths
# do sh bgp vrf TRANSIT_OBE ipv4 neighbors 46.227.64.56 advertised-routes wide
BGP table version is 966863, local router ID is 185.243.23.254, vrf id 17
Default local pref 100, local AS 203038
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network                                      Next Hop                                  Metric LocPrf Weight Path
 *> 185.243.23.0/24                              0.0.0.0                                        0    500  32768 i
 *> 193.182.61.0/24                              0.0.0.0                                        0    500  32768 i

Total number of prefixes 2
# do sh bgp vrf TRANSIT_OBE ipv4 185.243.23.0/22
BGP routing table entry for 185.243.20.0/22, version 959776
Paths: (1 available, best #1, vrf TRANSIT_OBE)
  Not advertised to any peer
  Imported from 203038:101:185.243.20.0/22
  200490
    185.243.23.253 from 0.0.0.0 (185.243.23.254) vrf CUST_BODEMS(11)
      Origin IGP, localpref 300, valid, sourced, local, best (First path received)
      Community: 0:100 0:101
      Extended Community: RT:203038:100
      Originator: 185.243.23.254
      Last update: Thu Apr 18 07:39:38 2024

The /22 is inserted in the VRF and added to the kernel, it's also matching the correct outgoing route-map. But it's not advertised.

One Step forward, One Step backward....

4 replies

f0o Apr 18, 2024
Author

After multiple restarts and reboots and shutting down all peers and starting them one by one I can say that 10.x is not production ready.

It's literally a coinflip if it will advertise your prefixes or not.

I have two peers that utilize identical route-map for outgoing advertisements. One advertises all prefixes (mine+downstream) the other does not advertise any at all on v4 but all of them on v6.

f0o Apr 18, 2024
Author

turns out on start-up bgpd just didnt apply all import/export vpn statements everywhere.

This makes no sense... WIll reboot the box once again just to make sure that the config is stable now after re-writing it.

f0o Apr 18, 2024
Author

ok so 10.x rewrote my config quite a bit. Just finished cleaning up all those no neighbor 1.2.3.4 enforce-first-as entries that it put on every neighbor.

I suspect when the upgrade rewrote the config it messed up some of the rt import/export and vpn import/export lines which explained why all of a sudden no routes where advertised.

The intermittent issue of #15786 (comment) is still unexplained. This could be because I changed the blackhole routes into their own VRF in runtime? But that sounds very fragile then if moving static routes could nuke advertisements elsewhere...

7th reboot now, hoping that the config gets applied correctly and all sessions turn up and advertisements are correct.

f0o Apr 18, 2024
Author

And I'm back onto the issue with prefixes not being advertised despite being in kernel and matching the route-map.

Could this be related to bgp suppress-fib-pending?

f0o · 2024-04-18T10:13:39Z

f0o
Apr 18, 2024
Author

To summarize:

10.x Solves VyOS Peerings
Breaks blackhole route leak into VRFs; Workaround for that Peering full-tables with VyOS leads to flaps/crashes #15786 (reply in thread)
Breaks bgp suppress-fib-pending which leads to prefixes not being advertised ever (even when peer was shut/noshut)
Upgrade adds no neighbor xyz enforce-first-as and in the process breaks other parts of the config (name rt import/export statements missing or incomplete randomly) leading to unusable state

This was a very bumpy upgrade and I hope nobody else has to go through it. Or if you do, rather start with a blank config as you will end up rewriting or correcting a lot anyway.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Peering full-tables with VyOS leads to flaps/crashes #15786

{{title}}

Replies: 4 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Peering full-tables with VyOS leads to flaps/crashes #15786

f0o Apr 18, 2024

Replies: 4 comments · 10 replies

ton31337 Apr 18, 2024 Collaborator

ton31337 Apr 18, 2024 Collaborator

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o Apr 18, 2024 Author

f0o
Apr 18, 2024

Replies: 4 comments 10 replies

ton31337
Apr 18, 2024
Collaborator

ton31337 Apr 18, 2024
Collaborator

f0o Apr 18, 2024
Author

f0o Apr 18, 2024
Author

f0o Apr 18, 2024
Author

f0o Apr 18, 2024
Author

f0o
Apr 18, 2024
Author

f0o
Apr 18, 2024
Author

f0o Apr 18, 2024
Author

f0o Apr 18, 2024
Author

f0o Apr 18, 2024
Author

f0o Apr 18, 2024
Author

f0o
Apr 18, 2024
Author