Improve networks_file storage and load/query algorithm for IPv6 networks

[rodecker]

Description
I have some hosts that cannot receive full routing tables via BGP, so I wish to load them via a networks file. I've noticed a large performance difference between loading the IPv4 and IPv6 full routing tables from file. Loading the IPv6 table takes a very long time.

Build and config

pmacctd 1.7.9-git (20230121-0 (b5637b0))`
./configure --enable-jansson --enable-kafka

! pmacctd.conf
daemonize: false
pre_tag_map: /etc/pmacct/pretag.map
networks_file: /etc/pmacct/networks.lst
networks_file_no_lpm: true
!
!
aggregate: label, sampling_rate, in_iface, out_iface, src_mac, dst_mac, src_as, dst_as, src_net, dst_net, src_host, dst_host, etype, timestamp_arrival
!
! pcap capturing config
pcap_ifindex: map
pcap_interfaces_map: /etc/pmacct/iface.map
sampling_rate: 10 
pmacctd_renormalize: true
pmacctd_as: longest
pmacctd_net: longest
!
! export config
plugins: print
print_refresh_time: 60
print_output: json
!
timestamps_rfc3339: true
timestamps_secs: true

Steps to reproduce

Download IPv4 prefix list
# whois -h riswhois.ripe.net -- -F -M 0/0 | egrep -v "^%" | egrep -v "^$" | awk '{ print $1","$2; }' > networks.lst
# wc -l networks.lst
1093778 networks.lst

Start pmacctd
# /usr/local/sbin/pmacctd -f /etc/pmacct/pmacctd.conf -d

Wait until the loading is done
WARN ( default_print/print ): no print_output_file and no print_output_lock_file defined.

The load time is +/- 1 minute. During this time v4 nh: debug messages are continuously written to the screen while prefixes are being loaded.

Download IPv6 prefix list
# whois -h riswhois.ripe.net -- -F -M ::/0 | egrep -v "^%" | egrep -v "^$" | awk '{ print $1","$2; }' > networks.lst
# wc -l networks.lst
224898 networks.lst

Start pmacctd
# /usr/local/sbin/pmacctd -f /etc/pmacct/pmacctd.conf -d

Wait until the loading is done.
WARN ( default_print/print ): no print_output_file and no print_output_lock_file defined.

The load time is +/- 13 minutes. Most of this time no output is written to the screen and the pmacctd process is spinning at 99.9% CPU. The long processing time is in step 4a, bulding hierarchies: https://github.com/pmacct/pmacct/blob/master/src/net_aggr.c#L1532-L1559

Is this expected?

[paololucente]

Hi Martin ( @rodecker ),

Thank you for reporting this issue. You are indeed right, i could reproduce the issue. The root cause is very simple, the algorithm to load data was written by these very hands 15+ years ago and is very inefficient (and very possibly, i bet, if you let it load, it will be rather inefficient / less scalable also at lookup time).

I would like to mark this issue as an Enhancement and see what is possible to improve the situation, very likely the algorithm has to be swapped for something more optimized where a tree is built. Should you be looking for a quick solution, though, the only recommendation i can offer is to load all the prefixes in something like ExaBGP and make it BGP Peer with your pmacct daemon. The config would need a couple of lines more a-la:

bgp_daemon: true
bgp_agent_map: /path/to/bgp_agent.map

And the bgp_agent_map would be a one liner a-la: bgp_ip=<IP address used by ExaBGP to peer up> ip=0.0.0.0/0. Although not much involved, for sure it's more involved than just using a networks_file & i am sorry i can't offer any different quick solution.

Paolo