Source IP allowlist is not making it from RSS config file to Wicket UI

[bnaecker]

During customer install, the correct IP allowlist in the RSS config TOML file included many IPs. The wicket UI showed the default of Any, indicating it doesn't seem to be populated from the file in some way. No further details yet.

[bnaecker]

Debugging notes to self:

Print the RSS configuration from the wicketd endpoint handler shows that the allowlist is indeed received correctly:

bnaecker@shale : ~/omicron $ rg -A 10 "allowed_source" ./wicket/tests/output/example_non_empty.toml
54:[allowed_source_ips]
55-# Any external IPs to make requests. This is the default.
56-#allow = "any"
57-
58-# Use the below two lines to only allow requests from the specified IP subnets.
59-# Requests from any other source IPs are refused. Note that individual addresses
60-# must include the netmask, e.g., `1.2.3.4/32`.
61-allow = "list"
62-ips = [ "1.2.3.4/5", "5.6.7.8/10" ]
63-
64-# TODO: docs on network config
bnaecker@shale : ~/omicron $ SSH_ORIGINAL_COMMAND="setup set-config" cargo run -p wicket < wicket/tests/output/example_non_empty.toml
    Finished dev [unoptimized + debuginfo] target(s) in 0.73s
     Running `target/debug/wicket`
May 09 22:17:29.492 INFO reading config from stdin..., file: wicket/src/cli/rack_setup.rs:110
May 09 22:17:29.492 INFO parsing config..., file: wicket/src/cli/rack_setup.rs:115
May 09 22:17:29.494 INFO uploading config to wicketd..., file: wicket/src/cli/rack_setup.rs:120
May 09 22:17:29.498 INFO config upload complete, file: wicket/src/cli/rack_setup.rs:126

And from the wicketd side, where I'm just println!()-ing the received body:

PutRssUserConfigInsensitive {
    bootstrap_sleds: {
        1,
    },
    ntp_servers: [
        "ntp1.com",
        "ntp2.com",
    ],
    dns_servers: [
        1.1.1.1,
        2.2.2.2,
    ],
    internal_services_ip_pool_ranges: [
        V4(
            Ipv4Range {
                first: 10.0.0.1,
                last: 10.0.0.5,
            },
        ),
    ],
    external_dns_ips: [
        10.0.0.1,
    ],
    external_dns_zone_name: "oxide.computer",
    rack_network_config: UserSpecifiedRackNetworkConfig {
        infra_ip_first: 172.30.0.1,
        infra_ip_last: 172.30.0.10,
        switch0: {
            "port0": UserSpecifiedPortConfig {
                routes: [
                    RouteConfig {
                        destination: V4(
                            Ipv4Network {
                                addr: 0.0.0.0,
                                prefix: 0,
                            },
                        ),
                        nexthop: 172.30.0.10,
                        vlan_id: Some(
                            1,
                        ),
                    },
                ],
                addresses: [
                    V4(
                        Ipv4Network {
                            addr: 172.30.0.1,
                            prefix: 24,
                        },
                    ),
                ],
                uplink_port_speed: Speed400G,
                uplink_port_fec: Firecode,
                autoneg: true,
                bgp_peers: [
                    UserSpecifiedBgpPeerConfig {
                        asn: 47,
                        port: "port0",
                        addr: 10.2.3.4,
                        hold_time: Some(
                            6,
                        ),
                        idle_hold_time: Some(
                            3,
                        ),
                        delay_open: Some(
                            0,
                        ),
                        connect_retry: Some(
                            3,
                        ),
                        keepalive: Some(
                            2,
                        ),
                        remote_asn: None,
                        min_ttl: None,
                        auth_key_id: Some(
                            BgpAuthKeyId(
                                Name(
                                    "bgp-key-1",
                                ),
                            ),
                        ),
                        multi_exit_discriminator: None,
                        communities: [],
                        local_pref: None,
                        enforce_first_as: false,
                        allowed_import: NoFiltering,
                        allowed_export: Allow(
                            [
                                V4(
                                    Ipv4Net(
                                        Ipv4Network {
                                            addr: 127.0.0.1,
                                            prefix: 8,
                                        },
                                    ),
                                ),
                            ],
                        ),
                        vlan_id: None,
                    },
                    UserSpecifiedBgpPeerConfig {
                        asn: 28,
                        port: "port0",
                        addr: 10.2.3.5,
                        hold_time: Some(
                            10,
                        ),
                        idle_hold_time: Some(
                            20,
                        ),
                        delay_open: Some(
                            40,
                        ),
                        connect_retry: Some(
                            30,
                        ),
                        keepalive: Some(
                            50,
                        ),
                        remote_asn: Some(
                            200,
                        ),
                        min_ttl: Some(
                            90,
                        ),
                        auth_key_id: Some(
                            BgpAuthKeyId(
                                Name(
                                    "bgp-key-2",
                                ),
                            ),
                        ),
                        multi_exit_discriminator: Some(
                            100,
                        ),
                        communities: [
                            60,
                            70,
                        ],
                        local_pref: Some(
                            80,
                        ),
                        enforce_first_as: true,
                        allowed_import: Allow(
                            [
                                V6(
                                    Ipv6Net(
                                        Ipv6Network {
                                            addr: 64:ff9b::,
                                            prefix: 96,
                                        },
                                    ),
                                ),
                                V4(
                                    Ipv4Net(
                                        Ipv4Network {
                                            addr: 255.255.0.0,
                                            prefix: 16,
                                        },
                                    ),
                                ),
                            ],
                        ),
                        allowed_export: Allow(
                            [],
                        ),
                        vlan_id: None,
                    },
                ],
            },
        },
        switch1: {
            "port0": UserSpecifiedPortConfig {
                routes: [
                    RouteConfig {
                        destination: V4(
                            Ipv4Network {
                                addr: 0.0.0.0,
                                prefix: 0,
                            },
                        ),
                        nexthop: 172.33.0.10,
                        vlan_id: Some(
                            1,
                        ),
                    },
                ],
                addresses: [
                    V4(
                        Ipv4Network {
                            addr: 172.32.0.1,
                            prefix: 24,
                        },
                    ),
                ],
                uplink_port_speed: Speed400G,
                uplink_port_fec: Firecode,
                autoneg: true,
                bgp_peers: [
                    UserSpecifiedBgpPeerConfig {
                        asn: 47,
                        port: "port0",
                        addr: 10.2.3.4,
                        hold_time: Some(
                            6,
                        ),
                        idle_hold_time: Some(
                            3,
                        ),
                        delay_open: Some(
                            0,
                        ),
                        connect_retry: Some(
                            3,
                        ),
                        keepalive: Some(
                            2,
                        ),
                        remote_asn: None,
                        min_ttl: None,
                        auth_key_id: Some(
                            BgpAuthKeyId(
                                Name(
                                    "bgp-key-1",
                                ),
                            ),
                        ),
                        multi_exit_discriminator: None,
                        communities: [],
                        local_pref: None,
                        enforce_first_as: false,
                        allowed_import: Allow(
                            [
                                V4(
                                    Ipv4Net(
                                        Ipv4Network {
                                            addr: 224.0.0.0,
                                            prefix: 4,
                                        },
                                    ),
                                ),
                            ],
                        ),
                        allowed_export: NoFiltering,
                        vlan_id: None,
                    },
                ],
            },
        },
        bgp: [
            BgpConfig {
                asn: 47,
                originate: [
                    Ipv4Network {
                        addr: 10.0.0.0,
                        prefix: 16,
                    },
                ],
                shaper: None,
                checker: None,
            },
        ],
    },
    allowed_source_ips: List(
        IpAllowList(
            [
                V4(
                    Ipv4Net(
                        Ipv4Network {
                            addr: 1.2.3.4,
                            prefix: 5,
                        },
                    ),
                ),
                V4(
                    Ipv4Net(
                        Ipv4Network {
                            addr: 5.6.7.8,
                            prefix: 10,
                        },
                    ),
                ),
            ],
        ),
    ),
}
May 09 22:17:29.497 INFO request completed, latency_us: 1697, response_code: 204, uri: /rack-setup/config, method: PUT, req_id: 05b9a646-0e73-4e28-9281-3475ce39fc94, remote_addr: [::1]:47116, local_addr: [::1]:12226, component: dropshot (wicketd), file: /home/bnaecker/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849

However, getting back the config seems to return something different:

bnaecker@shale : ~/omicron $ SSH_ORIGINAL_COMMAND="setup get-config" cargo run -p wicket | rg -A 10 allowed_source
    Finished dev [unoptimized + debuginfo] target(s) in 0.75s
     Running `target/debug/wicket`
[allowed_source_ips]
# Any external IPs to make requests. This is the default.
allow = "any"

# Use the below two lines to only allow requests from the specified IP subnets.
# Requests from any other source IPs are refused. Note that individual addresses
# must include the netmask, e.g., `1.2.3.4/32`.
# allow = "list"
# ips = [ "1.2.3.4/5", "5.6.7.8/10" ]

# TODO: docs on network config

So either the configuration isn't being persisted correctly (it's just in memory), or we're passing back something that's not reading that updated field.

The first bug is pretty obvious. In this snippet of CurrentRssConfig::update(), we're setting most of the values. But suspiciously absent is the allowlist...

omicron/wicketd/src/rss_config.rs

Lines 523 to 529 in 4dda5cb

	self.bootstrap_sleds = bootstrap_sleds;
	self.ntp_servers = value.ntp_servers;
	self.dns_servers = value.dns_servers;
	self.internal_services_ip_pool_ranges =
	value.internal_services_ip_pool_ranges;
	self.external_dns_ips = value.external_dns_ips;
	self.external_dns_zone_name = value.external_dns_zone_name;

Patching that to update the allowlist, we now receive it correctly and wicketd caches it. Here is some dbg!() output in wicketd's handlers for setting / getting the configuration:

May 09 22:31:46.213 INFO accepted connection, remote_addr: [::1]:62620, local_addr: [::1]:12226, component: dropshot (wicketd), file: /home/bnaecker/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765
List(
    IpAllowList(
        [
            V4(
                Ipv4Net(
                    Ipv4Network {
                        addr: 1.2.3.4,
                        prefix: 5,
                    },
                ),
            ),
            V4(
                Ipv4Net(
                    Ipv4Network {
                        addr: 5.6.7.8,
                        prefix: 10,
                    },
                ),
            ),
        ],
    ),
)
May 09 22:31:46.215 INFO request completed, latency_us: 1411, response_code: 204, uri: /rack-setup/config, method: PUT, req_id: 6b3c1fbe-58c0-4ff1-be15-7d6ce077cd1a, remote_addr: [::1]:62620, local_addr: [::1]:12226, component: dropshot (wicketd), file: /home/bnaecker/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849
May 09 22:31:47.383 DEBG client request, body: None, uri: http://[::1]:12225/ignition, method: GET, component: wicketd MgsManager
May 09 22:31:47.384 DEBG client response, result: Ok(Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv6(::1)), port: Some(12225), path: "/ignition", query: None, fragment: None }, status: 200, headers: {"content-type": "application/json", "x-request-id": "5188185f-b463-4f5c-ac51-32ea04f98c05", "content-length": "476", "date": "Thu, 09 May 2024 22:31:47 GMT"} }), component: wicketd MgsManager
May 09 22:31:47.482 WARN Failed to get prefixes from ddm, err: Failed making HTTP request to ddmd: Communication Error: error sending request for url (http://[::1]:8000/prefixes): error trying to connect: tcp connect error: Connection refused (os error 146), component: BootstrapPeers, file: wicketd/src/bootstrap_addrs.rs:146
May 09 22:31:49.904 INFO accepted connection, remote_addr: [::1]:53008, local_addr: [::1]:12226, component: dropshot (wicketd), file: /home/bnaecker/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765
[wicketd/src/http_entrypoints.rs:200:5] &config.allowed_source_ips = Some(
    List(
        IpAllowList(
            [
                V4(
                    Ipv4Net(
                        Ipv4Network {
                            addr: 1.2.3.4,
                            prefix: 5,
                        },
                    ),
                ),
                V4(
                    Ipv4Net(
                        Ipv4Network {
                            addr: 5.6.7.8,
                            prefix: 10,
                        },
                    ),
                ),
            ],
        ),
    ),
)
May 09 22:31:49.906 INFO request completed, latency_us: 975, response_code: 200, uri: /rack-setup/config, method: GET, req_id: f0cd98c0-39c4-4c1c-a0b4-4791396a5519, remote_addr: [::1]:53008, local_addr: [::1]:12226, component: dropshot (wicketd), file: /home/bnaecker/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849

However, we still get back the wrong thing in wicket itself:

bnaecker@shale : ~/omicron $ SSH_ORIGINAL_COMMAND="setup get-config" cargo run -p wicket | rg -A 10 allowed_source
    Finished dev [unoptimized + debuginfo] target(s) in 0.76s
     Running `target/debug/wicket`
[allowed_source_ips]
# Any external IPs to make requests. This is the default.
allow = "any"

# Use the below two lines to only allow requests from the specified IP subnets.
# Requests from any other source IPs are refused. Note that individual addresses
# must include the netmask, e.g., `1.2.3.4/32`.
# allow = "list"
# ips = [ "1.2.3.4/5", "5.6.7.8/10" ]

# TODO: docs on network config

Still digging.

[bnaecker]

Ok, it looks like we're definitely passing back the right thing to wicket now. The output of the wicketd endpoint handler get_rss_config() has the correct allow-list. So that means we're not messing up the conversion to the client type or serialization on the server side. Also, just curling the endpoint gets back the right IP list as well.

[bnaecker]

Still not sure why the commands using the CLI report the wrong values, but the Wicket UI itself shows the correct thing now:

[bnaecker]

Ok, I found that, it's here:

omicron/wicket/src/cli/rack_setup.rs

Lines 95 to 106 in 4dda5cb

	SetupArgs::GetConfig => {
	let config = client
	.get_rss_config()
	.await
	.context("error fetching current config from wicketd")?
	.into_inner();
	let template = TomlTemplate::populate(&config.insensitive);
	// This is intentionally not `println`; our template already
	// includes the final newline.
	print!("{template}");

dbg!() printing the allow-list there also shows the right bit, so I think the TomlTemplate() bits are probably not respecting the returned values. Checking that next.

[bnaecker]

Yep, the TomlTemplate bit was not doing anything with this field. Fixing and pushing soon.