test-dgram-udp6-link-local-address failing on AIX 7.1 Node.js 12

[richardlau]

• Test: test-dgram-udp6-link-local-address
• Platform: AIX 7.1
• Console Output:

14:39:08 not ok 573 parallel/test-dgram-udp6-link-local-address
14:39:08   ---
14:39:08   duration_ms: 0.196
14:39:08   severity: crashed
14:39:08   exitcode: -6
14:39:08   stack: |-
14:39:08     out/Release/node[26214558]: ../src/tcp_wrap.cc:363:v8::Local<v8::Object> node::AddressToJS(node::Environment*, const sockaddr*, v8::Local<v8::Object>): Assertion `(r) == (0)' failed.
14:39:08   ...

• Build Links: https://ci.nodejs.org/job/node-test-commit-aix/37263/nodes=aix71-ppc64/console

This test started failing 5 days ago (19 June 2021) in https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix71-ppc64/37210/ and has been consistently failing in every build since.

We only build/test on AIX 7.1 for Node.js 12 now, so every run is Node.js 12. Furthermore the last passing build, https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix71-ppc64/37194/, built the same commit 4cdb240 as the first failing build, so this wasn't due to a code change (although it's possibly an exposure of a latent bug we hadn't seen before).

I forced recent builds onto test-ibm-aix71-ppc64_be-4 (as Jenkins had stickied runs onto test-ibm-aix71-ppc64_be-3 and the test fails there too so it's not machine specific.

cc @nodejs/platform-aix

[richardlau]

out/Release/node[26214558]: ../src/tcp_wrap.cc:363:v8::Local<v8::Object> node::AddressToJS(node::Environment*, const sockaddr*, v8::Local<v8::Object>): Assertion `(r) == (0)' failed.

Line 363 of tcp_wrap.cc is:

node/src/tcp_wrap.cc

Lines 360 to 363 in 4cdb240

	const int r = uv_if_indextoiid(a6->sin6_scope_id,
	ip + addrlen + 1,
	&scopeidlen);
	CHECK_EQ(r, 0);

[richardlau]

Added some printf statements to get the value of r... -22 which is EINVAL.

[richardlau]

So far I've been able to determine that we get an IPv6 link local address with scope id 0 and that causes if_indextoname to return EINVAL.

For the last working build the test was actually skipped:
https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix71-ppc64/37194/testReport/(root)/test/parallel_test_dgram_udp6_link_local_address_/

cannot find any IPv6 interfaces with a link local address

Somehow between 18 June and 19 June we gained an IPv6 link local address.

[richardlau]

libuv test fails on the same machines:
https://ci.nodejs.org/view/libuv/job/libuv-test-commit-aix-cmake/nodes=aix71-ppc64/347/console

not ok 141 - ip6_addr_link_local
# exit code 393222
# Output from process `ip6_addr_link_local`:
# Assertion failed in /home/iojs/build/workspace/libuv-test-commit-aix-cmake/nodes/aix71-ppc64/test/test-ip6-addr.c on line 75: 0 == uv_if_indextoname(iface_index, scoped_addr, &scoped_addr_len)

[richardlau]

I'm not entirely sure the assumption in de174cd about link local addresses always being scoped is correct -- on AIX the link local address I can see on the interface does not appear to be scoped and https://www.ibm.com/docs/en/aix/7.2?topic=6-ipv6-multihomed-link-local-site-local-support seems to indicate options to control how AIX handles link local addresses without any mention of scoping.

In any case I had a discussion with @AshCripps who remembered that he had to remove the en1 network interface on our AIX 7.1 machines to get the Node.js tests to pass. On the current set up en1 is the interface with the unscoped IPv6 link local address. uptime shows the machine restarted 6 days ago (when the problem started appearing):

# uptime
  02:24PM   up 6 days,  11:13,  1 user,  load average: 1.70, 1.76, 2.08
#

the /etc/rc.tcpip script uses autoconf6 to create the en1 interface:

# grep autoconf6 /etc/rc.tcpip
# Start up autoconf6 process
start /usr/sbin/autoconf6 "" " -i en1"
#

I'm going to remove the interface so that we're back to the state we were in before and we can get a clean CI run for Node.js 12 for the upcoming security release next week but I'll keep this issue open as I think there's a mismatch somewhere between assumptions in libuv/Node.js and AIX with regards to IPv6 link local addresses and scope ids.

[richardlau]

FWIW Node.js 14 compiled on the AIX 7.1 machine (we can't run the binaries compiled for AIX 7.2 on AIX 7.1 so need to compile from source) exhibits the same test failure. We don't see failures with Node.js 14, 16 and master as they run on the AIX 7.2 machines at OSUOSL and those do not have any IPv6 addresses configured on their network interfaces.

[richardlau]

With the interface disabled CI now passes: https://ci.nodejs.org/job/node-test-commit-aix/37278/nodes=aix71-ppc64/

[richardlau]

I've also removed the sit0 interface as its presence was causing a libuv test failure.

[RaisinTen]

@richardlau this looks like the same crash as the one reported in #41500 which got fixed in #41547. Do we need to wait for the commits to land on v12.x before we can close this issue?

[richardlau]

@RaisinTen I haven't had the time to reenable the en1 and sit0 interfaces (i.e. the ones with IPv6 addresses) that were triggering the issue to verify the problem has been resolved.

[targos]

Closing this issue because v12.x goes EOL at the end of this month and no more releases are planned.