CI Failure (key symptom) in RandomNodeOperationsTest.test_node_operations

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/13607
https://buildkite.com/redpanda/vtools/builds/13607

Module: rptest.tests.random_node_operations_test
Class: RandomNodeOperationsTest
Method: test_node_operations
Arguments: {
    "num_to_upgrade": 3,
    "enable_failures": false,
    "with_tiered_storage": false
}

test_id:    RandomNodeOperationsTest.test_node_operations
status:     FAIL
run time:   57.132 seconds

RemoteCommandError({'ssh_config': {'host': 'ip-172-31-11-60', 'hostname': '172.31.11.60', 'user': 'root', 'port': 22, 'password': None, 'identityfile': '/home/ubuntu/.ssh/id_rsa'}, 'hostname': 'ip-172-31-11-60', 'ssh_hostname': '172.31.11.60', 'user': 'root', 'externally_routable_ip': '34.222.249.118', '_logger': <Logger rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.num_to_upgrade=3.with_tiered_storage=False-261 (DEBUG)>, 'os': 'linux', '_ssh_client': <paramiko.client.SSHClient object at 0x7f7440fdcaf0>, '_sftp_client': <paramiko.sftp_client.SFTPClient object at 0x7f7440ec7fd0>, '_custom_ssh_exception_checks': None}, 'curl -fsSL https://vectorized-public.s3.us-west-2.amazonaws.com/releases/redpanda/24.1.1/redpanda-24.1.1-amd64.tar.gz --create-dir -o /opt/redpanda_installs/v24.1.1/redpanda.tar.gz && gunzip -c /opt/redpanda_installs/v24.1.1/redpanda.tar.gz | tar -xf - -C /opt/redpanda_installs/v24.1.1 && rm /opt/redpanda_installs/v24.1.1/redpanda.tar.gz', 35, b'')
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 103, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/tests/random_node_operations_test.py", line 319, in test_node_operations
    self._start_redpanda(num_to_upgrade,
  File "/home/ubuntu/redpanda/tests/rptest/tests/random_node_operations_test.py", line 147, in _start_redpanda
    installer.install(
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 609, in install
    self._install_unlocked(nodes, install_target)
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 658, in _install_unlocked
    raise e
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 638, in _install_unlocked
    self.wait_for_async_ssh(self._redpanda.logger,
  File "/home/ubuntu/redpanda/tests/rptest/services/redpanda_installer.py", line 165, in wait_for_async_ssh
    for l in ssh_out_per_node[node]:
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 687, in next
    return next(self.iter_obj)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 354, in output_generator
    raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: root@ip-172-31-11-60: Command 'curl -fsSL https://vectorized-public.s3.us-west-2.amazonaws.com/releases/redpanda/24.1.1/redpanda-24.1.1-amd64.tar.gz --create-dir -o /opt/redpanda_installs/v24.1.1/redpanda.tar.gz && gunzip -c /opt/redpanda_installs/v24.1.1/redpanda.tar.gz | tar -xf - -C /opt/redpanda_installs/v24.1.1 && rm /opt/redpanda_installs/v24.1.1/redpanda.tar.gz' returned non-zero exit status 35.

JIRA Link: CORE-2852

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13628
*https://buildkite.com/redpanda/vtools/builds/13628
*https://buildkite.com/redpanda/vtools/builds/13628
*https://buildkite.com/redpanda/vtools/builds/13628

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13649

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13649
*https://buildkite.com/redpanda/vtools/builds/13649
*https://buildkite.com/redpanda/vtools/builds/13649

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13663
*https://buildkite.com/redpanda/vtools/builds/13663
*https://buildkite.com/redpanda/vtools/builds/13663
*https://buildkite.com/redpanda/vtools/builds/13663

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13676
*https://buildkite.com/redpanda/vtools/builds/13676
*https://buildkite.com/redpanda/vtools/builds/13676
*https://buildkite.com/redpanda/vtools/builds/13676

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13715
*https://buildkite.com/redpanda/vtools/builds/13715
*https://buildkite.com/redpanda/vtools/builds/13715
*https://buildkite.com/redpanda/vtools/builds/13715