failed: I/O exception during sandboxed execution: No such file or directory

[Ryang20718]

Description of the bug:

Periodically, the following error would occur when running tests

 Testing <blah> failed: I/O exception during sandboxed execution: /dev/shm/bazel-sandbox.34e9fe25bb0c6624a8ba8f5a00a18c3243dfc943119e415e09934c77b955441f/linux-sandbox/148/stats.out (No such file or directory)

we're on bazel 6.5.0 with spawn strategy linux-sandbox, jobs set to 1:1 with vcpus, sandbox mounted at /dev/shm

Whenever this occurs, we see system memory usage at 82-83% with cpu maxed at 100%.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I don't have a reliable repro. it happens sporadically

Which operating system are you running Bazel on?

ubuntu 20.04

What is the output of bazel info release?

release 6.5.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

this has been occuring more frequently since we switched to 6.5.0 from 6.3.2

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[iancha1992]

@Ryang20718 Could you please provide sample code and complete steps to reproduce this issue?
Also, please try updating your Bazel to one of our latest releases (See https://github.com/bazelbuild/bazel/releases). Thank you.

[Ryang20718]

I don't have a reliable repro; it just periodically happens when running large amounts of tests. we're in the process of upgrading to bazel 7, but still need to upgrade some dependencies to get there

[mattyclarkson]

To provide another data point: I have hit the similar error message in rules_git log. That does the git checkout in the action into a declared directory.

It is reproducible when the same cache is used but not consistent across executions. For example, the remote execution passed fine.

That run used Bazel 7.1.1.

[nikhilkalige]

#20976

[oquenchil]

Does everyone affected by this NOT have dynamic execution enabled?

[mattyclarkson]

No dynamic execution was enabled for rules_git

[Ryang20718]

no dynamic execution (this is with a local execution with remote cache)