Nondeterministic "No such file or directory" failure when cleaning project #2156
Comments
|
Does anyone have an idea what may be causing this error? It's giving me nightmares - happens only sometimes, can't reliably reproduce it. |
|
Not an sbt expert but it seems like maybe it isn't creating parent directories or something? Also your path is under /var/lib is it a permissions issue on some parent folder you can't modify? |
|
This only happens sometimes, so it's not a permission issue. |
|
Do you have any plugins that checks for test coverage or coding styles? |
|
No. The only plugins I have enabled are these: |
|
even I am seeing this failure randomly in my sbt builds. |
|
I would like to bring this again into attention. This error completely ruins our builds and I haven't got a slightest idea how to debug or workaround it. I tried to do this: and then adding to each subproject: But it doesn't work - still crashes randomly on various subprojects. |
|
For us to work on this, we need to reproduce this on our machine (or Travis) by one way or another. The error says: |
|
There's about 20 subprojects that depend on each other and the issue happens randomly in all subprojects, including the root one. Sometimes it even fails in more than one subproject at once. Invoked tasks are I still have some random ideas for workarounds - I'll try them and maybe narrow the circumstances of that error a bit. |
|
first things first. does the file's parent directory exist when the exception is thrown? |
|
SBT is supposed to create it. This is what it does (from |
|
So, I modified my workaround to do this: and it still doesn't work. Here's what's happening now (project names and URLs have been edited): It's as if someone deleted the files that I just created... |
|
You may want to check your filesystem /tmp directory restrictions. It's possible you are unable to create the files and/or your filesystem is immediately cleaning them up. |
|
/tmp directory does not seem to be used there or at least no error message mentions it. Is it used internally by SBT in a way that could cause these errors? |
|
another question: on which operating system do you see this behaviour? |
|
Ubuntu 12.04 x86_64, kernel 3.2.0-72-generic, ext3 filesystem |
|
Well, the problems seems to be gone after I disabled all concurrency in SBT: There's still a chance it will fail, but I've had about 10 successful builds from then. We don't need parallel builds, so it works for me, but I wouldn't count that as a solution or even workaround. |
|
I got a trace on CircleCI: https://circleci.com/gh/pathikrit/better-files/331 |
|
Can somebody who has this problem run strace when you launch sbt? Maybe that will shed light on what's happening... |
|
if I could reproduce it easily I would but as it's random... :/ |
|
How often does it happen? Could you live with the slowdown strace gives you until it happens randomly? Just change it to write to a file so it doesn't get lost in the noise haha |
|
See my comment above. You can git clone https://github.com/pathikrit/better-files at hash = 4597e81e9faecc1676c85e7cbc1e5be8506b879a and run |
|
@pathikrit Thanks for the repro. I saved the stacktrace to the gist just in case https://gist.github.com/eed3si9n/c53a4dcc2f3df58d93bd
See #1673 |
|
Is there a specific version of JDK 8 that you guys are using? |
|
@eed3si9n : If you have a CircleCI, account, you can retry this build yourself with SSH enabled. See: https://circleci.com/gh/pathikrit/better-files/331#ssh-info - it will bring up a VM and rerun for you. If it does not work, let me know, I can give you collab access to that repo. |
|
Had similar issue, problem was with insufficient disk space on Jenkins volume. |
|
Maybe this helps in finding the problem: This did not work: Only difference |
|
Our build just failed and I was complaining vocally and my team mate just hit the exact same problem on her macbook with a different repository. The only thing we seemed to be doing in common was running sbt with multiple tasks in one go, starting with clean. |
|
Same issue; we've been running our build script 100s of time without issue, and just ran into this. We're unable to reproduce. Our build script starts with a clean |
|
Does anyone notice that file that was reported that could not be created is actually there and the contents of this file is exactly the same message - which is very weird? It is something like this: from the stdout/stderr: SomeException: "I could not create file somefolder/somefile.txt" and then we notice file exists so we execute: $ cat somefolder/somefile.txt and the result is: SomeException: "I could not create file somefolder/somefile.txt" |
|
I wonder if all or the strong majority of people hitting this are hitting it due to using sbt-buildinfo and sbt-buildinfo using Extracted.runTask. See sbt/sbt-buildinfo#87 / #2970. |
|
@dwijnand not in my case - we're not using this plugin |
|
@dwijnand same here. We're seeing the issue but are not using sbt-buildinfo in the relevant project. |
|
We also see this at ~5% rate on our Bamboo builds, using the Play Framework 2.5.9 which builds with; Can't find any pattern to it. Sometimes 10 simultaneous builds work fine. Sometimes 2 of 10 fail. Sometimes a single build is happening with no other activity and fails. |
|
Probably not something you guys want to hear; but even though we ran into it three times in December, since then even though the team size has grown, we never ran into it again. Maybe by adding additional libraries/dependencies we slightly altered the timings of things to where we don't see this. I hope I didn't jinx the team. |
eed3si9n
changed the title
Fixes sbt#2156 This is a workaround for `clean` and logging clashing when run concurrently on the same subproject.
Fixes sbt#2156 cleanFiles is now a task that lists exactly what will be deleted recursively. It will now exclude target.value / "streams" directory thereby avoiding clash with logging. To clean the logs and task caches there's now a new task called `cleanBuild`.
Fixes sbt#2156 cleanFiles is now a task that lists exactly what will be deleted recursively. It will now exclude target.value / "streams" directory thereby avoiding clash with logging. To clean the logs and task caches there's now a new task called `cleanBuild`.
Fixes sbt#2156 This is a workaround for `clean` and logging clashing when run concurrently on the same subproject.
|
I am getting this error - Could not create file /target/streams/compile/run/$global/streams/outjava.io.IOException: Too many open files . Is this the same problem ? |
|
@ganeshkaspate if you're on linux you probably have to bump the file handle limit: https://unix.stackexchange.com/questions/36841/why-is-number-of-open-files-limited-in-linux |
|
We recently started seeing the same issue on a builds using Jenkins on Ubuntu 16.04 & java 1.8.0_131 in both sbt 0.13.13 and 0.13.15. The rate is about 10% of the builds. Sometimes it's |
|
After debugging , I found that 4096 was limit for my application. So, whenever it reaches to that limit I use to get that error. |
|
I get the following error too: [error] at sbt.MainLoop$.$anonfun$runWithNewLog$1(MainLoop.scala:76) See complete log in file:/home/gakuo/.IdeaIC2017.2/system/log/sbt.last.log |
To fix random SBT build errors as described in: sbt/sbt#2156
and others
I'm not sure if this is the right place to report this - I'd rather put this first on some forum, but since SBT does not have any mailing list for users, I'm forced to put it here.
I'm randomly getting this error when trying to build my project:
Can anyone direct me how do I even troubleshoot/debug this?
The text was updated successfully, but these errors were encountered: