Earthly randomly hangs on Init step in v0.6.24+
#2247
Assignees
Labels
type:bug
Something isn't working
Comments
|
Trying to emulate @jazzdan's usage (running Earthly via Tilt, which kills it frequently) - I tried running earthly in a loop and killing it randomly. I could never get it to hang. @jazzdan's docker version is the same as mine and we both use M1. This is the script I used: #!/bin/bash
set -eux -o pipefail
earthly=./build/darwin/arm64/earthly
one_test() {
it="$1"
echo "Iteration $it"
"$earthly" --no-output +for-darwin-m1 &
pid=$!
sleep $(( ( RANDOM % 10 ) + 0 ))
kill "$pid" || true
}
for i in {1..100}; do
one_test "$i" &
done
wait |
|
This script does cause package main
import (
"context"
"fmt"
"os"
"os/exec"
"strings"
"time"
)
type commmandContextOutput struct {
stdout strings.Builder
stderr strings.Builder
}
func command(ctx context.Context, suffix int) {
output := &commmandContextOutput{}
cmd := exec.CommandContext(ctx, "docker", "info")
cmd.Env = os.Environ() // Ensure all shellouts are using the current environment, picks up DOCKER_/PODMAN_ env vars when they matter
cmd.Stdout = &output.stdout
cmd.Stderr = &output.stderr
err := cmd.Run()
if err != nil {
out := output.stdout.String()
e := output.stderr.String()
if false {
fmt.Printf("%s, %s", out, e)
}
// fmt.Printf("error during run: %s\n", err)
} else {
// fmt.Printf("got info %d\n", suffix)
}
}
func forever() {
var i = 0
for {
i++
ctx, _ := context.WithTimeout(context.Background(), time.Minute*5)
go func() {
<-ctx.Done()
if ctx.Err() != nil {
fmt.Printf("context error: %s\n", ctx.Err())
}
}()
command(ctx, i)
}
}
func main() {
for i := 0; i < 200; i++ {
go forever()
}
forever()
}But not this bash script for i in {1..100000}; do
docker info &
done |
|
I've started getting this myself on my M1 mac. No repeated restarts or anything. |
|
It seems that this only affects earthly installed via A workaround is to CC @jazzdan |
|
FTR, this reproduces the issue very consistently: It doesn't matter what the build is, so it can be something really simple like |
vladaionescu
added a commit
that referenced
this issue
Oct 14, 2022
Also, add verbose printing on executed frontend-related commands. Re #2247 Co-authored-by: Vlad A. Ionescu <vladaionescu@users.noreply.github.com>
|
I noticed this happening again on an And now no matter how many times I send SIGHUP it doesn't even print a stack trace. But maybe I"m sending it to the wrong process somehow? |
|
I got a stack trace from a more recent version of earthly that exhibited this problem. This was installed with |
|
Hmm... my little script that used to reproduce it for me very consistently no longer reproduces it on either homebrew or earthly/earthly/earthly version. The stack trace points to the same thing: a I noticed that you're on Mac 12.6. Any chance you'd be able to upgrade to Mac 13 and let us know if this still reproducing? When it was reproducing for me in the past, it would reproduce highly consistently with a script like this: Where |
|
Oh I also just got it on |
|
Some more experimentation - it seems that this is related to I verified this fix by running Earthly 500 times - with Script I used for reference: for i in {1..500}; do ./build/earthly +t; if [ "$?" != 0 ]; then break; fi; done |
vladaionescu
added a commit
to vladaionescu/homebrew-core
that referenced
this issue
Nov 10, 2022
vladaionescu
added a commit
to earthly/homebrew-earthly
that referenced
this issue
Nov 10, 2022
This was referenced Nov 10, 2022
BrewTestBot
pushed a commit
to Homebrew/homebrew-core
that referenced
this issue
Nov 10, 2022
Fixes earthly/earthly#2247 Closes #115350. Signed-off-by: Sean Molenaar <1484494+SMillerDev@users.noreply.github.com> Signed-off-by: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com>
vladaionescu
added a commit
to earthly/homebrew-earthly
that referenced
this issue
Nov 10, 2022
|
CGO strikes again, great find! |
vladaionescu
added a commit
that referenced
this issue
Nov 10, 2022
This is normally not necessary, but in some cases, it does default to 1, and that can be problematic sometimes (an example: #2247). Co-authored-by: Vlad A. Ionescu <vladaionescu@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
SIGABRTstack trace: https://gist.github.com/jazzdan/505c9211c30efc533ab687831a169306The stack trace shows hanging on
docker info. Though that doesn't make a lot of sense.The fact that this does not reproduce in
v0.6.23seems to point to this possible change: #2207.EDIT: A workaround has been identified:
brew uninstall earthly && brew install earthly/earthly/earthly. This only happens on the homebrew core version of Earthly, but not on the version from our own tap.The text was updated successfully, but these errors were encountered: