Daisy cannot recover from failed GetSerialPortOutput failures.

[jeremyje]

It looks like if there's multiple GetSerialPortOutput failures, Daisy will halt the process.

compute-image-tools/daisy/step_wait_for_instances_signal.go

Line 144 in dc64ab8

// Retry up to 3 times in a row on any error if we successfully got InstanceStatus.

[EricEdens]

Hi Jeremy, this is correct. Are you affected by this? If so, thoughts on alternatives?

[jeremyje]

Interim fix:

• Use exponential backoff with randomized jitter. It looks like something sorta like that is going on:

  compute-image-tools/daisy/compute/compute.go

  Line 210 in 9ae2722

  func shouldRetryWithWait(tripper http.RoundTripper, err error, multiplier int) bool {

  but I'm not sure I'd roll my own retry loops there's plenty of golang libraries for this.
• Make the retry logic not fail after 3 times. Make it time bound like say if I keep getting failures for 5 minutes then give up.

Permanent Fix:
Read from Stackdriver logs as the primary source. If there's no access fallback to GetSerialPortOutput with a less aggressive polling internal. Once per minute instead of once every 10 seconds.

[EricEdens]

These all sound like reasonable approaches. If you're interested in writing a PR, we could chat about specifics, and I think @adjackura would have some thoughts too.

Before that, I'm curious: How often do your builds fail due to this?