Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: headless startup without break-on-panic #3469

Open
devnev opened this issue Aug 16, 2023 · 16 comments · May be fixed by #3632
Open

Feature request: headless startup without break-on-panic #3469

devnev opened this issue Aug 16, 2023 · 16 comments · May be fixed by #3632

Comments

@devnev
Copy link

devnev commented Aug 16, 2023

  1. What version of Delve are you using (dlv version)?
Delve Debugger
Version: 1.21.0
Build: $Id: fec0d226b2c2cce1567d5f59169660cf61dc1efe $
  1. What version of Go are you using? (go version)?
go version go1.20.5 darwin/amd64
  1. What operating system and processor architecture are you using?
darwin/amd64
  1. What did you do?

We make local development convenient by starting the program under development with both a restart-on-change and debug-server setup by default. For Go, this looks like wrapping the process in gow and dlv debug, e.g.

CMD ["gow", "-g=dlv", "-s", "-r=false", "-w=protos", "-w=server", "debug", "--headless", "--listen=0.0.0.0:9002", "--accept-multiclient", "--continue", "./server"]

This has been working fine for a while but we recently realised it has an unfortunate side effect. Its standard practice to have package-level initialisers panic if they fail for some reason, e.g.

var somePattern = regexp.MustCompile(`(??)`)

In the above example, the regular expression is invalid, and will cause a panic. In this case, with the setup we have, the break-on-panic kicks in and the process is stopped, without any indication as to what is happening.

I've also tried adding --init <(echo clear) but --init is not allowed in headless mode.

Related (I think): #3028

  1. What did you expect to see?

We expect, or rather would like, a way to have uncaught panics not trigger a breakpoint when dlv is started in headless mode, until the developer connects and alters the behaviour.

  1. What did you see instead?

Timeouts or even just no output instead of the errors the process should be showing due to the panic.

@western-hoolock
Copy link

I think I'm experiencing the same thing/have the same request.
I have a container that compiles my app and runs dlv --listen=:40000 --headless=true --api-version=2 --accept-multiclient exec --continue /usr/local/bin/app every time the source files change.

Most of the time I'm not attaching the debugger and I'm just watching the stdout/stderr of the docker container running dlv. It all works fine until there's an unhandled panic. e.g.:

func main() {
    panic("Panic!")
}

When started with the dlv options as shown above I get this output:

API server listening at: [::]:40000
2023-08-18T15:41:27+02:00 warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2023-08-18T15:41:27+02:00 error layer=rpc writing response:write tcp 127.0.0.1:40000->127.0.0.1:55790: use of closed network connection

There's no indication here what even happened let alone line numbers.

Just running the app outright, without dlv, produces this:

panic: Panic!

goroutine 1 [running]:
main.main()
        C:/Users/WH/Desktop/app/src/main.go:34 +0x25

It would be very helpful if dlv can show me that information as well. I'm assuming this is the same thing you are after @devnev ?

@aarzilli
Copy link
Member

I was wondering: what even is the point of running your application inside a debugger? You're not setting breakpoints and you don't want it to stay around if it crashes, what does it do? If you want to attach to a running application there's dlv attach and it doesn't keep an open tcp port all the time.

@western-hoolock
Copy link

western-hoolock commented Aug 18, 2023

I was wondering: what even is the point of running your application inside a debugger? You're not setting breakpoints and you don't want it to stay around if it crashes, what does it do? If you want to attach to a running application there's dlv attach and it doesn't keep an open tcp port all the time.

I have a Linux VM running Docker that hosts a container that watches the files and rebuilds and runs dlv. Code editing happens remotely on a Windows machine with VSCode. That VSCode has a launch.json that runs dlv attach if I ever want to attach to do some serious debugging, but most of the time just watching the container's output suffices. I just wish it would show those unhandled panics because i get 0 useful information right now unless I attach every time which I'm not doing because it's hard to synchronize that in VSCode. If you have any other suggestions I'm all ears of course.

If I run the app without the debugger I can clearly see the panic, is dlv unable to capture those? I naively assumed those lines would be no different from me doing a log.Println, which shows up fine. But it seems that is not the case?

@devnev
Copy link
Author

devnev commented Aug 21, 2023

@aarzilli Developers will be iterating on a component and making changes over a longer period, and then some issue will crop up that they want to look at in a debugger, and rather than requiring them to restart the process with a different configuration, we'd rather they could immediately attach their IDE to the process they already have running.

It also occurs to me that this is somewhat related to the use of the --continue flag, where we instruct dlv to not wait for an attachment and start the process anyway. I'd be interested to understand why the break-on-panic default is needed given the general dlv default of not starting the process and waiting for input instead (which allows the user to set up these breakpoints), and the --continue behaviour of running the process without waiting (at which point why have any breakpoints by default).

@aarzilli
Copy link
Member

Developers will be iterating on a component and making changes over a longer period, and then some issue will crop up that they want to look at in a debugger, and rather than requiring them to restart the process with a different configuration

You don't need to restart anything with dlv attach.

@devnev
Copy link
Author

devnev commented Aug 21, 2023

dlv attach seems way more complex to orchestrate, as it needs to be launched inside the already-runnnig docker container, and requires figuring out the PID of the process to debug

@devnev
Copy link
Author

devnev commented Aug 21, 2023

Also, how would dlv attach deal with process restarts from gow?

@aarzilli
Copy link
Member

Also, how would dlv attach deal with process restarts from gow?

Presumably if you are attached with a debugger you would just use the restart command of the debugger. If you aren't then dlv doesn't need to run.

Tangentially related: I'm curious how are you securing delve's port?

@western-hoolock
Copy link

Developers will be iterating on a component and making changes over a longer period, and then some issue will crop up that they want to look at in a debugger, and rather than requiring them to restart the process with a different configuration

You don't need to restart anything with dlv attach.

Oh wow that insight actually helps me. I was under the assumption that the error layer=rpc writing response:write tcp 127.0.0.1:40000->127.0.0.1:58674: use of closed network connection message meant dlvstopped but I can just dlv connect to it and read the stack trace/continue and get it to print the panic.
So that takes care of a lot of the issue I thought I had. (Still not sure what that message means though? I haven't connected myself, is it an internal control connection or something?)

I want to be able to debug from VSCode when an uncaught panic happens so I have the following code in my launch.json:

{
  "name": "Attach to Container",
  "type": "go",
  "request": "attach",
  "mode": "remote",
  "remotePath": "",
  "port": 40000,
  "host": "192.168.56.10",
  "showLog": true,
  "trace": "log",
  "logOutput": "rpc",
}

With that conf I can just dlv connect with the click of a button but it seems to always immediately continue rather than stay halted at the breakpoint like dlv connect seems to do. I should probably take this question to the VSCode repo but do you happen to know how I can make VSCode not immediately continue but just connect and then let me control the process?

@devnev
Copy link
Author

devnev commented Aug 21, 2023

Also, how would dlv attach deal with process restarts from gow?

Presumably if you are attached with a debugger you would just use the restart command of the debugger. If you aren't then dlv doesn't need to run.

In my setup, gow is restarting the target process on changes to the source files.

Tangentially related: I'm curious how are you securing delve's port?

Err, AFAICT I'm using the setup from the delve docs for running inside a container and connecting from the host, does something additional need to be done?

@devnev
Copy link
Author

devnev commented Aug 21, 2023

Tangentially related: I'm curious how are you securing delve's port?

Err, AFAICT I'm using the setup from the delve docs for running inside a container and connecting from the host, does something additional need to be done?

From the docs at https://docs.docker.com/network/ it looks like we maybe need to include a127.0.0.1 prefix with the docker port mapping to restrict it to localhost

@aarzilli
Copy link
Member

With that conf I can just dlv connect with the click of a button but it seems to always immediately continue rather than stay halted at the breakpoint like dlv connect seems to do. I should probably take this question to the VSCode repo but do you happen to know how I can make VSCode not immediately continue but just connect and then let me control the process?

You have to set stopOnEntry in launch.json.

@western-hoolock
Copy link

With that conf I can just dlv connect with the click of a button but it seems to always immediately continue rather than stay halted at the breakpoint like dlv connect seems to do. I should probably take this question to the VSCode repo but do you happen to know how I can make VSCode not immediately continue but just connect and then let me control the process?

You have to set stopOnEntry in launch.json.

Yep, that did the trick!

In the end I decided to have two container configurations, the container watches source files, rebuilds on change and runs a %cmd% after the build. The %cmd% is set to the app's binary for normal development so that unhandled panics get printed and when I want to debug I set %cmd% to dlv exec ... so the container starts the app in the debugger and I can connect to it from VSCode.

Ideally one would always want to have dlv connected but it's not easy to synchronize it with the watch/rebuild going on in the container. At least the way I have it now I get to see the panic's file:line and then I can run the container in debug mode and dive in with delve.

@devnev
Copy link
Author

devnev commented Oct 10, 2023

@aarzilli any thoughts on how to orchestrate dlv attach for development happening inside a container?

@yetone
Copy link

yetone commented Dec 16, 2023

‌‌I have the same needs as well. I am in the same situation as everyone else. I have developed an online development and debugging tool for internal use within our company. I also implemented the logic of listening to source code changes in the container, automatically compiling, and executing dlv exec. Due to its use within the company's internal intranet, there are no security issues, and it makes the development process very smooth for internal staff. However, panic automatically breaks when it occurs, causing developers to not know what happened. This is because, unless they encounter a debugging situation, they rarely use vscode to connect to the dlv server for debugging.

@twpayne
Copy link

twpayne commented Feb 1, 2024

Related feature request: #3653.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants