Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http2_client: fix reader segfault on PROTOCOL_ERRORs #3926

Merged
merged 1 commit into from Oct 6, 2020

Conversation

sorah
Copy link
Contributor

@sorah sorah commented Oct 4, 2020

http2.Framer ErrorDetail() might return nil even in case of http2.StreamError, as documented.


(Note: This problem was initially reported to grpc-security@ just in case, but they assessed this segfault isn't a security issue.)

What did you do?

Wrote a gRPC server may send an invalid HEADERS frame to a gRPC client process written with grpc-go. The frame is set length to 0. https://img.sorah.jp/x/20200917_185642_kdwgHWb8KG.png

(a server is written using our original protocol implementation, which is I'm maintaining and sending an invalid HEADERS frame is a different bug. Found during investigation of server process crash)

What did you expect to see?

HTTP/2 session would properly shut down with PROTOCOL_ERROR.

What did you see instead?

a client process would crash with SEGV (segmentation fault).

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x144f08f]
goroutine 772 [running]:
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0012f4380)
/.../pkg/mod/google.golang.org/gr...@v1.31.0/internal/transport/http2_client.go:1309 +0x2af
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/.../pkg/mod/google.golang.org/gr...@v1.31.0/internal/transport/http2_client.go:310 +0x1071

This happens because http2.Framer#ErrorDetail() returned nil (it may return, as documented):

msg := t.framer.fr.ErrorDetail().Error()

It might return nil even in case of StreamError.

For instance, when a server sent an invalid HEADERS frame where a length is zero, http2.Framer#ReadFrame returns an StreamError
without setting http2.Framer#lastError (which is a struct field back of ErrorDetail).
https://github.com/golang/net/blob/62affa334b73ec65ed44a326519ac12c421905e3/http2/frame.go#L1022

Due to this crash existing in http2Client Reader goroutine, this is unavoidable and a process would result in a segmentation fault.

This bug seems to be introduced at commit 40a879c which changed to call Error() of a return value of ErrorDetail().

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Oct 4, 2020

CLA Check
The committers are authorized under a signed CLA.

@dfawley dfawley self-requested a review October 5, 2020 21:05
@dfawley dfawley self-assigned this Oct 5, 2020
@dfawley dfawley added this to the 1.33 Release milestone Oct 5, 2020
Copy link
Member

@dfawley dfawley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the patch! One minor comment.

msg := ""
if errorDetail != nil {
msg = errorDetail.Error()
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some text to msg when errorDetail is nil.

	} else {
		msg = "unknown framer error"  // ??
	}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was unsure what comment is appropriate when errorDetail is unavailable...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds fine to me.

http2.Framer ErrorDetail() might return nil even in case of
http2.StreamError, as documented.
msg := ""
if errorDetail != nil {
msg = errorDetail.Error()
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds fine to me.

@dfawley dfawley merged commit 9a3c02f into grpc:master Oct 6, 2020
@dfawley
Copy link
Member

dfawley commented Oct 6, 2020

Thanks again!

@sorah sorah deleted the fix-framer-crash branch October 8, 2020 16:03
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 25, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants