Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test flake: invalid memory address in metrics test #971

Closed
pohly opened this issue May 28, 2021 · 2 comments · Fixed by #972 or #984
Closed

test flake: invalid memory address in metrics test #971

pohly opened this issue May 28, 2021 · 2 comments · Fixed by #972 or #984
Assignees
@pohly
Labels
1.0

Comments

@pohly
Copy link
Contributor

pohly commented May 28, 2021

https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/blue/organizations/jenkins/pmem-csi/detail/devel/367/tests

/mnt/workspace/pmem-csi_devel/test/e2e/metrics/metrics.go:50
Test Panicked
/go/src/runtime/panic.go:212
Panic: runtime error: invalid memory address or nil pointer dereference
Full stack:
github.com/intel/pmem-csi/test/e2e/metrics.glob..func1.2.1()
	/mnt/workspace/pmem-csi_devel/test/e2e/metrics/metrics.go:81 +0x7f0
github.com/onsi/gomega.InterceptGomegaFailures(0xc004d0e090, 0xc00373bc10, 0x0, 0xc002d66000)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/gomega@v1.10.5/gomega_dsl.go:107 +0xbb
github.com/intel/pmem-csi/test/e2e/metrics.glob..func1.2.2(0x0, 0x0)
	/mnt/workspace/pmem-csi_devel/test/e2e/metrics/metrics.go:106 +0x2a
reflect.Value.call(0x1af1fa0, 0xc00373bc00, 0x13, 0x1de44d6, 0x4, 0xc0011c0f10, 0x0, 0x0, 0x1af1fa0, 0x1, ...)
	/go/src/reflect/value.go:476 +0x8e7
reflect.Value.Call(0x1af1fa0, 0xc00373bc00, 0x13, 0xc0011c0f10, 0x0, 0x0, 0x11a90db8bc47, 0xc00005e6d8, 0xc00005c000)
	/go/src/reflect/value.go:337 +0xb9
github.com/onsi/gomega/internal/asyncassertion.(*AsyncAssertion).pollActual(0xc0030b4980, 0xc001a48000, 0x2e20fe0, 0x78c782, 0x2e52b10)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/gomega@v1.10.5/internal/asyncassertion/async_assertion.go:78 +0xee
github.com/onsi/gomega/internal/asyncassertion.(*AsyncAssertion).match(0xc0030b4980, 0x21064b0, 0x2e52b10, 0x3b9aca01, 0x0, 0x0, 0x0, 0x989680)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/gomega@v1.10.5/internal/asyncassertion/async_assertion.go:112 +0x92
github.com/onsi/gomega/internal/asyncassertion.(*AsyncAssertion).Should(0xc0030b4980, 0x21064b0, 0x2e52b10, 0x0, 0x0, 0x0, 0x20e32d0)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/gomega@v1.10.5/internal/asyncassertion/async_assertion.go:51 +0x85
github.com/intel/pmem-csi/test/e2e/metrics.glob..func1.2()
	/mnt/workspace/pmem-csi_devel/test/e2e/metrics/metrics.go:107 +0x2b4
github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc000113980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/leafnodes/runner.go:113 +0xa3
github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc000113980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/leafnodes/runner.go:64 +0x15c
github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0xc00015ce60, 0x20d98c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/leafnodes/it_node.go:26 +0x87
github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc0017deff0, 0x0, 0x20d98c0, 0xc000110840)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/spec/spec.go:215 +0x72f
github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc0017deff0, 0x20d98c0, 0xc000110840)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/spec/spec.go:138 +0xf2
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc0018a4140, 0xc0017deff0, 0x0)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/specrunner/spec_runner.go:200 +0x111
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc0018a4140, 0x1)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/specrunner/spec_runner.go:170 +0x147
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc0018a4140, 0xc00180a630)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/specrunner/spec_runner.go:66 +0x117
github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc000192070, 0x7fe4fc107c28, 0xc000582a80, 0x1df30f7, 0xe, 0xc0003ccaa0, 0x2, 0x2, 0x211dd58, 0xc000110840, ...)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/internal/suite/suite.go:79 +0x546
github.com/onsi/ginkgo.RunSpecsWithCustomReporters(0x20db800, 0xc000582a80, 0x1df30f7, 0xe, 0xc0003cca80, 0x2, 0x2, 0x2)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/ginkgo_dsl.go:229 +0x218
github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters(0x20db800, 0xc000582a80, 0x1df30f7, 0xe, 0xc00021e7a0, 0x1, 0x1, 0x1)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/ginkgo@v1.14.2/ginkgo_dsl.go:217 +0xad
github.com/intel/pmem-csi/test/e2e.RunE2ETests(0xc000582a80)
	/mnt/workspace/pmem-csi_devel/test/e2e/e2e.go:169 +0xcf
github.com/intel/pmem-csi/test/e2e.TestE2E(0xc000582a80)
	/mnt/workspace/pmem-csi_devel/test/e2e/e2e_test.go:71 +0x2b
testing.tRunner(0xc000582a80, 0x1ec79c0)
	/go/src/testing/testing.go:1194 +0xef
created by testing.(*T).Run
	/go/src/testing/testing.go:1239 +0x2b3
@pohly pohly self-assigned this May 31, 2021
@pohly pohly added the 1.0 label May 31, 2021
@pohly pohly mentioned this issue Jun 1, 2021
Merged
@pohly pohly closed this as completed in #972 Jun 4, 2021
@pohly pohly reopened this Jun 7, 2021
@pohly
Copy link
Contributor Author

pohly commented Jun 7, 2021

Still not solved. resp is nil?!

https://cloudnative-k8sci.southcentralus.cloudapp.azure.com/blue/organizations/jenkins/pmem-csi/detail/PR-977/1/tests

Panic: runtime error: invalid memory address or nil pointer dereference
Full stack:
github.com/intel/pmem-csi/test/e2e/metrics.glob..func1.2.1()
	/mnt/workspace/pmem-csi_PR-977/test/e2e/metrics/metrics.go:81 +0x7f0
github.com/onsi/gomega.InterceptGomegaFailures(0xc0052d4ab0, 0xc0014b2660, 0x0, 0xc002627800)
	/mnt/workspace/gopath/pkg/mod/github.com/onsi/gomega@v1.10.5/gomega_dsl.go:107 +0xbb

pmem-csi/test/e2e/metrics/metrics.go

Line 81 in 9cc771f

Expect(resp.Body).NotTo(BeNil(), "have response body")

@pohly pohly mentioned this issue Jun 7, 2021
Merged
@pohly
Copy link
Contributor Author

pohly commented Jun 8, 2021

I managed to reproduce this in a debugger by deleting the deployment and then running the direct-testing.*metrics test.

    80:								framework.ExpectNoError(err, "GET failed")
=>  81:								Expect(resp.Body).NotTo(BeNil(), "have response body")
    82:								data, err := ioutil.ReadAll(resp.Body)
    83:								framework.ExpectNoError(err, "read GET response")
    84:								name := pod.Name + "/" + container.Name
    85:								if strings.HasPrefix(container.Name, "pmem") {
    86:									Expect(data).To(ContainSubstring("go_threads "), name)
(dlv) p err
error(*net/url.Error) *{
	Op: "Get",
	URL: "http://default.pmem-csi-intel-com-node-74hlk:10011/metrics",
	Err: error(*errors.errorString) *{s: "EOF"},}
(dlv) p resp
*net/http.Response nil
(dlv) p err
error(*net/url.Error) *{
	Op: "Get",
	URL: "http://default.pmem-csi-intel-com-node-74hlk:10011/metrics",
	Err: error(*errors.errorString) *{s: "EOF"},}

So there has been an error, but framework.ExpectNoError didn't abort because of that as it usually does.

The reason is InterceptGomegaFailures: onsi/gomega#198 (comment)

@pohly pohly closed this as completed in #984 Jun 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pohly pohly

Labels
1.0
Projects
None yet
Milestone
No milestone
1 participant
@pohly