tparse ignoring some test runs?

[titpetric]

I find that some files that are present in the JSON get omitted from tparse. Would you be able to tell me why some are omitted?

{"Time":"2022-08-17T18:20:05.260649294Z","Action":"fail","Package":"github.com/TykTechnologies/tyk/gateway","Test":"TestOldMockResponse/whitelist","Elapsed":1.42}

I'm using -all, -nocolor, -format markdown. I don't find TestOldMockResponse in the markdown output.

Source: https://tyk-automated-tests.cdn.si/log/1660759791.json
tparse output: https://tyk-automated-tests.cdn.si/log/1660759791.md
pandoc md2html: https://tyk-automated-tests.cdn.si/log/1660759791.html

[mfridman]

Curious, before looking into it a bit. Are you using a testing package, such as https://github.com/matryer/is ?

[titpetric]

Testify/assert, and/or matryer/is, it's a bit fragmented so we're possibly using both in different packages.

[mfridman]

I ask because I was seeing something similar in 2 separate projects, specifically with this package.

I filed an issue here:

matryer/is#47

Once I removed it, I had no issues seeing all the expected output.

[titpetric]

I'm not sure this applies. The JSON has TestOldMockResponse failures, the produced tparse output does not. We're not using matryer/is here, but we are using testify/assert. If we filter by Action==fail, we can find these test failures in json.

[mfridman]

Gotcha, if the entry is in the .json output then it should be captured.

I'll write a test case and get a repro.

Can you confirm the Go version, and the tparse version ?

[titpetric]

go 1.17, tparse@latest (go install-ed today).

[mfridman]

One more question, when you say omitted, do you mean omitted from the Failure output and/or the tests table (the one displayed when running -all) ?

There is usually special handling of panic output, so want to understand what you're expecting

[titpetric]

It's omitted from the failure output and the tests table, can't find it at all in the markdown output. Please check that linked .json in the issue and the generated .md with tparse. I'm using a makefile to generate a backlog of logs, as I'm using the JSON to track flaky test data and then link failures. The flaky test surfaces a bunch of these omitted failed tests, check the index page of that host if you like. https://tyk-automated-tests.cdn.si/ - all single test failures are candidates that possibly don't show up in the .md output.

.PHONY: all clean

all: $(shell ls *.json | sed -e 's/.json/.html/')

clean:
	rm -f *.html

%.html:
	cat $*.json | tparse -all -nocolor -format markdown > $*.md || true
	pandoc $*.md -f markdown -t html -s -o $@

[mfridman]

Thanks. There is a slightly different code path as soon as a panic is detected by tparse. I suspect I'll have to fix that, but even then you won't get all test output because in go panics are end of the world afaik.

There might be running tests and as soon as a panic occurs we'll have a bunch of pending events for those running tests we'll never receive. tparse doesn't have a complete representation of the test, and the only thing we know for certain is a panic happened.

Very early on I made an assumption that if a panic happens you probably don't care about other output as it'll be incomplete.

[titpetric]

Yeah, i'd just like the tests listed as failed, the panic output will need to be inspected ourside of tparse for now if that's the case. I treated each line as json, and i suspect you attached some of that json to a panic text if i'm following you correctly? Tparse would need to break down the panic and link it from multiple test failures with an anchor perhaps. I'd parse the stack output to link from all the tests that show up in the panic. I don't know how good markdown is for that (on github?). I'm also asuming -failfast is suggested, to perhaps avoid multiple panics in the output? I'm asuming a panic is a stop-the-world event for most people 😬

…

On Sat, Aug 20, 2022, 15:11 Michael Fridman ***@***.***> wrote: Thanks. There is a slightly different code path as soon as a panic is detected by tparse. I suspect I'll have to fix that, but even then you won't get all test output because in go panics are end of the world afaik. There might be running tests and as soon as a panic occurs we'll have a bunch of pending events for those running tests we'll never receive. tparse doesn't have a complete representation of the test, and the only thing we know for certain is a panic happened. Very early on I made an assumption that if a panic happens you probably don't care about other output as it'll be incomplete. — Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABY7EBKUWL5V5XZSDUZA4DV2DKPPANCNFSM57ALYIYA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[lbergesio]

I'm not sure this applies. The JSON has TestOldMockResponse failures, the produced tparse output does not. We're not using matryer/is here, but we are using testify/assert. If we filter by Action==fail, we can find these test failures in json.

+1 same situation with go 1.21.4 and tparse 0.13.2

[mfridman]

Is it an open source project by any chance?

Could you provide a minimal go test -json output reproducible example?

[lbergesio]

Could it be that tparse has problems with the output of go test when count != 0?

The output of tparse --smallscreen shows the module failed but it does nto show the failing test:

┌─────────────────────────────────────────────────────────────────────────────────────┐
│  STATUS │ ELAPSED │             PACKAGE              │ COVER  │ PASS │ FAIL │ SKIP  │
│─────────┼─────────┼──────────────────────────────────┼────────┼──────┼──────┼───────│
│  PASS   │  1.34s  │ ZZZZ                │ 94.4%  │  32  │  0   │  0    │
│  FAIL   │ 26.25s  │ XXXX                      │ 92.0%  │  47  │  0   │  0    │
│  PASS   │  4.50s  │ YYYY                  │ 92.8%  │  49  │  0   │  2    │
└─────────────────────────────────────────────────────────────────────────────────────┘

I see it in the output of test:

=== RUN   TestSudoPerformAuditNoTunnels
    tunnel_test.go:576: 
        	Error Trace:	github.yyyy.go:576
        	Error:      	"SOME ERROR STRING"
        	Test:       	TestSudoPerformAuditNoTunnels
--- PASS: TestProcessConfigWithInvalidPolicy (0.06s)
=== RUN   TestPerformAuditIncompleteRule
--- PASS: TestApplyFwPoliciesWith2Ports (0.07s)
=== RUN   TestReadFwChainsIncompleteCounterIPAppBindIngress
--- PASS: TestAuditConfigFailureMissingFunctionalRules (0.33s)
=== RUN   TestCheckOSPFNeighborStateParamsAdd
--- PASS: TestCheckOSPFNeighborStateParamsAdd (0.00s)
=== RUN   TestMinId
--- PASS: TestPerformAuditIncompleteRule (0.04s)
=== RUN   TestBindings2NATMapsNilApps
--- PASS: TestBindings2NATMapsNilApps (0.00s)
=== RUN   TestProcessConfigWithMssRemoved
--- FAIL: TestSudoPerformAuditNoTunnels (0.10s)

But notice the RUN/PASS RUN/FAIL are not in order and also (not shown here) there several runs of the same tests