Update error handling on ingestion

[jeffmendoza]

Bulk emitter will return errors now. It continues to try to ingest all verbs even if one fails. It stops ingestion on a single noun failure. Guacone files command will log all errors and continue to ingest all files found even if one errors. Summary should list errors. Guacingest service will log errors and continue, it will ack messages even if errored.

Description of the PR

PR Checklist

• All commits have a Developer Certificate of Origin (DCO) -- they are generated using -s flag to git commit.
• All new changes are covered by tests
• If GraphQL schema is changed, make generate has been run
• If OpenAPI spec is changed, make generate has been run
• If collectsub protobuf has been changed, make proto has been run
• All CI checks are passing (tests and formatting)
• All dependent PRs have already been merged

[mihaimaruseac]

I like the

It stops ingestion on a single noun failure.

part. We need to make sure we never ingest a verb if the nouns are missing. 👍

[jeffmendoza]

guacone example output:

{"level":"error","ts":1712686862.01563,"caller":"helpers/bulk.go:189","msg":"ingestBulkHasMetadata failed with error: HasMetadataPkgs - specific version failed with error: input: ingestBulkHasMetadata random error generated\n","stacktrace":"github.com/guacsec/guac/pkg/assembler/clients/helpers.GetBulkAssembler.func1\n\t/home/jeffm/git/guac/pkg/assembler/clients/helpers/bulk.go:189\ngithub.com/guacsec/guac/pkg/ingestor.Ingest\n\t/home/jeffm/git/guac/pkg/ingestor/ingestor.go:61\ngithub.com/guacsec/guac/cmd/guacone/cmd.glob..func6.1\n\t/home/jeffm/git/guac/cmd/guacone/cmd/files.go:125\ngithub.com/guacsec/guac/pkg/handler/collector.Collect\n\t/home/jeffm/git/guac/pkg/handler/collector/collector.go:111\ngithub.com/guacsec/guac/cmd/guacone/cmd.glob..func6\n\t/home/jeffm/git/guac/cmd/guacone/cmd/files.go:144\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/jeffm/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/jeffm/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/jeffm/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\ngithub.com/guacsec/guac/cmd/guacone/cmd.Execute\n\t/home/jeffm/git/guac/cmd/guacone/cmd/root.go:56\nmain.main\n\t/home/jeffm/git/guac/cmd/guacone/main.go:23\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:267"}
{"level":"info","ts":1712686862.0156918,"caller":"helpers/bulk.go:193","msg":"assembling HasSBOM: 1"}
{"level":"info","ts":1712686862.0200574,"caller":"helpers/bulk.go:204","msg":"assembling VEX : 0"}
{"level":"info","ts":1712686862.0200763,"caller":"helpers/bulk.go:210","msg":"assembling HashEqual : 0"}
{"level":"info","ts":1712686862.0200846,"caller":"helpers/bulk.go:216","msg":"assembling PkgEqual : 0"}
{"level":"info","ts":1712686862.0200922,"caller":"helpers/bulk.go:222","msg":"assembling CertifyLegal : 4"}
{"level":"error","ts":1712686862.0207903,"caller":"collector/collector.go:112","msg":"emit error: unable to ingest document: error assembling graphs for \"file:///../guac-data/docs/spdx/syft-spdx-k8s.gcr.io-kube-apiserver.v1.24.4.json\" : HasMetadataPkgs - specific version failed with error: input: ingestBulkHasMetadata random error generated\n","stacktrace":"github.com/guacsec/guac/pkg/handler/collector.Collect\n\t/home/jeffm/git/guac/pkg/handler/collector/collector.go:112\ngithub.com/guacsec/guac/cmd/guacone/cmd.glob..func6\n\t/home/jeffm/git/guac/cmd/guacone/cmd/files.go:144\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/jeffm/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/jeffm/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/jeffm/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\ngithub.com/guacsec/guac/cmd/guacone/cmd.Execute\n\t/home/jeffm/git/guac/cmd/guacone/cmd/root.go:56\nmain.main\n\t/home/jeffm/git/guac/cmd/guacone/main.go:23\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:267"}

...

{"level":"fatal","ts":1712686869.3312323,"caller":"cmd/files.go:149","msg":"completed ingestion with error, 32 of 33 were successful - the following files did not ingest successfully:  file:///../guac-data/docs/spdx/syft-spdx-k8s.gcr.io-kube-apiserver.v1.24.4.json","stacktrace":"github.com/guacsec/guac/cmd/guacone/cmd.glob..func6\n\t/home/jeffm/git/guac/cmd/guacone/cmd/files.go:149\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/jeffm/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/jeffm/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/jeffm/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\ngithub.com/guacsec/guac/cmd/guacone/cmd.Execute\n\t/home/jeffm/git/guac/cmd/guacone/cmd/root.go:56\nmain.main\n\t/home/jeffm/git/guac/cmd/guacone/main.go:23\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:267"}

guacingest example output:

{"level":"info","ts":1712686767.5338917,"caller":"helpers/bulk.go:181","msg":"assembling PointOfContact: 0"}
{"level":"info","ts":1712686767.5338986,"caller":"helpers/bulk.go:187","msg":"assembling HasMetadata: 278"}
{"level":"error","ts":1712686767.5401647,"caller":"helpers/bulk.go:189","msg":"ingestBulkHasMetadata failed with error: HasMetadataPkgs - specific version failed with error: input: ingestBulkHasMetadata random error generated\n","stacktrace":"github.com/guacsec/guac/pkg/assembler/clients/helpers.GetBulkAssembler.func1\n\t/home/jeffm/git/guac/pkg/assembler/clients/helpers/bulk.go:189\ngithub.com/guacsec/guac/pkg/ingestor.Ingest\n\t/home/jeffm/git/guac/pkg/ingestor/ingestor.go:61\ngithub.com/guacsec/guac/cmd/guacingest/cmd.ingest.func1\n\t/home/jeffm/git/guac/cmd/guacingest/cmd/ingest.go:93\ngithub.com/guacsec/guac/pkg/handler/processor/process.Subscribe.func1\n\t/home/jeffm/git/guac/pkg/handler/processor/process/process.go:111\ngithub.com/guacsec/guac/pkg/emitter.(*subscriber).GetDataFromSubscriber\n\t/home/jeffm/git/guac/pkg/emitter/emitter.go:131\ngithub.com/guacsec/guac/pkg/handler/processor/process.Subscribe\n\t/home/jeffm/git/guac/pkg/handler/processor/process/process.go:122\ngithub.com/guacsec/guac/cmd/guacingest/cmd.ingest.func2\n\t/home/jeffm/git/guac/cmd/guacingest/cmd/ingest.go:104"}
{"level":"info","ts":1712686767.5401978,"caller":"helpers/bulk.go:193","msg":"assembling HasSBOM: 1"}
{"level":"info","ts":1712686767.5511336,"caller":"helpers/bulk.go:204","msg":"assembling VEX : 0"}
{"level":"info","ts":1712686767.551158,"caller":"helpers/bulk.go:210","msg":"assembling HashEqual : 0"}
{"level":"info","ts":1712686767.5511634,"caller":"helpers/bulk.go:216","msg":"assembling PkgEqual : 0"}
{"level":"info","ts":1712686767.5511684,"caller":"helpers/bulk.go:222","msg":"assembling CertifyLegal : 99"}
{"level":"error","ts":1712686767.5563598,"caller":"cmd/ingest.go:94","msg":"unable to ingest document \"file:///../guac-data/docs/spdx/syft-spdx-k8s.gcr.io-kube-proxy.v1.24.5.json\" : error assembling graphs for \"file:///../guac-data/docs/spdx/syft-spdx-k8s.gcr.io-kube-proxy.v1.24.5.json\" : HasMetadataPkgs - specific version failed with error: input: ingestBulkHasMetadata random error generated\n","stacktrace":"github.com/guacsec/guac/cmd/guacingest/cmd.ingest.func1\n\t/home/jeffm/git/guac/cmd/guacingest/cmd/ingest.go:94\ngithub.com/guacsec/guac/pkg/handler/processor/process.Subscribe.func1\n\t/home/jeffm/git/guac/pkg/handler/processor/process/process.go:111\ngithub.com/guacsec/guac/pkg/emitter.(*subscriber).GetDataFromSubscriber\n\t/home/jeffm/git/guac/pkg/emitter/emitter.go:131\ngithub.com/guacsec/guac/pkg/handler/processor/process.Subscribe\n\t/home/jeffm/git/guac/pkg/handler/processor/process/process.go:122\ngithub.com/guacsec/guac/cmd/guacingest/cmd.ingest.func2\n\t/home/jeffm/git/guac/cmd/guacingest/cmd/ingest.go:104"}
{"level":"info","ts":1712686767.5563924,"caller":"process/process.go:117","msg":"[processor: c9813663-28b9-4c19-9cb9-02757aaa6d0c] message acknowledged in pusbub"}

[jeffmendoza]

Addressing #1812
Some history #1275

comment

[pxp928]

One comment to tie together the work being done by Nathan's PR