Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Correctly handle --first/--last when reading flows from a stdin #958

Merged
merged 1 commit into from
Mar 31, 2023
Merged

Correctly handle --first/--last when reading flows from a stdin #958

merged 1 commit into from
Mar 31, 2023

Conversation

chancez
Copy link
Contributor

@chancez chancez commented Mar 23, 2023 •
edited

As the title says, when reading flows from stdin, hubble CLI does not correctly handle --first/--last, but this fixes it.

here's a rough demonstration (pardon the jq, it's to condense the JSON output so it's a bit easier to see).

(⎈|kind-kind:N/A) ~/p/w/hubble ❯❯❯ ./hubble observe --first 5 < flows.json                                                                                                pr/chancez/file_input_handle_first_last ✭ ◼
Mar 20 16:16:36.501: 10.0.0.139:35044 (host) <- monitoring/prometheus-prometheus-k8s-0:9090 (ID:23751) to-stack FORWARDED (TCP Flags: ACK, PSH)
Mar 20 16:16:36.501: 10.0.0.139:35044 (host) <- monitoring/prometheus-prometheus-k8s-0:9090 (ID:23751) to-stack FORWARDED (TCP Flags: ACK, FIN)
Mar 20 16:16:36.501: 10.0.0.139:35038 (host) <- monitoring/prometheus-prometheus-k8s-0:9090 (ID:23751) to-stack FORWARDED (TCP Flags: ACK, PSH)
Mar 20 16:16:36.501: 10.0.0.139:35038 (host) <- monitoring/prometheus-prometheus-k8s-0:9090 (ID:23751) to-stack FORWARDED (TCP Flags: ACK, FIN)
Mar 20 16:16:36.515: monitoring/prometheus-prometheus-k8s-0:56640 (ID:23751) <- 172.18.0.2:10250 (host) to-endpoint FORWARDED (TCP Flags: ACK)
(⎈|kind-kind:N/A) ~/p/w/hubble ❯❯❯ head -n5 flows.json | jq '"src id: \(.flow.source.identity) dst id: \(.flow.destination.identity)" '                                   pr/chancez/file_input_handle_first_last ✭ ◼
"src id: 23751 dst id: 1"
"src id: 23751 dst id: 1"
"src id: 23751 dst id: 1"
"src id: 23751 dst id: 1"
"src id: 1 dst id: 23751"
(⎈|kind-kind:N/A) ~/p/w/hubble ❯❯❯ ./hubble observe --last 5 < flows.json                                                                                                 pr/chancez/file_input_handle_first_last ✭ ◼
Mar 20 16:16:37.345: monitoring/prometheus-adapter-75b56bdb5b-svjdg:33732 (ID:8198) <- monitoring/prometheus-prometheus-k8s-0:9090 (ID:23751) to-endpoint FORWARDED (TCP Flags: ACK)
Mar 20 16:16:37.780: monitoring/prometheus-prometheus-k8s-0:55926 (ID:23751) -> monitoring/kube-state-metrics-56c49c7f8c-9xlw8:8080 (ID:36611) to-endpoint FORWARDED (TCP Flags: SYN)
Mar 20 16:16:37.780: monitoring/prometheus-prometheus-k8s-0:55926 (ID:23751) <- monitoring/kube-state-metrics-56c49c7f8c-9xlw8:8080 (ID:36611) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Mar 20 16:16:37.780: monitoring/prometheus-prometheus-k8s-0:55926 (ID:23751) -> monitoring/kube-state-metrics-56c49c7f8c-9xlw8:8080 (ID:36611) to-endpoint FORWARDED (TCP Flags: ACK)
Mar 20 16:16:37.780: monitoring/prometheus-prometheus-k8s-0:55926 (ID:23751) -> monitoring/kube-state-metrics-56c49c7f8c-9xlw8:8080 (ID:36611) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
(⎈|kind-kind:N/A) ~/p/w/hubble ❯❯❯ tail -n5 flows.json | jq '"src id: \(.flow.source.identity) dst id: \(.flow.destination.identity)" '                                   pr/chancez/file_input_handle_first_last ✭ ◼
"src id: 23751 dst id: 8198"
"src id: 23751 dst id: 36611"
"src id: 36611 dst id: 23751"
"src id: 23751 dst id: 36611"
"src id: 23751 dst id: 36611"

@chancez chancez requested a review from a team as a code owner March 23, 2023 17:02
@chancez chancez self-assigned this Mar 23, 2023
@chancez chancez requested a review from a team as a code owner March 23, 2023 17:02
@chancez chancez requested review from rolinh and removed request for a team March 23, 2023 17:02
@maintainer-s-little-helper maintainer-s-little-helper bot added the dont-merge/needs-release-note-label PR is blocked until the release note is set label Mar 23, 2023
@chancez
Copy link
Contributor Author

chancez commented Mar 23, 2023

Closes #957

@chancez chancez force-pushed the pr/chancez/file_input_handle_first_last branch from a08f910 to 708310b Compare March 23, 2023 17:30
kaworu
kaworu requested changes Mar 27, 2023
View reviewed changes
Copy link
Member

@kaworu kaworu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @chancez, couple of nitpicking comments but overall LGTM. Requesting changes because I would like to see tests covering the --first and --last case.

Also the patch introduce a lot of state tracking in ioReaderClient.Recv in the --last case (scanning, c.resps is empty, popping up flows) in addition to handling non --last cases. Have you considered splitting --first, --last, and "default" in their own helper func?

cmd/observe/agent_events.go Outdated Show resolved Hide resolved
cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
@kaworu kaworu added the release-note/minor This PR introduces functionality that users may find relevant to operating Hubble. label Mar 27, 2023
@maintainer-s-little-helper maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label PR is blocked until the release note is set label Mar 27, 2023
@kaworu kaworu mentioned this pull request Mar 27, 2023
Merged
@chancez chancez force-pushed the pr/chancez/file_input_handle_first_last branch from 708310b to a67172e Compare March 27, 2023 22:47
@chancez
Copy link
Contributor Author

chancez commented Mar 27, 2023

@kaworu PR updated based on review and tests added. Thanks!

kaworu
kaworu requested changes Mar 28, 2023
View reviewed changes
Copy link
Member

@kaworu kaworu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update and the tests @chancez! One last comment/question about checking --last n bound.

cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
cmd/observe/io_reader_observer.go Outdated Show resolved Hide resolved
@rolinh rolinh added the :basecamp: needs-rebase This PR needs to be rebased because it has merge conflicts. label Mar 28, 2023
@chancez chancez force-pushed the pr/chancez/file_input_handle_first_last branch 2 times, most recently from 1413762 to 600c069 Compare March 28, 2023 16:48
@chancez chancez removed the :basecamp: needs-rebase This PR needs to be rebased because it has merge conflicts. label Mar 28, 2023
@chancez chancez requested a review from kaworu March 28, 2023 16:48
kaworu
kaworu reviewed Mar 29, 2023
View reviewed changes
Copy link
Member

@kaworu kaworu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @chancez! Patch LGTM once CI is passing and commit history cleaned up.

cmd/observe/io_reader_observer.go Outdated
Comment on lines 73 to 76
// Used for --last
buffer *container.RingBuffer
resps []*observer.GetFlowsResponse
// Used for --first/--last
flowsReturned uint64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

cmd/observe/io_reader_observer_test.go Outdated Show resolved Hide resolved
@chancez chancez force-pushed the pr/chancez/file_input_handle_first_last branch from 600c069 to 7718220 Compare March 29, 2023 15:31
@chancez chancez requested a review from kaworu March 29, 2023 16:18
@chancez chancez force-pushed the pr/chancez/file_input_handle_first_last branch from 7718220 to b2d5e3b Compare March 29, 2023 16:19
@chancez
Correctly handle --first/--last when reading flows from a file/stdin
10feccf 
Signed-off-by: Chance Zibolski <chance.zibolski@gmail.com>
@chancez chancez force-pushed the pr/chancez/file_input_handle_first_last branch from b2d5e3b to 10feccf Compare March 29, 2023 16:53
@chancez chancez merged commit 4656704 into master Mar 31, 2023
@chancez chancez deleted the pr/chancez/file_input_handle_first_last branch March 31, 2023 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaworu kaworu kaworu approved these changes

@rolinh rolinh Awaiting requested review from rolinh rolinh was automatically assigned from cilium/vendor

Assignees

@chancez chancez

Labels
release-note/minor This PR introduces functionality that users may find relevant to operating Hubble.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@chancez @kaworu @rolinh