Prototype a MQTT Source #7919

ctmphuongg · 2024-05-10T23:12:42Z

Proposed Changes

(The main working files are cmd/mqttsource/main.go, config/tools/mqttsource/mqttsource.yaml, the rest is packages installed to use mqtt_paho, generated by using go get paho and go mod vendor)

Install packages for mqtt_paho
Prototype an mqtt container source

Notes: For testing

MQTT needs a source to listen from. For local testing, run this command to open the mosquitto mqtt broker for source.

docker run -it --rm --name mosquitto -p 1883:1883 eclipse-mosquitto:2.0 mosquitto -c /mosquitto-no-auth.conf

Keep the mqttsource running by go build the file and run

./main -source localhost:1883 -eventType mqtt.event -eventSource mqtt://localhost:1883 -topic "test-topic"

Send a message to the same topic, using

mosquitto_pub -t 'test-topic' -m '{"specversion" : "1.0","type" :"com.example.someevent", "id" : "1234-1234-1234","source" : "/mycontext/subcontext","data":{"msg":"hello world!"}}' -D PUBLISH user-property Content-Type application/cloudevents+json; charset=utf-8

The message will be logged and sent to sink.

knative-prow · 2024-05-10T23:12:51Z

Hi @ctmphuongg. Thanks for your PR.

I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

knative-prow · 2024-05-10T23:12:52Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ctmphuongg
Once this PR has been reviewed and has the lgtm label, please assign pierdipi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2024-05-10T23:19:58Z

Codecov Report

Attention: Patch coverage is 0% with 56 lines in your changes are missing coverage. Please review.

Project coverage is 69.27%. Comparing base (7e1c082) to head (d456eb6).
Report is 61 commits behind head on main.

Files	Patch %	Lines
cmd/mqttsource/main.go	0.00%	56 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7919      +/-   ##
==========================================
+ Coverage   69.22%   69.27%   +0.05%     
==========================================
  Files         339      345       +6     
  Lines       19494    16050    -3444     
==========================================
- Hits        13494    11119    -2375     
+ Misses       5337     4245    -1092     
- Partials      663      686      +23

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Cali0707 · 2024-05-13T19:24:45Z

/cc @matzew

Not sure if you have any initial feedback here...

ctmphuongg · 2024-05-14T15:07:47Z

/cc @Leo6Leo

cmd/mqttsource/main.go

Leo6Leo

Great job @ctmphuongg ! The PR looks great overall, and I have tested it locally, everything works perfectly.

Currently the CI is failing due to the check failure on codegen. Please run

./hack/update-codegen.sh

to fix the problem!

cmd/mqttsource/main.go

config/tools/mqttsource/mqttsource.yaml

Leo6Leo · 2024-05-27T22:02:11Z

/ok-to-test

Leo6Leo · 2024-05-28T13:48:26Z

Thanks for fixing the review comment! @ctmphuongg
/lgtm
/cc @pierDipi @creydr @Cali0707

Cali0707

Hey @ctmphuongg awesome start! I left a few comments regarding error handling/setting CE attribute properly.

Otherwise, we generally use camel case variable names, not snake case in Knative. So, I went through and suggested a new variable name on the first occurrence of each variable I saw that was snake case. I would recommend going through and renaming those in your IDE instead of just committing my suggestions, as I didn't actually correct every instance of the variables, just the first

Cali0707 · 2024-05-28T15:05:40Z

cmd/mqttsource/main.go

+	log.Printf("Received event: %s", event)
+	data := event.Data()
+	newEvent := cloudevents.NewEvent(cloudevents.VersionV1)
+	newEvent.SetType(eventType)


Shouldn't we use the event type from the input event, rather than overriding it on our end?

Suggested change

newEvent.SetType(eventType)

newEvent.SetType(event.Type())

Cali0707 · 2024-05-28T15:08:54Z

cmd/mqttsource/main.go

+	newEvent.SetType(eventType)
+	newEvent.SetSource(eventSource)
+	newEvent.SetID(event.ID())
+	_ = newEvent.SetData(cloudevents.ApplicationJSON, data)


We should probably handle the error here, if we fail to set the data does it make sense to dispatch the event?

Cali0707 · 2024-05-28T15:18:34Z

cmd/mqttsource/main.go

+func main() {
+	flag.Parse()
+
+	k_sink := os.Getenv("K_SINK")


Suggested change

k_sink := os.Getenv("K_SINK")

kSink := os.Getenv("K_SINK")

Cali0707 · 2024-05-28T15:18:49Z

cmd/mqttsource/main.go

+	"net"
+	"os"
+
+	mqtt_paho "github.com/cloudevents/sdk-go/protocol/mqtt_paho/v2"


Suggested change

mqtt_paho "github.com/cloudevents/sdk-go/protocol/mqtt_paho/v2"

mqttv2 "github.com/cloudevents/sdk-go/protocol/mqtt_paho/v2"

Cali0707 · 2024-05-28T15:19:53Z

cmd/mqttsource/main.go

+		},
+	}
+
+	p_receive, err := mqtt_paho.New(ctx, config, mqtt_paho.WithSubscribe(subscribeOpt))


Suggested change

p_receive, err := mqtt_paho.New(ctx, config, mqtt_paho.WithSubscribe(subscribeOpt))

mqttReceiver, err := mqtt_paho.New(ctx, config, mqtt_paho.WithSubscribe(subscribeOpt))

Cali0707 · 2024-05-28T15:20:04Z

cmd/mqttsource/main.go

+	}
+	defer p_receive.Close(ctx)
+
+	c_receive, err := cloudevents.NewClient(p_receive)


Suggested change

c_receive, err := cloudevents.NewClient(p_receive)

ceReceiver, err := cloudevents.NewClient(p_receive)

Cali0707 · 2024-05-28T15:20:14Z

cmd/mqttsource/main.go

+		log.Fatalf("failed to create client: %v", err)
+	}
+
+	c_send, err := cloudevents.NewClientHTTP()


Suggested change

c_send, err := cloudevents.NewClientHTTP()

ceClient, err := cloudevents.NewClientHTTP()

ctmphuongg added 2 commits May 10, 2024 19:06

imp mqtt source

f926ddb

package installs

2e36a40

knative-prow bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 10, 2024

knative-prow bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 10, 2024

knative-prow bot requested review from Cali0707 and odacremolbap May 10, 2024 23:12

format

f91dbad

knative-prow bot requested a review from matzew May 13, 2024 19:24

fix mqtt sending

86b00ed

knative-prow bot requested a review from Leo6Leo May 14, 2024 15:07

del debug stm

d9d62a3

Leo6Leo reviewed May 14, 2024

View reviewed changes

cmd/mqttsource/main.go Show resolved Hide resolved

add boiler template

cd40d41

ctmphuongg marked this pull request as ready for review May 15, 2024 15:58

knative-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 15, 2024

ctmphuongg requested a review from Leo6Leo May 15, 2024 15:58

Merge branch 'knative:main' into mqtt

5038750

Leo6Leo suggested changes May 27, 2024

View reviewed changes

cmd/mqttsource/main.go Outdated Show resolved Hide resolved

cmd/mqttsource/main.go Outdated Show resolved Hide resolved

config/tools/mqttsource/mqttsource.yaml Outdated Show resolved Hide resolved

knative-prow bot assigned Leo6Leo May 27, 2024

update codegent

d456eb6

knative-prow bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 27, 2024

knative-prow bot requested review from creydr and pierDipi May 28, 2024 13:48

knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label May 28, 2024

Cali0707 requested changes May 28, 2024

View reviewed changes

knative-prow bot assigned Cali0707 May 28, 2024

knative-prow bot removed the lgtm Indicates that a PR is ready to be merged. label May 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prototype a MQTT Source #7919

Prototype a MQTT Source #7919

ctmphuongg commented May 10, 2024 •

edited

knative-prow bot commented May 10, 2024

knative-prow bot commented May 10, 2024

codecov bot commented May 10, 2024 •

edited

Cali0707 commented May 13, 2024

ctmphuongg commented May 14, 2024

Leo6Leo left a comment

Leo6Leo commented May 27, 2024

Leo6Leo commented May 28, 2024 •

edited

Cali0707 left a comment

Cali0707 May 28, 2024

Cali0707 May 28, 2024

Cali0707 May 28, 2024

Cali0707 May 28, 2024

Cali0707 May 28, 2024

Cali0707 May 28, 2024

Cali0707 May 28, 2024

	mqtt_paho "github.com/cloudevents/sdk-go/protocol/mqtt_paho/v2"
	mqttv2 "github.com/cloudevents/sdk-go/protocol/mqtt_paho/v2"

	p_receive, err := mqtt_paho.New(ctx, config, mqtt_paho.WithSubscribe(subscribeOpt))
	mqttReceiver, err := mqtt_paho.New(ctx, config, mqtt_paho.WithSubscribe(subscribeOpt))

	c_receive, err := cloudevents.NewClient(p_receive)
	ceReceiver, err := cloudevents.NewClient(p_receive)

	c_send, err := cloudevents.NewClientHTTP()
	ceClient, err := cloudevents.NewClientHTTP()

Prototype a MQTT Source #7919

Are you sure you want to change the base?

Prototype a MQTT Source #7919

Conversation

ctmphuongg commented May 10, 2024 • edited

Proposed Changes

Notes: For testing

knative-prow bot commented May 10, 2024

knative-prow bot commented May 10, 2024

codecov bot commented May 10, 2024 • edited

Codecov Report

Cali0707 commented May 13, 2024

ctmphuongg commented May 14, 2024

Leo6Leo left a comment

Choose a reason for hiding this comment

Leo6Leo commented May 27, 2024

Leo6Leo commented May 28, 2024 • edited

Cali0707 left a comment

Choose a reason for hiding this comment

Cali0707 May 28, 2024

Choose a reason for hiding this comment

Cali0707 May 28, 2024

Choose a reason for hiding this comment

Cali0707 May 28, 2024

Choose a reason for hiding this comment

Cali0707 May 28, 2024

Choose a reason for hiding this comment

Cali0707 May 28, 2024

Choose a reason for hiding this comment

Cali0707 May 28, 2024

Choose a reason for hiding this comment

Cali0707 May 28, 2024

Choose a reason for hiding this comment

ctmphuongg commented May 10, 2024 •

edited

codecov bot commented May 10, 2024 •

edited

Leo6Leo commented May 28, 2024 •

edited