Go Change from github.com/Shopify/sarama to github.com/IBM/sarama

[exadevt]

Description

My go pacgae coded with Shopify/sarama is not working anymore. I followed the recommended change to replace github.com/Shopify/sarama by github.com/IBM/sarama and enned up with the following error when runnin go mod tidy.

/go/pkg/mod/github.com/exadevt/go-commont@v0.10.1/events/consumer.go:48:56: cannot use cm (variable of type *"github.com/IBM/sarama".ConsumerMessage) as *"github.com/Shopify/sarama".ConsumerMessage value in argument to kafka_sarama.NewMessageFromConsumerMessage
#0 58.51 /go/pkg/mod/github.com/exadevt/go-commont@v0.10.1/events/sender.go:44:61: cannot use config.SaramaConfig (variable of type *"github.com/IBM/sarama".Config) as *"github.com/Shopify/sarama".Config value in argument to kafka_sarama.NewSender

==================consumer.go
cm is a variable declared here in consumer.go

func (c *CloudEventsMessageConsumer) HandleKafkaMessage(cm *sarama.ConsumerMessage) error {
message := kafka_sarama.NewMessageFromConsumerMessage(cm)
if rs, rserr := binding.ToEvent(context.Background(), message); rserr == nil {
c.handleCloudEvent(*rs)
}
return nil
}

sender.go================================
func newKafkaCloudEventsProducerWithTopic(config *CloudEventsConfig, topic string) (*KafkaCloudEventsProducer, error) {
// We are using a sync producer which requires setting the variables below
config.SaramaConfig.Producer.Return.Errors = true
config.SaramaConfig.Producer.Return.Successes = true

sender, err := kafka_sarama.NewSender(config.KafkaBrokers, config.SaramaConfig, topic)
if err != nil {
	return nil, err
}

client, err := cloudevents.NewClient(sender, cloudevents.WithTimeNow(), cloudevents.WithUUIDs())
if err != nil {
	return nil, err
}

return &KafkaCloudEventsProducer{
	client: client,
	source: config.EventSource,
}, nil

}

Versions

github.com/Shopify/sarama v1.27.0

Sarama	Kafka	Go
1.27		1.20

Configuration

Logs

logs: CLICK ME

Additional Context

[dnwe]

cannot use cm (variable of type *"github.com/IBM/sarama".ConsumerMessage) as *"github.com/Shopify/sarama".ConsumerMessage value in argument to kafka_sarama.NewMessageFromConsumerMessage

This message shows that you appear to have updated the import path in your main module, but not in the github.com/exadevt/go-commont library that you're calling into

[exadevt]

The library github.com/exadevt/go-commont was updated.

However there is a go module from "github.com/cloudevents" that's still has an "import github.com/Shopify/sarama".

There is a fix mentioned here at : cloudevents/sdk-go@7f5ef39

What do I ned to do to implement that fix?

[dnwe]

Looks like they haven't cut a new release since making that change

In the short term you can bump the dependency via go get github.com/cloudevents/sdk-go/protocol/kafka_sarama/v2@HEAD, but I'm not sure how/when they plan to cut a new release version as it would ultimately need a semver bump

[dnwe]

Raised an upstream issue to query that cloudevents/sdk-go#938

[exadevt]

Thank you for raising the issue.

the go get ...@Head does not seem to work with my dockerfile config.

[exadevt]

Here is the result of the go get.

go: downloading github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8
go: downloading github.com/Shopify/sarama v1.41.2
go: downloading github.com/avast/retry-go v2.7.0+incompatible
go: found github.com/Shopify/sarama in github.com/Shopify/sarama v1.41.2
go: found github.com/avast/retry-go in github.com/avast/retry-go v2.7.0+incompatible
go: found github.com/xdg/scram in github.com/xdg/scram v1.0.5
go: found github.com/globalsign/mgo in github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8
go: github.com/exadevt/hydrophone imports
github.com/exadevt/go-commont/events imports
github.com/Shopify/sarama: github.com/Shopify/sarama@v1.41.2: parsing go.mod:
module declares its path as: github.com/IBM/sarama
but was required as: github.com/Shopify/sarama
ubuntu@ip-10-0-1-197:/rb/rb6-i18n/hydro-exadevt-v1/hydrophone$ go get github.com/cloudevents/sdk-go/protocol/kafka_sarama/v2@HEAD
ubuntu@ip-10-0-1-197:/rb/rb6-i18n/hydro-exadevt-v1/hydrophone$

[exadevt]

Still failing on Shopify/sarama...

[exadevt]

[development 19/19] RUN ./build.sh:
#0 5.876 go: writing stat cache: mkdir /go/pkg/mod/cache/download/github.com/exadevt: permission denied
#0 5.876 go: downloading github.com/exadevt/go-commont v0.10.1
#0 5.877 go: github.com/exadevt/go-commont/events@v0.10.1: mkdir /go/pkg/mod/cache/download/github.com/exadevt: permission denied

---

Dockerfile:49

[dnwe]

The error in #2652 (comment) still suggests that whilst you have `"github.com/IBM/sarama" in your go.mod file and/or in one of more of your source files, you must still have an import of the old path:

import (
   "github.com/Shopify/sarama"
)

somewhere in you Go code in a file in the github.com/exadevt/go-commont/events module

The error in #2652 (comment) I can't easily help you with as your code repos are private, but the go get should just be done in your own repo to bump the go.mod to the newer version rather than doing it at build time

[dnwe]

@exadevt so adding a go.mod/go.sum with https://github.com/exadevt/go-commont/pull/1 allows the library module to build and tests to pass

[exadevt]

I couldn't find an import to Shopify/sarama to exadevt/go-commont/events.

I made exadevt/go-comments/events public

[exadevt]

I have two modules:
Module 1 is module github.com/exadevt/hydrophone
Module 2 is github.com/exadevt/go-commont

On Module1
I import github.com/exadevt/go-commont
package main

import (
"context"
"crypto/tls"
"encoding/json"
"log"
"net/http"
"path"
"time"

clinicsClient "github.com/exadevt/clinic/client"
"go.uber.org/zap"

ev "github.com/exadevt/go-commont/events"
"github.com/exadevt/hydrophone/events"
"github.com/exadevt/hydrophone/localize"

"github.com/gorilla/mux"
"go.uber.org/fx"

"github.com/kelseyhightower/envconfig"
"github.com/exadevt/go-commont/clients"
"github.com/exadevt/go-commont/clients/disc"
"github.com/exadevt/go-commont/clients/highwater"
"github.com/exadevt/go-commont/clients/shoreline"
"github.com/exadevt/hydrophone/api"
sc "github.com/exadevt/hydrophone/clients"
"github.com/exadevt/hydrophone/models"
"github.com/exadevt/hydrophone/templates"

)
On Module 2:
I import cloudevents and IBM/sarama:

import (
"context"
"github.com/IBM/sarama"
"github.com/cloudevents/sdk-go/protocol/kafka_sarama/v2"
cloudevents "github.com/cloudevents/sdk-go/v2"
"github.com/cloudevents/sdk-go/v2/binding"
"log"
)

On which module should I run go get github.com/cloudevents/sdk-go/protocol/kafka_sarama/v2@HEAD? on hydrophone or from go-commont ?

Hydrophone uses a Dockerfile to build a docker image.

Module 2 does not use Dockerfile or docker. It's mainly a set a common librairies used by other modules such as hydrophone.

[exadevt]

Please find enclosed the go mod tidy for hydrophone. I noticed it contains some references to Shopify.

Although hydrophone has no direct references to Shopify

[exadevt]

hydrophone go mod tidy.txt

[exadevt]

References to Shopify from hydrophone mod tidy

go: downloading github.com/Shopify/goreferrer v0.0.0-20220729165902-8cddb4f5de06
go: downloading github.com/Shopify/toxiproxy/v2 v2.5.0

[exadevt]

I have made public exadevt/hydrophone, exadevt/clinic and exadevt/go-commont

[dnwe]

Repos seem to have been made private again, but the fix I shared with you is what you need basically — go mod init where you don't currently have a module file in the github.com/exadevt/go-commont library, then go mod tidy followed by go get github.com/cloudevents/sdk-go/protocol/kafka_sarama/v2@HEAD

[exadevt]

I followed the instructions, run the Dockerfile inside hydrophone (module 1 folder)
and ended up with this error message.

[development 12/18] RUN go mod download:
#0 0.880 verifying github.com/exadevt/go-commont@v0.10.1/go.mod: checksum mismatch
#0 0.880 downloaded: h1:xjJZQ9Q7jDGL8VHdkikJTZb24ceS+457I6YjWCYGQWo=
#0 0.880 go.sum: h1:2vgV79Ze03+RdeLcqjyN96uErqtk12p19o1f4DMxqI0=
#0 0.880
#0 0.880 SECURITY ERROR
#0 0.880 This download does NOT match an earlier download recorded in go.sum.
#0 0.880 The bits may have been replaced on the origin server, or an attacker may
#0 0.880 have intercepted the download attempt.
#0 0.880
#0 0.880 For more information, see 'go help module-auth'.

[exadevt]

I've made the repo public...

[exadevt]

Inside the Dockerfile I ran the go get...@Head.

It fails with the message I sent you before.