The results are in! See what nearly 90,000 developers picked as their most loved, dreaded, and desired coding languages and more in the 2019 Developer Survey.

Questions tagged [sarama]

sarama is a Golang client library for Apache Kafka.

2
votes
1answer
28 views

how to handle consumer recovery by offset with shopify sarama

I've read that kafka provides a consumer client library that allows recovery by saving the last offset read in zookeeper (not 100% sure about where it's stored). Is it possible to do the same with ...
0
votes
0answers
15 views

Sarama Producer needs to be recreated/reinstantiated when kafka is recycled

I have created Kafka Producer using sarama with below code. var producer sarama.AsyncProducer //...this is Global declaration config := sarama.NewConfig() //config.Producer.Return.Successes = false /...
1
vote
2answers
40 views

Do I need one or many sarama.SyncProducer inside my Golang app?

I am new in Golang, I need to write the app that will publish events to kafka, I can't find the answers on the following questions: How many sarama.SyncProducer do I need? Is it fine to use one ...
0
votes
0answers
25 views

Sarama Kafka consumer didn't use up all bandwidth

We use Sarama's kafka-console-consumer to run a performance test. Without changing any Sarama configuration, above program uses up 1/4 of the EC2 instance bandwidth. If we create 3 more sarama....
0
votes
0answers
33 views

Running Sarama tests on a live kafka cluster

In the GitHub repo for Sarama https://github.com/Shopify/sarama there are some *_test.go files. How to run these tests on a live kafka cluster?
0
votes
0answers
98 views

Delay in Golang Consumer Receiving Kafka Messages After Connecting to Kafka

I'm new to Golang and Kafa so this might seem like a silly question. After my Kafka consumer first connects to the Kafka server, why is there a delay (~ 20 secs) between establishing connection to ...
1
vote
0answers
65 views

package unable to consume from a partition

I'm not able to consume from a topic and I'm not sure if there's something wrong with my code or my kafka configuration. The problem I'm getting is it's stuck on the print statement of "Starts" so it'...
0
votes
1answer
382 views

Invalid Timestamp when writing a Kafka producer with sarama

I have a Kafka instance running (locally, in a Docker) and I created a producer in Go, using the sarama package. As I want to use Kafka Streams on my topic, the producer has to embed a timestamp in ...
0
votes
0answers
68 views

Sarama production to compacted topic not compacting

High Level Issue I am running kafka locally and I am using a compacted topic. When I run the command line producer and consumer I can verify that compaction is occurring but when I use sarama ("...
0
votes
1answer
392 views

Unable to create Kafka producer client with Sarama Golang package-“client/metadata got error from broker while fetching metadata: EOF”

Versions: GoLang 1.10.2 Kafka 4.4.1 Docker 18.03.1 I'm trying to use Shopify's Sarama package to test out my Kafka instance. I used Docker compose to stand up Kafka/Zookeeper and ...
2
votes
1answer
241 views

how group id can be used in Sarama

As you probably know, Sarama's documentation is so bad. Nothing is clear. I have just found this example: https://godoc.org/github.com/Shopify/sarama#example-Consumer It consumes the messages. That's ...
0
votes
1answer
77 views

How to get multiple subscriptions in sarama

Someone knows how to make a multi-subscription topics in Apache Kafka using Sarama I have a simple consumer, and I need to subscribe to three different topics topic := "Payments" // need to be "...
0
votes
0answers
167 views

Kafka messages not delivered to consumer

I am facing a very weird problem over here. I started a Kafka consumer using sarama-cluster library in go to consume some messages from a kafka topic. But the messages are not being received by the ...
0
votes
1answer
187 views

I always get 0 partitions when I call SendMessage(msg). I am specifying 12 via command line

What configuration values are you using for Sarama and Kafka? Kafka Version: kafka_2.12-1.1.0.tgz Go Version: 1.9.1 package kafka import ( "flag" "fmt" "log" "strings" "github....
0
votes
2answers
1k views

kafka server: Offset's topic has not yet been created

I'm using Kafka server 0.9 + zookeper. I'm new with Kafka. It's running in virtualbox , I can connect to it using public IP so it's working ... more or less: can get topics and messages. So for now I ...
-2
votes
1answer
125 views

Why my kafka cosumer group does not work?

I'm using sarama-cluster (written by Golang kafka consumer client) In broker, my topic's partition offset was 11000 and my consumer group's partition offset was 10100. Then I run my cluster-consumer,...
1
vote
0answers
267 views

Understanding GoLang Profiling output

Some context: I'm trying to debug a GoGRPC server, a specific API call seems to take a lot of time. This call does a number of reads to Kafka(lets say 10-20) so I expected it to take some time, just ...
0
votes
1answer
179 views

What is the proper way to read from the errors channel in Sarama?

I am using the Sarama library written in Go to read from an error channel when I produce a message. The overall code looks like this which is enclosed within a function: producer.AsyncProducer.Input()...
3
votes
1answer
650 views

Sarama cannot talk to the Kafka server

So I am trying to configure a Sarama (a native go client for kafka) producer client. I have configured my TLS accordingly making sure the client certs were generated using proper ciphers. My Go code ...
0
votes
1answer
188 views

Apache Kafka consumer implemented in Golang panics when deployed to Docker

here is my attempt in implementing a simple microservice that is supposed to read messages from a kafka server and send it via HTTP. It works fine when i run it from the terminal, but when deployed on ...
1
vote
1answer
411 views

Unable to consume messages from locally running Kafka server, using Golang Sarama Package

I am making a simple Telegram bot that would read messages from a local Kafka server and print it out to a chat. Both zookeeper and kafka server config files are at their defaults. Console consumer ...
3
votes
1answer
594 views

Mocking NewConsumer in sarama-cluster

Is there a way to test/mock sarama-cluster's NewConsumer function without having actual brokers set up? What am I missing here? Code I'm trying to test: import cluster "github.com/bsm/sarama-cluster"...
1
vote
0answers
401 views

Kafka Consumer not consuming messages after upgrade

I use Go and did a server upgrade to kafka 0.11 and sarama client to v1.13.0. I am also using sarama-cluster instead of wvanbergen/kafka for consumer group management now. Note that kafka upgrade ...
0
votes
2answers
691 views

How to notify consumer that new topic has been created in Kafka?

I am trying to get my consumer to dynamically update its consumption. Let me give you a more concrete example using animals. Imagine that I have a pet store, every topic is a type of animal (e.g. ...
1
vote
0answers
142 views

Upgrading consumers from zk based offset storage to kafka based storage

I am using Golang and Sarama client. Kafka version is 0.9 which I plan to upgrade. I am planning to upgrade sarama clients to latest version and use sarama-cluster instead of wvanbergen/kafka. I see ...
0
votes
1answer
424 views

Kafka 0.11 / Golang Sarama version support

I spent some time finding that my Go application connecting to a Kafka 0.11 cluster was using the old 0.8.2 version of the library, which is missing the Timestamp value in the response. I then found ...
0
votes
1answer
425 views

How does offset work when I have multiple topics on one partition in Kafka?

I am trying to develop a better understanding of how Kafka works. To keep things simple, currently I am running Kafka on one Zookeeper with 3 brokers and one partition with duplication factor of 3. I ...
1
vote
1answer
473 views

Why partition is needed in Shopify Sarama consumer to consume messages

I am sorry for posting a question related to a Kafka Library as not many people are interested in Library specific questions. But this library is one of the most used library for golang-Kafka ...
7
votes
2answers
820 views

Creating Kafka topic in sarama

Is it possible to create kafka topic in sarama? I know java API enables you do create topic but I couldn't find any information on how to do that in sarama. if it's possible, an example or explanation ...
0
votes
1answer
3k views

How to create a kafka consumer group in Golang?

An available library is sarama (or its expansion sarama-cluster) however no consumer group example are provided, not in sarama nor in sarama-cluster. I do not understand the API. May I have an ...
1
vote
1answer
344 views

How to get GroupID in Golang Kafka 10?

I am using Kafka 10.0 and https://github.com/Shopify/sarama. I am trying to get the offset of the latest message that a consumer processed. To do so I've found the method NewOffsetManagerFromClient(...
1
vote
1answer
1k views

How to set consumer to start from a specific offset in Golang Kafka 10

My need is to make the producer to start from the last message it processed before it crashed. Fortunately I am in the case of having only one topic, with one partition and one consumer. To do so I ...