Questions tagged [apache-kafka-connect]

Apache Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems. - https://kafka.apache.org/documentation.html#connect

0
votes
0answers
4 views

Kafka Connect ignoring the Subject Strategies specified

I want to publish multiple table data on to same Kafka topic using the below connector config, but I am seeing below exception Exception Caused by: io.confluent.kafka.schemaregistry.client.rest....
1
vote
1answer
12 views

Kafka gives Invalid receive size with Hyperledger Fabric Orderer connection

I was setting up a new cluster for Hyperledger Fabric on EKS. The cluster has 4 kafka nodes, 3 zookeeper nodes, 4 peers, 3 orderers, 1 CA. All the containers come up individually, and the kafka/...
0
votes
0answers
14 views

How to have byte array data produced by RabbitMQ Kafka connector get a key?

I use the official Kafka RabbitMQ connector to get JSOn messages from a RabbitMQ instance. Currently, this connector if configured like this name=RabbitMQConsentConsumer connector.class=io.confluent....
-1
votes
1answer
24 views

How to solve error which arrives when I am trying to connect Kafka with the help of “ Kafka Connect ”?

I got the error while bin/connect-distributed.sh config/connect-distributed.properties config/connect-distributed.properties config/connect-distributed.properties: line 23: bootstrap.servers=...
0
votes
0answers
23 views

Debezium is failing to snapshot big table size

I think I might have missed some configuration, but we're trying to use Debezium to snapshot all the rows from a table that has about 8 million records and after some time it stops. The connector ...
0
votes
1answer
19 views

Kafka spring boot application producer and unable to reflect that with Kafka Sink Connecyor with Avro format

My target is I have spring boot application kafka producer with Avro serializing property and i am excepting the message which is pushed to respective topic that should access by confluent Sink ...
0
votes
1answer
12 views

Kafka Connect S3 Dynamic S3 Folder Structure Creation?

I have manually installed Confluent Kafka Connect S3 using the standalone method and not through Confluent's process or as part of the whole platform. I can successfully launch the connector from the ...
0
votes
1answer
48 views

How get the stream from kafka topic to elasticsearch with confluent?

I'm read data from machine and stream it as JSON to a kafka topic. I would like to read this topic and store the streamdata into elasticsearch with confluent. My steps: 1. Create KSQL Streams to ...
0
votes
1answer
24 views

How to commit Kafka messages to HDFS sink on reaching a specific size (128 Mb)

My configuration: Confluent (5.0.0) Kafka produces some avro messages. Connect worker (HDFS connector sink) streams these messages to HDFS node in Parquet format. I configured connect worker to commit ...
0
votes
1answer
27 views

Kafka connector and Schema Registry - Error Retrieving Avro Schema - Subject not found

I have a topic that will eventually have lots of different schemas on it. For now it just has the one. I've created a connect job via REST like this: { "name":"com.mycompany.sinks.GcsSinkConnector-...
0
votes
1answer
43 views

org.apache.kafka.connect.errors.DataException: Struct schemas do not match

Trying to create and populate an array schema of type string inside an array schema of particular object type using kafka.connect.data.Schema in java. But getting error org.apache.kafka.connect....
0
votes
0answers
32 views

Connect for MQ, SSL connection

I am trying to connect to the remote queues using secured SSL connection and Kafka Connect for MQ. I have all the details provided by third party regarding SSL connection and Queue Manager details. ...
1
vote
0answers
35 views

How to use the Kafka Connect JDBC to source PostgreSQL with multiple schemas that contain tables with the same name?

I need to source data from a PostgreSQL database with ~2000 schemas. All schemas contain the same tables (it is a multi-tenant application). The connector is configured as following: { "name": "...
0
votes
1answer
23 views

Replying kafka template connect headers(CorrelationId) is not sent to Google pub sub

I am trying to use Replyingkafkatemplate, where correlationid is automatically added but when the message is posted to Google pubsub, using kafka connector-Sink connector, the headers are missing. ...
0
votes
0answers
19 views

How to use custom avro schema with dockerized connect-datagen?

I'm trying to generate test data from custom avro schema's using dockerized connect-datagen. Here is an example on Github: https://github.com/damc-dev/kafka-docker-datagen I mapped the directory to ...
0
votes
1answer
49 views

Debezium SQLServerConnector for Kakfa Connect not creating topics in Kafka

I'm trying to get the SQLServerConnector from Debezium working in Kafka Connect and everything seems to be running fine, except there are no topics created. Am I missing something? It starts the task ...
3
votes
1answer
88 views

How to add a new Kafka topic for every new producer from IoT Hub?

I'm working on Azure cloud solution. I'm using IoT Hub connected to Kafka in order to process data coming from various IoT devices. What I'm facing is that all the data coming from multiple devices ...
0
votes
2answers
76 views

Kafka Connect: how to parse String to Map

Let's suppose I have a file filled with JSON objects/lines separated by new line characters (\n). When a FileStreamSource based connector would read this file it would consider each line as java.lang....
0
votes
1answer
29 views

KafkaConnect produces CDC event with null value when reading from mongoDB with debezium

When reading the kafka topic which contains lots of CDC events produced by Kafka-Connect using debezium and the data source is in a mongodb collection with TTL, I saw some of the CDC events are null, ...
0
votes
0answers
24 views

Data Loss in Kafka S3 Connector

We are using Kafka S3 Connector for log pipeline, as it guarantees exactly-once semantics. However, we've experienced two data loss events on different topics. We found a suspicious error message in ...
0
votes
1answer
29 views

Configuring Kafka connect Postgress Debezium CDC plugin

I am trying to use kafka connect to read changes in postgress DB. I have Kafka running on my local system and i want to use the Kafka connect API in standalone mode to read the postgress server DB ...
0
votes
0answers
18 views

Errors/Issues using different converters in kafka-connect to S3

I have been trying to implement the confluent kafka-connect image to connect or our on prem S3. We have successfully written to s3 from the box using Boto3. So we know it is not a connection issue. ...
0
votes
0answers
30 views

Kafka Connect - How to customize json using transformation

i'm trying to achieve { "source": "NEWS", "metadata": { "publishTime": "02/06/2019 09:56:24.317", "channel": "paper", "status":"active" }, "Data": { "NAME": 67, "GENDER": "MALE", ... } } but i'...
0
votes
2answers
33 views

Is it possible to apply SMT (Single Message Transforms) to messages from specified topics only

I'm streaming database change events from MySql database to Kafka using Debezium MySql connector. I need to apply specific transformations to records from some specified tables (but not from the ...
0
votes
0answers
83 views

Kafka JDBC Sink Connector: no tasks assigned

I try to start JDBC sink connector with following configuration: { "name": "crm_data-sink_hh", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "...
0
votes
1answer
26 views

Vault for Kafka distributed connectors

I am using a JBoss based vault to secure sensitive data such as the database credentials. I use a Java based HTTP REST client to create distributed Kafka connectors but ended up with a security ...
0
votes
1answer
31 views

kafka connect - How to filter schema metadata from payload

I'm trying to remove schema from the payload and here are the configurations connector.properties name=test-source-mysql-jdbc-autoincrement connector.class=io.confluent.connect.jdbc....
0
votes
0answers
31 views

Kafka Connect concatenate fields and convert to timestamp

I'm a total newbie with Kafka Connect, but trying to use it to load some data into Postgres. My message includes two fields "epochSeconds" and "nanos", however I would like to add a Timestamp column ...
0
votes
1answer
17 views

Error when used org.apache.kafka.connect.json.JsonConverter in Kafka Connect

I am trying to use Json converter in Kafka connect but it is throwing below error : {"type":"log", "host":"connecttest6-ckaf-connect-84866788d4-p8lkh", "level":"ERROR", "neid":"kafka-connect-...
0
votes
0answers
38 views

Adding new custom properties in Kafka Connect

I am trying to add new custom Kafka Converter which is a modification of JsonConverterConfig in connect-json. I am trying to add some new custom property say "schemas.modifications.enable" in the ...
0
votes
1answer
36 views

How to track rows (by id) with a specific column value using Kafka JDBC Connector?

I have a table containing a large number of records. There's a column defining a type of the record. I'd like to collect records with a specific value in that column. Kind of: Select * FROM ...
0
votes
1answer
36 views

How does a Solace source connector “tasks.max” connect to a queue?

We are implementing a Kafka topic that is a 3 machine cluster. We are going to be pulling off of a Solace queue and are wondering what the tasks.max parameter does and how it divides the messages. We ...
1
vote
1answer
60 views

Kafka Connect - Failed to flush, timed out while waiting for producer to flush outstanding messages

I am trying to use the Kafka Connect JDBC Source Connector with following properties in BULK mode. connector.class=io.confluent.connect.jdbc.JdbcSourceConnector timestamp.column.name=timestamp ...
0
votes
0answers
20 views

Ordered starting and waiting for containers

I have two containers in my tests using @Testcontainers with Junit5, a Kafka and a KafkaConnect. @Container private final KafkaContainer kafka = new KafkaContainer() .withNetwork(network) ...
0
votes
1answer
37 views

push messages from kafka consumer to mongodb

I have created kafka consumer using 'kafka-node', on the event consumer.on('message' ()=>{ connecting to mongodb and inserting to a collection. }) mongo.js file used to create connection to ...
0
votes
2answers
33 views

kafka_connect not finding its connectors

I have a confluent kafka cluster running inside docker containers on EC2 machines as below 3 docker containers on separate EC2 instances for zookeepers and kafka-brokers 1 docker container on a ...
1
vote
1answer
603 views

Using a connector with Helm-installed Kafka/Confluent

I have installed Kafka on a local Minikube by using the Helm charts https://github.com/confluentinc/cp-helm-charts following these instructions https://docs.confluent.io/current/installation/...
0
votes
0answers
59 views

How to transform all timestamp fields when using Kafka Connect?

I am trying to convert all timestamp fields to a string type with the format yyyy-MM-dd HH:mm:ss. To transform multiple fields, I have to create a transform for each one individually. ... "...
0
votes
0answers
44 views

Kafka Connect HDFS (Azure) Persist Avro Values AND String Keys

I have configured Kafka Connect HDFS to work on Azure Datalake, however I just noticed that the keys (Strings) are not being persisted in anyway, only the Avro values. When I think about this I ...
2
votes
1answer
595 views

Kafka connect: The configuration XXX was supplied but isn't a known config in AdminClientConfig

When starting Kafka-Connect, I saw lots of warnings 10:33:56.706 [DistributedHerder] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'config.storage.topic' was supplied but ...
0
votes
0answers
42 views

NullPointerException for kafka-connect-jdbc (postgres) without proper trace (WorkerSinkTask.deliverMessages)

I'm using JdbcSinkConnector with postgres. Config: { "name":"jdbc", "config":{ "topics":"some_topic", "connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector", "...
0
votes
1answer
248 views

Kafka Connect JDBC Connector - Exiting WorkerSinkTask due to unrecoverable exception

I am using the JDBC sink connector and have a bad message in the topic. I know why the message is bad (it is failing due to a FK constraint violation because of a issue with a producer). The error ...
0
votes
1answer
58 views

Kafka Connect JDBC connector query + incrementing mode chokes with large datasets at initial poll

I'm using the JDBC connector to move data from MySQL into Kafka. The data I'm interested in is coming from a select joining 3 tables, therefore I've configured my connector with mode:incrementing and ...
0
votes
1answer
52 views

Kafka Connect sink tasks ignore tolerance limits

I try to ignore bad messages in sink connector with errors.tolerance: all option. Full connector configuration: { "name": "crm_data-sink_pandora", "config": { "connector.class": "io....
1
vote
1answer
60 views

Can I have Kafka consumers/sink connects to skip specific partitions within a topic?

Any option in Kafka Connect to specify from which partition specifically to read the messages. Basically, I am looking for an option in Kafka Connects to manually assign a list of partitions to read. ...
1
vote
2answers
55 views

Kafka and Kafka Connect deployment environment

if I already have Kafka running on premises, is Kafka Connect just a configuration on top of my existing Kafka, or does Kafka Connect require it's own Server/Environment separate from that of my ...
0
votes
2answers
48 views

Kafka Connect S3 sink - how to use the timestamp from the message itself [timestamp extractor]

I've been struggling with a problem using kafka connect and the S3 sink. First the structure: { Partition: number Offset: number Key: string Message: json string Timestamp: timestamp }...
2
votes
1answer
50 views

How to connect kafka with MySQL on another Host?

My database is in another Linux node lets suppose xxx.xx.xxx.xxx:3306 and I am running the standalone command of kafka connect from my Windows machine. My server.properties code is name=test-source-...
0
votes
1answer
48 views

Kafka use case: continuously read from kafka, perform decryption on message, then insert to db

Kafka use case: continuously read from Kafka (perhaps a stream), perform decryption on message value with Java method, then insert to db (either sink connector or I have a Java REST API that can ...
0
votes
1answer
25 views

How to fix Couldn't open connection to jdbc:oracle:thin:@ldap://etc\nQuery mode must be specified

I am trying to create a Kafka-connect Source Instance using a rest API. I keep getting the above exception. I post this Connection information (Not including password) { "name": "example-...