Questions tagged [apache-kafka]
Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming applications.
apache-kafka
33,276
questions
0
votes
0
answers
8
views
Kafka. Partition leader election
Does kafka automatically perform unclean leader election for partitions?
I have topic
./kafka-topics.sh --bootstrap-server XXX --describe --topic my-topic
Topic: my-topic TopicId: ...
0
votes
0
answers
5
views
conflit class loader between quarkus and springframework
I'm actually working in project to test quarkus kafka and in the same time I tried CXFclient in quarkus also.
When I run my unit test, I get the following error:
Runtime java.lang.RuntimeException: io....
0
votes
0
answers
8
views
How to create a custom deserialiser for Apache Flink
I am building a Flink Project using Python that streams events from a Kafka topic. The payload in the topics has been encoded and the SimpleStringSchema is unable to decode them. I have create a ...
0
votes
0
answers
7
views
Confluent-Kafka: no broker available for coordinator query: intervaled in state query-coord
I am using python-confluent-kafka to create a producer.
The Kafka cluster is on MSK 3.7.x KRaft with IAM enabled and TLS enabled, both within the cluster and between clients and brokers.
Any ideas on ...
- 195
0
votes
0
answers
6
views
Kafka message order while processing the message
In Kafka there are 3 topics, but the messages are like order ready, ship and pickup.. There are huge requests coming for these messages. These messages are related to each other, so how do you make ...
0
votes
0
answers
7
views
Strimzi Kafka Connect Cluster - Resource planning
I'm planing resources which will be dedicated to run Kafka Connect Clusters in Strimzi Distribution.
My workloads are pretty small, there will be probably only 1 task per connector needed.
I would go ...
- 134
0
votes
0
answers
7
views
Failed to find any class that implements Connector and which name matches io.debezium.connector.postgresql.PostgresConnector
getting below error while trying to connect debezium with kafka
.\bin\windows\connect-standalone.bat config/connect-standalone.properties config/postgres.properties
INFO Kafka Connect started (org....
0
votes
1
answer
15
views
Kafka-streams filter messages using headers
We're trying to use kafka streams for our project to read data from one topic and write to another, and we have a use case to use KafkaHeaders as a mechanism to filter our certain records.
Example, in ...
- 2,503
1
vote
0
answers
11
views
Cannot connect with kafka-ui to kafka
I start kafka locally with
$ docker run -p 9092:9092 apache/kafka:3.7.1
$ docker exec -it gifted_wright /bin/bash
e8bebba64b30:/$ cd /opt/kafka/bin
e8bebba64b30:/opt/kafka/bin$ sh kafka-topics.sh --...
- 33
-1
votes
1
answer
28
views
"Microservices Using Database Per Service" pattern in practice
I am working on a project where I would like to use this pattern. I have read for days, but I am still not sure how to implement this pattern properly and keep the multiple DBs in sync.
Let's suppose ...
- 21.9k
0
votes
0
answers
8
views
Filebeat output using SAS token to azure eventhub
I want to send my filebeat logs to eventhubs using SAS token, earlier we wanted to use Kafka output but we came to know Kafka doesn't support SAS tokens, so now is there some way we can send data ...
- 13
1
vote
0
answers
17
views
Kafka UI message count
I am using kafka UI as a visual interface for viewing topics. I wanted to check if my service produced correct number of messages to the topic. When i used live mode in given topic i was able to ...
- 81
0
votes
0
answers
20
views
Pyflink 1.18 checkpointing (not starting from last checkpoint)
I’m using PyFlink (1.18v) on Kubernetes pods and want to ensure that checkpointing is enabled so that if a pod restarts, the process continues from the last checkpoint. I have configured checkpointing ...
- 41
0
votes
0
answers
11
views
Unexpected Kafka request of type METADATA during SASL handshake while trying to connect with kafka server by a consumer
I'm trying to setup SASL plain authentication in kafka. But when I try to read a message or send a message, the Kafka Server logs the following error:
INFO [SocketServer listenerType=ZK_BROKER, nodeId=...
- 3
0
votes
0
answers
10
views
Can debezium/Kafka connect ignore or resize too large messages?
We're using debezium via AWS MSK serverless and MSK Connect to monitor the binlog of RDS Aurora MySQL.
For 99% of our data this all works fine, but very occasionally debezium fails with:
[Worker-...
- 161