Questions tagged [apache-kafka]
Apache Kafka is a distributed streaming platform that is used for building real-time data pipelines and streaming applications.
apache-kafka
33,265
questions
1
vote
0
answers
14
views
Kafka UI message count
I am using kafka UI as a visual interface for viewing topics. I wanted to check if my service produced correct number of messages to the topic. When i used live mode in given topic i was able to ...
- 81
0
votes
0
answers
9
views
Pyflink 1.18 checkpointing (not starting from last checkpoint)
I’m using PyFlink (1.18v) on Kubernetes pods and want to ensure that checkpointing is enabled so that if a pod restarts, the process continues from the last checkpoint. I have configured checkpointing ...
- 41
0
votes
0
answers
11
views
Unexpected Kafka request of type METADATA during SASL handshake while trying to connect with kafka server by a consumer
I'm trying to setup SASL plain authentication in kafka. But when I try to read a message or send a message, the Kafka Server logs the following error:
INFO [SocketServer listenerType=ZK_BROKER, nodeId=...
- 3
0
votes
0
answers
7
views
Can debezium/Kafka connect ignore or resize too large messages?
We're using debezium via AWS MSK serverless and MSK Connect to monitor the binlog of RDS Aurora MySQL.
For 99% of our data this all works fine, but very occasionally debezium fails with:
[Worker-...
- 161
0
votes
0
answers
11
views
High producer latency in AWS MSK Kafka
I am trying to run performance test script against my test kafka cluster in AWS. It looks like something is restricting the performance to about 2 MB/sec.
I have tried following:
Increased the type ...
- 1,897
0
votes
1
answer
8
views
How to Isolate Developers' Kafka Message Consumption in MassTransit Without Using Temporary Topics?
We are using MassTransit to consume Kafka messages in our application. However, we've encountered an issue where different service instances on developers' machines consume each other's messages ...
- 231
0
votes
0
answers
11
views
Kafka producer ordering
My apology if this is a question which is already been answered before.
Use Case:
I am reviewing an application (producer) which is using librdKafka as a 3rd party library to interact with Kafka ...
-1
votes
0
answers
12
views
Custom vault kafka configProvider for Mongo sink Connector
I implemented a custom vault configProvider with dynamic credentials rotation using configData(Map<String, String> data, Long ttl) for a MongoSink Kafka connect.
The first version of plugin ...
- 1
0
votes
0
answers
17
views
Spring kafka @KafkaListener missing several messages
I have KafkaListener
@KafkaListener(
idIsGroup = false,
id = "#{T(java.util.UUID).randomUUID().toString()}",
topics = "${kafka.topic}",
...
1
vote
0
answers
22
views
Data Prepper Using Kakfa as Source with KeyTab Authentication
We are trying to use Data Prepper with Kafka as a source and OpenSearch as a Sink. But the Kafka cluster authenticates only using KeyTab. In the documentation, it says SASL but it supports only ...
- 675
-4
votes
0
answers
16
views
Spark and kafka integration in anaconda or docker [closed]
For my final year project, I need to use apache spark and kafka for data streaming. I have already set up pyspark environment in anaconda, but as for kafka I have no clue as there are no guides that ...
- 13
0
votes
0
answers
30
views
spring boot kafka listener consuming messages multiple times
I have a spring boot + kafka application, which consumes the messages from kafka, process and update DB. We have set "manual auto commit" and the max poll records as "500" and max ...
- 53
0
votes
0
answers
30
views
Why I am facing [Errno 32] Broken pipe when I start new process?
I am using confluent-kafka to process messages from kafka topic.
Here is the code template:
from multiprocessing import Process, Manager
def handle_records(records):
for record in records:
...
- 53
0
votes
0
answers
15
views
How a Kafka consumer asyncCommit is asynchronous on a single thread
As I know Kafka consumer is not thread safe. So how can asyncCommit be asynchronous if it uses one main application thread?
The documentation is unclear about this. I expect that asyncCommit must be ...
0
votes
0
answers
15
views
How to prevent regular loss of heartbeat in spring kafka
There is a service built with Springboot(~3.0.1) which consumes events from a Kafka topic. This service regularly loses heartbeat and becomes non-functional.
The service does several writes to a MySQL ...
