Questions tagged [amazon-kinesis-analytics]
Amazon Kinesis Data Analytics is the way to analyze streaming data, gain actionable insights, and respond in real time. SQL users can query streaming data or build entire streaming applications using templates and an interactive SQL editor. Java developers can build streaming applications using open source Java libraries and AWS integrations to transform and analyze data in real-time.
amazon-kinesis-analytics
133
questions
0
votes
0
answers
49
views
AWS managed apache flink increasing checkpoint size
I am using AWS managed apache flink, using scala 2.12 and flink 1.15 (flink-scala) .
i added a keyed process function and a ValueState.
ever since, the checkpoint size is slowly increasing, eventually ...
- 1
0
votes
1
answer
77
views
Adding external dependencies in Pyflink and use it in UDF
I am writing a UDF which will make an API call to get back JSON payload. Here is what it looks like -
@udf(result_type=DataTypes.STRING())
def get_data():
response = requests.get("https:...
- 57
0
votes
1
answer
146
views
AWS Apache flink, application not running in aws, But works in local. What could be the issue?
I am using to build an application which gets data from kinesis data stream and sending that data to an external database using endpoint. I used python table API to achieve this, When i run the ...
0
votes
0
answers
14
views
FlinkKinesisProducer unable to publish events to kinesis stream in AWS me-central-1(UAE) region
I am having a Flink KDA (now Managed Apache Flink) application in one of the AWS account.
Here we are publishing some events from KDA (us-west-2 region) to kinesis stream in other account on me-...
- 132
0
votes
1
answer
146
views
Resolve Flink SQL windows tumble performance drop
I have a flink sql application which group the data from Kafka into database in real-time. The data are aggregated with interval: [1, 5, 10, 30, 60min, daily] and write to the corresponding DB table.
...
- 1
1
vote
1
answer
190
views
Question about reducing impact as part of Flink application update in Amazon Managed Service for Apache Flink
We deployed a Flink application through Amazon Managed Service for Apache Flink. The overall application performs as expected during runtime. But we faced challenges in maintaining the throughput ...
- 35
0
votes
0
answers
63
views
How to handle invalid JSON in the Apache Flink / AWS Kinesis Connector?
I have a simple Apache Flink (PyFlink) application which reads data from an AWS Kinesis stream using the Official flink connector and the Flink TableAPI https://nightlies.apache.org/flink/flink-docs-...
0
votes
0
answers
110
views
Implement Flink Async I/O with Redisson Redis client has caused an error "maximum number of DatagramSockets reached"
I have my Flink app deployed to AWS Managed Flink, while trying to implement async I/O with Redisson (Redis client) to connect to AWS MemoryDB (a Redis Cluster), I got an error (see below). It isn't ...
- 31
0
votes
0
answers
222
views
Flink: "Unable to Shutdown Event Loop" when stopping with savepoint on Managed Apache Flink
I'm running a Flink (1.15) application on AWS Managed Service for Apache Flink. It's a relatively simple job topology with a single Kinesis source that utilizes EFO, a map and filter operation, and ...
- 1
0
votes
2
answers
208
views
Apache flink seems buffering events before processing and streaming to the sink
I was wondering if someone can provide some insights on this. We are building a low latency processor using flink and we use kinesis streams and the managed apache flink in AWS to do that.
We are ...
- 94
0
votes
1
answer
54
views
Do not see data in AWS Kinesis
I created a stream in AWS Kinesis, straightforward:
I put records into the stream (I took the script from AWS documentation):
import datetime
import json
import random
import boto3
STREAM_NAME = &...
0
votes
0
answers
68
views
AWS Kinesis streaming application failed to sink data to Redis
I am using AWS Kinesis Streaming Application to process streaming data.
The application is running but I found nothing was sank to my Redis. Related code:
public class SpyProcessor {
private ...
0
votes
0
answers
109
views
Refreshing Kafka security certificates in Apache Flink application running on Amazon KDA
I am building a Flink Application and currently planning on deploying using Amazon Kinesis Analytics. I need to connect to a Kafka topic using SSL truststore/keystore certificates and am looking at ...
0
votes
0
answers
32
views
Most suitable architecture for AWS realt-time ETL pipeline with transactional tables as a sink
The task was as following - build a near real-time pipeline for the project in AWS infrastructure. Having Nosql(DynamoDB) and SQL(RDS databases) (S3 stored data can be added in the future) sources we ...
0
votes
2
answers
203
views
How to upgrade Amazon Kinesis Analytics Apache Flink runtime?
We have an application running on Apache Flink 1.13, presume we have a branch we have updated the code to Flink 1.15.2 and built a jar and put that on S3.
What do we do in the AWS Console to change an ...
- 31.3k