Questions tagged [amazon-kinesis]
Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale.
amazon-kinesis
1,829
questions
0
votes
1
answer
28
views
Streaming audio to Amazon Connect through Kinesis Video Streams
What I am trying to achieve is the following.
In Amazon Connect, when I create a Flow with Start Streaming Data block, it starts streaming to a Kinesis Video Stream, and I can call a lambda function ...
- 550
0
votes
0
answers
14
views
Improve the ingestion strategy for my IoT Sensor data
We have a fleet of about 2000 IoT sensors (this number is increasing rapidly) whose data we need to ingest into an AWS S3 bucket for further analysis and display in the UI. The sensors are used by ...
0
votes
0
answers
18
views
How can I use "SchemaName.TableName" partition key in AWS DMS to migrate data to an Amazon Kinesis data stream?
I'm trying to set up DMS to push data to kinesis stream, with partition keys based on the schema and table name. The intent is to get all cdc events for a given table hitting a single shard, to better ...
- 151
1
vote
0
answers
21
views
Firehose data backup add newline after every record?
I have a kinesis streamer with EventBridge as the Producer and Firehose Data as the consumer. Firehose converts the event json in a parquet file, which can be queried in an athena table.
This is done ...
- 333
0
votes
1
answer
30
views
Trigger Lambda function every 15 minutes to process Kinesis data by device ID, calculate average, and store in DB
I'm working on a serverless architecture using AWS Kinesis Data Streams and Lambda functions. I want to achieve the following:
Read data from the Kinesis stream in 15-minute blocks, starting from
00:...
0
votes
1
answer
24
views
AWS Architecture for Splitting and streaming data to multiple 3rd party Accounts
There are no doubt many ways to skin a cat. But usual one is better than the others.
If I had a stream of data which is captured by a rule and sent to a lambda function for processing. The lambda ...
- 1,508
3
votes
1
answer
45
views
Why is this Lambda function not running more concurrent executions?
I have an AWS Lambda function connected to an AWS Kinesis data stream. The volume of data in the stream has recently scaled up significantly, but the number of concurrent executions of the Lambda ...
- 108
0
votes
0
answers
19
views
Kinesis data stream - TRIM_HORIZON not showing recently added data
I am using boto3 to put data in Kinesis data stream. The code is running successfully and returns response with 200 status. But when I check the data in AWS console, am not able to see the recently ...
- 321
0
votes
0
answers
28
views
Kinesis Lambda Event Source Mapping Concurrent Invocation
I am running a Lambda function to read data from a Kinesis data stream and perform some processing. The stream has 4 shards and 10 data records. Shard01 has 1 record, Shard02 has no records, Shard03 ...
- 321
0
votes
0
answers
35
views
Getting only control record for some DMS Task TableMappings
AWS DMS
Source endpoint: Aurora Postgres
Target Endpoint: Kinesis Data stream
Task with these settings
MigrationType: cdc
TableMappings: |
{"rules":[{"rule-type":"selection&...
- 123
0
votes
0
answers
34
views
AWS Managed AWS apache flink application writes to kinesis output stream, but kinesis monitoring doesnt show any updates in graph
We are trying to capture some data using kinesis as input and output stream using AWS Managed apache flink as per steps in https://github.com/aws-samples/pyflink-getting-started/tree/main/getting-...
- 17
0
votes
0
answers
30
views
Read Data From Kinesis with Apache Beam Python SDK
I'm trying to use Apache Beam's Python SDK to read data from a Kinesis Strem. I am using ReadDataFromKinesis. This should use an external service writtent in Java to read data from the Kinesis stream. ...
0
votes
0
answers
51
views
How to define custom/user defined partition key in AWS DMS
I want to define custom/user defined partition key to the records in AWS DMS before pushing them to AWS Kinesis.
The idea is to keep same partition key for table 1 and table 2 and push them into ...
0
votes
0
answers
37
views
Java Kinesis Producer Library (KPL) try to access appData folder of unexisting username
When running Java Spring application with Kinesis Producer configured with KPL following exception appear during class loading:
Factory method 'kinesisProducer' threw exception with message:
nCaused ...
- 1
0
votes
0
answers
19
views
AWS Kinesis Stream Put Record
aws_templates % aws kinesis put-record --stream-name KenesisStream --cli-binary-format raw-in-base64-out --data '{"number1":1, "number2":100}' --partition-key 2
How can we define ...
- 1