Questions tagged [amazon-kinesis-firehose]
Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations
amazon-kinesis-firehose
628
questions
-2
votes
0
answers
26
views
Optimizing data processing with AWS Lambda and DynamoDB [closed]
I'm currently working on a C++ codebase that processes high-throughput data, generating around 5000 records per second. These records are sent in batches to AWS Kinesis Firehose. The data has a column ...
0
votes
0
answers
36
views
JQ Parsing a json stream input - Amazon Firehose
0
Hello,
I have a json stream coming into my aws firehose that can contain either metrics or report key-value pairs based on the data source. They look like this -
metrics stream
{ "metric":...
- 1,213
1
vote
0
answers
21
views
Firehose data backup add newline after every record?
I have a kinesis streamer with EventBridge as the Producer and Firehose Data as the consumer. Firehose converts the event json in a parquet file, which can be queried in an athena table.
This is done ...
- 333
0
votes
1
answer
24
views
AWS Architecture for Splitting and streaming data to multiple 3rd party Accounts
There are no doubt many ways to skin a cat. But usual one is better than the others.
If I had a stream of data which is captured by a rule and sent to a lambda function for processing. The lambda ...
- 1,508
0
votes
0
answers
29
views
Loss of Logs when using fluentbit storage type as filesystem
I am sending the applications related logs to two different
destinations one is file and other through firehose to opensearch to three different indices based on tags and parsing. Defined with ECS aws ...
- 1
0
votes
0
answers
13
views
AWS Kinesis Firehose, Lambda event match ovservability
I have standard AWS architecture to capture data from my source that streams data into a Firehose endpoint. Firehose streams data into s3, which again ahs a PUT event that triggers Lambda to clean ...
- 1,213
0
votes
0
answers
27
views
fluent-bit different timestamp in logs
I am trying to push my application logs from ec2 machines to Kinesis firehose and from there to Opensearch.
This is one sample log:
127.0.0.1 - - - 2024-05-22T17:16:01.570+0000 "GET /server-...
- 528
1
vote
0
answers
28
views
Glue Schema registry and Firehose
I want to explore how I can utilize Glue Schema Registry in our application and, if not feasible, what alternative options we have.
Currently, we stream data from DynamoDB tables to an S3-based data ...
- 4,791
1
vote
1
answer
112
views
How to find the direct put address for firehose?
I created a aws firehose stream with source input: direct put.
Where do i see the address now, i can send the data to. I can not find it.
- 2,702
1
vote
0
answers
54
views
How to configure firehose client to put record in a stream
I am trying to use firehose data stream and specifically putRecord api of the firehose service. I have created a class which takes firehose client and delivery stream name in constructor and a method ...
- 109
0
votes
0
answers
62
views
Why are my messages not getting from SNS Subscription to Firehose Endpoint?
I am looking to log sent emails from Amazon SES to an S3 bucket. To do this I am trying to leverage Amazon SNS to push data through a Firehose Stream and into the S3 bucket.
The problem that I am ...
- 1
0
votes
1
answer
68
views
Does firehose with MSK source and lambda transfomer support dynamic partitioning?
As per documentation I do not see mentioned anywhere that MSK does not support dynamic partitioning.
However when I am creating firehose stream from console, I do not see enable dynamic partitioning ...
- 31
0
votes
1
answer
42
views
Write incremental load data from multiple DynamoDB tables to single S3 bucket
I am working on use case where I need to transfer data from multiple DynamoDB tales to single S3 bucket in an event based manner. The design I had can support data from one DynamoDB table to s3 and is ...
- 51
0
votes
0
answers
18
views
Conversion of Syslogs to Json
from __future__ import print_function
import base64
import json
import re
print('Loading function')
def lambda_handler(event, context):
output = []
succeeded_record_cnt = 0
...
- 1
0
votes
0
answers
46
views
Data Preprocessing for splitting into multiple records with AWS Lambda at AWS Data Firehose
I need to split the data coming from AWS IoT Core by time and publish them into s3 in real-time.
For this, I searched other stack overflow pages and also read AWS's reference page but an error ...
- 43