Questions tagged [data-stream]
In Connection-oriented communication, a data stream is a sequence of digitally encoded coherent signals (packets of data or data packets) used to transmit or receive information that is in the process of being transmitted.
data-stream
276
questions
-1
votes
0
answers
14
views
Updating Postgres Table using a Datastream in spark
I am using scala Api "2.12.15" of spark "3.5.0" and i am trying to use a data stream to update the rows of my Postgres table :
my data stream and my table both has the same columns
...
- 5
0
votes
1
answer
101
views
Airbyte Postgres Kafka connection failed to run schema discovery
I'm trying to create a connection in Airbyte 0.61.0 (deployed in Docker containers in a Linux Server) using PostgreSQL (conector version 3.4.10) as source and Kafka (conector version 0.1.10) as ...
0
votes
1
answer
50
views
Datastream - AWS AURORA MySQL to BigQuery
I'm trying to stream/migrate data from AWS aurora mysql to BigQuery. By following this documentation Create an HA VPN gateway to a peer VPN gateway I able to ping private and public subnet vice versa -...
- 3
0
votes
1
answer
17
views
What does flink store as a state for SlidingWindows?
When I use Aggregation function over SlidingWindow what does flink store in state?
For example, if I aggregate count over sliding window with size 1 hour, and slide of 5 minutes. Does it mean that ...
1
vote
0
answers
185
views
Is there any way to stream to a parquet file in Ruby?
I am trying to create an archival tool for a Ruby On Rails app.
To this end, I wish to store the data in parquet files, ideally with one parquet file per table per time interval.
However, I do not ...
- 11
1
vote
1
answer
60
views
implementation of RoundRobin partitioning in Apache Flink
Hi I would like to implement a RoundRobin implementation for an operator in ApacheFlink, before I continue I would like to preface that I'm well aware that this already is implemented in Flink but ...
0
votes
1
answer
91
views
How do I store a stream of data that is scarcely changing in Apache Flink
Essentially, I have a Flink DataStream which reads from a Kafka topic which rarely changes.
This topic holds records like
userConsumerIdentifier
{
"user_id":1,
"consumer_id": 1
}
...
- 125
0
votes
1
answer
44
views
Sliding window pyflink
I am new to PyFlink and I have a kafka stream which has phone_number, host_name and event_time all in string formats. How can I compute number of visits for each pair phone_number, host_name, during ...
- 3
0
votes
0
answers
58
views
Best way to send huge data in streaming mode through Python flask application?
I want to create flask application which returns the huge data but the problems is, my api is crashing when it is ask for huge dataset like 6 million records. I want to make my flask application which ...
- 57
0
votes
0
answers
79
views
Compute average of past k-values in a data stream with limited memory
I am not looking for particular code, more of general theory of what could be done for the problem.
We have a stream of incoming data (data entries are quite large, thus it is infeasible to store more ...
1
vote
0
answers
57
views
how to export mysql's data to browser with saving memory in golang web application
sometimes maybe data is huge in mysql,so hope to save memory to export data to browser in go web application,I guess application/octet-stream is a good way but don't how to do it.
so how to select ...
- 11
0
votes
1
answer
21
views
How to dump clustering results in MOA
Please let me know after using data stream clustering algorithm from MOA like Clustream, how can I dump the clustering Result into csv file, which shows each row of arff data file belong to finally ...
- 1
0
votes
0
answers
65
views
Datastream not pulling change data from RDS Postgresql
I followed the on screen instruction to connect datastream from AWS RDS Postgresql to Bigquery.
Enabled logical replication, added
CREATE PUBLICATION [MY_PUBLICATION] FOR ALL TABLES;
SELECT ...
0
votes
0
answers
292
views
Problems with sink on flink
[enter image description here][1] I could do this to try first, this works well this just print the data on another topic I trie to follow the same logic but don get the result
package org.example;
...
- 3
0
votes
0
answers
24
views
How can i get the start and the end indexes of a drift while using StreamGenerator Class of the stream-learn module?
I have generated some data with drift (abrupt, gradual, recurrent, and incremental). I want to compare some drift detectors algorithm on each but for that, I need the specific starting and ending ...
