Questions tagged [google-datastream]
Use this tag when facing Google Datastream issues or questions like: set up, running, viewing logs, handle error, recovery, working with datastream public API
google-datastream
66
questions
0
votes
1
answer
49
views
Reducing GCP DataStream Sync Latency from PostgreSQL to BigQuery
I'm using GCP DataStream to sync data from PostgreSQL to BigQuery. However, I've noticed a delay of just over ten minutes in the data sync process. After digging into the documentation, I found that ...
- 2,927
0
votes
1
answer
21
views
Datastream validation issues 'One or more tables in the stream's include list weren't found or Datastream couldn't read from them.'
I have set up a google datastream postgres to bigquery. My publication, replication slot area all created as well as my replication user, also my replication user has been assigned select access and ...
- 175
0
votes
0
answers
41
views
Datastream using private connectivity
You can help me, please?
I have try to create the reverse proxy to make connection from Datastream VPC to CloudSQL.
The VM is close connection with CloudSQL database, but when i run the "Test ...
0
votes
0
answers
24
views
Big Query export not able to exclude Events
I have configured a BQ export to get app_remove event and exclude all.
But the firebase_campaign event is not getting filtered to be excluded. I have even excluded the same by specifying the event by ...
0
votes
1
answer
116
views
Datastream - AWS to GCP
I been trying to connect two cloud providers AWS and GCP and found this documentation
Create an HA VPN gateway to a peer VPN gateway
I already config and set it up. And the result was good, ...
- 3
1
vote
0
answers
34
views
Source tables getting deselected in Data Stream
I have a Datastream enabled which replicates data from CloudSQL ( PostgreSQL ) to BigQuery. After the stream is created with automatic backfill, it runs for few hours and then the data replication ...
0
votes
0
answers
48
views
Google DataStream not connecting to Google CloudSQL instance
I am setting up Google DataStream and connecting to a Google CloudSQL database. The DataStream requires IP address, port, username and pw to connect to the db. I am supplying the correct information. ...
0
votes
0
answers
31
views
Datastream unstable connection when connected with private site-to-site connection to RDS
We have set up a private site-to-site connection between our Google Cloud project and AWS RDS (MySQL), which serves as our main database. The connection is established through datastream, allowing us ...
- 611
0
votes
1
answer
147
views
How to remove Datastream UUID from BigQuery tables during MySQL CDC replication?
I'm using Datastream to replicate data from MySQL to BigQuery, but I want to exclude the Datastream UUID column that's added to my BigQuery tables. However, I still need to maintain CDC (Change Data ...
0
votes
1
answer
135
views
GCP Datastream doesn't replicate new tables from RDS Postgres
I have a datastream that replicate RDS postgres to Bigquery dataset.
Everything is set correctly, the datastream works. But we add new tables in RDS postgres and we don't see these new tables neither ...
- 26
0
votes
0
answers
38
views
Validation stream workaround through UI
I'm using Google Cloud Datastream to replicate some tables from Postgresql into BigQuery. Setting up the source connection and destination worked well and passed the validations, but when I almost end ...
- 11
1
vote
0
answers
108
views
GCP Dataflow Changing Date Format Causing Errors/Retries
Replicating a mysql db using cloud_datastream_to_bigquery template in dataflow. So mysql->datastream->gcs->dataflow->bigquery
I'm seeing this in my DLQ/retry
"error_message":{&...
- 11
0
votes
0
answers
47
views
Datastream datatype changes
I have a Datastream job running that is connected to my-sql instance and writing events to gcs bucket .
Now the source my sql server has datatype change in of its column .. how can we handle the ...
0
votes
0
answers
23
views
Limiting specific schemas in BigQuery Datastream
We want to use Datastream to extract data only from some customer databases contained in an RDS instance with multiple schemas, since the company does not allow sharing data from all of them as they ...
- 1
1
vote
1
answer
453
views
Error when creating materialized view on datastream enabled Bigquery tables
We have tables in bigquery that are populated using datastream from postgres. We want to create materialized views on top of these bigquery tables but we run into the following error
Materialized ...
- 85