Questions tagged [etl]
ETL is an acronym for Extract, Transform, and Load. It refers to a process of extracting data from source systems, transforming the data in some way (manipulating it, filtering it, combining it with other sources), and finally loading the transformed data to target system(s).
5,958
questions
0
votes
0
answers
9
views
Not able to create job in dataflow for streaming data
I am executing my Apache-beam code in google cloud shell, I am able to execute code without errors, but jobs not creating in data flow.
**below roles I assigned to service account
**
Dataflow Worker, ...
- 13
-1
votes
0
answers
28
views
Choosing good approach to copy multiple tables in Azure Data Factory
I need to copy hundreds of tables (full or delta) from source to target using Azure Data Factory (ADF). I have two options:
Option A: 1 Pipeline per Table
Pros
Uses native ADF functionality.
...
- 1,589
0
votes
0
answers
37
views
Using Powershell to data drop a csv file from a WebEx API to SQL Server: Exception calling "WriteToServer" with "1" argument(s):
I am trying to drop WebEx meeting data to my company's SQL server. After running line by line, the csv data and datatable are loaded. However when I run the bulk copy method, it returns this error.
...
0
votes
0
answers
14
views
Dynamically pass file name to FTP task in Realisable Iman
I want to generate a dynamic file name and pass it to the 'Remote file' fied of an FTP task in Realisable Iman. I have used a script task (Vb) to generate the file name but I can't find away to pass ...
0
votes
1
answer
33
views
Extracting json array in Postgres
My Postgres database contains a list of json objects as records. I am trying to extract an array from the record, and Postgres does not seem to like what I'm proposing.
Here's an example of a record.
...
- 31
0
votes
1
answer
54
views
How I can do this data transformation in Azure Data Factory? [closed]
So I have this table in a SQL Server Database as a source table:
Customers
Id
Name
Address
City
State
Zip
BillingAddress
BillingCity
BillingState
BillingZip
1
Customer A
123 Ocean Drive
Miami
FL
...
- 309
2
votes
1
answer
60
views
Whats is a good way to add a column to an existing sql database table using spark?
I have an existing Postgres SQL table with some features.
I want to use spark to :
Read that table
Create some additional columns
Add those columns to the table.
Is there any way to make spark add ...
- 412
-1
votes
1
answer
29
views
How to import only the new data and new records that have changed using the ETL process in a data warehouse?
I have an ETL process that allows me to load data from one database to another, applying transformations along the way. The process currently starts by deleting all records from all tables, and then ...
0
votes
1
answer
51
views
Unable to use both SQL Server and Postgres connection together on the same job - Talend
For a test, I created this simple Talend job:
The tRowGenerator generates a row with int column and is staged to temporary database in Postgres.
The issue occurs whenever I run the job. The ...
- 2,982
1
vote
0
answers
25
views
Error parsing CSV File when copying Data to Snowflake after July 3rd Incident
I am encountering an error while trying to copy a CSV file into Snowflake from an S3 bucket. This process was functioning correctly until an incident occurred in Snowflake on July 3rd. The error ...
- 43
-1
votes
0
answers
13
views
SQL: How to convert iCal event blob to start and end timestamp
We have a production service that schedules events (both ad-hoc and recurring) via the iCal format. We then replicate these events to a Snowflake data warehouse as the event "blob" itself. ...
- 185
0
votes
1
answer
34
views
Drop message in function execution time - Spark
I'm trying to run some functions for my ETL pipelines and log them in the process, the issue is that when I call my function my log message is instantly shown, I want to display it during the ...
0
votes
0
answers
28
views
ETL design over an existing DDD aggregate
I hope you can help me with the design of a data ingestion process.
Currently, I have an existing aggregate called ExperiencePricing and an existing command called SetExperiencePricingCommand. The ...
- 966
0
votes
0
answers
35
views
AWS Glue - o109.pyWriteDynamicFrame. ERROR: relation "xyz" already exists
I have a really simple aws glue visual etl which reads data from a file on an s3 bucket, and then copies it into an aws rds postgresql db.
The script for that:
import sys
from awsglue.transforms ...
0
votes
1
answer
55
views
AWS MWAA -- dags not appearing in Airflow UI
I have created an AWS MWAA environment and correctly pointed it to an S3 bucket as well as an "dags" folder inside the S3 bucket.
I have created a simple dag file (python) and added it to my ...
- 21