Questions tagged [azure-data-factory]
Microsoft Azure Data Factory is a cloud service used to invoke (orchestrate) other Azure or external services in a controlled way using the concept of time slices. ADF also provides means to transform data in a parallel way. Data factories are predominately developed using hand crafted JSON, this provides the tool with instructions on what activities to perform.
azure-data-factory
9,412
questions
0
votes
0
answers
11
views
Error Loading ORC Files into BigQuery: "Invalid timestamp value" after 17th July 2024
I've been successfully fetching ORC files from an Azure Blob Storage to a GCS bucket using the Storage Transfer Service. This process handles around 100 files. After loading the files into GCS, I use ...
- 37
0
votes
1
answer
24
views
Azure Data Factory: Property selection is not supported on values of type 'Integer'
I am trying to implement a load that is split into batches.
I have used the following manual:
https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/extracting-sap-data-using-odata-part-4-...
- 145
-1
votes
1
answer
51
views
Choosing good approach to copy multiple tables in Azure Data Factory
I need to copy hundreds of tables (full or delta) from source to target using Azure Data Factory (ADF). I have two options:
Option A: 1 Pipeline per Table
Pros
Uses native ADF functionality.
...
- 1,589
0
votes
0
answers
16
views
IBMi Driver for Azure Data Factory - Integration Runtime Script
I've been trying to create a script to install a IBMi driver for Azure Data Factory Integration Runtime SSIS Lift and Shift
The driver is named ACS Windows App Pkg English (64bit)
https://www.ibm.com/...
- 1
0
votes
1
answer
45
views
How to connect, using terraform, the Azure Analysis Services Server (AASS) with an Azure SQL Database and Azure Data Factory (ADF)
Im planning to connect, using terraform, the azure analysis services server with an Azure SQL Database and Azure Data factory.
So far, the docs were not that useful, and I was wondering if you can ...
- 498
0
votes
1
answer
49
views
Unable to Execute Databricks Notebook from Azure Data Factory
When I attempt to execute a Databricks notebook from Azure Datafactory with a Base parameter 'entityName',see image, I get the error:
My Base parameters are as follows:
I also have the following ...
- 2,603
0
votes
1
answer
18
views
Azure Data Factory - Private Endpoint under Azure Portal vs Managed Private Endpoints
I have a quick question on the difference between the following network features in Azure Data Factory.
Managed Private Endpoints - I understood that this feature is used to establish private ...
- 977
0
votes
1
answer
42
views
How to redirect files to multiple folders in ADF
I have a folder in Datalake Gen 2 which contains multiple files. Now I want to redirect all the files into multiple folders based on that file name. I tried with IF and Switch conditon in ADF but that ...
0
votes
0
answers
24
views
"Detect Datetime" setting ignored when using Managed Identity for connecting to CosmosDB Source in ADF Copy activity
In several Azure Data Factory pipelines one of the activity is Copy Activity that uses Azure Cosmos DB for NoSQL as source and Azure Data Lake Storage Gen2 with JSON format as sink. The dates stored ...
- 31.2k
0
votes
0
answers
25
views
Azure Data Fatory + Azure Batch and Python Script
I have been trying for a few days to create a pipeline to execute my python script, which connects to a Mysql Flexible Server database and stores some data.
I'm really new to this and it's my first ...
0
votes
1
answer
37
views
Azure Data Factory Pipeline Return Value to Parent Pipeline and Switch
I have the following ADF Child Pipeline that is executed from a Parent Pipeline via a Switch Activity.
And in the Set Row Count, I am setting a pipeline return value of:
@activity('Copy Delta to Blob'...
- 565
0
votes
1
answer
42
views
Iterate through an API until no records found through ADF
I have an ADF pipeline where I have an API call where it returns data in chunks (by 1000 max). This pipeline works if you only need 1000 users. I have been trying to figure a way to call the API again ...
- 35
1
vote
1
answer
39
views
how to add 'OR' condition in Wildcard file path in Synapse
I am trying to filter files containing str or containing stn.
if i type -str in the wildcard file path it returns me files containing str
and if type -stn, it returns me files containing stn
how can I ...
- 15
-1
votes
0
answers
24
views
How to extract SAP ECC data using ADF? [closed]
How to generate URL while creating linked service using sap ecc option .
- 39
0
votes
1
answer
46
views
How to implement incremental load on ADF
We have 2 folders in adls source and destination. source folder has 4 file and destination has 3. How can we implement the Increment load in ADF. What all activities we have to perform to achieve ...
- 343