Questions tagged [azure-data-factory]
Microsoft Azure Data Factory is a cloud service used to invoke (orchestrate) other Azure or external services in a controlled way using the concept of time slices. ADF also provides means to transform data in a parallel way. Data factories are predominately developed using hand crafted JSON, this provides the tool with instructions on what activities to perform.
azure-data-factory
9,411
questions
-1
votes
0
answers
24
views
Choosing good approach to copy multiple tables in Azure Data Factory
I need to copy hundreds of tables (full or delta) from source to target using Azure Data Factory (ADF). I have two options:
Option A: 1 Pipeline per Table
Pros
Uses native ADF functionality.
...
0
votes
0
answers
15
views
IBMi Driver for Azure Data Factory - Integration Runtime Script
I've been trying to create a script to install a IBMi driver for Azure Data Factory Integration Runtime SSIS Lift and Shift
The driver is named ACS Windows App Pkg English (64bit)
https://www.ibm.com/...
0
votes
1
answer
35
views
How to connect, using terraform, the Azure Analysis Services Server (AASS) with an Azure SQL Database and Azure Data Factory (ADF)
Im planning to connect, using terraform, the azure analysis services server with an Azure SQL Database and Azure Data factory.
So far, the docs were not that useful, and I was wondering if you can ...
0
votes
1
answer
47
views
Unable to Execute Databricks Notebook from Azure Data Factory
When I attempt to execute a Databricks notebook from Azure Datafactory with a Base parameter 'entityName',see image, I get the error:
My Base parameters are as follows:
I also have the following ...
0
votes
1
answer
18
views
Azure Data Factory - Private Endpoint under Azure Portal vs Managed Private Endpoints
I have a quick question on the difference between the following network features in Azure Data Factory.
Managed Private Endpoints - I understood that this feature is used to establish private ...
0
votes
1
answer
40
views
How to redirect files to multiple folders in ADF
I have a folder in Datalake Gen 2 which contains multiple files. Now I want to redirect all the files into multiple folders based on that file name. I tried with IF and Switch conditon in ADF but that ...
0
votes
0
answers
22
views
"Detect Datetime" setting ignored when using Managed Identity for connecting to CosmosDB Source in ADF Copy activity
In several Azure Data Factory pipelines one of the activity is Copy Activity that uses Azure Cosmos DB for NoSQL as source and Azure Data Lake Storage Gen2 with JSON format as sink. The dates stored ...
0
votes
0
answers
23
views
Azure Data Fatory + Azure Batch and Python Script
I have been trying for a few days to create a pipeline to execute my python script, which connects to a Mysql Flexible Server database and stores some data.
I'm really new to this and it's my first ...
0
votes
1
answer
33
views
Azure Data Factory Pipeline Return Value to Parent Pipeline and Switch
I have the following ADF Child Pipeline that is executed from a Parent Pipeline via a Switch Activity.
And in the Set Row Count, I am setting a pipeline return value of:
@activity('Copy Delta to Blob'...
0
votes
1
answer
38
views
what is adf and data etl and ingetsion and more
Azure Data Factory (ADF) is a cloud-based data integration service provided by Microsoft Azure. It allows you to create, schedule, and manage data pipelines that can ingest data from various sources, ...
0
votes
1
answer
41
views
Iterate through an API until no records found through ADF
I have an ADF pipeline where I have an API call where it returns data in chunks (by 1000 max). This pipeline works if you only need 1000 users. I have been trying to figure a way to call the API again ...
1
vote
1
answer
38
views
how to add 'OR' condition in Wildcard file path in Synapse
I am trying to filter files containing str or containing stn.
if i type -str in the wildcard file path it returns me files containing str
and if type -stn, it returns me files containing stn
how can I ...
-1
votes
0
answers
24
views
How to extract SAP ECC data using ADF? [closed]
How to generate URL while creating linked service using sap ecc option .
0
votes
1
answer
45
views
How to implement incremental load on ADF
We have 2 folders in adls source and destination. source folder has 4 file and destination has 3. How can we implement the Increment load in ADF. What all activities we have to perform to achieve ...
1
vote
1
answer
59
views
Converting a flat file into JSON structure using ADF (Azure Data Factory)
I have a flat file which contains information such as names, addresses, mobile numbers, email addresses, etc. We want to convert this into a JSON-structured output. I have created a data flow which is ...