Questions tagged [airflow]
Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks.
airflow
10,714
questions
0
votes
0
answers
6
views
Can't expose Airflow UI with Classic Load Balancer
I have an airflow helm release running on EKS cluster
NAME READY STATUS RESTARTS AGE LABELS
pod/airflow-postgresql-0 1/1 Running ...
- 95
0
votes
0
answers
9
views
What is the configuration for Apache Airflow, Django, PySpark together in Systemd services?
We are using Apache airflow, Django (python), and Spark (pySpark) in our project. Our DAGs are running fine when run 'airflow scheduler' is run from command line. However, these DAGs are not working ...
0
votes
0
answers
14
views
Loading Joblib model in Airflow dag
I'm trying to use joblib inside an airflow dag, when loading a model file I run joblib.load(model_name) but I get an error
File "/usr/local/lib/python3.9/pickle.py", line 331, in ...
- 7,319
0
votes
0
answers
8
views
Apache Airflow service start in wrong folder
When I start the following airflow service, it create a new folder in /home/airflow/airflow/ instead of using the configuration from /home/airflow/ and initialize a new DB.
Description=Airflow ...
- 318
0
votes
0
answers
16
views
Airflow metrics not showing in datadog
I don't see any logs coming to the datadog for airflow, I enabled the logs and even added the DD_DOGSTATSD_MAPPER_PROFILES in the helm values for agent. I am looking to send the job logs aswell, which ...
- 29
0
votes
1
answer
9
views
Retrieve result status from AzureContainerInstancesOperator into XCOM in Airflow
I'd like to have an Airflow DAG get some status information back after running an AzureContainerInstancesOperator task in Airflow. With the equivalent docker operator, DockerOperator, it is possible ...
- 3,086
2
votes
0
answers
26
views
Airflow pods wil only run on one k8s node (probably not an affinity issue)
I am running Airflow (deployed via Helm chart) using the KubernetesExecutor on a microk8s cluster that's also runnig Ceph.
We have 4 nodes. A control node and three identical data/processing nodes.
I ...
- 21
0
votes
0
answers
18
views
Airflow is triggering concurrent lambda runs
I have two AWS lambda functions that run in docker containers that I trigger using Airflow, one at a time. I currently am triggering the data pipeline using my local MWAA for testing purposes. I want ...
- 348
0
votes
1
answer
32
views
How to send Airflow Failed Notifications to Microsoft Teams using Microsoft 365 Worfklow
I've got an Airflow function who send failed notification task in Teams group. With new Microsoft politics my old webhook call will not function anymore in few months so i want to work right now to ...
- 35
0
votes
0
answers
15
views
Issue with Airflow Variables Adding Extra Backslashes to Private Key
I'm encountering a peculiar issue with Airflow. I've stored a private key as a variable in Airflow, but it seems to be adding an extra backslash (\) to newline characters (\n).
Here's an example of ...
- 155
0
votes
1
answer
16
views
How to retrive which DAG has updated a Composer Airflow Dataset
Regarding Google Cloud Composer, I have defined a DAG in this way:
dataset = Dataset("//my_Dataset")
dag = DAG(
dag_id='my_dag',
default_args=default_args,
schedule=[dataset],
catchup=...
- 460
0
votes
0
answers
31
views
Deploying "Variables" in Cloud Composer via Terraform
I have set up a minimal version of a Composer environment in GCP. Through Terraform, I have deployed this thus far:
resource "google_composer_environment" "airflow-poc-env" {
...
- 1
0
votes
1
answer
14
views
Airflow run same task with pause for 24 hours
Is there a way to pause a task for 24 hours then run the same task again?
I want to write a dag that pull data from api, but the api have limit of data I can pull in a day so I need to wait for 24 ...
0
votes
0
answers
14
views
Airflow error while creating EMR cluster via DAG
I am looking to create an EMR cluster via airflow DAG using EmrCreateJobFlowOperator using a role called dev-emr-ec2-profile-role for jobFlow. This role is used to provision EMR cluster via Terraform ...
- 1
0
votes
0
answers
89
views
KubernetesPodOperator don't see PersistentVolume and PersistentVolumeClaim
I have an issue with KubernetesPodOperator.
Maybe someone could give me an advice or know the answer.
I'm implementing Liquibase tool on my project. And we are going to trigger Liquibase from Airflow ...
