Questions tagged [directed-acyclic-graphs]
Directed acyclic graphs appear in many data structures such a changeset graphs in distributed version control systems.
directed-acyclic-graphs
1,451
questions
0
votes
1
answer
16
views
How to retrive which DAG has updated a Composer Airflow Dataset
Regarding Google Cloud Composer, I have defined a DAG in this way:
dataset = Dataset("//my_Dataset")
dag = DAG(
dag_id='my_dag',
default_args=default_args,
schedule=[dataset],
catchup=...
- 460
0
votes
0
answers
18
views
LLVM DAG Pattern Instruction Selection Fails after introducing a new intrinsic
I am currently expanding a RISC-V core with a custom load instruction. For this question, the specifics are unimportant. It functionally behaves like a normal load instruction and should use the same ...
0
votes
1
answer
36
views
Apache airflow: No module name 'airflow'. Task failed
I installed Airflow by docker according to this video. (Windows+WSL+Ubuntu+Docker)
I have my_dag.py file (replaced APIKEY):
from airflow import DAG
from airflow.operators.python import PythonOperator
...
- 163
0
votes
1
answer
14
views
How can I include a project's folder structure in my dbt Cloud project's DAG?
I'm looking for a way to include folder structure and other available metadata into my dbt Cloud project's DAG. While the lineage is clearly demonstrated in the DAG generated, I'm looking for a way to ...
0
votes
0
answers
37
views
Airflow DAG Import Error Avro - installed everywhere
import avro.schema
ModuleNotFoundError: No module named 'avro'
I have this error when I try to import a DAG in Airflow.
I have avro installed in the docker image as shown below:
Installed /home/...
- 3
0
votes
0
answers
43
views
Is it possible to orchestrate snowflake Notebook Job?
I have 3 note book created for different data operation . I want to orchestrate the notebook execution as per below flow
Execute Notebook 1
Execute Notebook2
Execute Notebook3
Based on certain flag in ...
- 13
0
votes
0
answers
9
views
How to have runtime visualization of code symbols
Is there a way to generate a graph view of a run, similar to stacktrace but just have it as a graph view.
Stack traces could be visualized as graphs by having the file source as the node and then ...
- 104
1
vote
2
answers
36
views
Clear and automatic DAG visualization
I am trying to draw a DAG (directed acyclic graph) starting from a dataframe with 212 attributes.
I am able to draw it using graphviz
import graphviz
G = graphviz.Digraph()
for col in graph_df....
- 78
1
vote
0
answers
35
views
Airflow script to schedule dag for every hour has loaded data for next 5 days instead of live updates every hour
[![Postgres][1]][1]
[![Airflow][2]][2]
I created an Airflow script that pulls info from Open weather api and loads into Postgres database locally.The Dag is running successfully and executes every ...
- 93
0
votes
1
answer
125
views
How to execute a DAG of tasks using async.io?
I have written the following code:
`
async def execute_task(self, task_id):
await self.tasks[task_id]()
self.task_status[task_id] = "done"
async with self.lock:
...
2
votes
0
answers
53
views
LOD algorithm nanite's style: lod selection issue
I'm implementing a LOD algorithm nanite's style. So, the selection is for group of meshlets. I use the meshoptimizer lib to perform the simplification and meshlets creation and the METIS lib to group ...
- 41
0
votes
0
answers
34
views
Topological Sorting question. Does the values of an edge impact the order in which they are sorted?
I am currently working on the "Longest Road" Dag problem, and finished writing my topological sorting algorithm in Java. Currently I am using edges with no value associated to them, and I am ...
0
votes
1
answer
47
views
Airflow sensor task, run for 3 days straight waiting for file to get added to GCS bucket
I have a DAG that currently checks a GCS bucket for a specific file that comes once a month. The problem I have is that his file will come on the 6th "Working day" of the month, which could ...
- 169
0
votes
0
answers
12
views
Traverse a graph to minimize the amount of nodes that are alive
A node becomes alive when it is visited and can be killed when all of it's children have been visited. A child can only be visited if all of it's parents have been visited For example for this graph
...
- 566
0
votes
0
answers
26
views
Airflow takes a random python file as a DAG and breaks it
I have a pipeline forlder with this structure:
project forlder 1
|--project forlder 2
|-- DAGS
|--dag1
|--dag2
|--utils
|-- __init__.py
|--python folder
...
- 55