I have 3 note book created for different data operation . I want to orchestrate the notebook execution as per below flow
Execute Notebook 1
Execute Notebook2
Execute Notebook3
Based on certain flag in Notebook1, either Notebook2 or Notebook3 should be executed.
i found below example from the document using DAG
with DAG("mydag", schedule=timedelta(days=1), warehouse="my_warehouse_name") as dag:
dag_task1 = DAGTask("LOAD_EXCEL_FILES_TASK", definition=f'''EXECUTE NOTEBOOK "{database_name}"."{schema_name}"."{env}_06_load_excel_files"()''', warehouse=warehouse_name)
dag_task2 = DAGTask("LOAD_DAILY_CITY_METRICS", definition=f'''EXECUTE NOTEBOOK "{database_name}"."{schema_name}"."{env}_07_load_daily_city_metrics"()''', warehouse=warehouse_name)
# Define the dependencies between the tasks
dag_task1 >> dag_task2 # dag_task1 is a predecessor of dag_task2
# Create the DAG in Snowflake
dag_op.deploy(dag, mode="orreplace")
but when i execute with above example, it throwing syntax error
Unexpected '='
i also want to know if there any option run this notebook job parallel..
i mean, reducing number of task. so how to achieve such scenario within Snowflake?
EXECUTE NOTEBOOK
is a direct command, and you could create a task for each notebook and then make those tasks dependent on each other in a series.