0

I have 3 note book created for different data operation . I want to orchestrate the notebook execution as per below flow

Execute Notebook 1

Execute Notebook2

Execute Notebook3

Based on certain flag in Notebook1, either Notebook2 or Notebook3 should be executed.

i found below example from the document using DAG

with DAG("mydag", schedule=timedelta(days=1), warehouse="my_warehouse_name") as dag:
    dag_task1 = DAGTask("LOAD_EXCEL_FILES_TASK", definition=f'''EXECUTE NOTEBOOK "{database_name}"."{schema_name}"."{env}_06_load_excel_files"()''', warehouse=warehouse_name)
    dag_task2 = DAGTask("LOAD_DAILY_CITY_METRICS", definition=f'''EXECUTE NOTEBOOK "{database_name}"."{schema_name}"."{env}_07_load_daily_city_metrics"()''', warehouse=warehouse_name)

    # Define the dependencies between the tasks
    dag_task1 >> dag_task2 # dag_task1 is a predecessor of dag_task2

# Create the DAG in Snowflake
dag_op.deploy(dag, mode="orreplace")

but when i execute with above example, it throwing syntax error

Unexpected '='

i also want to know if there any option run this notebook job parallel..

i mean, reducing number of task. so how to achieve such scenario within Snowflake?

6
  • Have you tried simply using tasks and executing the appropriate notebook via each task? You can then make the task dependencies. Commented Jul 7 at 15:27
  • Hi @MikeWalton so you mean there is no such option within notebook to orchestrate the process without using Stored procedure and then task
    – hari_azure
    Commented Jul 8 at 4:36
  • You don't need a stored procedure. EXECUTE NOTEBOOK is a direct command, and you could create a task for each notebook and then make those tasks dependent on each other in a series. Commented Jul 9 at 1:02
  • @MikeWalton .. Ok .. do you have any sample code to showcase about this?
    – hari_azure
    Commented Jul 10 at 6:01
  • Sample code to show a task executing a command or sample
    – NickW
    Commented Jul 11 at 11:25

0