Mark tasks as critical

[mporracindie]

Description
The idea is to be able to mark specific tasks as critical making them define the output of the entire dag.

Use case / motivation
I'll use a specific example but this can be extended to many use cases. Let's say you use an operator to create an EMR cluster one for summiting code to the cluster and finally a cleanup one for killing the cluster when all the previous have finished.
The task submitting code to EMR is critical and I would want the whole DAG run to be marked as failed if any of them fails. However, this doesn't happen cause the cleanup always runs successfully. (which is expected since we want the cluster finished regardless of the output of the previous tasks)

With this example, I think I'm being clear. The end idea is to have more control over the dag run state when you have things like this:

Are you willing to submit a PR?
Yes

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[eladkal]

Not ideal but you can solve it by placing PythonOperator after the terminate task.
The operator can look for the main task status and set to fail/success accordingly.

[mporracindie]

Not ideal but you can solve it by placing PythonOperator after the terminate task.
The operator can look for the main task status and set to fail/success accordingly.

And that's how I'm solving it right now but personally, it seems like something that is adding tasks to my DAG in order to solve a problem that should be there in the first place. Being able to fully control the behavior of the DAG final status based on the task's executions.

[nathadfield]

With the release of Airflow 2.7 and the introduction of setup and teardown tasks, you now have much more control in situations such as this.

[potiuk]

With the release of Airflow 2.7 and the introduction of setup and teardown tasks, you now have much more control in situations such as this.

Good point @nathadfield . Marking the cleanup task as teardown, should solve the problem.