Azure Data Factory Activity to Stop a Trigger

In real life projects there are scenarios where ETL pipelines scheduled, for example each hour, process data in a given hour, taking into account also data previously processed in other time-slots. If those previous data have been lost or not processed, the consequence is inconsistency.

To avoid inconsistency it’s possible first of all to block automatically the Data Factory trigger that runs all the time-slices. The idea is that when an Activity fails, another pipeline is automatically triggered to block the scheduler. To do that it’s important to have a Service Principal and implement its authentication within Data Factory. (In this previous article I’ve shown how to achieve this)

Once the login web activity has succeeded it is possible to take the access_token output and use it in another web activity.
The TriggerBlocker web activity makes a POST call exploiting that access_token in the Authorization header.

The URL is built exploiting the Data Factory v2 API:
“https://management.azure.com/subscriptions/<SubscriptionId>/
resourceGroups/<ResourceGroupName>/providers/Microsoft.DataFactory/
factories/<ADF_Name>/triggers/<TriggerName>/
stop?api-version=2018-06-01″

The Content-Type header value is:
Application/json

The Autorization header value is:
Bearer @{activity(‘login‘).output.access_token}

The Body is simply:
{}

Below you can see a simple example on how to use the pipeline that blocks the trigger. The databricks notebook service is called in the first activity. If it fails, then the Execute Pipeline activity is run invoking the trigger2block pipeline explained before.

At this point the Data Factory Trigger is blocked preventing the following time-slices to run. Of course that’s an important step to avoid inconsistency, so that we are sure that as soon as the risk arises, it won’t become an issue. It’s important by the way to implement a recovery strategy to process the data not processed after the failure (this task totally depends on the specific scenario taken into consideration and the ETL process logic)