Scheduling and Triggering Data Pipelines in ADF

Pipelines in ADF are logical constructs that define a sequence of data transformation activities that are executed on the data stored in various data sources. Triggers are used to create configurations for running these pipelines on a specific schedule. In this lesson, we’ll discuss the scheduling and triggering options available in ADF.

Scheduling and triggering data pipelines in ADF

A pipeline can consist of various activities, including data transformation, data movement, and control flow activities. These activities can be executed in parallel or sequentially, depending on the requirement. Each activity in a pipeline performs a specific task, such as copying data from a source to a destination, transforming data, or running a custom code. Creating triggers for data pipelines ensures that these individual activities are performed in the desired sequence.

Scheduling data pipelines

ADF provides different scheduling options to run the data pipelines at regular intervals. The scheduling options available in ADF are as follows:

1. Tumbling window triggers run pipelines at fixed intervals, like every hour or every day. They have a predefined interval and execute on schedule, such as running a pipeline daily at 10:00 a.m.

2. Event-based triggers execute pipelines in response to specific events, such as file uploads or messages in a service bus queue. They trigger the pipeline when the event occurs.

3. Schedule triggers run pipelines at specified times and dates. Users can set up a schedule trigger to execute a pipeline on a particular day and time, such as the first day of every month at 12:00 p.m.

4. Tumbling window triggers with offset are similar to tumbling window triggers, but they allow customization of the start time and duration. They enable running pipelines at specific intervals with a customized start time and duration.

Triggering data pipelines

In addition to scheduling, triggering data pipelines can be done both manually using the Azure portal or programmatically using tools such as Azure CLI or PowerShell. Here, we’ll demonstrate how to run an existing pipeline using Azure CLI.

The az datafactory pipeline-run create command to start a pipeline run. The syntax for the command is as follows: