Key Concepts and Terminology

This lesson defines various terms that will be used throughout the course and are always used in conjunction with ADF offerings, such as pipelines, linked services, data sets, activities, triggers, control flow, transformations, data flow, and monitoring and management.

Key terminology

Below, we’ll cover concepts that are central to the functioning of Azure Data Factory:

1. Pipelines: Azure Data Factory is a cloud-based data integration service that allows users to create, schedule, and orchestrate data pipelines. In ADF, pipelines are used to define a sequence of activities or steps that are required to transform, process, and move data from one or more sources to one or more destinations. Pipelines consist of several components including linked services, data sets, activities, triggers, and control flow.

2. Linked services: Linked services define the connections to the various data stores that are involved in the data movement and transformation activities. This includes connections to on-premises data sources, cloud-based data sources, and third-party services.

3. Datasets: Datasets represent the logical data that will be processed by the activities in the pipeline. They contain metadata about the data, including its structure and location, and are used by the activities to determine the data to be processed.

4. Activities: Activities are tasks or operations that are executed on the data as it moves through the data integration pipeline. Activities can include a wide range of tasks such as data transformations, copying data from one location to another, or executing custom code using an Azure Function. Activities are the building blocks of data pipelines and can be arranged in a sequence or parallel to create a complete data integration workflow.

ADF provides a wide range of pre-built activities that can be used out of the box to build a data pipeline. Additionally, users can create custom activities using Azure Functions or Azure BatchAzure Batch is a compute management platform from Azure that allows for large-scale parallel batch workloads to be run in the cloud. to execute specific tasks. Activities can be executed on-demand or can be scheduled to run on a specific interval or triggered by an event using Triggers. ADF also provides monitoring and management features to track the status of each activity and troubleshoot any issues that may arise during pipeline execution.