What is AWS Step Functions? How it works and use cases

Imagine you’re building a warehouse robot that sorts packages based on size and destination. Its arms pick up items, its camera identifies labels, and its wheels transport packages to the correct conveyor belts. For the robot to perform these tasks efficiently, all its parts must work together seamlessly, following a well-coordinated plan. But how do you ensure that every action happens in the right order at the right time?

AWS Step Functions serves as the coordinator for orchestrating complex workflows like this. While it doesn’t perform the tasks directly, it ensures that every component—such as Lambda functions, databases, and messaging services—works harmoniously. For developers, mastering Step Functions is essential for building scalable and reliable systems, simplifying distributed application management, and automating processes. This blog will explore how AWS Step Functions works and discover its powerful use cases in real-world scenarios.

What is AWS Step Functions?#

AWS Step Functions is a service that allows developers to build distributed applications, automate processes, orchestrate microservices, and create machine learning (ML) pipelines. It can be integrated with other Amazon services, such as Lambda and SNS. Using Step Functions, we can create workflows—known as state machines—where each state can be integrated with a service.

State machines#

A state machine is a series of event-driven steps. Each step in a state machine is known as a state. A state can make decisions based on the input it receives, process it, and pass an output to other states. Following are the states supported by AWS Step Functions.

Control states#

These states control the flow of the state machine without performing specific actions:

• Pass: This state passes the input it receives as an output without performing any action on it.

• Wait: This state is used to add delay into our state machine.

• Succeed: This state successfully stops the state machine. It is usually used with conditional states where we have nothing to do if a condition is met.

• Fail: This state ends the state machine in failure.

Action states#

These states perform tasks or handle data dynamically:

• Task: This state performs actions using AWS services, such as Lambda functions or other third-party services.

• Choice: This state introduces conditional logic into the state machine, enabling it to make decisions based on input values. For example, this state can determine whether an object uploaded to an S3 bucket should be routed to a Lambda function for image resizing or object detection or another for document text extraction and analysis.

• Parallel: This state adds multiple branches to our state machine, which can be executed in parallel. For example, this state can simultaneously process different files in an S3 bucket to save time and effort.

• Map: In this state, a set of workflow states is run in parallel for each item in a dataset. This dataset can be a JSON array, a list of objects in Amazon S3, or a CSV file. For example, this state can be used to analyze customer feedback by iterating through each review in a list and using a sentiment analysis pipeline for each one.

Workflows in Step Functions#

State machines are called workflows in AWS Step Functions, which support two types of workflows: standard and express.

• In a standard workflow, each step is executed exactly once (unless retrying is specified in the ASL), and these workflows can run for up to one year. They are ideal for long-running auditable workflows.

• In an express workflow, each step is executed at least once, and there is a chance some steps are executed more than once. These workflows can run up to five minutes and are used in applications where high-event-rate workloads are required, such as streaming data processing applications. The express workflow can be both asynchronous and synchronous.

Note: We cannot change the type of the workflow after it is created.

Use case for Step Functions#

AWS Step Functions allow us to build applications by converting their components into individual steps. Each step can invoke a Lambda function, update our database, or perform other actions according to our requirements. Let’s look at some of the use cases of step functions:

1. Processing SQS messages with Step Functions#

Imagine you are managing a book club, and your readers frequently provide feedback about their reading experiences. You want to automate the feedback analysis process to determine whether the sentiment is positive, neutral, or negative. This can help you quickly gauge customer satisfaction and respond to the feedback without manually sifting through countless messages.

By combining Amazon EC2, SQS, SNS, and AWS Step Functions, we can build an automated, scalable system to handle this feedback analysis efficiently.

An overview of workflow:#

1. Setting up an EC2 instance: We’ll launch an EC2 instance to host a web application where readers can submit feedback about the books they read. This application will expose a simple REST API, allowing users to enter feedback comments directly from the interface.

2. Submitting feedback to SQS: Once a user submits feedback through the REST API, the feedback message is sent to an Amazon SQS queue. SQS acts as a buffer, storing feedback messages and decoupling the submission process from the analysis workflow. This ensures that your system can handle fluctuating workloads without overloading downstream services.

3. Processing with AWS Step Functions: AWS Step Functions then takes over the workflow. It picks up messages from the SQS queue and begins processing each one. This state machine is designed to:

• Fetch a feedback message from SQS.

• Pass the message to an AWS Lambda function, which uses Amazon Comprehend, AWS’s natural language processing service, to detect the feedback’s sentiment (positive, negative, neutral, or mixed).

• Store the sentiment detected in your database (such as DynamoDB), and if the feedback is negative, send an SNS notification to take the appropriate steps to address the issue the user faced.

The following diagram gives a high-level overview of the steps performed by Step Functions:

2. Document processing pipeline#

Step Functions has a variety of application patterns and features that can be very useful for orchestrating Lambda functions. Some of these application patterns include parallelismIn this pattern, we run multiple functions in parallel., chainingIn this pattern, the output of one function is used as the input in another function., and branchingIn this pattern, a workflow is selected based on the output/results of other fuctions.. An example of using these application patterns would be creating a document processing pipeline where we see if the document contains any inappropriate text or images using AWS Step Functions. A Step Function state machine is triggered in this pipeline when a PDF file is uploaded in an S3 bucket. The following steps are performed in this state machine:

An overview of workflow:#

1. Extract images and text: We’ll first use parallel processing to extract images and text from the PDF file uploaded to the S3 bucket in parallel. We’ll use two Lambda functions. One extracts images from the document, and another extracts the text.

2. Parallel processing of images: In the image processing pipeline, the function to extract images from the PDF file returns the number of images in the file in a JSON array. To check if the image is offensive, we can use the count to implement dynamic parallelism, using the Map state of the state machine, which can run a flexible number of parallel processes based on the number of items in a JSON array.

3. Choice state: Next, we’ll use the concept of branching to create two branches of action depending on whether the image is offensive. If the image is offensive, we’ll email the user to notify them using Amazon SNS. The workflow will simply end if we don’t find an offensive image.

4. Examine the text: In the text processing pipeline, we’ll use chaining to provide the extracted text to a Lambda function that will analyze the text to ensure it isn’t offensive. If the text is offensive, we’ll email the user to notify them using Amazon SNS. The workflow will simply end if the text is found to be non-offensive.

Following is a high-level overview of the steps given above:

Best practices #

The following practices can help us optimize the implementation of Step Functions:

• Time-outs: By default, the AWS Step Function relies on the output it receives to move ahead. However, providing an explicit timeout in our states is better to avoid getting stuck if the output is not generated due to an erroneous condition.

• Using S3 buckets: Rather than providing large data payloads to our state machine, it is advised to upload data into S3 buckets and then use them in the state machine.

• Lambda exceptions: It is suggested to properly handle the errors faced by Lambda functions, such as execution timeout errors, by providing a retry policy or a catch block in the state machine.

Final word#

AWS Step Functions is a powerful tool for building distributed applications, automating processes, orchestrating microservices, and creating machine learning (ML) pipelines. By defining state machines, developers can automate complex processes, handle parallel execution, incorporate conditional logic, and manage long-running workflows efficiently.

Educative offers a wide range of Cloud Labs with access to the AWS console and step-by-step instructions that will help you get hands-on experience building solutions using AWS Step Functions.