type

project_id

private_key_id

private_key

client_email

client_id

auth_uri

token_uri

auth_provider_x509_cert_url

client_x509_cert_url

kaggle_username

kaggle_key

Gain insights into building scalable data and model pipelines, explore different cloud environments, delve into streaming workflows, and discover essential tools for creating real-time data products.

dsp-new.tar.gz

PROJECT_ID

aws_access_key_id

aws_secret_access_key

aws_account_id

jupyter-hello

code_widget_advanced

spa_job

spa_job_model_persistence

spa_job_gcloud

spa_job_gcloud_authen

spa_gcloudAuthen_sklearnWorkflows

spa_job_apacheKafka

spa_job_jupyter_gcloudAuth_pubsub

code_widget_basic

spa_job_aws

spa_job_gcloudAuth_GCP_Kubernetes

spa_job_jupyter

spa_job_gcloud_authen_model

spa_job_aws_model

spa_job_aws_echo_service

spa_job_Airflow

spa_job_aws_kaggle_nhl_data

spa_job_kaggle_nhl_data_jupyter

airflow_liveApp

spa_job_heroku

test-athar

spa_job_heroku-copy

core-jupyter

core-jupyter-copy

spa-jupyter

jupyter-hello-spa

osama

sklearn_workflow_job

spa_job_model_endpoint_keras

spa_job_aws-copy

spa_job_aws_model-copy

spa_job_aws_echo_service-copy

spa_job_aws_kaggle_nhl_data-copy

The goal of this course is to provide you with a set of tools that can be used to build predictive model services for product teams.

In this course, you’ll start by covering the different cloud environments and tools for building scalable data and model pipelines. You’ll then learn the different data sets and types of models that will be used heavily in everyday production. Throughout the course, you’ll have plenty of exercises and challenges to get you comfortable working with the diverse toolset.

Lastly, you’ll explore streaming model workflows which is crucial for building real-time data pipelines that move data between different components in a cloud environment. 


After working through this course, you will have gained valuable hands-on experience with many of the tools needed to build data products. You will also have a better understanding of how to build scalable machine learning pipelines in a cloud environment.

Data Science in Production: Building Scalable Model Pipelines

![](/api/collection/10370001/6068402050301952/page/6012545497300992/image/5323616330711040)

PubSub is a fully-managed streaming platform available on GCP. It provides similar functionality to Kafka for achieving high throughput and low latency when handling large volumes of messages. However, it reduces the amount of DevOps work needed to maintain the pipeline. One of the benefits of PubSub is that the APIs map well to common use cases for streaming Dataflow pipelines.

## Setting up PubSub
Unlike Kafka, PubSub uses separate concepts for producer and consumer data sources. In Kafka, you can publish and subscribe to a topic directly, while in PubSub consumers subscribe to subscriptions rather than directly subscribing to topics. With PubSub, you first set up a topic and then create one or more subscriptions that listen to this topic.

### Creating a topic
To create a topic with PubSub, perform the following steps:
1. Browse to the PubSub UI in the GCP Console.
2. Click "Create Topic".
3. Enter "natality" for the topic ID, and click "Create Topic". 

The result of performing these actions is that we now have a topic called `natality` that we can use for publishing messages. 

### Creating a subscription
Next, we'll create a subscription that listens for messages on this topic by performing these steps: 
1. Click on "Subscriptions" in the navigation pane.
2. Click "Create Subscription".
3. Assign a subscription ID "dsp".
4. Select the "natality" topic. 
5. Click "Create". 

We now have a topic and subscription set up for streaming messages in a pipeline. Before setting up a Dataflow pipeline, we'll first create message consumers and producers in Python. 

> We will create consumer and producer messages in the Jupyter notebook [below](https://www.educative.io/courses/data-science-in-production-building-scalable-model-pipelines/pubsub#try-it-out).

### Consumer
The code snippet below shows how to read messages from the subscription using the Google Cloud library. 

We first create a subscriber client, then set up the subscription and assign a callback function. 

 

Introduction to PubSub and its producer and consumer APIs.

- PubSub

Introduction to Building Scalable Model Pipelines

Models as Web Endpoints

Models as Serverless Functions

Create an Echo Function in Lambda

Working with S3 in Lambda

Working with API in Lambda

Containers for Reproducible Models

Working with AWS Container Registry

Workflow Tools for Model Pipelines

PySpark for Batch Pipelines

Cloud Dataflow for Batch Modeling

Streaming Model Workflows

Course Conclusion

- PubSub

What is PubSub?

Setting up PubSub