- Datastore Publish

Publishing results to BigQuery is useful for ETLs and other applications that are part of batch data pipelines. However, it doesn’t work well for use cases where applications need to retrieve a prediction for users with low latency.

Cloud Datastore

GCP provides two NoSQL databases that provide a solution for this use case, in which you need to retrieve information for a specific user with minimal latency.

In this lesson, we’ll explore Cloud Datastore, which provides some querying capabilities for an application database. We’ll build upon our prior pipeline and add an additional step that publishes to Datastore while also publishing the predictions to BigQuery. The resulting DAGDAG is shown in the figure below.