Workflows automation

One of the ways to automate workflows authored in Python is to directly connect to data sources.

For databases, you can use connectors based on JDBC or native connectors, such as the bigquery module provided by the Google Cloud library. This connector enables Python applications to send queries to BigQuery and load the results as a Pandas dataframe. This process involves setting up a GCP project, installing the prerequisite Python libraries, setting up the Google Cloud command line tools, creating GCP credentials, and finally sending queries to BigQuery programmatically.

If you do not already have a GCP account set up, you’ll need to create a new account. Google provides a $300 credit for getting up and running with the platform.

In our pre-configured execution environment, the gcloud library and CLI are already installed. To skip local installation instructions and set up your credentials, click here.

Installing Google cloud library

The first step is to install the Google Cloud library by running the following steps:

Get hands-on with 1200+ tech skills courses.