Creating a dbt Project
Learn to create a service account in BigQuery, which dbt will use to run your models.
Creating a Google Cloud project
If you don’t already have a Google Cloud project, create a Google Cloud account using the free trial. Then, create your first project if you don’t already have one.
Creating a BigQuery service account
To create a service account in BigQuery, which dbt will use to run our models, we need to follow the steps below:
Open the BigQuery console and navigate to IAM & Admin > Service Accounts in the left menu.
Create a new service account and grant it the “Job User” and “BigQuery Admin” roles.
The service account should be visible on the accounts list:
Click the email address of the service account and go to the “Keys” tab. Add a new JSON key, which will download a JSON file.
Creating a dbt project
To create our first dbt project, use the terminal at the bottom of the page.
Run the following command in the terminal. This will create a brand new dbt project.
dbt init
💡
dbt initis a shortcut to automatically generate the files we need, but it’s also completely valid to create those files manually!
Next, we need to supply several values:
Name: The name we want to give to the project
Database: In our case, BigQuery
Authentification method: In our case, we'll use service-account
Path: The location of our service account file (in our case, it will be:
/usercode/credentials.json)Project: The name of our Google Cloud project (make sure the service account has access to that project)
Dataset: The name of our BigQuery dataset (if we don’t already have one, we can pass the desired name for it and dbt will create it)
Dataset location: The region where our dataset is located (EU or US, for example)
Threads: The number of models dbt can run at the same time (we can keep the default value for this)
job_execution_timeout: The number of seconds after which dbt returns an error if the model has not finished running (we can also keep the default value for this)
Once our project is created, we move to the directory of our project using the following command:
cd <project_name>
Then, we can check that everything works fine by running the following command:
dbt debug
In case something is wrong, dbt will return an error message. In our case, everything should be fine.
We can finally run our new models by running the following command:
dbt run
Now, if we check our BigQuery dataset, we’ll have one new view and one new table:
We just learned how to create and run a dbt project!
Try it yourself
🧪 This is your first playground! First, change values in
credentials.jsonby using the “Import Values from JSON” button. Then, click the “Run” button. Create a project and use commands likels(list) andcd(change directory) to see what files dbt created.
{
"type": "{{type}}",
"project_id": "{{project_id}}",
"private_key_id": "{{private_key_id}}",
"private_key": "{{private_key}}",
"client_email": "{{client_email}}",
"client_id": "104553242413497706108",
"auth_uri": "{{auth_uri}}",
"token_uri": "{{token_uri}}",
"auth_provider_x509_cert_url": "{{auth_provider_x509_cert_url}}",
"client_x509_cert_url": "{{client_x509_cert_url}}"
}