Terraform for Data Teams: BigQuery, Cloud Run, and the Blueprint That Scales

Data teams start by clicking through the console — create a dataset, grant an IAM role, deploy a Cloud Run job. It works, until six months later nobody remembers who created what, staging and prod have drifted, and the on-call engineer has no idea which service accounts can access which datasets.

Terraform is not about being fancy — it's about having one place where the truth lives. Every dataset, table, service account, IAM binding, Cloud Run job, and pub/sub topic in one repo, reviewed via PR, applied via CI.

terraform/
├── modules/
│   ├── bigquery_dataset/       # reusable module with variables
│   ├── cloud_run_job/
│   ├── service_account/
│   └── pubsub_topic/
├── envs/
│   ├── dev/
│   │   ├── main.tf             # instantiates modules for dev
│   │   ├── backend.tf          # remote state config
│   │   └── terraform.tfvars
│   ├── stage/
│   └── prod/
└── shared/                     # cross-env resources (DNS, org policies)

This layout scales from one engineer to a team of twenty. Each environment has its own state file, but shares the same modules. A change to modules/bigquery_dataset is applied to all envs on merge, environment-by-environment via CI.

# modules/bigquery_dataset/main.tf
resource "google_bigquery_dataset" "this" {
  dataset_id                 = var.dataset_id
  location                   = var.location
  default_table_expiration_ms = var.default_expiration_ms
  labels                     = merge(var.labels, { managed_by = "terraform" })
}

resource "google_bigquery_dataset_iam_binding" "readers" {
  dataset_id = google_bigquery_dataset.this.dataset_id
  role       = "roles/bigquery.dataViewer"
  members    = var.reader_members
}

output "dataset_id" { value = google_bigquery_dataset.this.dataset_id }

Notice: the module never reads secrets, never hard-codes members, always tags with managed_by. Those three habits prevent 80% of the operational pain.

Two ways to separate environments: Terraform workspaces (one dir, many state files) or directory-per-env (one dir per env, one state file each). I prefer directory-per-env for data teams because it's more explicit — a PR touching envs/prod is visibly a prod change and gets extra review.

Whichever you pick, keep it consistent across the repo. Mixed styles are worse than either style.

Local state is fine for a demo, disastrous for a team. Store state in GCS (or S3) with versioning and encryption enabled. Enable state locking (GCS does this automatically with a temporary lock object).

# envs/prod/backend.tf
terraform {
  backend "gcs" {
    bucket = "acme-terraform-state"
    prefix = "data-platform/prod"
  }
}

A pragmatic GitHub Actions workflow for a data team:

1. On PR: terraform plan for each affected env, post the plan output as a PR comment.
2. Require review from a data lead for any prod plan.
3. On merge to main: terraform apply against dev automatically.
4. Stage and prod apply are manual approvals (Environments in GitHub, or a separate workflow trigger).
5. Nightly: terraform plan against all envs; if diff is non-empty, alert Slack (drift detection).

This gets you 95% of what Terraform Cloud offers without the license cost, and stays inside your existing dev workflow.

Never put a service account JSON key in the repo or in CI secrets. Use Workload Identity Federation to let GitHub Actions assume a GCP identity via OIDC. Rotate nothing, revoke via IAM, done.

For local development, use gcloud auth application-default login. Every engineer authenticates as themselves; the state doesn't contain any long-lived secrets.