Setting Up Google Cloud Composer in a Shared VPC Network

Plenty of resources are available for setting up a Cloud Composer environment in a single GCP project. However, there is a lack of unified resources that walk through how to integrate Cloud Composer into a professional enterprise environment. This usually entails setting up a subnet in a Shared VPC network exported to a service project, often managed by another team. This poses challenges of cross-team communication and permissions boundaries. This article walks through setting up a Cloud Composer environment in a Shared VPC network. Most importantly, I will point out some pitfalls around Google-generated service accounts which require the appropriate permissions to set up the environment.

Project Structure

Two GCP projects will be used in this system: the service project and host project. The service project houses the Cloud Composer environment and the host project contains the Shared VPC network. Beginning with the host project ensures all networking pieces are in place. This section of the tutorial guides you through setting up a Shared VPC network and subnet via the UI, typically managed by a Terraform stack in most enterprises. As a data/infrastructure engineer, defining CIDR ranges becomes crucial during this setup. Focusing on the Google console allows you to grasp the essentials for successfully connecting your Cloud Composer Environment to the Shared VPC network. While many enterprises have a networking team managing this aspect, understanding the process is crucial for data and infrastructure engineers. If you are setting up the networking for your team then this will inform how you will write your Infrastructure as Code (IaC).

Permissions Preparation
In order to follow along, your IAM user must have the following roles:
Compute Shared VPC Admin

Project IAM Admin

Note: The Project IAM Admin binding is at the organizational level or the folder level, not the project level.

Before we proceed, enable the GKE API in both your service and host projects (Cloud Composer is essentially an Apache Airflow install that runs on a GKE Autopilot cluster).

Creating the VPC Network

1. From the GCP UI verify the host project is selected
2. Go to the VPC networks page
3. Click Create VPC Network
   - Set the name
4. Make sure the Subnet Creation Mode is set to Custom
5. Click Add Subnet
   - Make sure to give this subnet a sensible name; it will be the primary subnet that holds the GKE nodes for your Cloud Composer cluster. Also make sure that it has a large enough CIDR range to account for your workloads. See the guidelines here.
6. Click Create Secondary IPV4 Range
   - You will do this step three times: once to define the CIDR range for your GKE cluster’s pods, one for your services and one for the GKE Control Plane.
   - Again give these ranges sensible names they will be used in your Terraform code when you are standing up your Cloud Composer Environment.
7. Turn Private Google Access on
   - This will make your cluster private and is recommended for enterprise clusters.
8. Click done

Things to look out for when creating the CIDR ranges for your cluster:

1. The secondary IP ranges cannot overlap with any other secondary ranges in the VPC network.
2. For a Cloud Composer environment, the minimum recommended range for Pods is /23. This range can support only a single small environment with a single scheduler and three workers. For production environments I would suggest nothing smaller than /22.
3. For most enterprise environments you will be spinning this cluster up within a Private IP environment, thus you will have to also establish the CIDR range for the GKE control plane, which lives in a Google-managed “tenant project”. This allows the GKE Cluster to communicate with the tenant project via VPC peering in the shared VPC. This CIDR range may not overlap with any subnet in the cluster’s VPC since this is a requirement for VPC peering.

Here is an example of a subnet CIDR range along with the defined secondary ranges for a Private IP environment:

See my article on setting up your Shared VPC with Terraform:

https://medium.com/real-kinetic-blog/using-terraform-to-build-a-shared-vpc-network-in-gcp-2077b0246a20

Attach the Service Project to the Shared VPC

In order for the Cloud Composer cluster to use the Shared VPC in the host project, the service project must be attached to the Shared VPC network by sharing the subnet with the service project.

1. In the cloud console make sure you have the host project selected and navigate to the Shared VPC page.
2. Navigate to Attached Projects and click Attach Projects.
3. Select the service project where the Cloud Composer environment will live in the projects list.
4. Under VPC network permissions keep the default permissions.
5. Under Kubernetes Engine Access check the Enabled box.
6. Under Sharing Mode select individual subnets (subnet-level permissions)
7. Under Subnets to share select the subnet you created in the previous step.
8. Save

Permissions for the Google APIs service account (project level)

After the service project has been attached to the Shared VPC network. On the Shared VPC page in the host project in the Google console.

1. Click in the upper right corner Show Permissions Panel
2. Under All subnet permissions (project-level-permissions) select the host project
3. In the Project level permissions panel on the right-hand side of the page click Add Principal
4. Add the Cloud Build Service Account as the principal:
   SERVICE_PROJECT_NUMBER@cloudservices.gserviceaccount.com

Here is a snapshot of the permissions UI:

Permissions for GKE Service Account (subnet level)

On the same Shared VPC page under the host project select the subnet that you created earlier.

1. In the permissions panel on the right-hand side of the page select Add Principal
2. Add the GKE service account as the principal:
   service-SERVICE_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com
3. Assign Role Compute Network User from the Select a role menu

Permissions for the Composer Agent Service Account (project level)

If this is the first Cloud Composer environment that you have set up, do step 1 first. If not, skip to step 2.

1. Enable the Cloud Composer API in your service project.
   - This will create the Composer Service Agent.
   - Verify it was created by going to the IAM page of the service project check the box for Include google-provided role grants.

1. Verify sure this service account was created: service-SERVICE_PROJECT_NUMBER@cloudcomposer.iam.gserviceaccount.com
2. Navigate back to the Shared VPC page on the host project
3. Click in the upper right corner Show Permissions Panel
4. Under All subnet permissions (project-level-permissions) select the host project
5. In the Project level permissions panel on the right-hand side of the page click Add Principal
6. Add the above created Cloud Composer Service Agent as the principal.
7. For Private IP environments add the Composer Shared VPC Agent Role
   - For Public IP environments, add the Compute Network User Role

These last three steps are a good place to verify with your network administrator. You will want to make sure that these service accounts are set up properly. You can do that by verifying they all exist in the service project as we discussed in the Permissions for the Composer Agent Service Account step 1c.

This completes the section for setting up the Shared VPC network. These steps can be verified with the google documentation found here. You will notice I have added a few steps for clarification and updated where I found the Google documentation lacking.

Create the Composer Environment

Next we will look at the service project and begin walking through the Cloud Composer setup in Terraform. I will use the Service Account and Cloud Composer Environment (V2) Terraform modules. These modules bring together all of the resources needed to spin up the Cloud Composer environment with a user-managed service account. The source code for these modules can be found here for Cloud Composer and here for Service Account.

Enable the Cloud Composer API

In order to spin up a Cloud Composer environment we must first enable the API.

resource "google_project_service" "composer_api" {
  project = "composer-project"
  service = "composer.googleapis.com"
  disable_on_destroy = false
}

Create Custom Service Account

By default Cloud Composer will attempt to use the default Compute Engine account. In more enterprise environments this account is disabled for security reasons. It is considered a best practice to create a user-managed service account with the appropriate role bindings.

The only role needed for this setup is the roles/composer.worker. However, it is almost always the case that your Composer environment will want to interact with either Google Cloud Storage or BigQuery. I have added two roles for Composer to interact with those services.

module "service_account_cloud_composer" {
  source        = "terraform-google-modules/service-accounts/google"
  version       = "~> 4.0"
  project_id    = "PROJECT_ID"
  names         = ["composer-service-account"]
  description   = "Service account for Cloud Composer environment"
  display_name  = "Cloud Composer service account"
  project_roles = [
    "composer-project=>roles/composer.worker",
    "composer-project=>roles/bigquery.user",
    "composer-project=>roles/storage.objectUser",
  ]
}

Create the Cloud Composer Environment

An important note when setting up the Cloud Composer module, the `master_ipv4_cidr` attribute is the CIDR block for the GKE control plane discussed in the Shared VPC Network section above. This CIDR range must not overlap with another in the subnet in the Shared VPC network.

Depending on your company’s policy on managing firewall rules, it is considered a best practice to utilize network tags to apply firewall rules.

Below is an example code snippet for utilizing the Cloud Composer module. You will also notice in it I have included some examples of adding additional python packages, environment variables and Airflow config overrides. In this instance the environment variable and pypi packages are needed for connecting to an on-premises Oracle database. This is to imitate a more real-world example.

module "composer" {
  source                           = "terraform-google-modules/composer/google//modules/create_environment_v2"
  version                          = "~> 4.0"
  project_id                       = "composer-project"
  composer_env_name                = "composer-dev"
  image_version                    = "composer-2.5.1-airflow-2.6.3"
  region                           = "us-central1"
  network                          = "vpc-name"
  network_project_id               = "vpc-host-project"
  subnetwork                       = "subnetwork-name"
  composer_service_account         = module.service_account_cloud_composer.email
  pod_ip_allocation_range_name     = "pod-ip-allocation-range-name"
  service_ip_allocation_range_name = "service-ip-allocation-range-name"
  master_ipv4_cidr                 = "10.1.2.0/28"
  environment_size                 = "ENVIRONMENT_SIZE_SMALL"
  use_private_environment          = "true"
  
  airflow_config_overrides = {
    core-dags_are_paused_at_creation = "true"
  }
  
  pypi_packages = {
    apache-airflow-providers-common-sql = ">=1.3.1"
    apache-airflow-providers-oracle     = ">=3.9.1"
    oracledb                            = ">=2.0.0"
  }

  env_variables = {
    LD_LIBRARY_PATH = "/home/airflow/gcs/data/oracle_client"
    AIRFLOW_EXTRA   = "/home/airflow/gcs/data"
  }

  tags = [
    "composer-egress",
    "google-apis",
    "oracle-egress"
  ]
}

Make sure to run the following from your terminal before applying this plan.

$ terraform init
$ terraform fmt
$ terraform validate
$ terraform plan

It will take approximately 25 minutes to stand up the Composer environment. Once it is stood up, from your service project navigate to the Composer page and click your new environment. Click on the Environment Configuration tab scroll down and under the Resources section and verify the Network Configuration is accurate.

Conclusion

In conclusion, this tutorial has covered the setup of a Cloud Composer environment within a Shared VPC using Private IPs. We’ve navigated through establishing a Shared VPC with a subnet and two secondary IP ranges tailored for the Cloud Composer Environment. Emphasis was placed on ensuring the necessary permissions were correctly configured on relevant service accounts, addressing common pitfalls encountered in such setups. The final step involved the streamlined setup of your Cloud Composer environment using Terraform, offering clarity and efficiency. While Google documentation serves as a reference, the use of Terraform modules showcased here proved to be more readable and effective. Happy building!

If you or your organization needs help working with Composer, or designing and implementing other aspects of your ETL processes in the cloud, please get in touch. We’d love to help however we can.