Ctrlk
Go to Workbench
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Google Kubernetes Engine Setup for Workbench

The Expel Google Kubernetes Engine (GKE) integration consumes audit logs from the Google Cloud Platform (GCP) through a log sink and pub/sub, which allows Workbench to identify activities of interest in GKE, investigate issues, and notify organizations if they need to take action or remediate.

Diagram indicates Kubernetes flows to the Log sink which flows to the Pub / Sub which flows to the Expel Integration, after which the SOC can Investigate and Respond.

Prerequisites

  1. Ensure you have the permissions to create the necessary resources in this process:

Resource
Required permissions
Purpose

Log sink

roles/logging.configWriter

MDR

Workbench GCP Service Account

roles/resourcemanager.projectIamAdmin

MDR

Google pub/sub

roles/pubsub.admin

MDR

Cloud infra custom role

roles/iam.roleAdmin

Inventory/metrics

Step 1: Enable Control Plane Logging for GKE Clusters

By default, GKE clusters are configured to send Kubernetes audit events to Cloud Logging. If the default state of clusters isn't changed, no additional configuration is needed.

Otherwise, at a minimum, Workbench requires logging of the Kubernetes control plane audit events for each cluster. Tracking activities that affect resources in the cluster provides necessary information for Expel: the “who”, “what”, and “when” that we need to detect, and take action.

Step 2: Enable Cloud Resource Manager API for Monitored Projects

For Expel to discover GKE clusters in the environment, the Cloud Resource Manager API must be enabled in monitored projects.

To enable Cloud Resource Manager API:

  1. In the Google Cloud console, navigate to APIs & Services library.

  2. From the library, select Cloud Resources Manager, and enable this API for all monitored projects.

Cloud Resource Manager API with API Enabled status as a check mark.

Step 3: Enable Kubernetes Engine API for Monitored Projects

For Expel to discover GKE clusters and get details of the clusters in the environment, the Kubernetes Engine API must be enabled in monitored projects. This is a must for inventory/metrics and security benchmarking, and a good-to-have for MDR investigations.

To enable the Kubernetes Engine API:

  1. In the Google Cloud console, navigate to APIs & Services library.

  2. From the library, select Kubernetes Engine API.

  3. Enable this API for all monitored projects.

Kubernetes Engine API status indicates it is enabled.

Step 4: Enable Data Access Auditing

Google Kubernetes Engine (GKE) separates the Kubernetes control plane logs into the following log types in Cloud Logging:

  • Admin Activity logs

    Capture write operations, like GET, UPDATE, or DELETE.

  • Optional Data Access logs

    Capture read operations, like GET or LIST.

For more information, see the GKE documentation.

Admin Activity logging is enabled by default. To increase logging visibility and include read operations, you may need to enable Data Access logging. You can enable auditing at a project, folder, or organization level.

To enable Data Access logging:

  1. Navigate to the IAM > Audit Logs page and search for "Kubernetes Engine API". Select these three boxes:

    • Admin Read

    • Data Read

    • Data Write

A screenshot of Data Access audit logs configuration with checks for Admin Read, Data Read, and Data Write.

Enabling additional logging can increase costs for Cloud Logging. For more information on how this is priced, see the Google Cloud documentation.

Step 5: Send GKE Logs to a Pub/Sub Topic

GKE begins routing Kubernetes logs to Cloud Logging automatically. In this step, we will create a pub/sub queue and a log sink to route logs to Workbench.

1. Create a New Pub/Sub Queue

Workbench collects GKE logs through a pub/sub subscription. You can create a subscription in the command line interface (CLI) with gcloud or the web console UI.

Be sure to:

  • Note the topic and subscription paths for later use.

  • Make sure the topic path has the following structure: projects/<project id>/topics/expel-k8s-integration-topic

Use the CLI

To create a pub/sub subscription using the CLI:

Use the web console UI

To create a pub/sub subscription using the web console UI:

  1. Navigate to Pub/Sub > Topics.

  2. Create a new topic with a default subscription:

    • Enter a name in the Topic ID field.

    • Select the Add a default subscription checkbox. The other checkboxes should be left unchecked. We don't recommend enabling retention duration because of additional cost. If it's needed, you can configure a customer-managed encryption key.

    • Under Encryption, select Google-managed encryption key.

    • Select Create.

A Create topic dialog with the following options selected: Add a default subscription, Google-managed encryption key.

2. Create a Log Sink to Route Logs to Pub/Sub

Steps for adding the log sink are different for the GCP Organization level and for the GCP Project level. Depending on your environmental needs, you have these options:

Option 1: Create an Organization-level log sink

Create a log sink to route logs to the pub/sub topic. To create a sink at the Organization level, you need to use the gcloud command line utility.

The Organization level is differentiated from the Project level by adding the --include-children and --organization=[org-id] parameters to the create sink command.

To create an Organization-level log sink:

  1. Log in to GCP:

    $ gcloud auth login

  2. Note the GCP org ID from the output of gcloud organizations list:

  3. Create the sink:

  4. Confirm the sink is successfully created:

Option 2: Create a Project-level log sink

Create a log sink to route logs to the pub/sub topic. You can use the command line interface (CLI) or the web console UI:

Use the CLI

To create a sink at the Project level, in the gcloud command line utility:

  1. Log in to GCP:

    $ gcloud auth login

  2. Note the GCP org ID from the output of gcloud organizations list:

  3. Create the sink:

  4. Confirm the sink is successfully created:

Use the Web console UI

  1. Navigate to Cloud Logging > Logs Router and create a new sink.

  2. Complete the fields:

    Name

    A name for the sink.

    We recommend expel-k8s-log-sink.

    Description

    A description for the sink.

    We recommend GKE logs for Expel monitoring.

    Destination

    The Cloud pub/sub topic created in Step 1:

    projects/${EXPEL_LOG_SINK_PROJECT_ID}/topics/expel-k8s-integration-topic

    Build inclusion filter

    resource.type=gke_cluster OR resource.type=k8s_cluster

A screenshot with sample sink field values. The values are like the ones provided in the table above.

Step 6: Enable Workbench Access

1. Create IAM Role for Workbench Kubernetes Access

Create a new custom IAM role to capture the required permissions for Workbench.

You can use the web console UI or the command line interface:

Use the CLI

To create a custom IAM role:

After you run the command, CLI generates a warning about some of these permissions being in testing. You can safely ignore this warning at this time. An example:

Note: permissions [container.bindings.get, container.bindings.list, container.initializerConfigurations.get, container.initializerConfigurations.list, container.petSets.get, container.petSets.list, container.podPresets.get, container.podPresets.list, container.scheduledJobs.get, container.scheduledJobs.list, container.thirdPartyResources.get, container.thirdPartyResources.list] are in 'TESTING' stage, which means the functionality is not mature and they can go away in the future. This can break your workflows, so do not use them in production systems!

Use the Web console UI

To create a custom IAM role:

  1. Navigate to IAM > Roles, and create a new role.

  2. Click the ADD PERMISSIONS button, and add these permissions to this role:

2. Create a GCP Service Account for Workbench

Create a service account for Workbench to collect GKE logs and monitor GKE clusters.

To create a service account:

  1. Navigate to IAM > Service Accounts, and create a new service account.

  2. Under Service account details, complete the following fields with your information:

    • Service account name - enter "expel-integration-gke-account".

    • Service account ID - enter "expel-integration-gke-account".

    • Service account description - enter "Expel GKE Integration Service Account".

    The Service account details dialog with the fields filled in like described above.

  3. Click Create and Continue.

  4. Add the following roles to the service account:

    • Browser (this allows Workbench to discover GKE clusters in the environment)

    • Pub/Sub Subscriber

    Screenshot of the Service account details dialog with the roles added, as described above.

We recommend adding a condition to limit the Custom GKE Reader role to the pub/resource you created earlier.

  1. For the last section, leave the blank defaults, and click Done.

To confirm that these roles are assigned correctly to the service account, run the following commands:

Step 7: Add Google Kubernetes Engine as a Security Device in Workbench

The Expel Workbench connection requires:

  • The service account key for the service account you created.

    The key is a JSON blob starting with "type": "service-account".

  • The full subscription name for the topic you created.

    For example, projects/<project id>/subscriptions/<subcription id>.

  • The 12-digit numeric organization (or project) ID for the project for monitoring.

1. Download the GCP Service Account Key

Workbench requires a service account key to authenticate to your GCP environment.

To download the GCP service account key:

  1. Navigate to IAM > Service Accounts.

  2. Locate the service account you created earlier for Workbench.

  3. From the Actions menu for that account, select Manage Keys.

  4. Create a new key.

  5. Download the key in the default JSON format. This key is used in Workbench to onboard the GKE integration.

2. Add GKE as a Security Device in Workbench

To configure Expel Workbench:

  1. Log in to Workbench.

  2. Navigate to Settings > Security devices.

  3. Click + Add security device.

  4. In the search field, enter "GKE".

  5. Select Google Kubernetes Engine.

    Google Kubernetes Engine Security Device config in Workbench.

  6. Enter the following information:

    • Name - enter a name that might help you more easily identify this integration, such as “CompanyName GKE”; this name will display in Workbench under the Name column, and is a text string that you can filter on.

    • Location - enter the location of your integration, for example “cloud;” this is also a text string that you can filter on, so we recommend being consistent with location naming across your Expel integrations.

    • Service account JSON - enter your GCP service account key.

    • Subscription name - enter the full subscription name for the topic you created.

      Example: projects/<project id>/subscriptions/<subcription id>

    • Organization ID - enter your Google Cloud Organization ID.

  7. Click Save.

Your device should be created successfully within a few seconds. A few reminders:

  • After your connection is healthy, it will take some time for your device to begin polling and receiving data.

  • To check on the status, select the downward arrow for your device in the first column and choose View details.

  • Polling will happen first; data will be received after that. You must refresh the page to see updates.

  • If your device does not begin polling within 15 minutes, and does not begin receiving data within 30 minutes, contact our support team for help.

  • To check if alerts are coming through, navigate to Dashboards > Alert Analysis. Scroll to the device you want to check, and select the Expel Alerts tab to reveal more alert information. It can take 36 to 72 hours for alerts to appear after setup, as we tune your device.

Last updated

Was this helpful?