Google Cloud Platform: Enable Data Access Log Support

This guide explains how to enable Data Access log support in BigQuery.

Data Access audit logs generate log events from the resource interaction plane of Google Cloud Platform. This includes event logging of requests to Google Cloud Platform resources such as object read/write operations, to answer questions such as “Who accessed file data in a storage bucket?” Expel doesn't offer monitoring and detection support for Data Access audit logs; however, Expel can use these logs to support alert triage and investigations.

To generate these logs, Google Cloud Platform customers must enable them. Review Google’s pricing reference for more information.

These logs can be long, especially if resources are being accessed by the public or other high-volume situations. Make sure you thoroughly understand the cost impact before enabling Data Access audit logs.

Prerequisites

1. You must have completed Google Cloud Platform setup for Workbench.
2. You must have the Google Cloud CLI installed.

Step 1: Configure BigQuery

1. Navigate to BigQuery in the expel-integration project and create a new dataset.

   • Dataset ID - expel_integration_dataset
   • Data location - Default
   • Default Table Expiration - 30 days
   • Encryption - Google-managed key
2. From the dataset settings, click SHARE DATASET, and grant the expel-gcp-integration service account (created in Step 3 of the Google Cloud Platform guide) the BigQuery Data Viewer and BigQuery User roles.

Step 2: Create the BigQuery Organization Log Sink

1. Log in to Google Cloud Platform.

   $ gcloud auth login

2. List the organizations and take note of the org ID.

   $ gcloud organizations list

3. Create the BigQuery organization log sink:

   • If your organization plans to use Event Threat Detection service, use this command:

     $ gcloud logging sinks create expel-org-sink-bigquery bigquery.
      googleapis.com/projects/[project-id]
     /datasets/expel_integration_ dataset --include-children
     --organization=[org-id] --log
     filter="logName=~\"cloudaudit\.googleapis\.com\%2Fdata_access\"
     OR  resource.type=\"threat_detector\""

   • If your organization does not plan to use the Event Threat Detection service, use this command:

     $ gcloud logging sinks create expel-org-sink-bigquery bigquery.
     googleapis.com/projects/[project-id]
     /datasets/expel_integration_ dataset --include-children
     --organization=[org-id] --log
     filter="logName=~\"cloudaudit\.googleapis\.com\%2Fdata_access\""

   This automatically creates a new service account which must be granted BigQuery Data Editor in the expel_integration_dataset in the console.

Step 3: Enable Data Access Logs

Admin Activity audit logs are enabled by default in Google Cloud Platform. During your initial onboarding, your organization was configured to send Admin Activity audit logs and Event Threat Detection alerts to Expel for monitoring and security detection support. Data Access audit logs are not enabled by default in Google Cloud Platform and have a cost to do so.

Reference:

• Google Cloud: Understanding Audit Logs
• Google Cloud: Enable Data Access audit logs

Default Configuration Option

A default audit log configuration can be set at the organization IAM so all new folders and projects inherit the configuration. This saves you from administrative hassle down the road.

Cost-Conscious Option

If you only want to collect Data Access audit logs from projects with particularly sensitive data or perhaps just production projects, you can configure audit logs in any IAM and IAM inheritance applies configuration as necessary. For example, if your organization has a production project with sensitive client data stored in Cloud Storage, you can consider only enabling the Google Cloud Storage Data Access audit logs in that particular project and leave it turned off in all of the other projects.

Enabling the Logs by Service

1. From the IAM console, click Audit Logs. Each service API can be enabled or disabled from this table. Some of the services to consider enabling:

   • Cloud Billing API
   • Cloud Build API
   • Cloud Functions API
   • Cloud Key Management Service (KMS) API
   • Cloud Logging API
   • Cloud Memorystore for Redis API
   • Cloud Pub/Sub API
   • Cloud Resource Manager API
   • Cloud SQL
   • Compute Engine API
   • Google App Engine Admin API
   • Google Cloud Deployment Manager V2 API
   • Google Cloud Storage
   • Identity and Access Management (IAM) API
   • Secret Manager API

Some services are prone to generate large volumes of log data (for example, Kubernetes, Compute Engine API and Google Cloud Storage), which can significantly affect cost.

1. Select your audit logging preference, and click Save.

As new projects are created and services are enabled by users, audit logging configuration should be inherited in IAM appropriately and data is recorded to your BigQuery table.