Amazon Elastic Kubernetes Service Setup for Workbench

The Expel Amazon Elastic Kubernetes Service (EKS) consumes audit logs from the AWS platform through Kinesis. This visibility allows Workbench to identify activity of interest in EKS, investigate, and notify organizations if action is recommended.

Step 1: Enable Control Plane Logging for Each EKS Cluster

At a minimum, Workbench requires logging of the audit events for each cluster. This provides visibility into activity affecting resources in the cluster: the “who, what, and when” we need to detect and take action. After control plane logging is enabled, EKS begins sending logs to CloudWatch.

To complete this step, update the logging configuration for each cluster in each region with the AWS CLI:

CLUSTER_REGION=<your-cluster-region> ; \
CLUSTER_NAME=<your-cluster-name> ; \
aws eks update-cluster-config \
  --region ${CLUSTER_REGION} \
  --name ${CLUSTER_NAME} \
  --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}'

For additional help, see the Amazon reference guide.

Step 2: Create a Kinesis Stream

Logs must be routed from CloudWatch to a Kinesis data stream so Workbench can consume the logs in real time.

1. Create a stream with the AWS CLI tool and note the following:

   • The value of K_STREAM_NAME must be different for each separate region.
   • We recommend creating an “ON_DEMAND” stream to allow the stream to adjust capacity based on demand.

CLUSTER_REGION=<your-cluster-region> ; \
K_STREAM_NAME=expel-aws-eks-kinesis-data-stream ; \
aws kinesis create-stream \
  --region ${CLUSTER_REGION} \
  --stream-name ${K_STREAM_NAME} \
  --stream-mode-details '{"StreamMode": "ON_DEMAND"}'

1. Note the Kinesis Stream ARN that the code CLI generates for use in later steps; the ARN is different for reach region:

CLUSTER_REGION=<your-cluster-region> ; \
K_STREAM_NAME=expel-aws-eks-kinesis-data-stream ; \
aws kinesis describe-stream-summary --stream-name=${K_STREAM_NAME} --region=${CLUSTER_REGION}

Step 3: Create an IAM Role for CloudWatch Log Delivery

An IAM role is required to allow CloudWatch to deliver logs to your Kinesis stream.

The sample code writes a policy JSON definition to expel_cw_assume_role_trust _policy_doc.json in the current directory. For convenience, the system creates a JSON document of the same name on the local machine.

1. Create a trust policy document that allows the CloudWatch service to assume the role:

AWS_REGION=<your-aws-region> ; \
AWS_ACCOUNT_ID=<your-aws-account-id> ; \
cat > expel_cw_assume_role_trust_policy_doc.json << EOF
{
  "Statement": {
    "Effect": "Allow",
    "Principal": { "Service": "logs.${AWS_REGION}.amazonaws.com" },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringLike": { "aws:SourceArn": "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:*" }
    }
  }
}

EOF

1. Create the IAM role with the expel_cw_assume_role_trust_policy_doc.json document:

EXPEL_CW_ASSUME_ROLE_NAME=expel-aws-eks-cloudwatch-assume-role ; \
aws iam create-role \
  --role-name ${EXPEL_CW_ASSUME_ROLE_NAME} \
  --assume-role-policy-document file://expel_cw_assume_role_trust_policy_doc.json  

Step 4: Update the IAM Policy for the Producer Role

1. Create an IAM policy document that grants the following role permissions to put records into your Kinesis stream:

K_STREAM_ARN=<kinesis stream ARN saved in earlier step> ; \
cat > expel_iam_policy_doc.json << EOF
{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "kinesis:PutRecord",
      "Resource": "${K_STREAM_ARN}"
    }
  ]
}

EOF

1. Apply the policy to the CloudWatch (producer) role:

EXPEL_CLOUDWATCH_ROLE_NAME=expel-aws-eks-cloudwatch-assume-role; \
EXPEL_IAM_POLICY_NAME=expel-aws-eks-producer-policy; \
aws iam put-role-policy \
   --role-name "${EXPEL_CLOUDWATCH_ROLE_NAME}" \
   --policy-name "${EXPEL_IAM_POLICY_NAME}" \
   --policy-document file://expel_iam_policy_doc.json

Step 5: Route Logs From CloudWatch to Kinesis

Logs must be routed to Kinesis with a CloudWatch log group subscription filter. Create the filter with the AWS CLI:

K_STREAM_ARN=<kinesis stream ARN saved in earlier step> ; \
EKS_LOG_GROUP_NAME=<your choice of log group name> ; \
CLUSTER_AWS_REGION=<your-aws-region> ; \
EXPEL_CLOUDWATCH_ROLE_NAME=expel-aws-eks-cloudwatch-assume-role ; \
aws logs put-subscription-filter \
  --region ${CLUSTER_AWS_REGION} \
  --log-group-name ${EKS_LOG_GROUP_NAME} \
  --filter-name "AllEKSLogs" \
  --filter-pattern "" \
  --destination-arn ${K_STREAM_ARN} \
  --role-arn ${EXPEL_CLOUDWATCH_ROLE_NAME}

Step 6: Create an IAM Role for Expel

If you already have an IAM role for another integration (like CloudWatch or Amazon GuardDuty), the existing role can be reused and you can skip to Step 7.

To authenticate to your AWS account and retrieve logs, Expel requires an IAM role.

For convenience, the system creates a JSON document named expel_wb_iam_role_policy_doc.json on the local machine.

1. Create a trust policy document that allows Workbench to assume the role. Make sure to use your own Workbench GUID:

ORG_GUID=<Your workbench org_guid(formerly customer id)> ; \
cat > expel_wb_iam_role_policy_doc.json << EOF
{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::012205512454:user/ExpelCloudService" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": { "sts:ExternalId": "${ORG_GUID}" }
      }
    }
  ]
}

EOF

1. Create the IAM role with the expel_wb_iam_role_policy_doc.json document:

EXPEL_ASSUME_ROLE_NAME=ExpelServiceAssumeRole ; \
      aws iam create-role \
        --role-name ${EXPEL_ASSUME_ROLE_NAME} \
        --assume-role-policy-document file://expel_wb_iam_role_policy_doc.json

Step 7: Grant the IAM Role Permissions

We require certain IAM permissions to retrieve logs from Kinesis and to investigate the activity of interest affecting your EKS clusters.

1. Create an IAM policy document that grants Workbench the required permissions for Kinesis and EKS:

K_STREAM_ARN=< kinesis stream ARN saved in earlier step > ; \
cat > expel_k8s_ro_policy_doc.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
      {
            "Effect": "Allow",
            "Action": [
                "kinesis:DescribeLimits",
                "kinesis:DescribeStream",
                "kinesis:DescribeStreamSummary",
                "kinesis:GetRecords",
                "kinesis:GetShardIterator",
                "kinesis:ListShards"
            ],
          "Resource": "${K_STREAM_ARN}"
        },
        {
            "Effect": "Allow",
            "Action": [
                "eks:AccessKubernetesApi",
                "eks:DescribeCluster",
                "eks:DescribeNodegroup",
                "eks:ListClusters",
                "eks:ListNodegroups",
                "eks:ListUpdates",
                "sts:GetCallerIdentity",
                "ec2:DescribeRegions",
                "autoscaling:DescribeAutoScalingGroups"
            ],
            "Resource": "*"
        }
    ]
}

EOF

1. Apply the policy to the Expel IAM role:

EXPEL_ASSUME_ROLE_NAME=ExpelServiceAssumeRole ; \
EXPEL_EKS_CONSUMER_POLICY_NAME=expel-aws-eks-eks-consumer-policy ; \
aws iam put-role-policy \
  --role-name ${EXPEL_ASSUME_ROLE_NAME} \
  --policy-name ${EXPEL_EKS_CONSUMER_POLICY_NAME} \
  --policy-document file://expel_k8s_ro_policy_doc.json

Step 8: Add Amazon EKS as a Security Device in Workbench

In this step we will add Amazon EKS as a security device and then enable network access by allowing specific IP addresses in AWS.

1. Log in to Workbench.
2. Navigate to Organization Settings > Security Devices.
3. Select Add Security Device.
4. Scroll down or search "Kubernetes (EKS)" and select Kubernetes Service (EKS).
5. Complete the fields as follows:

   • Name - enter a name that might help you more easily identify this integration, such as “CompanyName EKS”; this name will display in Workbench under the Name column, and is a text string that you can filter on.
   • Location - enter the location of your integration, for example “cloud;” this is also a text string that you can filter on, so we recommend being consistent with location naming across your Expel integrations.
   • Role ARN - enter the ARN for the cluster role you created in Step 6.
   • Role Session Name - we recommend "ExpelSession", but you may enter any name you like.
   • Region - enter the AWS region containing the Kubernetes cluster.
   • Kinesis Stream Name - enter the log stream name you created in Step 2.
6. Select Save.
7. Next, enable network access by adding the IP addresses listed in this article to your allow list in AWS.

Reference

This integration requires configuration of certain resources in order to function. Refer to the following list for details:

Resource	Purpose
Kinesis Stream	MDR
Kinesis Stream Filter	MDR
IAM Role	MDR
IAM Policy “Puts” (binds)	Inventory/metrics
Network Allow Rule	All Workbench investigation access