About Identity Classification

Identity Classification is a type of Investigative Action that leverages a machine learning model to help quickly assess incoming identity alerts. It allows our analysts to more efficiently perform initial triage so that they can spend more of their time on the investigation and response.

This technology will only run when there is a qualifying identity alert with high quality log data. It will show up in Workbench as an Investigative Action that is titled “Identity Classification” and has an "Expel AI" label.

What Qualifies

To qualify as an identity alert, the event must be associated with a SaaS application that contains authentication behaviors, events, and user activity within the cloud. This is defined as having log data from one of the following platforms:

• Office365
• Okta
• Duo
• G Suite

Model Training

The model is trained on a full year of historical cloud identity alerts, learning from past analyst decisions about which alerts were benign or malicious. By studying these outcomes, the model identifies the patterns and features that most strongly predict whether an identity alert is safe.

Assessment Criteria

The model relies on two main types of features to assess alerts.

Prevalence Features

These features measure how common or rare certain activities, locations, or entities are for a user or within an environment. High prevalence (i.e., repeated, consistent activity) typically suggests normal or expected behavior, while low prevalence (i.e., rare or first-time activity) may indicate something unusual or risky.

Examples:

• How often a user logs in from a particular IP address, region, or country
• How frequently a VPN or hosting provider is used

Behavioral Features

These features capture user behaviors or variations in system activity. They help the model distinguish between routine activity and actions that might be suspicious.

Examples:

• The volume and type(s) of actions performed (such as multiple account changes or file operations)
• Recent account or MFA registrations
• The presence of known suspicious behaviors (like use of a new device or user agent)

Alert Classifications

The model assigns one of the following classifications to indicate how strongly an identity alert leans benign or malicious.

Classification	Meaning
Benign	The model detected clear signals of safe behavior, with 97% or higher confidence.
Likely Benign	The model leans safe, but there is a small chance of risk.
Inconclusive	The model is uncertain and cannot make a strong call.
Suspicious	The model leans toward malicious behavior, but it is not definitive.
Malicious	The model detected strong indicators of a threat.

These classifications are meant to guide the investigation and to show you how confident the model is. For identity alerts classified as benign, Expel AI auto-closes them as benign and removes them from the queue.

Alerts classified as likely benign, inconclusive, suspicious, or malicious are not auto-closed.

Quality Control

Expel uses multiple quality control (QC) processes to ensure the technology does what it is supposed to do.

Randomized Auto-Close Exceptions

A subset of randomly selected identity alerts that are classified as benign will not be auto-closed. These alerts are instead redirected to Expel SOC analysts for triage, so that the human triage decision can be compared to the model's triage decision.

Protective Rules

The model contains protective rules to prevent the auto-close of identity alerts when certain post-processing indicators are evident. These mechanisms act as a safety net to ensure important identity alerts are always triaged by an analyst regardless of their assigned classification.

Arthur AI

We use Arthur AI to watch the model in production. We look at the model's inputs and outputs to identify any drifts in the data, which would indicate potential performance degradation, and to monitor precision. We have set up alerts to notify us of any problems.