Monitoring Alerts for GCP IAM Service Account Key Creation

In this quick demo, we are going setup Monitoring Alerts for GCP IAM service account key creation, meaning we will get alerts when some one creates a service account key.

To setup alerts for IAM service account key creation, first we need to create log based metric for logs which contains iam.serviceAccountKeys.create IAM permission.

I’m going to use following log filter to capture those type of logs.

protoPayload.authorizationInfo.permission = "iam.serviceAccountKeys.create"

Apply below terraform code to create a log based metric with above log filter.

Note: you can also set up the below log metric on aggregated logging log buckets, just add bucket_name option to below bloc.

resource "google_logging_metric" "sa_key_create_metric" {
  name    = "sa-key-create-metric"
  project = "devops-counsel-demo"
  filter  = "protoPayload.authorizationInfo.permission = \"iam.serviceAccountKeys.create\""
  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
    unit        = "1"
    labels {
      key        = "serviceAccount_project_id"
      value_type = "STRING"
    }
    labels {
      key        = "serviceAccount_email"
      value_type = "STRING"
    }
    labels {
      key        = "serviceAccount_key_creator"
      value_type = "STRING"
    }
    display_name = "IAM Change Metric"
  }
  label_extractors = {
    "serviceAccount_project_id"  = "EXTRACT(resource.labels.project_id)"
    "serviceAccount_email"       = "EXTRACT(resource.labels.email_id)"
    "serviceAccount_key_creator" = "EXTRACT(protoPayload.authenticationInfo.principalEmail)"
  }
}

When you apply above terraform code it will create a log based metric like below( you can find this metric on logging console under “Log Based Metrics” section.

Now we need to create monitoring alert policy, to create alert policy we need a notification channel, I have already created a Slack notification channel. to get the notification channel id, run below gcloud command.

gcloud alpha monitoring channels list

Apply below terraform code to create alert policy for above log based metric.

Note: Replace project and notification channel with your notification channel id.

resource "google_monitoring_alert_policy" "sa_key_create_alert_policy" {
  display_name = "sa-key-create-alert-policy"
  project      = "devops-counsel-demo"
  combiner     = "OR"
  conditions {
    display_name = "Service Account Key Creation Alert Policy"
    condition_threshold {
      filter     = "resource.type = \"global\" AND metric.type = \"logging.googleapis.com/user/sa-key-create-metric\""
      duration   = "0s"
      comparison = "COMPARISON_GT"
      trigger {
        count = 1
      }
      aggregations {
        alignment_period   = "60s"
        per_series_aligner = "ALIGN_COUNT"
      }
    }

  }
  alert_strategy {
    auto_close = "1800s"
  }
  notification_channels = ["projects/devops-counsel-demo/notificationChannels/2113444069280824342"]
}

After applying above terraform code you can see alert policy on monitoring console.

From now on when some one creates a service account key will get an alert to notification channel(here in demo I used a Slack channel)

To test this alert policy I have created a key for one of my service account called iam-alert-demo@devops-counsel-demo.iam.gserviceaccount.com with in a minute, I have received an alert to my Slack channel.

In the below alert message, you can see who created the service account key and for which service account the key was created and in which project the key was created.

You can also create these Monitoring Alerts for GCP IAM service account key creation with GCP logging “log based alerts” but at this moment, log based alerts does not support aggregated logging buckets, they work only with project scoped logs, read more here.

you may also be curious to know:

Monitoring Alerts for GCP IAM Policy Changes