Monitoring Alerts for GCP IAM Policy Changes

In this quick demo, we are going setup Monitoring Alerts for GCP IAM Policy changes, meaning we will get alerts for GCP IAM Role assignments and removals to users, service accounts and groups.

To setup alerts for IAM changes first we need to create log based metric for logs which contains setIamPolicy method.

I’m going to use following log filter to capture those type of logs.

protoPayload.authorizationInfo.permission =~ "setIamPolicy"

Apply below terraform code to create a log based metric with above log filter.

Note: you can also set up the below log metric on aggregated logging log buckets, just add bucket_name option to below bloc.

resource "google_logging_metric" "iam_change_metric" {
  name   = "iam-change-metric"
  project = "devops-counsel-demo"
  filter = "protoPayload.authorizationInfo.permission =~ \"setIamPolicy\""
  metric_descriptor {
    metric_kind = "DELTA"
    value_type  = "INT64"
    unit        = "1"
    labels {
      key        = "resource"
      value_type = "STRING"
    }
    labels {
      key        = "granter"
      value_type = "STRING"
    }
    labels {
      key        = "grantee"
      value_type = "STRING"
    }
    labels {
      key        = "action"
      value_type = "STRING"
    }
    labels {
      key        = "role"
      value_type = "STRING"
    }
    display_name = "IAM Change Metric"
  }
  label_extractors = {
    "resource" = "EXTRACT(protoPayload.authorizationInfo.resource)"
    "granter"  = "EXTRACT(protoPayload.authorizationInfo)"
    "grantee"  = "EXTRACT(protoPayload.serviceData.policyDelta.bindingDeltas.member)"
    "action"   = "EXTRACT(protoPayload.serviceData.policyDelta.bindingDeltas.action)"
    "role"     = "EXTRACT(protoPayload.serviceData.policyDelta.bindingDeltas.role)"
  }
}

When you apply above terraform code it will create a log based metric like below( you can find this metric on logging console under “Log Based Metrics” section.

Now we need to create monitoring alert policy, to create alert policy we need a notification channel, I have already created a Slack notification channel. to get the notification channel id run below gcloud command.

gcloud alpha monitoring channels list

Apply below terraform code to create alert policy for above log based metric.

Note: Replace notification channel with your notification channel id.

resource "google_monitoring_alert_policy" "iam_alert_policy" {
  display_name = "iam-change-alert-policy"
  project = "devops-counsel-demo"
  combiner     = "OR"
  conditions {
    display_name = "IAM Change Alert Policy"
    condition_threshold {
      filter     = "resource.type = \"global\" AND metric.type = \"logging.googleapis.com/user/iam-change-metric\""
      duration   = "0s"
      comparison = "COMPARISON_GT"
      trigger {
        count = 1
      }
      aggregations {
        alignment_period   = "60s"
        per_series_aligner = "ALIGN_COUNT"
      }
    }

  }
  alert_strategy {
    auto_close = "1800s"
  }
  notification_channels = ["projects/devops-counsel-demo/notificationChannels/2113444069280824342"]
}

after applying above terraform code you can see alert policy on monitoring console.

From now on when you make a change you will get an alert to notification channel(here in demo I used a Slack channel)

To test this alert policy I have added bigquery.admin role to one of my service account with in a minute I have received an alert to my Slack channel.

In the below alert message, you can see who made the policy change and what type of role was added/removed and on which resource’ the role’s policy was changed and type of action.

You can also create these Monitoring Alerts for GCP IAM Policy changes with GCP logging “log based alerts” but at this moment, log based alerts does not support aggregated logging buckets, they work only with project scoped logs, read more here.

you may also be curious to know:

Monitoring Alerts for GCP IAM Service Account Key Creation