Recently, I noticed a weird KMS key on an AWS account – what was weird about it, was the fact that it wasn’t marked as AWS-managed key, but no-one (not even root) could delete or modify the key!

The key itself has no alias, only a description which is the same as the description for default EBS AWS-managed keys (“Default master key that protects my EBS volumes when no other key is defined”). It also has the following access policy assigned to it:

{
  "Version": "2012-10-17",
  "Id": "auto-ebs-2",
  "Statement": [
    {
      "Sid": "Allow access through EBS for all principals in the account that are authorized to use EBS",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:CreateGrant",
        "kms:DescribeKey"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:CallerAccount": "xxx",
          "kms:ViaService": "ec2.region.amazonaws.com"
        }
      }
    },
    {
      "Sid": "Allow direct access to key metadata to the account",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxx:root"
      },
      "Action": [
        "kms:Describe*",
        "kms:Get*",
        "kms:List*",
        "kms:RevokeGrant"
      ],
      "Resource": "*"
    }
  ]
}

This is a policy that’s used for AWS-managed keys, which does not allow keys to be deleted. But this key was not marked as AWS-managed, so that made no sense! AWS-managed keys have aliases starting with “aws/” prefix, for example “aws/ebs” and have a small AWS logo next to them in the Console. You also cannot see the actual JSON access policy for the key in the Console.
This is how AWS-managed KMS keys look like in the Console:
KMS AWS-Managed keys in the Console

There was also no trace in CloudTrail of this key ever being created. Aside from all the above, this key also had exactly the same creation time as the “real” aws/ebs key. I got in touch with AWS support, and I have to say I was a bit disappointed with their initial responses… after a lot of messages back and forth and reporting this looks like a bug in KMS, the question was finally directed to the KMS service team and I received the following response:

Hi, yes, the customer have indeed encountered a known bug in the KMS system that can be triggered during key auto-creation. We sincerely apologize for the inconvenience. However, the customer is not being billed for the extra key so we ask the customer to simply ignore the key in question. Thank you!

According to the billing reports, the account is in fact not charged for this key, so its existence is basically an annoyance and potentially confusing to users, since it’s not possible to change the alias or description of the key to indicate to users not to use it. So if you have a key similar to this on your account, be aware of the bug and notify your users about it. You can potentially use this key if you’d like, thought given the fact that you cannot modify anything about this key, I’d probably discourage that.

Was this post helpful to you? Yes!


3 Comments

  1. Jakub

    Any news from AWS regarding this bug? It seems like it’s still there: I have the same issue on one of accounts and I could reproduce it by creating encrypted EBS volume in an unused region (default aws/ebs key is being generated then).

    • Paulina Budzon

      Hi Jakub,

      Nope, no updates so far. I’d recommend reporting this via support, so they’re aware that more people are complaining about it 🙂

  2. George L. Yermulnik

    Same issue here.
    Going to report to AWS Support.


Leave a comment