Nodes fail to launch with a cross-account KMS encrypted AMI

If your EDH nodes use a custom AMI whose EBS snapshot is encrypted with a KMS key that lives in a different AWS account, new instances can fail to launch. EDH provisions capacity through Amazon EC2 Auto Scaling, and Auto Scaling uses its service-linked role to launch instances. That role cannot decrypt the AMI unless the KMS key owner grants it access, so scaling activities fail and no node comes up.

In the EC2 console, open Auto Scaling groups, select the group for your EDH nodes, and go to the Activity tab. Under activity history you see a failed launch with a message similar to:

Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state

The fix is to let the Auto Scaling service-linked role use the KMS key. This follows the AWS procedure in Auto Scaling group fails to launch instances from an encrypted AMI.

Two accounts

In the steps below, Account A owns the KMS key that encrypts the AMI, and Account B is the account where EDH runs and Auto Scaling launches nodes. Run each command with credentials for the account named in that step.

Step 1: Allow the service-linked role in the KMS key policy (Account A)

In the account that owns the KMS key, edit the key policy to let Account B use the key and create grants. Add the following two statements, replacing <ACCOUNT_B_ID> with your EDH account ID.

{
    "Sid": "Allow use of the key from the EDH account",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::<ACCOUNT_B_ID>:root"
    },
    "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
    ],
    "Resource": "*"
},
{
    "Sid": "Allow grants for AWS resources",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::<ACCOUNT_B_ID>:root"
    },
    "Action": "kms:CreateGrant",
    "Resource": "*",
    "Condition": {
        "Bool": {
            "kms:GrantIsForAWSResource": "true"
        }
    }
}

Warning

Grant only the actions listed above, and keep the kms:GrantIsForAWSResource condition on the kms:CreateGrant statement. That condition limits grant creation to AWS services acting on your resources, which avoids opening broader cross-account access than you intend.

Step 2: Create a grant for the Auto Scaling service-linked role (Account B)

From the EDH account, create a grant that lets the Auto Scaling service-linked role use the key. Replace <KEY_ARN> with the full ARN of the KMS key in Account A and <ACCOUNT_B_ID> with your EDH account ID.

aws kms create-grant \
    --key-id <KEY_ARN> \
    --grantee-principal arn:aws:iam::<ACCOUNT_B_ID>:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling \
    --operations Decrypt Encrypt GenerateDataKey GenerateDataKeyWithoutPlaintext ReEncryptFrom ReEncryptTo DescribeKey CreateGrant

Note

AWSServiceRoleForAutoScaling is the Auto Scaling service-linked role. It is created automatically the first time Auto Scaling is used in the account.

Step 3: Allow the EDH roles to use the KMS key (Account B)

Two EDH components request capacity on your behalf, and each one runs with its own IAM role. Both roles must be allowed to use the KMS key:

Component	What it launches	IAM role
EDH Controller	Compute nodes for HPC jobs	soca-<cluster_id>-ControllerRole
<cluster_id>-CapacityExecutor Lambda function	Virtual desktops (VDI)	The execution role of the function

To find the role of the CapacityExecutor function, open the Lambda console, select the <cluster_id>-CapacityExecutor function, and go to Configuration > Permissions. The role name is shown under Execution role.

For each of the two roles, open the IAM console, select the role, and choose Add permissions > Create inline policy. Switch to the JSON editor and paste the following policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowUseOfCrossAccountAmiKey",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:DescribeKey",
                "kms:GenerateDataKeyWithoutPlaintext",
                "kms:ReEncrypt*",
                "kms:CreateGrant"
            ],
            "Resource": "*"
        }
    ]
}

Tip

This policy applies to any KMS key to keep the setup simple. If you need tighter access, replace * with the full ARN of the KMS key in Account A, so the roles can only use the key that encrypts your AMI.

Step 4: Verify the EC2 Fleet service-linked role (Account B)

EDH also uses Amazon EC2 Fleet, which relies on its own service-linked role. Check that the role exists in the EDH account:

aws iam get-role --role-name AWSServiceRoleForEC2Fleet

If the command returns NoSuchEntity, create the role once:

aws iam create-service-linked-role --aws-service-name ec2fleet.amazonaws.com

Note

Like the Auto Scaling role, the EC2 Fleet service-linked role is normally created automatically the first time the service is used. You only need to create it manually if it is missing.

Step 5: Retry

Once the grant and permissions are in place, trigger a new scaling activity (for example, submit a job or start a virtual desktop). Auto Scaling can now decrypt the AMI, and the node should launch successfully. If it still fails, confirm the key ARN, the account IDs, the inline policies on the Controller and CapacityExecutor roles, and that the key policy changes in Account A were saved.