Reporting and remediating EC2 instances that AWS Systems Manager doesn’t list as managed instances

One of the primary features of AWS Systems Manager is Run Command. Run Command lets you remotely and securely manage the configuration of your managed instances. A managed instance is any Amazon EC2 instance that has been configured for Systems Manager. Now that the service has released the ec2-instance-managed-by-ssm, this is a good time to discuss options for discovering and remediating EC2 instances that are not managed by AWS Systems Manager.

Prerequisites

For an EC2 instance to be considered a managed instance in AWS Systems Manager, it must:

Be running a supported operating system

Be in an AWS Region that supports AWS Systems Manager

Have an IAM instance profile with permission to perform AWS Systems Manager actions on your instance

Have the AWS Systems Manager Agent installed and the service running

Be able to reach the Systems Manager endpoints with internet access or VPC endpoints

To learn more about these and other requirements in the operating system (OS), please review this user guide.

Problem

The AWS Config managed rule ec2-instance-managed-by-ssm rule can help you identify which instances are not managed by AWS Systems Manager, however it can’t tell you why they aren’t managed. To obtain more information about why EC2 instances are not showing up in AWS Systems Manager, you can use a combination of AWS Lambda and AWS Systems Manager Automation to build a detailed report.

Solution

This AWS CloudFormation template creates the following resources in your account:

AWS Lambda function

IAM role

AWS Systems Manager Automation document

The Lambda function checks through each Region in your account and compares a list of EC2 instances to AWS Systems Manager Managed Instances. AWS Lambda will review the configuration of each unmanaged EC2 instance and determination why the instance isn’t listed in AWS Systems Manager. Depending on the numbers of EC2 instances in your account, you may need to adjust the memory size and timeout of the AWS Lambda function. The IAM role is assigned to the Lambda function giving it permission to perform describe calls to gather this information on your behalf.

The purpose of the Automation document is to provide a wrapper for the output provided by AWS Lambda. When the Lambda function is completed, the payload provided will be a JSON file containing details about each of the unmanaged instances along with a reason why it isn’t registered with AWS Systems Manager. The Automation document can be run manually or triggered on a schedule using Amazon CloudWatch Events.

Handling the output

You always have the option of viewing the output in the AWS Management Console. However the JSON content is difficult to read, and you would still need to take action on it. I recommend using the AWS SDKs to retrieve and parse the data. Here are a couple of examples:

$unmanagedinstances = ConvertFrom-JSON -InputObject (((Get-SSMAutomationExecution -AutomationExecutionId <AUTOMATIONEXECUTIONID> -region <REGION>).Outputs.Values).Replace('" ',''))

ForEach ($i in $unmanagedinstances) { $i.InstanceId }

import boto3
import json

ssmclient = boto3.client('ssm','<REGION>')
response = ssmclient.get_automation_execution(
AutomationExecutionId='<AUTOMATIONEXECUTIONID>'
)

data = json.loads(''.join(response['AutomationExecution']['Outputs']['invokeEvaluateUnmanagedInstancesFunction.Payload']))

for i in data:
print (i['InstanceId'])

In both of these examples, the data is retrieved using the Automation Execution ID from an execution of the custom Automation document. The data is then parsed and turned into an object the SDK recognizes.

You can expand on these examples significantly and make this a part of a more complicated workflow. Here we are only iterating through the data collected and reporting the identified instance IDs. You could choose to take corrective actions based on the data gathered. For example, you could use PowerShell or boto3 to associate an IAM role to any instances that come back with the “No role” reason.

Conclusion

Using the provided CloudFormation template you can use our services to discover EC2 instances that are not managed by AWS Systems Manager, and in addition you can evaluate why they are unmanaged. This will help you to reduce time when onboarding resources into AWS Systems Manager. It also gives you some examples you can use to develop more complicated workflows that also take action on the results, which would further reduce administrative effort to onboard.

About the Author

Dan Hammel is a Sr. Cloud Support Engineer in AWS Premium Support. He specializes in Amazon EC2 Windows, AWS Systems Manager, and Microsoft PowerShell. Outside of AWS, he has a passion for cooking, enjoys a wide variety of music, and has been writing the first 17 pages of a novel for over a decade.