Using the AWS CLI for investigative, forensics and for administrivia hygiene is something I use with many customers. The more traditional, evolved examples of cloud adoption often result in unmanaged environments that are hard to manage. This anti pattern is common in clickops and pre cloud environments. There is generally a lack of self documenting systems (automation, IaaC, DevOps).

Moving these environments to something more modern and away from anti patterns is a slow and tedious journey. The tech is the easy aspect. No consistent nomenclature, over reliance on out of date and disconnected documentation, legacy CMDB and related views of systems and poor information management hygiene are all common anti patterns.

I’ve included some useful cli commands that leverage the AWC CLI, JQ and basic scripting knowledge. While you can find a smattering of web articles I felt that describing the journey and rationale for using these commands was absent; and my experience has shown that the path we take is crucial in successfully raising the maturity of cloud administrators, their managers and how the organization supports them.

As a warmup I often start with listing all the AWS APIs. I would often use this in my training sessions where I would ask the attendees how many AWS services exist. I use the count of APIs as a proxy answer. Some of the responses are not strictly APIs, like the global pricing list which is a json list of prices for services. By sorting for the latest updates, I too learn something.

How many AWS APIs are there?

aws ssm get-parameters-by-path --path /aws/service/global-infrastructure/services --query 'Parameters[].Name' | wc -l

returns 332 as of 14Nov2022
returns 251 as of 30Nov2020

Here is another approach that is probably listing actual service APIs based on endpoints

curl -s https://raw.githubusercontent.com/boto/botocore/develop/botocore/data/endpoints.json | jq -r '.partitions[0].services | keys[]' | wc -l

returns 285 as of 15Nov2022

Find APIs with ’text string’ in the Name

What am I looking at?

It’s easy to get lost trolling through accounts and nested json. Before I start querying accounts I want a high level understanding of what I’m looking at. I also want to check and advise on basic administrivia like nomenclature.

{ aws sts get-caller-identity & aws iam list-account-aliases; } | jq -s ".|add”

This command succinctly lists:

  • The user id
  • Account number
  • The ARN of the user
  • The account alias

The enclosing curly braces group all the results in a neat little object.

What services are in use?

Using costs is a great way to see what AWS resources are in an account. AWS tracks all costs as they are incurred, which allows us to query them to see what resources and running and how costs are trending.

aws ce get-cost-and-usage --time-period Start=2022-09-01,End=2022-10-01 --granularity MONTHLY --metrics UsageQuantity --group-by Type=DIMENSION,Key=SERVICE | jq '.ResultsByTime[].Groups[] | select(.Metrics.UsageQuantity.Amount > 0) | .Keys[0]'

What Ports are enabled in which CIDRs?

It can be helpful to see what CIDRs are in use and what ports have ingress enabled. While this query doesn’t give me any detail about specific systems it will typically help ask some guided questions.

aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add'

List VPCs in a CSV file

Another way to demonstrate value quickly is to assist in documenting AWS resources. Here’s a cli call that outputs a csv of the VPCs in an account. There is no region parameter so here I’ve limited the call to a single region.

aws ec2 describe-vpcs —region | jq -r '[.Vpcs[] | {VpcId: .SubnetId, Name: (.Tags[]?|select(.Key=="Name")|.Value), CIDR: .CidrBlock, State: .State, VpcId: .VpcId, AWSAccount: .OwnerId}]|(.[0] | keys_unsorted) as $keys | $keys, map([.[ $keys[] ]])[]|@csv’ > vpc-ap-southeast-2.csv

Lamba Functions grouped by Runtime

This one lists Lambda functions in use and groups them by their runtime. This is a simple way to highlight older runtime that might need updating.

aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ runtime:.[0].Runtime, functions:[.[]|.FunctionName] }
]”

What is in Environment Variables?

Here I can look for keywords in environment variables to quickly troll through a large AWS Lambda function collection. Secrets, passwords, buckets, namespace and logging levels can be useful.

aws lambda list-functions | jq -r '[.Functions[]|{name: .FunctionName, env: .Environment.Variables}]|.[]|select(.env|length > 0)' | grep -in bucket | wc -l

What RDS Environments Exist?

Here I can get a quick overview of what RDS instances exist, what their status is and what instance sizes are they using.

aws rds describe-db-instances | jq -r '[.DBInstances[] | {DBName: .DBName, ID: .DBInstanceIdentifier, Status: .DBInstanceStatus, Class: .DBInstanceClass}]'

List EC2 Instances sorted by launch time

aws ec2 describe-instances | jq '.["Reservations"]|.[]|.Instances|.[]|.LaunchTime + " " + .InstanceId' | sort -n

Without using JQ, list EC2 instances sorted by launch time

aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId,LaunchTime]' --output text | sort -n -k 2

Other options in addition to InstanceId and LaunchTime include:

  • ImageId
  • KeyName - very useful to find forgotten keys. ’none’ indicates a more secure posture leveraging roles

Continue reading articles in my Amazon Web Services series

References