Toolbox for AWS Deep Dives

Disclaimer:

This page contains guidance for evaluating publically available AWS reference architectures and related material. It is the authors opinion only. No warranties or guarantees are provided or implied.

How many AWS Services Exist?

  • Try using AWS SSM to get a sorted list… aws ssm get-parameters-by-path –path /aws/service/global-infrastructure/services –output json | jq .Parameters[].Name | wc -l

from https://aws.amazon.com/blogs/aws/new-query-for-aws-regions-endpoints-and-more-using-aws-systems-manager-parameter-store/

Transcribing Podcasts

  • An asynchronous approach to transcribing podcasts using AWS Step Functions and AWS Lambda.
Name Discovering and indexing podcast episodes using Amazon Transcribe and Amazon Comprehend
Link https://amzn.to/2S4Wd75
From AWS Machine Learning Blog
Updated 20 Sep 2018
Services Amazon Comprehend, Amazon Transcribe, AWS Step Functions, Amazon Cognito, AWS Lambda, Amazon S3, Amazon Elasticsearch and Kibana
Build Time 20 min
Cost $0.43 / day for Elasticsearch
Caveats Lambda functions are Python 2.7 so watch for deprecation as per https://amzn.to/2slo1K0, Ensure you shrink the Elasticsearch cluster to reduce costs, Good example of loose coupling asynchronous architecture. Assumes US-EN dialect which can result in some interesting transcriptions.
TAG - Resource Stack Asynchronous and Serverless
TAG - Application Podcast Transcription
TAG - Source https://amzn.to/2S4Wd75
TAG - Language Python 2.7
TAG - Demo Status No Issues
TAG - Review By Jan 2020

Well Architected Observations

  • Security
    • (+) Serverless so no network access
    • (+) Uses Cognito for Authentication and IAM for Authorization [demo1-1]
    • (-) Elasticsearch cluster open to 0.0.0.0/0 by default. Let’s lock that down. [demo1-2]
  • Performance
    • (+) You can tune the individual Lambda functions independently [demo1-3]
    • (-) The use of managed services limits throughput OOTB
  • Reliability
    • (+) Amazon Step Functions loosely couples all integrations and processing. No practical limits on podcast length [demo1-4]
  • Cost
    • (-) Elasticsearch cluster
  • Ops
    • (+) Self contained user management, deployment and teardown of resources [demo1-5]
    • (-) Lamba function is Python 2.7 (support stops in 2020)
    • (-) podcast-transcribe-index insufficient permissions to add tags
  • Stats
    • 34 resources

Centralized Logging

  • A logging and monitoring service that supports multi account and one way data diodes.
Name Centralized Logging
Link https://amzn.to/2EwpsHW
From AWS Solution
Updated Dec 2019
Services Amazon Cloudwatch custom Alarms, Amazon Cognito, Amazon Elasticsearch and Kibana, AWS Lambda, Amazon SNS
Build Time 16 min
Cost $0.43 / day for Elasticsearch
Caveats Deploy the demo Wordpress App and distribute the public url to generate traffic, Check rejected IP reputation, Ensure you shrink the Elasticsearch cluster to reduce costs
TAG - Resource Stack Logging and Monitoring
TAG - Application Centralized Logging
TAG - Source https://amzn.to/2EwpsHW
TAG - Language Node JS 8.10
TAG - Demo Status Node JS 8.10 support ends 31Dec2019. Switch to nodejs12.x LTS in the Cloudformation template. Only two code changes required.
TAG - Review By Jan 2020

Well Architected Observations

  • Security
    • (+) Roles used for cross account access [demo2-1]
    • (+) Data diode feeds main logging account from logged accounts
    • (+) Uses Cognito for Authentication and IAM for Authorization
    • (-) Elasticsearch cluster open to 0.0.0.0/0 by default. Lock it down to a more restrictive CIDR.
  • Performance
    • (+) You can tune the individual Lambda functions in each logged account independently
    • (-) Log capture is mostly asynchronous so not sub second real time visibility [demo2-2]
  • Reliability
    • (+) Mix of asynchronous and managed services allows this solution to scale out seamlessly
  • Cost
    • (-) Elasticsearch cluster
  • Ops
    • (+) Self contained user management, deployment and teardown of resources
    • (+) Can add additional accounts to be logged. Ideal for standup and teardown of DR, POC, staging and other non prod deployments.
    • (+) Cloudformation Update Stack to NodeJS12.x update (was NodeJS8.10) in less than 10 minutes [demo2-3]
    • (+) Can use in conjunction with AWS Limit Monitor Solution. Visit DynamoDB LimitMonitor table and filter for Region=us-east-1 AND TimeStamp > 2020 AND CurrentUsage=Red or Green. [demo2-4]
    • (+) Creating Cloudwatch Alarms (for Elasticsearch cluster) using Cloudformation is a great start point that is rarely contentious
    • (-)Lamba function is NodeJS8.10 (support stops in 2020)
  • Stats
    • 28 Tagged resources [demo2-5]

Neptune and Notebook

  • Standalone grouping of database ETL, cleansing and querying using an Amazon Sagemaker Notebook.
Name Let Me Graph That For You – Part 1 – Air Routes
Link https://amzn.to/2S34XKZ
From AWS Database Blog
Updated 7 Nov 2018
Services Amazon Neptune, Amazon VPC, AWS Lambda, Amazon Sagemaker
Build Time 12 min
Cost $8.35 per day for Neptune. Assume Sagemaker Notebook instance is quiesced
Caveats Interesting use of Sagemaker to host a Notebook that loads data, runs graphdb queries and creates vizualizations. This is now an Amazon Neptune feature.
TAG - Resource Stack Big Data
TAG - Application Sagemaker and Neptune with Air Routes Dataset
TAG - Source https://amzn.to/2S34XKZ
TAG - Language Python
TAG - Demo Status Some Python 3.6 Import and Resource Warnings
TAG - Review By Jan 2020

Well Architected Observations

  • Security
    • (+) Sagemaker notebook only accessible when running
    • (+) IAM User created that limits access to the graph db [demo3-1]
    • (-) No VPC Interface Endpoint to Sagemaker notebook. Direct internet access to the Sagemaker Notebook. At least lock it down to a more restrictive IP range.
    • (-) DB Security Groups open to 0.0.0.0/0. Lock down SSH unless you need it for EC2 access.
  • Performance [demo3-2]
    • (+) You can tune the Neptune graph DB and Sagemaker notebook instance sizes
    • (+) Good example of fast, automated ETL with Jupyter Notebooks and Python code retrieved during Cloudformation Stack build
  • Reliability
    • (+) Sagemaker Notebook flushes and reloads the data set each execution[demo3-3]
  • Cost
    • (+) Cloudformation parameters restrict Sagemaker Notebook instance choices.
    • (-) Neptune Graph DB cost is persistent. Even when shutting down the Sagemaker notebook.
    • (-) Sagemaker Notebook flushes and reloads the data set each execution. This has data transfer cost implications.
  • Ops
    • (+) Self contained graph db data load, query and teardown of resources
    • (+) Contains additional notebooks for first time users (partial runbook)
    • (+) Contains additional notebooks of the Python code (embedded codebase)
    • (+) Automated ETL, query engine and visualization packaged in a single deployment
    • (-) Lamba function is Python 2.7 (support stops in 2020)
    • (-) Throws import and resource Python 3.6 warnings. The python dependency mismatches are described in the jupyter cloudwatch logs as ERROR. [demo3-4]
    • (-) Notebook start script has data and code dependencies[demo3-5]
  • Stats
    • 37 resources

Containers, Cognito, DynamoDB and CICD

  • Multi tenant container based product catalog with self contained CICD and state teardown.
Name SaaS identity and isolation with Amazon Cognito
Link https://amzn.to/2sHhLvV
From AWS Quickstart
Updated Dec 2017
Services Amazon VPC, AWS Lambda, ALB, Amazon API Gateway, Amazon DynamoDB, Amazon ECS, Amazon S3, Amazon SNS, AWS Codebuild, AWS CodePipeline, Amazon Cloudwatch
Build Time 98 min
Cost $0.54 / day for the ALB and $2.21 / day for EC2 and $0.29 / day for EBS
Caveats Uniquely builds CICD pipelines and the application with 43 Cloudformation stacks and more than 200 resources. It’s more than 2 years old but still impressive. Read the 37 page Deployment Guide.
TAG - Resource Stack Multi Tenant
TAG - Application Containers Cognito DynamoDB CICD
TAG - Source https://amzn.to/2sHhLvV
TAG - Language TODO
TAG - Demo Status No Issues
TAG - Review By Jan 2020

Well Architected Observations

  • Security
  • Performance
    • (+) 7 ECS and 1 UI uncoupled. This is a very scalable application with minimal modification
  • Reliability [demo4-1]
    • Serverless and Containers! Inherently reliable compared to running everything on traditional compute
  • Cost
    • (+) ECS Cluster and EBS volume costs are relatively low for this large deployment (200+ resources)
  • Ops
    • (+) Seperate Cloudformation stacks to teardown the resources [demo4-2]
    • (+) 7 ECS and 1 UI codebases have seperate Code Pipelines [demo4-3]
    • (-) S3 bucket sprawl. Some consolidation seems appropriate. [demo4-4]
    • (-) DynamoDB provisioned capacity is assumed and low. No Cloudwatch alarms are configured in the Cloudformation template. Some dashboards would be useful for this many resources.
    • (-) Lamba function is Python 2.7 (support stops in 2020)
  • Stats
    • 226 total resources from 43 Cloudformation stacks [demo4-5]

AWS Workshop for Kubernetes

  • Here we use the very comprehensive AWS Workshop for Kubernetes available on Github.
Name Kubernetes the AWSome Way!
Link https://bit.ly/2YZxYsH
From AWS Github
Updated 30 Oct 2018
Services Amazon EKS, AWS Cloud9, Amazon VPC
Build Time 2 hrs to 10 hours
Cost approx $20
Caveats This workshop utilizes AWS Cloud9 IDE and integrates EKS and native Kubernetes tooling. This workshop defaults to Kubernetes v1.10 so update as necessary. This workshop also allows you to use native Kubernetes tooling in conjunction with AWS. It’s self paced and composed of a 2 hr intro and 4 hr each Dev and Ops instructions.
TAG - Resource Stack Containers
TAG - Application Kubernetes the AWSome Way
TAG - Source https://bit.ly/2YZxYsH
TAG - Language NA
TAG - Demo Status Cloudwatch Logs Not Accessible
TAG - Review By Jan 2020
TAG - Platform Kubernetes

Well Architected Observations

  • Security

    • (+) No inbound Security Group access to the control plane SG
    • (-) Security groups are open to 0.0.0.0/0

  • Performance

  • Reliability

  • Cost

    • (-) Kubernetes does incur additional master overhead
    • (-) The need for a quorum also demands three Availability Zones which increases costs by 50% basedon availability alone

  • Ops

    • (+) The use of AWS Cloud9 to run the runbook is a neat seperation from the cluster [demo5-1]
    • (-) Kubernetes roadmap evolves rapidly (quarterly). Need to evaluate and update the runbook regularly [demo5-2]
    • (-) Need to enable the EKS Cloudwatch logs. Having prebuilt dashboards would help improve situational awareness.

  • Stats

    • 15 Tagged resources

  • Alternatively you can use the Modular and Scalable Amazon EKS Architecture Quickstart located at https://aws.amazon.com/quickstart/architecture/amazon-eks/

Bonus Examples

Big Data Examples

Redshift Step and Glue

  • Serverless ETL using AWS Glue and AWS Step Functions for Amazon Redshift data loading
Name Orchestrate Amazon Redshift-Based ETL workflows with AWS Step Functions and AWS Glue
Link https://amzn.to/2S5AnAt
From AWS Big Data Blog
Updated 11 Oct 2019
Services Amazon Redshift, AWS Step Functions, AWS Glue, Amazon SNS, AWS Secrets Manager, Amazon S3
Build Time approx 2 hours
Cost
Caveats Multiple Cloudformation templates to deploy which closer mimics production deployments. Uses Secrets Manager for DB passwords
TAG - Resource Stack ETL
TAG - Application Redshift Step and Glue
TAG - Source https://amzn.to/2S5AnAt
TAG - Language
TAG - Demo Status TODO Reevaluate
TAG - Review By Jan 2020

Six Degrees of Kevin Bacon

  • An Example of an ETL Process for Transforming and Loading Data Into Amazon Neptune
Name Six Degrees of Kevin Bacon
Link https://bit.ly/2MaGZKh
From AWS Github
Updated
Services
Build Time
Cost
Caveats
TAG - Resource Stack ETL
TAG - Application Six Degrees of Kevin Bacon
TAG - Source https://bit.ly/2MaGZKh
TAG - Language
TAG - Demo Status TODO Not yet evaluated
TAG - Review By Jan 2020

Networking and Multiple Account Examples

Some demos are logistically more complex. Any complex networking, multiple account or federated authentication examples typically require multiple accounts and the use of third party services.

Serverless Transit Network Orchestrator
Name Serverless Transit Network Orchestrator
Link https://bit.ly/2Q18l82
From AWS Solution
Updated
Services AWS Transit Gateway, AWS Organizations
Build Time
Cost $75 / month
Caveats Demonstrates AWS Transit Gateway and network automation for complex networks and account hierarchies. Requires at least two accounts linked hierarchically with AWS Organizations
TAG - Resource Stack Networking
TAG - Application Serverless Transit Network Orchestrator
TAG - Source https://docs.aws.amazon.com/solutions/latest/serverless-transit-network-orchestrator/welcome.html
TAG - Language
TAG - Demo Status TODO Not yet evaluated
TAG - Review By Jan 2020

Deep Dive Tips and Tricks

  • Demo integrations and not just services. (but specialist SAs, ProServe and Product team folks can support truly deep deep dives)
  • Manage sprawl of S3 buckets
    • Configure services like AWS Config, Amazon Cloudtrail and Amazon Cloudwatch to use a single bucket. This lets you use this data for interesting adhoc log queries without having to manage complex manifests.
  • Manage all those convenient 0.0.0.0/0 access configurations to apply the rule of least privilege. Use AWS Config to dive deeper. NOTE Avoid demonstrating any sensitive resources you may have in your account.
  • Tag everything so you can find, cull and track all supported resources. Anything not tagged is a candidate for automatic culling

Continue reading articles in my Amazon Web Services series