Architecting on AWS - Tips and Tricks
This is my list of hints and tips for this course. It’s markdown so you can save it, access it or store it anywhere. I might also give you other links that are course specific. I’ll add specific answers to questions I get during the course. I’ll share it with everyone on this page.
Your Instructors
- Ian Falconer https://www.linkedin.com/in/leftbrainstuff/
Administrivia
We need to jump through some hoops to get access to the labs and slide decks. Be consistent with the email address you use for all sites. There are three seperate sites you need to access and one bitly link which is this page:
- Join or login to https://www.aws.training/ to ensure your training and certifications are captured. No we don’t spam you or sell your details.
- Access Qwiklab (yes it is spelt INCORRECTLY)
- aws.qwiklabs.com for the labs in this class
- run.qwiklabs.com for outside of the class or to do other labs at your own pace.
- NOTE: Some are free others require course credits. Also check out the AWS Professional Developer Series of MOOCs on edX https://www.edx.org/aws-developer-professional-series and there are MOOCs on Coursera too.
- Access the course notes and slides. You’ll receive two emails. One confirming your attendance at this course and with the following links. The download link seems broken. You can download apps for phones, tablets and laptops. Or use your browser.
- www.vitalsource.com look for a signup link and download link. Or just go to https://evantage.gilmoreglobal.com/#/user/signin
- Once you’ve logged into Vitalsource (aka Bookshelf, Gilmore, eVantage) you can redeem your unique course materials code (in a seperate email) and update your book list. You should see a lab guide and student guide for DevOps on AWS, version 2.3 and 2.4. Just pick the latest version as they’ve bundled together by the vitalsource folks.
- The student guide is the powerpoint decks and notes and the lab guide is the step by step instructions for the labs. The lab guide is included in the labs so this document is somewhat redundant. You can download the Vitalsource Bookshelf app for Windows, Mac, IoS and Android at https://support.vitalsource.com/hc/en-us/articles/201344733-Bookshelf-Download-Page
- You can probably print the student and lab guides to pdf from the app. You’ll need to confirm this as the vitalsource folks update the app and website each week.
Group and Individual Exercises
What is DevOps?
- split into groups and agree on a definition of DevOps. Also describe key attributes of a DevOps culture. Also describe issues or challenges with DevOps. What DevOps metrics do you track?
- Time 20 - 30 minutes max
- Nominate a spokesperson and spend 2 or 3 minutes describing your findings to the class
Cool links
- AWS Global Infrastructure. Here are several videos that ‘open the kimono’ on how AWS is designed and built to support millions of customers across the globe.
- James Hamilton, AWS SVP and Distinguished Engineer, talks about the design decisions and inner workings of the AWS global infrastructure. James also provides the history behind major technological innovations like we’re seeing now in Cloud Computing. This deck is over 4 years old but still a good summary. This should be the first AWS video you watch. https://www.slideshare.net/AmazonWebServices/spot301-aws-innovation-at-scale-aws-reinvent-2014 . Here’s the youtube video. https://www.youtube.com/watch?v=JIQETrFC_SQ There are Youtube videos from more recent ReInvents with some updates too. Here is James in 2016. It’s titled as AWS re:Invent 2016: Amazon Global Network Overview with James Hamilton https://www.youtube.com/watch?v=uj7Ting6Ckk
- Here is a 4 min snippet from 2016 titled AWS re:Invent 2016: Introduction to Amazon Global Network and CloudFront PoPs with James Hamilton https://www.youtube.com/watch?v=FjHBGjLnou0&feature=youtu.be
- AWS re:Invent 2017 Keynote - Tuesday Night Live with Peter DeSantis, VP AWS Global Infrastructure talks about the AWS global infrastructure. Up to 15;46 minutes is about the infrastructure. https://www.youtube.com/watch?v=dfEcd3zqPOA&feature=youtu.be&t=1h17m0s
- https://www.infrastructure.aws/ now has an interactive map and animations describing the AWS Global Infrastructure. 100 GBps intercontinental network.
- This one is self explanatory AWS re:Invent 2018: Amazon VPC: Security at the Speed Of Light (NET313) https://www.youtube.com/watch?v=UP7wDBjZ37o&feature=youtu.be
- AWS re:Invent 2017: Scaling Up to Your First 10 Million Users (ARC201). This is like the Tech Essentials course in a single video. Well worth a watch. https://www.youtube.com/watch?v=w95murBkYmU
- AWS re:Invent 2016: Metering Big Data at AWS: From 0 to 100 Million Records in 1 Second (ARC308) another journey through choosing AWS services to build high scale near real time data management systems. https://www.youtube.com/watch?v=dD1wzEdQCb4
- Amazon EC2 Instance Types explained in neat tabular comparisons. https://aws.amazon.com/ec2/instance-types/ . Also here’s a third party site that has a table that lets you sort on memory, network performance, cost and instance type. You can also quickly compare costs here too. https://ec2instances.info/
- An external post about S3 data leaks. https://www.bleepingcomputer.com/news/security/amazon-aws-servers-might-soon-be-held-for-ransom-similar-to-mongodb/. Now consider features like AWS Organization and Service Control Policies and S3 object locking for implementing rule of least privilege security controls.
- Latency between AWS regions. Lot’s of good empirical data points. Note these are 24 hour averages of hourly averages. 95th percentile (or similar) values would be needed for real time troubleshooting. https://www.cloudping.co/
- Latency http://highscalability.com/latency-everywhere-and-it-costs-you-sales-how-crush-it
- List of CIDR ranges of AWS regions http://ec2-reachability.amazonaws.com/ which show you relative capacity between regions.
- How Amazon handles prime day (DynamoDB) https://www.youtube.com/watch?v=83-IWlvJ__8
- Chalice python based microservices framework using Lambda, API Gateway and IAM. http://chalice.readthedocs.io/en/latest/ Very fast, very lightweight and very extensible. Also look at SAM, the AWS Serverless Application Model and AWS Amplify.
- Benchmark tests of EC2 versus other bare metal and cloud servers. https://www.phoronix.com/scan.php?page=article&item=cloud-cpu-36&num=1
- Here’s a Lambda deep dive which more clearly explains some of the questions around managing state, retries, testing and handling large data sets with Lambda. https://www.youtube.com/watch?v=dB4zJk_fqrU
- Mechanical sympathy. Interesting concept for software defined assets but aligns with Werner Vogel’s catchy advice that ’everything fails all the time’. Therefore you need to architect resilient systems. http://infrastructure-as-code.com/book/2015/03/23/mechanical-sympathy.html
- AWS Open Guide on GitHub is a good summary of AWS Documentation and an attempt to summarize the wealth of AWS documentation on a single page. https://github.com/open-guides/ AWS has also open sourced all our documentation at https://github.com/awsdocs . The ENTRY page for all aws documentation at https://docs.aws.amazon.com/index.html#lang/en_us`
- You can check the overall health and availability of AWS globally at the Service Health Dashboard (SHD) https://status.aws.amazon.com/ You can also use the AWS Health API to programatically check for service health at https://docs.aws.amazon.com/health/latest/ug/getting-started-api.html
- Netflix have some cool tools that they’ve open sourced. https://netflix.github.io/
- All quickstarts now in github https://github.com/aws-quickstart and also check out the AWS solutions https://aws.amazon.com/solutions/ which are vetted technical reference architectures.
- Consider building load and performance test simulators with IaaC. Like IoT Device Simulator https://aws.amazon.com/solutions/iot-device-simulator/?trk=sl_card
- AWS Service Catalog Validation Pipeline https://aws.amazon.com/solutions/aws-service-catalog-validation-pipeline/?trk=sl_card
- Blue green quickstart at https://aws.amazon.com/quickstart/architecture/blue-green-deployment/
- Netflix Open Sourcing of many useful and interesting tools for running large AWS cloud environments. https://netflix.github.io/
- Researchers say Data61-backed blockchain platform delivers scalability, energy efficiency 1000 ec2 instances, 14 AWS Regions and 30k TPS https://www.computerworld.com.au/article/647265/researchers-say-data61-backed-blockchain-platform-delivers-scalability-energy-efficiency/
- Adrian Cockcroft AWS VP of Cloud Architecture Strategy and former CTO of Netflix. Here’s his Youtube playlist with talks about DevOps, migrations, Netflix lessons learned and digital transformation topics. https://www.youtube.com/playlist?list=PL_KXMLr8jNTnwkzV7SePa0jHFUG2qn0MA
- The Network is Reliable and other fallacies. Key performance and reliability concerns of distributed systems. https://blog.acolyer.org/2014/12/18/the-network-is-reliable/ . Also web search for Werner Vogels, Amazon CTO, who is acknowledged as a global expert on distributed systems.
- Architecture reviews are important. The cloud design principles, here is the 2011 AWS Whitepaper https://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf and the Well Architected Review are key inputs. https://aws.amazon.com/architecture/well-architected/
- AWS General Reference. https://docs.aws.amazon.com/general/latest/gr/Welcome.html This document is a key reference when architecting and designing AWS solutions. Be familiar with service limits.
- My favourite AWS blog post. Transcribe and Amazon Comprehend https://aws.amazon.com/blogs/machine-learning/discovering-and-indexing-podcast-episodes-using-amazon-transcribe-and-amazon-comprehend/
- AWS CloudFormation Masterclass https://www.youtube.com/watch?v=6R44BADNJA8
- https://aws.amazon.com/solutions/ which are vetted technical reference architectures. Like quickstarts.
- Here is a full serverless backend and front end app in github, https://github.com/aws-samples/aws-serverless-airline-booking and the full build on Twitch https://pages.awscloud.com/GLOBAL-devstrategy-OE-BuildOnServerless-2019-reg-event.html
Best Practice
- Amazon S3 Path Deprecation Plan – The Rest of the Story https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/
- AWS Answers provides prescriptive guidance or how tos. For example networking guidance as per https://aws.amazon.com/answers/networking/
- AWS Solutions provides reference architectures as IaaC. You can build them, reverse engineer them and reuse the code for your own solutions. https://aws.amazon.com/solutions/
- Centralized Logging allows you to monitor traffic across multiple accounts with a Kibana Dashboard fronting Amazon Elastisearch and serverless components in just a few minutes. https://aws.amazon.com/solutions/centralized-logging/?trk=sl_card
- Custom scheduler for EC2 and RDS is the AWS Instance Scheduler solution https://aws.amazon.com/solutions/instance-scheduler/?trk=sl_card
- Become familiar with the best practice for each AWS Service. AWS Lambda is a serverless compute environment that does in fact run in a container, on a Firecracker microVM which in turn runs on AWS compute infrastructure. Lambda cold starts can impact your applications and this can be particularly noticeable with Java runtimes. This link has a good explanation of Lambda cold starts and how you can mitigate them. https://aws.amazon.com/blogs/opensource/java-apis-aws-lambda/
- Converting monolithic Java applications to microservices
- Check out the blog titled Deploying Java Microservices on Amazon Elastic Container Service https://aws.amazon.com/blogs/compute/deploying-java-microservices-on-amazon-ec2-container-service/ . This example also includes many different branches with graphql and angular front ends. https://spring-petclinic.github.io/docs/forks.html
- Re-Writing a Mainframe Software Package to Java on AWS with Ippon Technologies https://aws.amazon.com/blogs/apn/re-writing-a-mainframe-software-package-to-java-on-aws-with-ippon-technologies/
- From Framework to Function: Deploying AWS Lambda Functions for Java 8 using Apache Maven Archetype https://aws.amazon.com/blogs/compute/from-framework-to-function-deploying-aws-lambda-functions-for-java-8-using-apache-maven-archetype/
- Deploying Java Microservices on Amazon Elastic Container Service https://aws.amazon.com/blogs/compute/deploying-java-microservices-on-amazon-ec2-container-service/
- A great way to understand how Cloudformation can build, update and delete immutable or mutable environments is to reverse engineer AWS Quickstarts (gold standard reference architectures). Check out https://aws.amazon.com/quickstart/saas/identity-with-cognito/ for the deployment guide and https://github.com/aws-quickstart/saas-identity-cognito for all the Cloudformation templates.
Migration Best Practice
-
Migrating to AWS - Best Practices and Strategies is a good starting point for execs and planners https://d1.awsstatic.com/Migration/migrating-to-aws-ebook.pdf
-
AWS Cloud Adoption Framework (CAF) https://aws.amazon.com/professional-services/CAF/
-
AWS Cloud Adoption Readiness Tool (CART) https://cloudreadiness.amazonaws.com/#/cart
-
AWS Server Migration Service requirements https://docs.aws.amazon.com/server-migration-service/latest/userguide/prereqs.html
-
Migrating to AWS https://aws.amazon.com/cloud-migration/
-
Cloud stages of adoption in the AWS blog titled Cloud Transformation Maturity Model: Guidelines to Develop Effective Strategies for Your Cloud Adoption Journey https://aws.amazon.com/blogs/publicsector/cloud-adoption-maturity-model-guidelines-to-develop-effective-strategies-for-your-cloud-adoption-journey/
-
Stephen Orban’s 2017 post on how Capital One journeyed through the Cloud stages of adoption titled Capital One’s Cloud Journey Through the Stages of Adoption https://medium.com/aws-enterprise-collection/capital-ones-cloud-journey-through-the-stages-of-adoption-bb0895d7772c
-
Check out the AWS Migration Hub https://aws.amazon.com/migration-hub/ and related tooling to support your Migrations
- AWS Database Migration Service Best Practices https://docs.aws.amazon.com/dms/latest/userguide/dms-ug.pdf#CHAP_BestPractices
- Getting Started with the Migration Hub https://docs.aws.amazon.com/migrationhub/latest/ug/getting-started.html
Multiple Account Best Practice and Examples
-
How about multi region CICD. Using AWS CodePipeline to Perform Multi-Region Deployments https://aws.amazon.com/blogs/devops/using-aws-codepipeline-to-perform-multi-region-deployments/
-
Multi cloud guardrails and IaaC using Turbot. https://turbot.com/features/ This team grew from the Johnson and Johnson xbot HPC orchestration and guardrail service. The original xbot technology is circa 2015 https://www.youtube.com/watch?v=za1EysyUVS0
-
Distributed Load Testing Using Fargate https://github.com/aws-samples/distributed-load-testing-using-aws-fargate Solution to setup AWS Fargate to run multi-region distributed performance testing. Runs Distributed Load Tests using AWS Fargate and Taurus. You can use it to test your services under high stress scenarios and understand it’s behavior and scalability.
-
AWS Landing Zone is designed to quickly set up a secure, multi-account AWS environment based on AWS best practiceshttps://aws.amazon.com/solutions/aws-landing-zone/?did=sl_card&trk=sl_card
-
Enabling self-service provisioning of AWS resources with AWS Control Tower https://aws.amazon.com/blogs/mt/enabling-self-service-provisioning-of-aws-resources-with-aws-control-tower/ . This solution uses the following AWS services:
- AWS Control Tower
- AWS Service Catalog
- AWS CloudFormation
- Amazon CloudWatch
- Amazon RDS
- AWS Organizations
-
AWS Service Catalog Integration with AWS Budgets. This integration lets you manage cost and manage centralized deployment assets. https://aws.amazon.com/blogs/aws-cost-management/launch-aws-service-catalog-integration-with-aws-budgets/
-
Automate account creation, and resource provisioning using AWS Service Catalog, AWS Organizations, and AWS Lambda https://aws.amazon.com/blogs/mt/automate-account-creation-and-resource-provisioning-using-aws-service-catalog-aws-organizations-and-aws-lambda/
Compute links
- Is AMI like Sysprep? https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ami-create-standard.html
- Debug and run Lambda Python locally https://medium.com/@bezdelev/how-to-test-a-python-aws-lambda-function-locally-with-pycharm-run-configurations-6de8efc4b206 This is another option for testing serverless. Also enable AWS Xray on your Lambdas and you can trace what is happening over the life of the Lambda.
- Amazon EC2 Spot introduces new pricing model and the ability to launch Spot instances via RunInstances API https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-ec2-spot-introduces-new-pricing-model-and-the-ability-to-launch-new-spot-instances-via-runinstances-api/
- EC2 Auto Recovery
- This non AWS video might help https://www.youtube.com/watch?v=hea5q_XYsIg
- https://www.slideshare.net/AmazonWebServices/deep-dive-amazon-ec2
Containers
- A deep dive on Fargate. Lot’s of feature updates around container orchestration so watch the AWS what’s new for the latest. Here’s a slide share that deep dives on Fargate. https://de.slideshare.net/AmazonWebServices/deep-dive-into-aws-fargate
- The Illustrated Children’s Guide to Kubernetes https://www.youtube.com/watch?v=4ht22ReBjno&feature=youtu.be
- Configuring Cloudwatch logs with containers. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_cloudwatch_logs.html
- ECS and Fargate blue green cloudformation templates (from the blog post) https://github.com/aws-samples/ecs-blue-green-deployment
- Deep dive into Kubernetes tooling and EKS with 2+4+4 hours (total 10 hours) of self paced DevOps deep dive into Kubernetes the AWSome Way! https://github.com/aws-samples/aws-workshop-for-kubernetes
- Here’s a much shorter EKS introduction https://github.com/aws-samples/eks-workshop
- Firecracker is next generation microvm open source technology designed to support serverless and container based workloads. AWS Lambda and AWS Fargate run on Firecracker. 5MB microvm that launches in less than 125 ms and has robust, secure and efficient control plane, data plane https://github.com/firecracker-microvm/firecracker
- Follow the AWS Container roadmap https://github.com/aws/containers-roadmap/projects/1?card_filter_query=eks
AWS has made available several product and service roadmaps for public consumption. We even mention our desire to provide beta endpoints for customers to trial new features. Check these out and keep an eye on them as we are constantly shipping new features:
- AWS Container Service Roadmap https://github.com/aws/containers-roadmap
- AWS App Mesh Service Roadmap https://github.com/aws/aws-app-mesh-roadmap/projects/1
Database and Storage links
- 24 Jul 2017 S3 Rate Request Performance Increase announcement https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ and notice the exponetial scaling possible with multiple prefixes. https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html but if using sse-kms this service will a limiting factor. https://docs.aws.amazon.com/kms/latest/developerguide/limits.html#requests-per-second-table
- S3 Transfer Acceleration Speed Checker http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html uses a multi part upload to check the speed difference when using S3 transfer acceleration between regions.
- S3 Deep Dive Mar 2017 https://www.slideshare.net/AmazonWebServices/deep-dive-on-amazon-s3-march-2017-aws-online-tech-talks
- Automated RDS failover if you enable Multi AZ. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html
- S3 bucket policy examples https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-8
- Amazon RDS Now Supports Database Storage Size up to 16TB and Faster Scaling for MySQL, MariaDB, Oracle, and PostgreSQL Engines (22 Nov 2017) https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-rds-now-supports-database-storage-size-up-to-16tb-and-faster-scaling-for-mysql-mariadb-oracle-and-postgresql-engines/
- S3 Puts: Under the section “Q: How will I be charged and billed for my use of Amazon S3?” in FAQS: https://aws.amazon.com/s3/faqs/ and in detail at https://aws.amazon.com/s3/pricing/ Request Example: Assume you transfer 10,000 files into Amazon S3 and transfer 20,000 files out of Amazon S3 each day during the month of March. Then, you delete 5,000 files on March 31st. Total PUT requests = 10,000 requests x 31 days = 310,000 requests
- Deep Dive on EBS Snapshots https://www.youtube.com/watch?v=TUJCQRejA28
- Looks like they started allowing S3 SSE with customer provided keys (SSE-C) in 2014 https://aws.amazon.com/about-aws/whats-new/2014/06/12/amazon-s3-now-supports-server-side-encryption-with-customer-provided-keys-sse-c/
- DynamoDB deep dive from ReInvent 2016 https://www.youtube.com/watch?v=bCW3lhsJKfw
- Determining volume IO performance https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html
- RDS Performance benchmarks on percona https://d0.awsstatic.com/product-marketing/Aurora/RDS_Aurora_Performance_Assessment_Benchmarking_v1-2.pdf
- RDS Deep Dive from ReInvent 2017. Watch to gain an appreciation of how RDS works https://www.youtube.com/watch?v=TJxC-B9Q9tQ
- Best Practices for Running Oracle Database on Amazon Web Services (Jan 2018) https://d0.awsstatic.com/whitepapers/best-practices-for-running-oracle-database-on-aws.pdf Also review the links in the appendix which dive deeper into running Oracle workloads on EC2 and advanced archictures for running Oracle databases on AWS. https://d0.awsstatic.com/enterprise-marketing/Oracle/AWSAdvancedArchitecturesforOracleDBonEC2.pdf
- Deep Dive on Amazon Neptune (circa Jan 2018) https://www.slideshare.net/AmazonWebServices/deep-dive-on-amazon-neptune-aws-online-tech-talks Look for updates at ReInvent
- Moving a Galaxy into the Cloud. Samsung’s experience migrating from Cassandra to DynamoDB with big cost savings and at very large scale. https://www.youtube.com/watch?v=Z-2UIrI9feQ
Network links
- Private routable CIDR ranges as per RFC 1918 https://en.wikipedia.org/wiki/Private_network ENAS:
- https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/enhanced-networking.html Elastic Network Adapter (ENA) The Elastic Network Adapter (ENA) supports network speeds of up to 25 Gbps for supported instance types. C5, F1, G3, H1, I3, m4.16xlarge, M5, P2, P3, R4, and X1 instances use the Elastic Network Adapter for enhanced networking.
- and the original Nov 2016 ENA announcement. https://aws.amazon.com/blogs/aws/elastic-network-adapter-high-performance-network-interface-for-amazon-ec2/
- NACLs for subnets are configurable. Rules are evaluated from top to bottom with the final rule (immutable) of deny all. See https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html for examples and also https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html
- The NAT Gateway is simple to create and use. Just create the NAT Gateway and update your route table to direct all 0.0.0.0/0 traffic to the UID of the NAT Gateway. AWS looks after the rest. Another fully managed service. https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html
- OSI model is used to describe the 7 layers of our networks. https://en.wikipedia.org/wiki/OSI_model
- Privatelink intro book https://aws.amazon.com/privatelink/
- VPC Summary including private endpoints supported and alternative privatelink endpoint support. https://aws.amazon.com/vpc/
- Privately connect to AWS Services without using an Internet gateway, NAT or firewall proxy through a VPC Endpoint. Available AWS services include S3, DynamoDB, Kinesis Streams, Service Catalog, EC2 Systems Manager (SSM), Elastic Load Balancing (ELB) API, Amazon Elastic Compute Cloud (EC2) API, and SNS.
- DX security concerns (Advanced Networking guide ppgs 255 to 290)
- Third party options for encryption over Direct Connect https://supportforums.cisco.com/legacyfs/online/csr-secure-directconnect-2014110501.pdf and https://aws.amazon.com/quickstart/architecture/aviatrix-user-vpn/
- Or VPN over AWS
- VPC Options for Resizing https://aws.amazon.com/about-aws/whats-new/2017/08/amazon-virtual-private-cloud-vpc-now-allows-customers-to-expand-their-existing-vpcs/ and https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html#vpc-resize
- Webinar slides with Route 53 regional failover examples including ELB, EC2 instance and fixed IP. https://www.slideshare.net/AmazonWebServices/webinar-route53-dnsfailoverfinal and here is a description of using Route53 to use latency based domains and weighted domains for multi region failover. https://www.sumologic.com/blog/amazon-web-services/aws-route-53-global-load-balancing/
- Using Route 53 private hosted zones across VPCs in a region. aws route53 create-vpc-association-authorization –hosted-zone-id ZONEID –vpc VPCRegion=land-of-oz,VPCId=vpc-xxxxxxx . NOTE: from the AWS Service Limits Amazon VPCs that you can associate with a private hosted zone is limited to 100. There is some additional hints, not called out in the AWS documentation at https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.boto3_route53.html. Specifically around associations within a single account and multiple accounts. Both the VPC and Private hosted zone must exist before creating the association. There are a number of 4xx errors called out in the documentation.
- How to evaluate custom NACLs https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html
- Operating Reverse Proxies and alternative methods of protecting backend services in AWS:
- Things to Consider When You Build REST APIs with Amazon API Gateway https://aws.amazon.com/blogs/architecture/things-to-consider-when-you-build-rest-apis-with-amazon-api-gateway/ . This post describes serverless architectures and loosely coupled patterns for Rest APIs. Look out for the GraphQL followup.
- Ten Things Serverless Architects Should Know https://aws.amazon.com/blogs/architecture/ten-things-serverless-architects-should-know/ . This blog post contains serverless and end to end security best practices. Also included are many great dive deep links.
- How to Design Your Serverless Apps for Massive Scale https://aws.amazon.com/blogs/architecture/how-to-design-your-serverless-apps-for-massive-scale/ . This post focusses on high volume loosely coupled serverless design patterns.
- Deploying an NGINX Reverse Proxy Sidecar Container on Amazon ECS https://aws.amazon.com/blogs/compute/nginx-reverse-proxy-sidecar-container-on-amazon-ecs/ . This post presents a more traditional reverse proxy implementation. This approach is not optimized for security in depth and also does not sufficiently provide security in depth by leveraging the wider AWS support for dealing with attack traffic and anomalies closer to the edge. Here is an example of questions posed online on this topic Is there a cloud-based reverse proxy solution in AWS? https://serverfault.com/questions/709806/is-there-a-cloud-based-reverse-proxy-solution-in-aws . Note the reference to Amazon API Gateway as an alternative.
- Check out the AWS and Partner solutions categorized as Security at the AWS Solutions reference architectures and POC page https://aws.amazon.com/solutions/?solutions-all.sort-by=item.additionalFields.sortDate&solutions-all.sort-order=desc&awsf.AWS-Product%20Category=categories%23security-identity-compliance
- Visit the AWS Architecture centre, https://aws.amazon.com/architecture , and dive deep. Check out the Finance whitepaper at https://aws.amazon.com/whitepapers/?whitepapers-main.sort-by=item.additionalFields.sortDate&whitepapers-main.sort-order=desc&awsf.whitepapers-content-type=*all&awsf.whitepapers-category=categories%23security-identity-compliance
Big Data Links
Migrating Relational Data to RDS and Redshift
- How to migrate a large data warehouse from IBM Netezza to Amazon Redshift with no downtime https://aws.amazon.com/blogs/big-data/how-to-migrate-from-ibm-netezza-to-amazon-redshift-with-no-downtime/
- Automate Amazon Redshift cluster creation using AWS CloudFormation https://aws.amazon.com/blogs/big-data/automate-amazon-redshift-cluster-creation-using-aws-cloudformation/ This blog post contains two Cloudformation launch buttons to build the data warehouse and management tools
- Orchestrate an ETL process using AWS Step Functions for Amazon Redshift https://aws.amazon.com/blogs/big-data/orchestrating-an-etl-process-using-aws-step-functions-for-amazon-redshift/ . This blog post also builds out the ETL environment.
- How to enable cross-account Amazon Redshift COPY and Redshift Spectrum query for AWS KMS–encrypted data in Amazon S3 https://aws.amazon.com/blogs/big-data/how-to-enable-cross-account-amazon-redshift-copy-and-redshift-spectrum-query-for-aws-kms-encrypted-data-in-amazon-s3/ . This blog post walks through configuring this capability in the management console.
- Migrate RDBMS or On-Premise data to EMR Hive, S3, and Amazon Redshift using EMR – Sqoop https://aws.amazon.com/blogs/big-data/migrate-rdbms-or-on-premise-data-to-emr-hive-s3-and-amazon-redshift-using-emr-sqoop/ . This blog post provides 3 scenarios for migrating from a relational database using EMR for custom ETL.
- Get sub-second query response times with Amazon Redshift result caching https://aws.amazon.com/blogs/big-data/get-sub-second-query-response-times-with-amazon-redshift-result-caching/
Security links
- AWS Compliance mapping to services https://aws.amazon.com/compliance/services-in-scope/
- S3 Access Control Lists (ACLs) explains how permissions are or can be applied to S3 buckets. This is a tedious read but worth while for anyone interested in simple permission management of cross account access or large number of accounts and say log consolidation to one account or bucket.
- Educate our customers and show them how to use services like Well Architected, Trusted Advisor, Inspector, Macie, Shield, WAF, Partner tooling, etc to get secure. Make sure your customers are fully conversant and implementing our guidance from https://aws.amazon.com/whitepapers/#essentials and get them to audit their use of our services as per https://d1.awsstatic.com/whitepapers/compliance/AWS_Auditing_Security_Checklist.pdf .
- We also have people reviewing our services for the upcoming GDPR legislation that will come into effect in Europe in May 2018. Perhaps we could have an update on what that impact will be. (positive for issues like this from my brief conversations)
- As Werner said ‘dance like nobody is watching and secure like everybody is watching’ [sic].
- S3 permissions can be on buckets, bucket contents and applied to objects say at upload. https://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html for a deeper dive.
- F5 WAF git hub https://github.com/f5devcentral/f5-aws-autoscale/tree/master/deployments/waf-sandwich-utility-only-immutable compare appliance based waf sandwiches to using native AWS services https://f5.com/resources/white-papers/load-balancing-101-firewall-sandwiches
- With NACLs (optional stateless firewall for a subnet boundary) rules are evaluated from lowest to highest. As soon as a match is found it is applied. https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html
- If you search the web for ‘aws deep dive’ AND sa ‘security’ you’ll find some great videos and slide decks from ReInvent, our public bootcamps and from many of AWS SMEs. Here’s one on security goverance https://www.youtube.com/watch?v=xjtSWd8z_bE and here’s another on a service GuardDuty. https://www.youtube.com/watch?v=o2YaIsps5LY
- A useful article on using parameter store to store secrets. https://aws.amazon.com/blogs/mt/the-right-way-to-store-secrets-using-parameter-store/
- Apple are publicaly mentioning their use of S3 in https://images.apple.com/business/docs/iOS_Security_Guide.pdf
- Security Assessments on Github (also AWS Services like Inspector too)
- https://github.com/awslabs/aws-security-benchmark/blob/master/aws_cis_foundation_framework/CIS_Amazon_Web_Services_Foundations_Benchmark_v1.1.0.pdf
- https://github.com/Alfresco/prowler
- Netflix Security Monkey. https://github.com/Netflix/security_monkey
- Lambda script to install the SSM agent https://github.com/awslabs/amazon-inspector-agent-autodeploy
- Inspector blog post https://aws.amazon.com/blogs/aws/scale-your-security-vulnerability-testing-with-amazon-inspector/
- Use Inspector to assess the NIST Quickstart for vulnerabilities -* https://docs.aws.amazon.com/inspector/latest/userguide/inspector_quickstart.html -* Install the Inspector agent. https://docs.aws.amazon.com/inspector/latest/userguide/inspector_installing-uninstalling-agents.html
- IAM Ninja and Deep Dives from ReInvents
- IAM Policy Ninja (300ish level) https://www.youtube.com/watch?v=aISWoPf_XNE
- Here is an IAM talk from ReInvent 2016 https://www.slideshare.net/AmazonWebServices/aws-reinvent-2016-iam-best-practices-to-live-by-sac317
- Multiple Account Deep Dives
- AWS re:Invent 2016: NEW SERVICE: Manage Multiple AWS Accounts with AWS Organizations (SAC323) https://www.youtube.com/watch?v=Oeb7PDyiT2A
- AWS re:Invent 2017: Architecting Security and Governance Across a Multi-Account Stra (SID331) https://www.youtube.com/watch?v=71fD8Oenwxc
- Neat explanation with graphics of signing of urls https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
Monitoring
- Other than VPC Flow Logs what other monitoring options are available on AWS?
- Amazon GuardDuty analyzes AWS CloudTrail, VPC Flow Logs, and AWS DNS logs. The service is optimized to consume large volumes of data for near real-time processing of security detections. GuardDuty gives you access to built-in detection techniques that are developed and optimized for the cloud and maintained and continuously improved upon by AWS Security. Amazon GuardDuty pulls independent streams of data directly from AWS CloudTrail, VPC Flow Logs, and AWS DNS logs. You don’t have to manage Amazon S3 bucket policies or modify the way you may collect and store your logs. GuardDuty permissions are managed as Service Linked Roles that you can revoke at any time by disabling GuardDuty. This makes it easy to enable the service without complex configuration and it eliminates the risk that an AWS IAM permission modification or S3 bucket policy change will affect the operation of the service. It also makes GuardDuty extremely efficient at consuming high-volumes of data in near real-time without affecting the performance or availability of your account or workloads. Also no performance impact. https://aws.amazon.com/guardduty/faqs/
- VPC Flow Log limitations https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html#flow-logs-limitations
- Cloudwatch logs also allows you to store and trigger events from application, service and custom logs. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html
- Cloudwatch logs metrics at varying intervals. Basis is 5 minute, detailed and basic custom metric is 1 minute and high resolution custom metrics (July 2017) is 1 sec. Refer to https://aws.amazon.com/cloudwatch/faqs/ for the varying retention windows for each logging type.
Visualize your AWS environment
- Visualize VPC Flow Logs using an ELK stack approach https://aws.amazon.com/blogs/security/how-to-optimize-and-visualize-your-security-groups/
- www.dome9.com gives most AWS employees their free DevOps tier. Great for visualizing, auditing and checking for compliance across an account or accounts. If you build the NIST QuickStart in your account you can see lot’s of cool outputs from dome9. Get the NIST QuickStart at https://aws.amazon.com/quickstart/architecture/accelerator-nist/
- Visualize Cloudtrail logs using Glue and Quicksight https://aws.amazon.com/blogs/big-data/streamline-aws-cloudtrail-log-visualization-using-aws-glue-and-amazon-quicksight/
Self paced Learning and Building
- AWS Certification roadmap https://aws.amazon.com/certification/ Check out the learning paths link at the bottom of the page.
- Read the service FAQ pages, http://aws.amazon.com/faqs/, and documentation for each of the services. Just search for AWS + + documentation in any search engine. You can keep the documentation as pdf, html online or even in your Kindle. You can also git clone the documentation for most services.
- Find and build interesting AWS and partner solutions you find the in AWS Blog https://aws.amazon.com/blogs/ . Any post you find with a yellow launch button will build that solution using Cloudformation.
- AWS free digital training is mostly 100 level but we also have over 40 hours of Machine Learning training available for free. You can search by topic, role or level. https://www.aws.training/LearningLibrary?src=courses You’ll find specialist deep dives from level 100 through 300 like this video describing the differences between NACLs and Security groups. https://www.aws.training/Details/Video?id=16486 NOTE: You’ll need to enroll and allow popups in your browser.
- You can also take AWS Qwiklabs Labs for free at https://aws.amazon.com/training/self-paced-labs/
- Get a sandbox or personal account. There are free tiers for many services. https://aws.amazon.com/free/
- http://run.qwiklabs.com and complete quests and labs. These enhance your familiarity with AWS services without you having to use your own account. Some labs are free. Others will require you to redeem Qwiklab credits. Reach out to your training manager or AWS account manager. Also check out the Exam guides for SA, SysOps and Advanced Networking https://www.amazon.com/Certified-Advanced-Networking-Official-Study/dp/1119439833/ref=sr_1_1?s=books&ie=UTF8&qid=1519925473&sr=1-1&keywords=advanced+networking
- Search github, https://github.com/aws , and the AWS blogs, https://aws.amazon.com/blogs/ , for solutions that interest you. Look for posts with a launch button. These will build a complete environment using Cloudformation. Retrieve the Cloudformation templates either from the built environment in your account or from Github. You can reverse engineer or use these templates as scaffolds for your own use.
- Visit Stackoverflow and the AWS discussion forum to pose questions or to contribute to answers about AWS
- You can also take a number of AWS MOOCs (Massive Open Online Courses) on EDx and Coursera including:
- There are many other self paced labs and solutions you can build on AWS. Try:
- Build a Serverless Web Application https://aws.amazon.com/getting-started/projects/build-serverless-web-app-lambda-apigateway-s3-dynamodb-cognito/
- How about AWS Developer Center https://aws.amazon.com/developer/ where you can build the Mythical Misfits app in your choice of programming language.
- The AWS Podcast has a monthly update which is a great way to keep up with the latest changes, releases and interviews with domain experts https://aws.amazon.com/podcasts/aws-podcast/
- AWS has released a number of webinars and now has a monthly cadence https://aws.amazon.com/about-aws/events/monthlywebinarseries/
- AWS Answers is now available to the public. It contains some interesting links. https://aws.amazon.com/answers/
- Get to know your AWS Solution Architects and your Technical Account Manager (TAM). The SAs help you to architect and understand best practice. The TAMs provide support for your applications running on AWS. They can help you prepare for major events like testing and scaling. They can also help troubleshoot and provide visibility into AWS infrastructure metrics for troubleshooting. https://aws.amazon.com/premiumsupport/faqs/
- AWS Glossary contains service names and nomenclature https://docs.aws.amazon.com/general/latest/gr/glos-chap.html
- Now go build stuff…
Continue reading articles in my Amazon Web Services series
- Data Warehousing on AWS
- Migrating to AWS
- AWS Business Essentials
- IAM Demo
- Architecting on AWS
- SysOps on AWS
- S3 Demo
- Predict the Future
- AWS Tech Essentials
- Developing on AWS
- DevOps on AWS
- Advanced Architecting on AWS
- Big Data on AWS
- AWS Deep Dive Toolbox
- Security Engineering on AWS
- Deep Learning on AWS
- AWS List of Services
- Networking on Aws
- AWS Data and Analytics
- Microsoft Immersion Day
- Adelaide Deep Racer Hints and Tips
- Deep Racer Awards
- Windows on AWS
- AWS Ask Me Anything
- Cloudwatch and Systems Manager Workshop
- Containers Immersion Day
- Redshift Immersion Day
- Innovation in Ambiguity
- AWS Contingency Planning
- AWS CLI Examples
- Migrating to Cloud in 2023
- Chaos Engineering Workshop
- Chatgpt Friend or Foe