AWS Technical Essentials - Tips and Tricks

This is my list of hints and tips for this course. It’s markdown so you can save it, access it or store it anywhere. I might also give you other links that are course specific. I’ll add specific answers to questions I get during the course. I’ll share it with everyone.

Your Instructor

• Ian Falconer https://www.linkedin.com/in/leftbrainstuff/

Cool links

• AWS Global Infrastructure. Here are several videos that ‘open the kimono’ on how AWS is designed and built to support millions of customers across the globe.

  • James Hamilton, AWS SVP and Distinguished Engineer, talks about the design decisions and inner workings of the AWS global infrastructure. James also provides the history behind major technological innovations like we’re seeing now in Cloud Computing. This deck is over 4 years old but still a good summary. This should be the first AWS video you watch. https://www.slideshare.net/AmazonWebServices/spot301-aws-innovation-at-scale-aws-reinvent-2014 . Here’s the youtube video. https://www.youtube.com/watch?v=JIQETrFC_SQ There are Youtube videos from more recent ReInvents with some updates too. Here is James in 2016. It’s titled as AWS re:Invent 2016: Amazon Global Network Overview with James Hamilton https://www.youtube.com/watch?v=uj7Ting6Ckk
  • Here is a 4 min snippet from 2016 titled AWS re:Invent 2016: Introduction to Amazon Global Network and CloudFront PoPs with James Hamilton https://www.youtube.com/watch?v=FjHBGjLnou0&feature=youtu.be
  • AWS re:Invent 2017 Keynote - Tuesday Night Live with Peter DeSantis, VP AWS Global Infrastructure talks about the AWS global infrastructure. Up to 15:46 minutes is about the infrastructure. https://www.youtube.com/watch?v=dfEcd3zqPOA&feature=youtu.be&t=1h17m0s
  • https://www.infrastructure.aws/ now has an interactive map and animations describing the AWS Global Infrastructure. 100 GBps intercontinental network.
  • This one is self explanatory AWS re:Invent 2018: Amazon VPC: Security at the Speed Of Light (NET313) https://www.youtube.com/watch?v=UP7wDBjZ37o&feature=youtu.be

• AWS re:Invent 2017: Scaling Up to Your First 10 Million Users (ARC201). This is like the Tech Essentials course in a single video. Well worth a watch. https://www.youtube.com/watch?v=w95murBkYmU

• AWS re:Invent 2016: Metering Big Data at AWS: From 0 to 100 Million Records in 1 Second (ARC308) another journey through choosing AWS services to build high scale near real time data management systems. https://www.youtube.com/watch?v=dD1wzEdQCb4

• Amazon EC2 Instance Types explained in neat tabular comparisons. https://aws.amazon.com/ec2/instance-types/ . Also here’s a third party site that has a table that lets you sort on memory, network performance, cost and instance type. You can also quickly compare costs here too. https://ec2instances.info/

• An external post about S3 data leaks. https://www.bleepingcomputer.com/news/security/amazon-aws-servers-might-soon-be-held-for-ransom-similar-to-mongodb/. Now consider features like AWS Organization and Service Control Policies and S3 object locking for implementing rule of least privilege security controls.

• Latency between AWS regions. Lot’s of good empirical data points. Note these are 24 hour averages of hourly averages. 95th percentile (or similar) values would be needed for real time troubleshooting. https://www.cloudping.co/

• Latency http://highscalability.com/latency-everywhere-and-it-costs-you-sales-how-crush-it

• List of CIDR ranges of AWS regions http://ec2-reachability.amazonaws.com/ which show you relative capacity between regions.

• How Amazon handles prime day (DynamoDB) https://www.youtube.com/watch?v=83-IWlvJ__8

• Chalice python based microservices framework using Lambda, API Gateway and IAM. http://chalice.readthedocs.io/en/latest/ Very fast, very lightweight and very extensible. Also look at SAM, the AWS Serverless Application Model and AWS Amplify.

• Benchmark tests of EC2 versus other bare metal and cloud servers. https://www.phoronix.com/scan.php?page=article&item=cloud-cpu-36#=1

• Here’s a Lambda deep dive which more clearly explains some of the questions around managing state, retries, testing and handling large data sets with Lambda. https://www.youtube.com/watch?v=dB4zJk_fqrU

• Mechanical sympathy. Interesting concept for software defined assets but aligns with Werner Vogel’s catchy advice that ’everything fails all the time’. Therefore you need to architect resilient systems. http://infrastructure-as-code.com/book/2015/03/23/mechanical-sympathy.html

• AWS Open Guide on GitHub is a good summary of AWS Documentation and an attempt to summarize the wealth of AWS documentation on a single page. https://github.com/open-guides/ AWS has also open sourced all our documentation at https://github.com/awsdocs . The ENTRY page for all aws documentation at https://docs.aws.amazon.com/index.html#lang/en_us`

• You can check the overall health and availability of AWS globally at the Service Health Dashboard (SHD) https://status.aws.amazon.com/ You can also use the AWS Health API to programatically check for service health at https://docs.aws.amazon.com/health/latest/ug/getting-started-api.html

• Netflix have some cool tools that they’ve open sourced. https://netflix.github.io/

• All quickstarts now in github https://github.com/aws-quickstart and also check out the AWS solutions https://aws.amazon.com/solutions/ which are vetted technical reference architectures.

  • Consider building load and performance test simulators with IaaC. Like IoT Device Simulator https://aws.amazon.com/solutions/iot-device-simulator/?trk=sl_card
  • AWS Service Catalog Validation Pipeline https://aws.amazon.com/solutions/aws-service-catalog-validation-pipeline/?trk=sl_card
  • Blue green quickstart at https://aws.amazon.com/quickstart/architecture/blue-green-deployment/

• Netflix Open Sourcing of many useful and interesting tools for running large AWS cloud environments. https://netflix.github.io/

• Researchers say Data61-backed blockchain platform delivers scalability, energy efficiency 1000 ec2 instances, 14 AWS Regions and 30k TPS https://www.computerworld.com.au/article/647265/researchers-say-data61-backed-blockchain-platform-delivers-scalability-energy-efficiency/

• Adrian Cockcroft AWS VP of Cloud Architecture Strategy and former CTO of Netflix. Here’s his Youtube playlist with talks about DevOps, migrations, Netflix lessons learned and digital transformation topics. https://www.youtube.com/playlist?list=PL_KXMLr8jNTnwkzV7SePa0jHFUG2qn0MA

• The Network is Reliable and other fallacies. Key performance and reliability concerns of distributed systems. https://blog.acolyer.org/2014/12/18/the-network-is-reliable/ . Also web search for Werner Vogels, Amazon CTO, who is acknowledged as a global expert on distributed systems.

• Architecture reviews are important. The cloud design principles, here is the 2011 AWS Whitepaper https://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf and the Well Architected Review are key inputs. https://aws.amazon.com/architecture/well-architected/

• AWS General Reference. https://docs.aws.amazon.com/general/latest/gr/Welcome.html This document is a key reference when architecting and designing AWS solutions. Be familiar with service limits.

• My favourite AWS blog post. Transcribe and Amazon Comprehend https://aws.amazon.com/blogs/machine-learning/discovering-and-indexing-podcast-episodes-using-amazon-transcribe-and-amazon-comprehend/

• AWS CloudFormation Masterclass https://www.youtube.com/watch?v=6R44BADNJA8

• https://aws.amazon.com/solutions/ which are vetted technical reference architectures. Like quickstarts.

• Here is a full serverless backend and front end app in github, https://github.com/aws-samples/aws-serverless-airline-booking and the full build on Twitch https://pages.awscloud.com/GLOBAL-devstrategy-OE-BuildOnServerless-2019-reg-event.html

Migrating to AWS

• AWS Cloud Adoption Framework (CAF) https://aws.amazon.com/professional-services/CAF/
• AWS Cloud Adoption Readiness Tool (CART) https://cloudreadiness.amazonaws.com/#/cart
• AWS Server Migration Service requirements https://docs.aws.amazon.com/server-migration-service/latest/userguide/prereqs.html
• Migrating to AWS https://aws.amazon.com/cloud-migration/
• Cloud stages of adoption in the AWS blog titled Cloud Transformation Maturity Model: Guidelines to Develop Effective Strategies for Your Cloud Adoption Journey https://aws.amazon.com/blogs/publicsector/cloud-adoption-maturity-model-guidelines-to-develop-effective-strategies-for-your-cloud-adoption-journey/
• Stephen Orban’s 2017 post on how Capital One journeyed through the Cloud stages of adoption titled Capital One’s Cloud Journey Through the Stages of Adoption https://medium.com/aws-enterprise-collection/capital-ones-cloud-journey-through-the-stages-of-adoption-bb0895d7772c

More Links to AWS Resources to help you Build

• Centralized Logging – AWS Answers | https://aws.amazon.com/answers/logging/centralized-logging/
• AWS Developer Forums: Discussion Forums | https://forums.aws.amazon.com/index.jspa
• Amazon Web Services - Labs · GitHub | https://github.com/awslabs
• GitHub - awslabs/aws-shell: An integrated shell for working with the AWS CLI. | https://github.com/awslabs/aws-shell
• Region Table | https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/
• AWS Regions and Endpoints - Amazon Web Services | https://docs.aws.amazon.com/general/latest/gr/rande.html
• AWS Service Limits - Amazon Web Services | https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html
• AWS IP Address Ranges - Amazon Web Services | https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html
• Error Retries and Exponential Backoff in AWS - Amazon Web Services | https://docs.aws.amazon.com/general/latest/gr/api-retries.html
• Cloud Solutions by Application - Amazon Web Services (AWS) | https://aws.amazon.com/solutions/
• AWS - Application Architecture Center | https://aws.amazon.com/architecture/
• AWS Simple Icons | https://aws.amazon.com/architecture/icons/
• Compliance Programs - Amazon Web Services (AWS) | https://aws.amazon.com/compliance/programs/
• Case Studies & Customer Success - Amazon Web Services (AWS) | https://aws.amazon.com/solutions/case-studies
• AWS Certification - AWS Cloud Computing Certification Program | https://aws.amazon.com/certification/
• Check out the architecture lens that describe best practice that is defined around the AWS Well Architected program. https://aws.amazon.com/architecture/well-architected/ Also check out AWS solutions https://aws.amazon.com/architecture/well-architected/

Compute links

• Amazon EC2 Spot introduces new pricing model and the ability to launch Spot instances via RunInstances API https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-ec2-spot-introduces-new-pricing-model-and-the-ability-to-launch-new-spot-instances-via-runinstances-api/

• EC2 Auto Recovery

  • This non AWS video might help https://www.youtube.com/watch?v=hea5q_XYsIg
  • https://www.slideshare.net/AmazonWebServices/deep-dive-amazon-ec2

• AWS provides SLAs for most compute related services. Details at https://aws.amazon.com/compute/sla/

Containers

• A deep dive on Fargate. Lot’s of feature updates around container orchestration so watch the AWS what’s new for the latest. Here’s a slide share that deep dives on Fargate. https://de.slideshare.net/AmazonWebServices/deep-dive-into-aws-fargate
• Configuring Cloudwatch logs with containers. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_cloudwatch_logs.html
• ECS and Fargate blue green cloudformation templates (from the blog post) https://github.com/aws-samples/ecs-blue-green-deployment

Database and Storage links

• 24 Jul 2017 S3 Rate Request Performance Increase announcement https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ and notice the exponetial scaling possible with multiple prefixes. https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html but if using sse-kms this service will a limiting factor. https://docs.aws.amazon.com/kms/latest/developerguide/limits.html#requests-per-second-table

• S3 Transfer Acceleration Speed Checker http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html uses a multi part upload to check the speed difference when using S3 transfer acceleration between regions.

• S3 Deep Dive Mar 2017 https://www.slideshare.net/AmazonWebServices/deep-dive-on-amazon-s3-march-2017-aws-online-tech-talks

• Automated RDS failover if you enable Multi AZ. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html

• S3 bucket policy examples https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-8

• Amazon RDS Now Supports Database Storage Size up to 16TB and Faster Scaling for MySQL, MariaDB, Oracle, and PostgreSQL Engines (22 Nov 2017) https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-rds-now-supports-database-storage-size-up-to-16tb-and-faster-scaling-for-mysql-mariadb-oracle-and-postgresql-engines/

• S3 Puts: Under the section “Q: How will I be charged and billed for my use of Amazon S3?” in FAQS: https://aws.amazon.com/s3/faqs/ and in detail at https://aws.amazon.com/s3/pricing/ Request Example: Assume you transfer 10,000 files into Amazon S3 and transfer 20,000 files out of Amazon S3 each day during the month of March. Then, you delete 5,000 files on March 31st. Total PUT requests = 10,000 requests x 31 days = 310,000 requests

• Announcement for Serverless Aurora: https://aws.amazon.com/rds/aurora/serverless/

  • If you want to participate in the preview, get more insight into the feature itself, or visibility into how serverless aurora is priced, here is a very helpful link: https://aws.amazon.com/rds/aurora/serverless/

• Deep Dive on EBS Snapshots https://www.youtube.com/watch?v=TUJCQRejA28

• Looks like they started allowing S3 SSE with customer provided keys (SSE-C) in 2014 https://aws.amazon.com/about-aws/whats-new/2014/06/12/amazon-s3-now-supports-server-side-encryption-with-customer-provided-keys-sse-c/

• DynamoDB deep dive from ReInvent 2016 https://www.youtube.com/watch?v=bCW3lhsJKfw

• Determining volume IO performance https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html

• RDS Performance benchmarks on percona https://d0.awsstatic.com/product-marketing/Aurora/RDS_Aurora_Performance_Assessment_Benchmarking_v1-2.pdf

• RDS Deep Dive from ReInvent 2017. Watch to gain an appreciation of how RDS works https://www.youtube.com/watch?v=TJxC-B9Q9tQ

• Best Practices for Running Oracle Database on Amazon Web Services (Jan 2018) https://d0.awsstatic.com/whitepapers/best-practices-for-running-oracle-database-on-aws.pdf Also review the links in the appendix which dive deeper into running Oracle workloads on EC2 and advanced archictures for running Oracle databases on AWS. https://d0.awsstatic.com/enterprise-marketing/Oracle/AWSAdvancedArchitecturesforOracleDBonEC2.pdf

• Deep Dive on Amazon Neptune (circa Jan 2018) https://www.slideshare.net/AmazonWebServices/deep-dive-on-amazon-neptune-aws-online-tech-talks Look for updates at ReInvent

• Moving a Galaxy into the Cloud. Samsung’s experience migrating from Cassandra to DynamoDB with big cost savings and at very large scale. https://www.youtube.com/watch?v=Z-2UIrI9feQ

• To keep up with the latest on any service keep abreast of the whats new. Here’s some interesting AWS database links. https://aws.amazon.com/blogs/database

• Amazon S3 Path Deprecation Plan – The Rest of the Story https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/
• Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region https://aws.amazon.com/message/41926/
• Using manifest files to group disparate files in S3. This allows you to uncouple the ‘where of your data’ from downstream processing, analysis and visualization.
  • Start with ‘Create an Analysis Using Your Own Amazon S3 Data’ https://docs.aws.amazon.com/quicksight/latest/user/getting-started-create-analysis-s3.html and read ;Supported Formats for Amazon S3 Manifest Files’ https://docs.aws.amazon.com/quicksight/latest/user/supported-manifest-file-format.html
  • Also check out ‘Creating a Data Set Using Amazon S3 Files’ on github at https://github.com/awsdocs/amazon-quicksight-user-guide/blob/master/doc_source/create-a-data-set-s3.md and follow the links to deep dive

ETL Links

• Need to ETL large amounts of files and want to avoid reprocessing files. Here are some best practice blog posts, with cloudformation scripts (just clicke the yellow launch button), that will build the solutions for you:
  • Orchestrate Amazon Redshift-Based ETL workflows with AWS Step Functions and AWS Glue https://aws.amazon.com/blogs/big-data/orchestrate-amazon-redshift-based-etl-workflows-with-aws-step-functions-and-aws-glue/
  • Trigger cross-region replication of pre-existing objects using Amazon S3 inventory, Amazon EMR, and Amazon Athena https://aws.amazon.com/blogs/big-data/trigger-cross-region-replication-of-pre-existing-objects-using-amazon-s3-inventory-amazon-emr-and-amazon-athena/
  • Build and automate a serverless data lake using an AWS Glue trigger for the Data Catalog and ETL jobs https://aws.amazon.com/blogs/big-data/build-and-automate-a-serverless-data-lake-using-an-aws-glue-trigger-for-the-data-catalog-and-etl-jobs/

Network links

• Private routable CIDR ranges as per RFC 1918 https://en.wikipedia.org/wiki/Private_network ENAS:

• https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/enhanced-networking.html Elastic Network Adapter (ENA) The Elastic Network Adapter (ENA) supports network speeds of up to 25 Gbps for supported instance types. C5, F1, G3, H1, I3, m4.16xlarge, M5, P2, P3, R4, and X1 instances use the Elastic Network Adapter for enhanced networking.

• and the original Nov 2016 ENA announcement. https://aws.amazon.com/blogs/aws/elastic-network-adapter-high-performance-network-interface-for-amazon-ec2/

• NACLs for subnets are configurable. Rules are evaluated from top to bottom with the final rule (immutable) of deny all. See https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html for examples and also https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html

• The NAT Gateway is simple to create and use. Just create the NAT Gateway and update your route table to direct all 0.0.0.0/0 traffic to the UID of the NAT Gateway. AWS looks after the rest. Another fully managed service. https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html

• OSI model is used to describe the 7 layers of our networks. https://en.wikipedia.org/wiki/OSI_model

• Privatelink intro book https://aws.amazon.com/privatelink/

• VPC Summary including private endpoints supported and alternative privatelink endpoint support. https://aws.amazon.com/vpc/

  • Privately connect to AWS Services without using an Internet gateway, NAT or firewall proxy through a VPC Endpoint. Available AWS services include S3, DynamoDB, Kinesis Streams, Service Catalog, EC2 Systems Manager (SSM), Elastic Load Balancing (ELB) API, Amazon Elastic Compute Cloud (EC2) API, and SNS.

• DX security concerns (Advanced Networking guide ppgs 255 to 290)

  • Third party options for encryption over Direct Connect https://supportforums.cisco.com/legacyfs/online/csr-secure-directconnect-2014110501.pdf and https://aws.amazon.com/quickstart/architecture/aviatrix-user-vpn/
  • Or VPN over AWS

• VPC Options for Resizing (specifically expanding) https://aws.amazon.com/about-aws/whats-new/2017/08/amazon-virtual-private-cloud-vpc-now-allows-customers-to-expand-their-existing-vpcs/ and https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html#vpc-resize

• Webinar slides with Route 53 regional failover examples including ELB, EC2 instance and fixed IP. https://www.slideshare.net/AmazonWebServices/webinar-route53-dnsfailoverfinal and here is a description of using Route53 to use latency based domains and weighted domains for multi region failover. https://www.sumologic.com/blog/amazon-web-services/aws-route-53-global-load-balancing/

• Using Route 53 private hosted zones across VPCs in a region. aws route53 create-vpc-association-authorization –hosted-zone-id ZONEID –vpc VPCRegion=land-of-oz,VPCId=vpc-xxxxxxx . NOTE: from the AWS Service Limits Amazon VPCs that you can associate with a private hosted zone is limited to 100. There is some additional hints, not called out in the AWS documentation at https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.boto3_route53.html. Specifically around associations within a single account and multiple accounts. Both the VPC and Private hosted zone must exist before creating the association. There are a number of 4xx errors called out in the documentation.

Security links

• AWS Compliance mapping to services https://aws.amazon.com/compliance/services-in-scope/

• Norse attack map http://map.norsecorp.com/#/

• S3 Access Control Lists (ACLs) explains how permissions are or can be applied to S3 buckets. This is a tedious read but worth while for anyone interested in simple permission management of cross account access or large number of accounts and say log consolidation to one account or bucket.

• How to use services like Well Architected, Trusted Advisor, Inspector, Macie, Shield, WAF, Partner tooling, etc to get secure.

• Make sure your customers are fully conversant and implementing our guidance from https://aws.amazon.com/whitepapers/#essentials and get them to audit their use of our services as per https://d1.awsstatic.com/whitepapers/compliance/AWS_Auditing_Security_Checklist.pdf .

• S3 permissions can be on buckets, bucket contents and applied to objects say at upload. https://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html for a deeper dive.

• F5 WAF git hub https://github.com/f5devcentral/f5-aws-autoscale/tree/master/deployments/waf-sandwich-utility-only-immutable compare appliance based waf sandwiches to using native AWS services https://f5.com/resources/white-papers/load-balancing-101-firewall-sandwiches

• With NACLs (optional stateless firewall for a subnet boundary) rules are evaluated from lowest to highest. As soon as a match is found it is applied. https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html

• If you search the web for ‘aws deep dive’ AND sa ‘security’ you’ll find some great videos and slide decks from ReInvent, our public bootcamps and from many of AWS SMEs. Here’s one on security goverance https://www.youtube.com/watch?v=xjtSWd8z_bE and here’s another on a service GuardDuty. https://www.youtube.com/watch?v=o2YaIsps5LY

• A useful article on using parameter store to store secrets. https://aws.amazon.com/blogs/mt/the-right-way-to-store-secrets-using-parameter-store/

• Apple are publicaly mentioning their use of S3 in https://images.apple.com/business/docs/iOS_Security_Guide.pdf

• Security Assessments on Github (also AWS Services like Inspector too)

  • https://github.com/awslabs/aws-security-benchmark/blob/master/aws_cis_foundation_framework/CIS_Amazon_Web_Services_Foundations_Benchmark_v1.1.0.pdf

  • https://github.com/Alfresco/prowler

  • Netflix Security Monkey. https://github.com/Netflix/security_monkey

  • Lambda script to install the SSM agent https://github.com/awslabs/amazon-inspector-agent-autodeploy

  • Inspector blog post https://aws.amazon.com/blogs/aws/scale-your-security-vulnerability-testing-with-amazon-inspector/

  • Use Inspector to assess the NIST Quickstart for vulnerabilities

    • https://docs.aws.amazon.com/inspector/latest/userguide/inspector_quickstart.html
    • Install the Inspector agent. https://docs.aws.amazon.com/inspector/latest/userguide/inspector_installing-uninstalling-agents.html

• IAM Ninja and Deep Dives from ReInvents

  • IAM Policy Ninja (300ish level) https://www.youtube.com/watch?v=aISWoPf_XNE
  • Here is an IAM talk from ReInvent 2016 https://www.slideshare.net/AmazonWebServices/aws-reinvent-2016-iam-best-practices-to-live-by-sac317

• Multiple Account Deep Dives

  • AWS re:Invent 2016: NEW SERVICE: Manage Multiple AWS Accounts with AWS Organizations (SAC323) https://www.youtube.com/watch?v=Oeb7PDyiT2A
  • AWS re:Invent 2017: Architecting Security and Governance Across a Multi-Account Stra (SID331) https://www.youtube.com/watch?v=71fD8Oenwxc

• Neat explanation with graphics of signing of urls https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html

Monitoring

• Other than VPC Flow Logs what other monitoring options are available on AWS?

  • Amazon GuardDuty analyzes AWS CloudTrail, VPC Flow Logs, and AWS DNS logs. The service is optimized to consume large volumes of data for near real-time processing of security detections. GuardDuty gives you access to built-in detection techniques that are developed and optimized for the cloud and maintained and continuously improved upon by AWS Security.
  • Amazon GuardDuty pulls independent streams of data directly from AWS CloudTrail, VPC Flow Logs, and AWS DNS logs. You don’t have to manage Amazon S3 bucket policies or modify the way you may collect and store your logs. GuardDuty permissions are managed as Service Linked Roles that you can revoke at any time by disabling GuardDuty. This makes it easy to enable the service without complex configuration and it eliminates the risk that an AWS IAM permission modification or S3 bucket policy change will affect the operation of the service. It also makes GuardDuty extremely efficient at consuming high-volumes of data in near real-time without affecting the performance or availability of your account or workloads.
  • Also no performance impact. https://aws.amazon.com/guardduty/faqs/

• VPC Flow Log limitations https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow-logs.html#flow-logs-limitations

• Cloudwatch logs also allows you to store and trigger events from application, service and custom logs. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html

• Cloudwatch logs metrics at varying intervals. Basis is 5 minute, detailed and basic custom metric is 1 minute and high resolution custom metrics (July 2017) is 1 sec. Refer to https://aws.amazon.com/cloudwatch/faqs/ for the varying retention windows for each logging type.

Visualize your AWS environment

• Visualize VPC Flow Logs using an ELK stack approach https://aws.amazon.com/blogs/security/how-to-optimize-and-visualize-your-security-groups/
• www.dome9.com gives most AWS employees their free DevOps tier. Great for visualizing, auditing and checking for compliance across an account or accounts. If you build the NIST QuickStart in your account you can see lot’s of cool outputs from dome9. Get the NIST QuickStart at https://aws.amazon.com/quickstart/architecture/accelerator-nist/
• Visualize Cloudtrail logs using Glue and Quicksight https://aws.amazon.com/blogs/big-data/streamline-aws-cloudtrail-log-visualization-using-aws-glue-and-amazon-quicksight/

Self paced Learning and Building

• AWS Certification roadmap https://aws.amazon.com/certification/ Check out the learning paths link at the bottom of the page.
• Read the service FAQ pages, http://aws.amazon.com/faqs/, and documentation for each of the services. Just search for AWS + + documentation in any search engine. You can keep the documentation as pdf, html online or even in your Kindle. You can also git clone the documentation for most services.
• Find and build interesting AWS and partner solutions you find the in AWS Blog https://aws.amazon.com/blogs/ . Any post you find with a yellow launch button will build that solution using Cloudformation.
• AWS free digital training is mostly 100 level but we also have over 40 hours of Machine Learning training available for free. You can search by topic, role or level. https://www.aws.training/LearningLibrary?src=courses You’ll find specialist deep dives from level 100 through 300 like this video describing the differences between NACLs and Security groups. https://www.aws.training/Details/Video?id=16486 NOTE: You’ll need to enroll and allow popups in your browser.
• You can also take AWS Qwiklabs Labs for free at https://aws.amazon.com/training/self-paced-labs/
• Get a sandbox or personal account. There are free tiers for many services. https://aws.amazon.com/free/
• http://run.qwiklabs.com and complete quests and labs. These enhance your familiarity with AWS services without you having to use your own account. Some labs are free. Others will require you to redeem Qwiklab credits. Reach out to your training manager or AWS account manager. Also check out the Exam guides for SA, SysOps and Advanced Networking https://www.amazon.com/Certified-Advanced-Networking-Official-Study/dp/1119439833/ref=sr_1_1?s=books&ie=UTF8&qid=1519925473&sr=1-1&keywords=advanced+networking
• Search github, https://github.com/aws , and the AWS blogs, https://aws.amazon.com/blogs/ , for solutions that interest you. Look for posts with a launch button. These will build a complete environment using Cloudformation. Retrieve the Cloudformation templates either from the built environment in your account or from Github. You can reverse engineer or use these templates as scaffolds for your own use.
• Visit Stackoverflow and the AWS discussion forum to pose questions or to contribute to answers about AWS
• You can also take a number of AWS MOOCs (Massive Open Online Courses) on EDx and Coursera including:
  • EDx https://www.edx.org/school/aws
  • Coursera https://www.coursera.org/aws
• There are many other self paced labs and solutions you can build on AWS. Try:
  • Build a Serverless Web Application https://aws.amazon.com/getting-started/projects/build-serverless-web-app-lambda-apigateway-s3-dynamodb-cognito/
  • How about AWS Developer Center https://aws.amazon.com/developer/ where you can build the Mythical Misfits app in your choice of programming language.
• The AWS Podcast has a monthly update which is a great way to keep up with the latest changes, releases and interviews with domain experts https://aws.amazon.com/podcasts/aws-podcast/
• AWS has released a number of webinars and now has a monthly cadence https://aws.amazon.com/about-aws/events/monthlywebinarseries/
• AWS Answers is now available to the public. It contains some interesting links. https://aws.amazon.com/answers/
• Get to know your AWS Solution Architects and your Technical Account Manager (TAM). The SAs help you to architect and understand best practice. The TAMs provide support for your applications running on AWS. They can help you prepare for major events like testing and scaling. They can also help troubleshoot and provide visibility into AWS infrastructure metrics for troubleshooting. https://aws.amazon.com/premiumsupport/faqs/
• AWS Glossary contains service names and nomenclature https://docs.aws.amazon.com/general/latest/gr/glos-chap.html
• Now go build stuff…

Continue reading articles in my Amazon Web Services series

• Data Warehousing on AWS
• Migrating to AWS
• AWS Business Essentials
• IAM Demo
• Architecting on AWS
• SysOps on AWS
• S3 Demo
• Predict the Future
• AWS Tech Essentials
• Developing on AWS
• DevOps on AWS
• Advanced Architecting on AWS
• Big Data on AWS
• AWS Deep Dive Toolbox
• Security Engineering on AWS
• Deep Learning on AWS
• AWS List of Services
• Networking on Aws
• AWS Data and Analytics
• Microsoft Immersion Day
• Adelaide Deep Racer Hints and Tips
• Deep Racer Awards
• Windows on AWS
• AWS Ask Me Anything
• Cloudwatch and Systems Manager Workshop
• Containers Immersion Day
• Redshift Immersion Day
• Innovation in Ambiguity
• AWS Contingency Planning
• AWS CLI Examples
• Migrating to Cloud in 2023
• Chaos Engineering Workshop
• Chatgpt Friend or Foe