System Operations on AWS - Tips and Tricks
This is my list of hints and tips for this course. It’s markdown so you can save it, access it or store it anywhere. I might also give you other links that are course specific. I’ll add specific answers to questions I get during the course. I’ll share it with everyone.
Pre Class Survey Results
- Response 1)- AWS Infrastructure (ASG and LBs) TODO Labs 2)- AWS automation technologies. TODO Broad service review in the console 3)- AWS GuardRails and Cost Optimization. TODO EC2 Velocity Migration Deep Dive
- How to effectively monitor and troubleshoot systems and EC2 instances. TODO AWS Systems Manager, Dome9 and Labs
- creating and managing cloud formation templates TODO Cloudformation Build out from scratch
- identifying strategies for applying autoscaling to messaging systems. TODO whiteboard architecture using messaging services
- security and delegation TODO IAM review specifically roles
- VPC and security groups and NACL to use them in combination to accomplish the design goals. TODO Review fully integrated NIST quickstart
- To understand best practices for setting up and configuring AWS services. Mostly in architecting resilient services powered by AWS. TODO Labs and Arch and Adv Arch courses
- It is important for me to understand how to create, test, deploy, and troubleshoot a CloudFormation template. To understand the whole pipeline. As much hands-on experience as possible. TODO Cloudformation Build out from scratch
- I come from the Infrastructure team predominantly using Vmware ESXi for our on-premise virtualization. As such I am most interested in the fundamental compute/storage/networking aspects of AWS like EC2/EBS/S3/VPC…etc TODO Labs and VMWare Cloud on AWS
- Use standard AWS infrastructure features such as VPC, EC2, Elastic Load Balancer, and auto scaling from the command line TODO Labs
- Use CloudFormation and other automation technologies to produce stacks of AWS resources that can be deployed in an automated, repeatable fashion TODO Review SaaS Quickstart
- Monitor the health of EC2 instances and other AWS services TODO Cloudwatch, Inspector, Labs
- Using cloudformation templates for deployments, also the usage of AWS SAM for serverless. TODO Add SAM to Cloudformation Demo
- General automation around deployments. TODO Codestar Intro
- Monitoring and troubleshooting. TODO AWS Systems Manager, Dome9 and Labs
- A very good understanding of creating VPC’s ground-up TODO Cloudformation Build out from scratch
- and how to use Cloudformation
- How to navigate the UI to piece all the pieces together.
- Use cases of companies having services in both public and private cloud (ie. hybrid infrastructure). The use cases should provide examples on how challenges of migrating services to public cloud were overcome or are still being worked through. It would be also good to show examples of how proprietary standardization of cloudformation templates is enforced. TODO Service Catalog
- Cloudformation.
- Details on properly setting up ACLs / Security Groups. TODO Review fully integrated NIST quickstart
- Good ways to manage large numbers of resources. TODO AWS Systems Manager
- What is most important for my role is to understand the architecture of AWS and how each component works with the others and where they lie within the infrastructure TODO Review SaaS Quickstart
Lunch Time Entertainment
- play a funny or interesting video. ** Like the New Zealand police recruitment, https://www.youtube.com/watch?v=f9psILoYmCc ** Perhaps even the Australia Day lamb video with the historical boat people and a history of Australia in 3 minutes. https://www.youtube.com/watch?v=yugymulPx9Y . Or how about learning Aussie slang. https://www.youtube.com/watch?v=yDb_WsAt_Z0
- Here’s a sample of the AWS Podcast http://d1le29qyzha1u4.cloudfront.net/AWS_Podcast_Episode_230.mp3
- Come visit Australia but be wary of the critters. Just kidding most are just cuddly except for the crocs. https://www.youtube.com/watch?v=iQSxuqWQ_4c
- Subjecting snowball to a Mil-Std 810 mine blast test. https://www.youtube.com/watch?v=__ooXhq5gZ4&feature=youtu.be
In an Emergency
- Ian Falconer (415)-797-9307 ifal@amazon.com ** if you get locked out, lost or something else disastrous you can call my cell phone. I sometimes miss it buzzing or beeping so be persistent.
Administrivia
We need to jump through some hoops to get access to the labs, notes and my hints and tips. Be consistent with the email address you use for all sites. There are three seperate sites you need to access and one bitly link which is this page:
- Join or login to https://www.aws.training/ to ensure your training and certifications are captured. No we don’t spam you or sell your details.
- Access Qwiklab (yes it is spelt INCORRECTLY) ** aws.qwiklabs.com for the labs in this class ** run.qwiklabs.com for outside of the class or to do other labs at your own pace. NOTE: Some are free others require course credits.
- Access the course notes and slides. You’ll receive two emails. One confirming your attendance at this course and with the following links. The download link seems broken. You can download apps for phones, tablets and laptops. Or use your browser.
- www.vitalsource.com look for a signup link and download link. Or just go to https://evantage.gilmoreglobal.com/#/user/signin
- Once you’ve logged into Vitalsource (aka Bookshelf, Gilmore, eVantage) you can redeem your unique course materials code (in a seperate email) and update your book list. You should see a lab guid and student guide for XXXXXXXXX, version 5. . The student guide is the powerpoint decks and notes and the lab guide is the step by step instructions for the labs. You can download the Vitalsource Bookshelf app for Windows, Mac, IoS and Android at https://support.vitalsource.com/hc/en-us/articles/201344733-Bookshelf-Download-Page
- If you have trouble printing the student and lab guides to pdf try using the 32 bit (not 64 bit) Bookshelf apps). https://support.vitalsource.com/hc/en-us/articles/201344733-Bookshelf-Download-Page
ReInvent
There are some proven strategies to get the most out of ReInvent. The event is spread out over many casinos so you want to minimize commuting. I prefer to walk than shuttle to get some sun.
- Visit the partner expo early for swag and preregistration for cool freebies
- Do some research before hand. Like this Your guide to Amazon DynamoDB sessions, workshops, and chalk talks at AWS re:Invent 2018 https://aws.amazon.com/blogs/database/your-guide-to-amazon-dynamodb-sessions-workshops-and-chalk-talks-at-aws-reinvent-2018/
Cool links
- James Hamilton, AWS SVP, talks about our infrastructure. This deck is over 3 years old but still a good summary. https://www.slideshare.net/AmazonWebServices/spot301-aws-innovation-at-scale-aws-reinvent-2014 . Here’s the youtube video. https://www.youtube.com/watch?v=JIQETrFC_SQ There are Youtube videos from more recent ReInvents with some updates too. Here is James in 2016. It’s titled as Global Innovation at Scale. https://www.youtube.com/watch?v=uj7Ting6Ckk
- AWS re:Invent 2017: Scaling Up to Your First 10 Million Users (ARC201). This is like the Tech Essentials course in a single video. Well worth a watch. https://www.youtube.com/watch?v=w95murBkYmU and Dynamo Deep Dive from 2017 https://duckduckgo.com/?q=dynamodb+deep+dive&t=ffab&atb=v94-3_w&ia=videos&iax=videos&iai=KL-9auR9fes and the DAX Deep Dive from 2017 https://duckduckgo.com/?q=dynamodb+deep+dive&t=ffab&atb=v94-3_w&ia=videos&iax=videos&iai=KL-9auR9fes and DynamoDB updates in 2018 and 2017 and Werner Vogel’s blog on picking the right database. Hint the answer aint always relational. https://www.allthingsdistributed.com/2018/06/purpose-built-databases-in-aws.html https://aws.amazon.com/blogs/database/amazon-dynamodb-highlights-from-the-last-18-months-you-may-have-missed/s
- It can be tricky to locate all information about AWS instance types in the documentation. Here’s a third party site that has a table that lets you sort on memory, network performance, cost and instance type. https://ec2instances.info/
- Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region https://aws.amazon.com/message/41926/
- An external post about S3 data leaks. https://www.bleepingcomputer.com/news/security/amazon-aws-servers-might-soon-be-held-for-ransom-similar-to-mongodb/
- AWS Regions and Endpoints (list of supported services) https://docs.aws.amazon.com/general/latest/gr/rande.html#ssm_region
- Latency between AWS regions. Lot’s of good empirical data points. Note these are averages of averages. 95th percentile values would be more useful. https://www.cloudping.co/
- Latency http://highscalability.com/latency-everywhere-and-it-costs-you-sales-how-crush-it
- List of CIDR ranges of AWS regions http://ec2-reachability.amazonaws.com/
- How we handle prime day (DynamoDB) https://www.youtube.com/watch?v=83-IWlvJ__8
- Chalice python based microservices framework using Lambda, API Gateway and IAM. http://chalice.readthedocs.io/en/latest/ Very fast, very lightweight and very extensible.
- Benchmark tests of EC2 versus other bare metal and cloud servers. https://www.phoronix.com/scan.php?page=article&item=cloud-cpu-36&num=1
- Here’s a Lambda deep dive which more clearly explains some of the questions around managing state, retries, testing and handling large data sets with Lambda. https://www.youtube.com/watch?v=dB4zJk_fqrU
- Mechanical sympathy http://infrastructure-as-code.com/book/2015/03/23/mechanical-sympathy.html
- AWS Open Guide on GitHub is a good summary of AWS Documentation https://github.com/open-guides/ We have also open sourced our documentation at https://github.com/awsdocs
- You can check the overall health and availability of AWS globally at the Service Health Dashboard (SHD) https://status.aws.amazon.com/ You can also use the AWS Health API to programatically check for service health at https://docs.aws.amazon.com/health/latest/ug/getting-started-api.html
- Netflix have some cool tools that they’ve open sourced. https://netflix.github.io/
- All quickstarts now in github https://github.com/aws-quickstart ** Compare the course notes (slides 25 thru 39 in 05_WebScaleApplications.ppt) with the blue green quickstart at https://aws.amazon.com/quickstart/architecture/blue-green-deployment/
- Netflix Open Sourcing of many useful and interesting tools for running large AWS cloud environments. https://netflix.github.io/
- Researchers say Data61-backed blockchain platform delivers scalability, energy efficiency 1000 ec2 instances, 14 AWS Regions and 30k TPS https://www.computerworld.com.au/article/647265/researchers-say-data61-backed-blockchain-platform-delivers-scalability-energy-efficiency/
- Adrian Cockcroft AWS VP of Cloud Architecture Strategy and former CTO of Netflix. Here’s his Youtube playlist with talks about DevOps, migrations, Netflix lessons learned and digital transformation topics. https://www.youtube.com/playlist?list=PL_KXMLr8jNTnwkzV7SePa0jHFUG2qn0MA
- The Network is Reliable and other fallacies. Key performance and reliability concerns of distributed systems. https://blog.acolyer.org/2014/12/18/the-network-is-reliable/
- Architecture reviews are important. The cloud design principles, here is the 2011 AWS Whitepaper https://media.amazonwebservices.com/AWS_Cloud_Best_Practices.pdf and the Well Architected Review are key inputs. https://aws.amazon.com/architecture/well-architected/
Ian’s list of links for weekly review of all stuff AWS. (trying to keep up with the firehose)
- Search for AWS and Deep Dive or Ninja and you’ll find lots of great videos or slideshares.
- AWS is continually sharing more good stuff on github. https://github.com/aws
- AWS What’s New: https://aws.amazon.com/new/
- And my favourite the weekly AWS Podcast https://aws.amazon.com/podcasts/aws-podcast/
- The first place to start on any of AWS’ more than 125 services is the the faq pages for each product (this is where I start reading) https://aws.amazon.com/faqs/
https://aws.amazon.com/podcasts/aws-podcast/ and all the faq pages for each product (this is where I start reading)
AWS has released a number of webinars and now has a monthly cadence https://aws.amazon.com/about-aws/events/monthlywebinarseries/
AWS Answers is now available to the public. It contains some interesting links. https://aws.amazon.com/answers/
Get to know your Technical Account Manager (TAM) The TAMs provide support for your applications running on AWS. They can help you prepare for major events like testing and scaling. They can also help troubleshoot and provide visibility into AWS infrastructure metrics for troubleshooting. https://aws.amazon.com/premiumsupport/faqs/
AWS Glossary contains service names and nomenclature https://docs.aws.amazon.com/general/latest/gr/glos-chap.html
Management tools
- AWS Systems Manager – A Unified Interface for Managing Your Cloud and Hybrid Resources (either start with this blog post or check out the FAQ or What is Systems Manager page) https://aws.amazon.com/blogs/aws/aws-systems-manager/ or https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html
- Deep Dive with Amazon EC2 Systems Manager [ENT401] https://www.youtube.com/watch?v=BmpxZsk9N48 or the London Summit (circa Jun 2017) slide share http://london-summit-slides-2017.s3.amazonaws.com/EC2_Systems_Manager_DeepDive.pdf
- Why AWS Service Catalog is so important to managing at scale. https://devops.com/5-reasons-aws-service-catalog-is-sexy-for-devops-and-cloud/ and the hub and spoke service catalog blog post https://aws.amazon.com/blogs/mt/aws-service-catalog-hub-and-spoke-model-how-to-automate-the-deployment-and-management-of-service-catalog-to-many-accounts/
- Key Systems Manager Blog Posts ** New – S3 Sync capability for EC2 Systems Manager: Query & Visualize Instance Software Inventory https://aws.amazon.com/blogs/aws/new-s3-sync-capability-for-ec2-systems-manager-query-visualize-instance-software-inventory/
- CLI Cheat Sheet from github has a sample of interesting CLI commands using JMESPath and filter queries. https://github.com/leftbrainstuff/aws-stuff/blob/master/aws-cli-cheat-sheet.md
Compute links
- Is AMI like Sysprep? https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ami-create-standard.html
- Debug and run Lambda Python locally https://medium.com/@bezdelev/how-to-test-a-python-aws-lambda-function-locally-with-pycharm-run-configurations-6de8efc4b206 This is another option for testing serverless. Also enable AWS Xray on your Lambdas and you can trace what is happening over the life of the Lambda.
- Amazon EC2 Spot introduces new pricing model and the ability to launch Spot instances via RunInstances API https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-ec2-spot-introduces-new-pricing-model-and-the-ability-to-launch-new-spot-instances-via-runinstances-api/
- EC2 Auto Recovery ** This non AWS video might help https://www.youtube.com/watch?v=hea5q_XYsIg ** https://www.slideshare.net/AmazonWebServices/deep-dive-amazon-ec2
- ALB, NLB or Classic Load Balancer? https://aws.amazon.com/elasticloadbalancing/faqs/ walks you through the choices and use cases.
- Quickly find AMI IDs in multiple regions. Cloud Formation AMI Mapping Builder. https://github.com/kennyk65/aws-teaching-demos
- To install the cloudwatch agent on on prem servers you can now use SSM. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-onprem.html You can also find a number of (unverified) methods on github like https://github.com/cgswong/chef-cloudwatch-logs
Serverless
- 20 min video walk thru using SAM Local from the AWS Summit - London | twitch.tv/aws | Deeper Dive into AWS SAM Local https://youtu.be/wd-9XTeKSks
- Another video covering most of the serverless offerings from the AWS Builders’ Day | Serverless Development Deep Dive https://youtu.be/Gh3QE0tLaZs
- AWS SAM CLI on github https://github.com/awslabs/aws-sam-cli
- Investigating spikes in AWS Lambda function concurrency https://aws.amazon.com/blogs/compute/investigating-spikes-in-aws-lambda-function-concurrency/
- API Gateway has lot’s of interesting features including private endpoints https://aws.amazon.com/blogs/compute/introducing-amazon-api-gateway-private-endpoints/
- API Gateway Regional End Points described in How ipdata uses AWS to serve a global, highly-scalable IP geolocation API https://aws.amazon.com/blogs/startups/how-ipdata-uses-aws-to-serve-a-global-highly-scalable-ip-geolocation-api/ and some interesting uses of API Gateway, Lambda and Step Functions in Building a Real World Evidence Platform on AWS https://aws.amazon.com/blogs/big-data/building-a-real-world-evidence-platform-on-aws/ and
Containers
- A deep dive on Fargate. Lot’s of feature updates around container orchestration so watch the AWS what’s new for the latest. Here’s a slide share that deep dives on Fargate. https://de.slideshare.net/AmazonWebServices/deep-dive-into-aws-fargate
- Configuring Cloudwatch logs with containers. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_cloudwatch_logs.html
- ECS and Fargate blue green cloudformation templates (from the blog post) https://github.com/aws-samples/ecs-blue-green-deployment
- AWS Podcast #265: The State of Containers on AWS | September 30, 2018 https://aws.amazon.com/podcasts/aws-podcast/?ref=wnĉ
Database and Storage links
- 24 Jul 2017 S3 Rate Request Performance Increase announcement https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/ and notice the exponetial scaling possible with multiple prefixes. https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html but if using sse-kms this service will a limiting factor. https://docs.aws.amazon.com/kms/latest/developerguide/limits.html#requests-per-second-table
- S3 Transfer Acceleration Speed Checker http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html uses a multi part upload to check the speed difference when using S3 transfer acceleration between regions.
- S3 Deep Dive Mar 2017 https://www.slideshare.net/AmazonWebServices/deep-dive-on-amazon-s3-march-2017-aws-online-tech-talks
- Automated RDS failover if you enable Multi AZ. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html
- S3 bucket policy examples https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-8
- Amazon RDS Now Supports Database Storage Size up to 16TB and Faster Scaling for MySQL, MariaDB, Oracle, and PostgreSQL Engines (22 Nov 2017) https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-rds-now-supports-database-storage-size-up-to-16tb-and-faster-scaling-for-mysql-mariadb-oracle-and-postgresql-engines/
- S3 Puts: Under the section “Q: How will I be charged and billed for my use of Amazon S3?” in FAQS: https://aws.amazon.com/s3/faqs/ and in detail at https://aws.amazon.com/s3/pricing/ Request Example: Assume you transfer 10,000 files into Amazon S3 and transfer 20,000 files out of Amazon S3 each day during the month of March. Then, you delete 5,000 files on March 31st. Total PUT requests = 10,000 requests x 31 days = 310,000 requests
- Announcement for Serverless Aurora: https://aws.amazon.com/rds/aurora/serverless/ ** If you want to participate in the preview, get more insight into the feature itself, or visibility into how serverless aurora is priced, here is a very helpful link: https://aws.amazon.com/rds/aurora/serverless/
- Deep Dive on EBS Snapshots https://www.youtube.com/watch?v=TUJCQRejA28
- Looks like they started allowing S3 SSE with customer provided keys (SSE-C) in 2014 https://aws.amazon.com/about-aws/whats-new/2014/06/12/amazon-s3-now-supports-server-side-encryption-with-customer-provided-keys-sse-c/
- DynamoDB deep dive from ReInvent 2016 https://www.youtube.com/watch?v=bCW3lhsJKfw
- Determining volume IO performance https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html
- RDS Performance benchmarks on percona https://d0.awsstatic.com/product-marketing/Aurora/RDS_Aurora_Performance_Assessment_Benchmarking_v1-2.pdf
- RDS Deep Dive from ReInvent 2017. Watch to gain an appreciation of how RDS works https://www.youtube.com/watch?v=TJxC-B9Q9tQ
- Best Practices for Running Oracle Database on Amazon Web Services (Jan 2018) https://d0.awsstatic.com/whitepapers/best-practices-for-running-oracle-database-on-aws.pdf Also review the links in the appendix which dive deeper into running Oracle workloads on EC2 and advanced archictures for running Oracle databases on AWS. https://d0.awsstatic.com/enterprise-marketing/Oracle/AWSAdvancedArchitecturesforOracleDBonEC2.pdf
- Deep Dive on Amazon Neptune (circa Jan 2018) https://www.slideshare.net/AmazonWebServices/deep-dive-on-amazon-neptune-aws-online-tech-talks Look for updates at ReInvent
- Moving a Galaxy into the Cloud. Samsung’s experience migrating from Cassandra to DynamoDB with big cost savings and at very large scale. https://www.youtube.com/watch?v=Z-2UIrI9feQ
Edge and IoT Computing links
- Deep Dive on AWS IoT Core (circa May 2018) https://www.slideshare.net/AmazonWebServices/deep-dive-on-aws-iot-core
Account and Network links
- Network deep dive https://youtu.be/b1gq9jTqInA This 30 min video from Mid 2018 succinctly describes new networking features like resizing CIDRs, Private Link, LBs and Direct Connect Gateway
- AWS Organizations on steroids. https://turbot.com/features/
- Private routable CIDR ranges as per RFC 1918 https://en.wikipedia.org/wiki/Private_network ENAS:
- https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/enhanced-networking.html Elastic Network Adapter (ENA) The Elastic Network Adapter (ENA) supports network speeds of up to 25 Gbps for supported instance types. C5, F1, G3, H1, I3, m4.16xlarge, M5, P2, P3, R4, and X1 instances use the Elastic Network Adapter for enhanced networking.
- and the original Nov 2016 ENA announcement. https://aws.amazon.com/blogs/aws/elastic-network-adapter-high-performance-network-interface-for-amazon-ec2/
- NACLs for subnets are configurable. Rules are evaluated from top to bottom with the final rule (immutable) of deny all. See https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html for examples and also https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html
- The NAT Gateway is simple to create and use. Just create the NAT Gateway and update your route table to direct all 0.0.0.0/0 traffic to the UID of the NAT Gateway. AWS looks after the rest. Another fully managed service. https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-nat-gateway.html
- OSI model is used to describe the 7 layers of our networks. https://en.wikipedia.org/wiki/OSI_model
- Want to understand how the AWS Network scales. A Day in the Life of a Billion Packets (CPN401) | AWS re:Invent 2013 https://www.youtube.com/watch?v=Zd5hsL-JNY4
- A neat visual subnet calculator http://www.davidc.net/sites/default/subnets/subnets.html
- AWS Landing Zone https://aws.amazon.com/answers/aws-landing-zone/ Quickly set up a secure, multi-account AWS environment based on AWS best practices. This is a deployable reference architecture. It does not have it’s own API but uses AWS Organizations, AD, SSO, S3, VPCs and VPC peering
- how to check which resources are using VPC IP addresses from StackOverFlow https://serverfault.com/questions/831722/in-aws-how-to-check-which-resources-are-using-vpc-ip-addresses Also a concise summary of IP allocation and ELB ENI (and IP) consumption during scaling that you need to watch out for.
- Privatelink can now be used with API Gateway
- Amazon Virtual Private Cloud ebook with AWS Privatelink (Privatelink is not listed in service limits and can be hard to find documentation) https://pages.awscloud.com/vpc-ebook.html?&trk=ba_a131L000005OSLAQA4&trkCampaign=pac_vpc_ebook_download_page&sc_channel=ba&sc_campaign=pac_q3_09-2018_mid-page-banner_VPC_ebook_PrivateLink_webpage&sc_outcome=CSI_Digital_Marketing&sc_publisher=Others
Security links
- HA Active Directory. Always a challenge but this quickstart describes and builds an architecture that can meet that requirement. https://aws.amazon.com/quickstart/architecture/active-directory-ds/
- AWS Compliance mapping to services https://aws.amazon.com/compliance/services-in-scope/
- Norse attack map http://map.norsecorp.com/#/
- S3 Acess Control Lists (ACLs) explains how permissions are or can be applied to S3 buckets. This is a tedious read but worth while for anyone interested in simple permission management of cross account access or large number of accounts and say log consolidation to one account or bucket.
- Educate our customers and show them how to use services like Well Architected, Trusted Advisor, Inspector, Macie, Shield, WAF, Partner tooling, etc to get secure. Make sure your customers are fully conversant and implementing our guidance from https://aws.amazon.com/whitepapers/#essentials and get them to audit their use of our services as per https://d1.awsstatic.com/whitepapers/compliance/AWS_Auditing_Security_Checklist.pdf . ** We also have people reviewing our services for the upcoming GDPR legislation that will come into effect in Europe in May 2018. Perhaps we could have an update on what that impact will be. (positive for issues like this from my brief conversations) ** As Werner said ‘dance like nobody is watching and secure like everybody is watching’ [sic].
- S3 permissions can be on buckets, bucket contents and applied to objects say at upload. https://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html for a deeper dive.
-
- F5 WAF git hub https://github.com/f5devcentral/f5-aws-autoscale/tree/master/deployments/waf-sandwich-utility-only-immutable compare appliance based waf sandwiches to using native AWS services https://f5.com/resources/white-papers/load-balancing-101-firewall-sandwiches
- With NACLs (optional stateless firewall for a subnet boundary) rules are evaluated from lowest to highest. As soon as a match is found it is applied. https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_ACLs.html
- If you search the web for ‘aws deep dive’ AND sa ‘security’ you’ll find some great videos and slide decks from ReInvent, our public bootcamps and from many of AWS SMEs. Here’s one on security goverance https://www.youtube.com/watch?v=xjtSWd8z_bE and here’s another on a service GuardDuty. https://www.youtube.com/watch?v=o2YaIsps5LY
- A useful article on using parameter store to store secrets. https://aws.amazon.com/blogs/mt/the-right-way-to-store-secrets-using-parameter-store/
- Apple are publicaly mentioning their use of S3 in https://images.apple.com/business/docs/iOS_Security_Guide.pdf
- Security Assessments on Github (also AWS Services like Inspector too) ** https://github.com/awslabs/aws-security-benchmark/blob/master/aws_cis_foundation_framework/CIS_Amazon_Web_Services_Foundations_Benchmark_v1.1.0.pdf ** https://github.com/Alfresco/prowler ** Netflix Security Monkey. https://github.com/Netflix/security_monkey ** Lambda script to install the SSM agent https://github.com/awslabs/amazon-inspector-agent-autodeploy ** Inspector blog post https://aws.amazon.com/blogs/aws/scale-your-security-vulnerability-testing-with-amazon-inspector/ ** Use Inspector to assess the NIST Quickstart for vulnerabilities *** https://docs.aws.amazon.com/inspector/latest/userguide/inspector_quickstart.html *** Install the Inspector agent. https://docs.aws.amazon.com/inspector/latest/userguide/inspector_installing-uninstalling-agents.html
- IAM Ninja and Deep Dives from ReInvents ** IAM Policy Ninja (300ish level) https://www.youtube.com/watch?v=aISWoPf_XNE ** Here is an IAM talk from ReInvent 2016 https://www.slideshare.net/AmazonWebServices/aws-reinvent-2016-iam-best-practices-to-live-by-sac317
- Multiple Account Deep Dives ** AWS re:Invent 2016: NEW SERVICE: Manage Multiple AWS Accounts with AWS Organizations (SAC323) https://www.youtube.com/watch?v=Oeb7PDyiT2A ** AWS re:Invent 2017: Architecting Security and Governance Across a Multi-Account Stra (SID331) https://www.youtube.com/watch?v=71fD8Oenwxc
- Encryption of EC2 instance Storage ** using linux dm-crypt Jan 2017 https://aws.amazon.com/blogs/security/how-to-protect-data-at-rest-with-amazon-ec2-instance-store-encryption/ ** Marketplace option for encrypting boot volumes. Old, 32 bit m1, m2 and c1 instances only supported https://aws.amazon.com/marketplace/pp/B00DR0EVUU/ ** encrypted EBS boot volumes supported from Dec 2015 https://aws.amazon.com/blogs/aws/new-encrypted-ebs-boot-volumes/ ** linux options are dm-crypt and loop-AES are both OSS options. Very slow to boot AMIs. ** Windows bitlocker supported by third party options including (Safenet KeySecure / ProtectV, Trend Micro SecureCloud)
- List of HIPAA compliant AWS services. https://aws.amazon.com/compliance/hipaa-compliance/
- Software scanning service https://ionchannel.io/ will scan OSS for vulnerabilities as a per package service. An interesting option for validation of imports in your code before production release. Thanks Mark…
- OWASP Top 10 security vulnerabilities. https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project from 2010 through to current year -1.
- AWS WAF, Shield and Firewall Manager are described in the same developer guide. It’s a little confusing as Firewall Manager works with Organizations to apply rules to multiple accounts and Shield and WAF can also work together. https://github.com/awsdocs/aws-waf-and-shield-advanced-developer-guide for the markdown or https://www.amazon.com/AWS-Firewall-Manager-Shield-Advanced-ebook/dp/B07641Q364 for the Kindle version.
Visualize your AWS environment
- Visualize VPC Flow Logs using an ELK stack approach https://aws.amazon.com/blogs/security/how-to-optimize-and-visualize-your-security-groups/
- Great for visualizing, auditing and checking for compliance across an account or accounts. If you build the NIST QuickStart in your account you can see lot’s of cool outputs from dome9. Get the NIST QuickStart at https://aws.amazon.com/quickstart/architecture/accelerator-nist/
- Visualize Cloudtrail logs using Glue and Quicksight https://aws.amazon.com/blogs/big-data/streamline-aws-cloudtrail-log-visualization-using-aws-glue-and-amazon-quicksight/
- Using Looker and SQL to analyze your AWS Cost and Usage Reports. From blog post Analyzing AWS Cost and Usage Reports with Looker and Amazon Athena at https://aws.amazon.com/blogs/big-data/analyzing-aws-cost-and-usage-reports-with-looker-and-amazon-athena/ . Also find the scripts at https://github.com/DillonMorrison/aws_cost_and_usage
- Analyze and visualize your VPC network traffic using Amazon Kinesis and Amazon Athena https://aws.amazon.com/blogs/big-data/analyze-and-visualize-your-vpc-network-traffic-using-amazon-kinesis-and-amazon-athena/?nc1=b_rp
- Visual CIDR calculator http://www.davidc.net/sites/default/subnets/subnets.html
IaaC
The Cloudformation resource type reference is a great single place to dive deep on which services are supported as IaaC https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html
Run Cloudformation template from the command line
aws cloudformation create-stack --stack-name 'my-yaml-cloudformation-demo' --template-body file://cfn-manual-build.yaml
YAML Cloudformation Template
AWSTemplateFormatVersion: "2010-09-09"
Description: >
# We are manually building a VPC, subnet, IGW, IGW Attachment, Security Group and EC2 Instance
# ami-0b59bfac6be064b78 is hardcoded to us-central-1 ohio
Resources:
MyVPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.23.0.0/16 # required
Tags: # optional, list of Tag
- Key: Environment # optional
Value: Demo # optional
- Key: Customer
Value: CVent
MySubnet:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: 10.23.0.0/17 # required
VpcId: !Ref MyVPC # required
MyIGW:
Type: AWS::EC2::InternetGateway
MyIGWAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
InternetGatewayId: !Ref MyIGW # optional
VpcId: !Ref MyVPC # required
MyWebServerSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Allow http to client host
VpcId: !Ref MyVPC
SecurityGroupIngress:
- CidrIp: 0.0.0.0/0
FromPort: 80
ToPort: 80
IpProtocol: tcp
SecurityGroupEgress:
- CidrIp: 0.0.0.0/0
FromPort: 0
ToPort: 65535
IpProtocol: tcp
MyWebServer:
Type: AWS::EC2::Instance
Properties:
ImageId: ami-0b59bfac6be064b78
InstanceType: t3.small
SecurityGroupIds:
- Ref: MyWebServerSG
SubnetId: !Ref MySubnet
UserData:
Fn::Base64: !Sub |
"#!/bin/bash
yum install -y httpd
service httpd start"
Continue reading articles in my Amazon Web Services series
- Data Warehousing on AWS
- Migrating to AWS
- AWS Business Essentials
- IAM Demo
- Architecting on AWS
- SysOps on AWS
- S3 Demo
- Predict the Future
- AWS Tech Essentials
- Developing on AWS
- DevOps on AWS
- Advanced Architecting on AWS
- Big Data on AWS
- AWS Deep Dive Toolbox
- Security Engineering on AWS
- Deep Learning on AWS
- AWS List of Services
- Networking on Aws
- AWS Data and Analytics
- Microsoft Immersion Day
- Adelaide Deep Racer Hints and Tips
- Deep Racer Awards
- Windows on AWS
- AWS Ask Me Anything
- Cloudwatch and Systems Manager Workshop
- Containers Immersion Day
- Redshift Immersion Day
- Innovation in Ambiguity
- AWS Contingency Planning
- AWS CLI Examples
- Migrating to Cloud in 2023
- Chaos Engineering Workshop
- Chatgpt Friend or Foe