5

things
you dont know about
Amazon Web Services
Simone Brunozzi
Sr. Technology Evangelist, AWS

@simon
1

AWS CLI

01

Delete a bucket and all its

content with just one
command

aws s3 rb s3://bucket-name --force

AWS CLI

02

Recursive copy of a directory

and its subfolders to Amazon
S3 (and vice versa)

aws s3 cp MyFolder s3://bucket-name -recursive [--region us-west-2]

Autoscaling

03

Hmm...

10

Autoscaling
Amazon
Amazon
Amazon
Amazon
Amazon
Etc.

S3
DynamoDB
ELB (Elastic Load Balancing)
Glacier
EB (Elastic Beanstalk)

11

EC2 Autoscaling

03

Automatically scale out EC2

within ELB
(old version)

12

AutoScaling (AS):
Four main components
Get the AS Command Line Tools:
aws.amazon.com/developertools/2535
1) Launch config: AMI to be used
2) Autoscaling group: where/how to launch
3) Autoscaling policy: what should AS do?
4) Autoscaling trigger: what will activate AS
13
13

14

AutoScaling tools
14

15

CloudWatch API Tools

15

Windows setup
FILE: ec2.bat
@echo off
set EC2_HOME=c:\ec2
set PATH=%PATH%;%EC2_HOME%\bin
set JAVA_HOME=C:\Program Files\Java\jre6
"%JAVA_HOME%\bin\java" -version
set JAVA_HOME=C:\Program Files (x86)\Java\jre6 (Windows 7)
set EC2_PRIVATE_KEY=c:\ec2\PrivateKey.pem
set EC2_CERT=c:\ec2\509Certificate.pem
set AWS_AUTO_SCALING_HOME=c:\ec2\AutoScaling
set AWS_CLOUDWATCH_HOME=c:\ec2\CloudWatch
set PATH=%PATH%;%AWS_AUTO_SCALING_HOME%\bin;%AWS_CLOUDWATCH_HOME%\bin
16
16

Linux setup
FILE: .bash_profile
export EC2_HOME=~/ec2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=pk-[removed].pem
export EC2_CERT=cert-[removed].pem
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/
export AWS_AUTO_SCALING_HOME=~/ec2/AutoScaling
export AWS_CLOUDWATCH_HOME=~/ec2/CloudWatch
export PATH=$PATH:$AWS_AUTO_SCALING_HOME/bin:$AWS_CLOUDWATCH_HOME/bin

17
17

Test if it works
% as-version
Amazon AutoScaling CLI version 1.0.61.3 (API 2011-01-01)

18
18

AutoScaling + ELB
(1, 2)
STEP 1: Create Launch Config
as-create-launch-config LC1 --image-id ami-31814f58 --instance-type m1.small
[--region us-east-1] --group SG

STEP 2: Autoscaling with ELB

as-create-auto-scaling-group ASG1 --launch-configuration LC1 --availabilityzones us-east-1b us-east-1c --min-size 2 --max-size 20 --load-balancers aws201

Check if group is created, any running EC2

as-describe-auto-scaling-groups --headers
as-describe-auto-scaling-instances --headers
19
19

AutoScaling + ELB
Emergency break
Scale in EC2 instances to ZERO, if needed
as-update-auto-scaling-group ASG1 --min-size 0 --max-size 0

20
20

AutoScaling + ELB
(3)
STEP 3: Create Scaling Out Policy
as-put-scaling-policy SOut1 --auto-scaling-group ASG1 --adjustment=1 --type
ChangeInCapacity --cooldown 120
arn:aws:autoscaling:useast-1:132891460296:scalingPolicy:a4a80ac4-3e3e-46a7-9b4a-72d893803618:autoSca
lingGroupName/ASG1:policyName/SP1

21
21

AutoScaling + ELB
(4)
STEP 4: Create Metric Out Alarm
mon-put-metric-alarm MOutA1

--comparison-operator GreaterThanThreshold --

evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" -period 60 --statistic Average --threshold 70 --alarm-actions
arn:aws:autoscaling:useast-1:132891460296:scalingPolicy:a4a80ac4-3e3e-46a7-9b4a-72d893803618:autoSca
lingGroupName/ASG1:policyName/SOut1 --dimensions "AutoScalingGroupName=ASG1"

22
22

AutoScaling + ELB
(5)
STEP 5: Create Scaling In Policy
as-put-scaling-policy SInP1 --auto-scaling-group ASG1 --adjustment=-1 --type
ChangeInCapacity --cooldown 120
arn:aws:autoscaling:us-east-1:132891460296:scalingPolicy:5ddb0046-a571-417fb025-4268baf9d860:autoScalingGroupName/ASG1:policyName/SInP1

If you are using Windows, wrap the --adjustment parameter in quotation marks:
"--adjustment=-1".

23
23

AutoScaling + ELB
(6)
STEP 6: Create Metric In Alarm
mon-put-metric-alarm MInA1 --comparison-operator LessThanThreshold -evaluation-periods 1 --metric-name CPUUtilization --namespace "AWS/EC2" -period 60 --statistic Average --threshold 30 --alarm-actions
arn:aws:autoscaling:us-east-1:132891460296:scalingPolicy:5ddb0046-a571-417fb025-4268baf9d860:autoScalingGroupName/ASG1:policyName/SInP1 --dimensions
"AutoScalingGroupName=ASG1"

24
24

Script you can use

within each EC2 instance, at boot
#!/bin/bash
aws_instance=$(wget -q -O- http://169.254.169.254/latest/meta-data/instanceid)
aws_region=$(wget -q -O- http://169.254.169.254/latest/meta-data/hostname)
aws_region=${aws_region#*.}
aws_region=${aws_region%%.*}
aws_zone=`ec2-describe-instances $aws_instance --region $aws_region`
aws_zone=`expr match "$aws_zone" ".*\($aws_region[a-z]\)"`
print "Instance: $aws_instance<br>Region: $aws_region<br>Availability Zone:
$aws_zone<br><br>END" > index.html

25
25

EC2 Autoscaling

03

Automatically scale out EC2

within ELB
(NEW version)

26

27

EC2 metadata

04

How to get details related to

the EC2 machine that Im
running (PHP)

28

29

30

VPC + Cloudformation

05

Using CF to create a 3tier architecture within

VPC.
(first steps)

31

{
"AWSTemplateFormatVersion" : "2010-09-09",
!
"Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture",
!
"Parameters" : {
!
!
"VpcCidr" : {
!
!
!
"Description" : "CIDR for the VPC",
!
!
!
"Type" : "String",
!
!
!
"MinLength": "9",
!
!
!
"MaxLength": "18",
!
!
!
"Default" : "10.0.0.0/16",
!
!
!
"AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\
\d{1,2})",
!
!
!
"ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/
x."
!
!
}
},
!
"Resources" : {
"VPC" : {
"Type" : "AWS::EC2::VPC",
"Properties" : {
"CidrBlock" : {"Ref" : "VpcCidr"},
"Tags" : [
{"Key" : "Application", "Value" : {"Ref" : "AWS::StackId"} }
]
}
}
!
}
}
32

{
"AWSTemplateFormatVersion" : "2010-09-09",
!
"Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture",
!
"Parameters" : {
!
!
"VpcCidr" : {
!
!
!
"Description" : "CIDR for the VPC",
!
!
!
"Type" : "String",
!
!
!
"MinLength": "9",
!
!
!
"MaxLength": "18",
!
!
!
"Default" : "10.0.0.0/16",
!
!
!
"AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\
\d{1,2})",
!
!
!
"ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/
x."
!
!
}
},
!
"Resources" : {
"VPC" : {
"Type" : "AWS::EC2::VPC",
"Properties" : {
"CidrBlock" : {"Ref" : "VpcCidr"},
"Tags" : [
{"Key" : "Application", "Value" : {"Ref" : "AWS::StackId"} }
]
}
}
!
}
}

Do you hate me?

Too small?

33

{
"AWSTemplateFormatVersion" : "2010-09-09",
!
"Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture",
!
"Parameters" : {
!
!
"VpcCidr" : {
!
!
!
"Description" : "CIDR for the VPC",
!
!
!
"Type" : "String",
!
!
!
"MinLength": "9",
!
!
!
"MaxLength": "18",
!
!
!
"Default" : "10.0.0.0/16",
!
!
!
"AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\
\d{1,2})",
!
!
!
"ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/
x."
!
!
}
},
!
"Resources" : {
"VPC" : {
"Type" : "AWS::EC2::VPC",
"Properties" : {
"CidrBlock" : {"Ref" : "VpcCidr"},
"Tags" : [
{"Key" : "Application", "Value" : {"Ref" : "AWS::StackId"} }
]
}
}
!
}
}
34

{
"AWSTemplateFormatVersion" : "2010-09-09",
!
"Description": "Creates a Multi-AZ VPC with subnets for a three tier architecture",
!
"Parameters" : {
!
!
"VpcCidr" : {
!
!
!
"Description" : "CIDR for the VPC",
!
!
!
"Type" : "String",
!
!
!
"MinLength": "9",
!
!
!
"MaxLength": "18",
!
!
!
"Default" : "10.0.0.0/16",
!
!
!
"AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\
\d{1,2})",
!
!
!
"ConstraintDescription" : "must be a valid CIDR range of the form x.x.x.x/
x."
!
!
}
},
!
"Resources" : {
"VPC" : {
"Type" : "AWS::EC2::VPC",
"Properties" : {
"CidrBlock" : {"Ref" : "VpcCidr"},
"Tags" : [
{"Key" : "Application", "Value" : {"Ref" : "AWS::StackId"} }
]
}
}
!
}
}
35

cfn-create-stack CustomerLittleNetwork --tag

"Key=env; Value=temp" --template-file
stack_cfn_learn_001_vpc_step1.template
cfn-describe-stacks
cfn-describe-stacks CustomerLittleNetwork
cfn-list-stack-resources CustomerLittleNetwork

36

37

38

cfn-update-stack CustomerLittleNetwork
--template-file
stack_cfn_learn_001_vpc_step2.template
cfn-describe-stacks CustomerLittleNetwork
cfn-list-stack-resources CustomerLittleNetwork

39

{
"AWSTemplateFormatVersion" : "2010-09-09",
! "Description": "Creates a Multi-AZ VPC
with subnets for a three tier architecture",
!
!

"Parameters" : {
! "VpcCidr" : {

40

"Parameters" : {
! !
"VpcCidr" : {
! ! !
"Description" : "CIDR for the VPC",
! ! !
"Type" : "String",
! ! !
"MinLength": "9",
! ! !
"MaxLength": "18",
! ! !
"Default" : "10.0.0.0/16",
! ! !
"AllowedPattern" : "(\\d{1,3})\\.(\
\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
! ! !
"ConstraintDescription" : "must be a
valid CIDR range of the form x.x.x.x/x."
! !
},
41

! !
"PublicSubnetCidrA" : {
! ! !
"Description" : "Subnet for public
resources in the first AZ",
! ! !
"Type" : "String",
! ! !
"MinLength": "9",
! ! !
"MaxLength": "18",
! ! !
"Default" : "10.0.1.0/24",
! ! !
"AllowedPattern" : "(\\d{1,3})\\.(\
\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})",
! ! !
"ConstraintDescription" : "must be a
valid CIDR range of the form x.x.x.x/x."
! !
},
42

43

CloudFormer

06

CloudFormation
templates from an
existing deployment

44

45

Custom errors

07

Enabling custom error

pages on CloudFront

46

47

48

49

ZFS on Amazon EC2

08

How to install and run

ZFS on Amazon EC2
(yes, you can!)
Thanks to:
Constantin Gonzalez
Dougal Ballantyne
(demo coming soon)
50

What is ZFS?
ZFS is a file system + logical volume manager.
Features:
- protection against data corruption
- support for high storage capacities
- integration of filesystem and volume management
- snapshots and copy-on-write clones
- continuous integrity checking and automatic repair
- RAID-Z
- native NFSv4 ACLs.
OpenZFS is an Open Source implementation of it.
51

How can I use it?

Launch an EC2 instance with an AMI which supports
ZFS.
1) Linux
2) OmniOS (native ZFS support based on a fork of the
original OpenSolaris project that open-sourced ZFS)
3) FreeBSD (native ZFS support based on a port of the
OpenSolaris ZFS code)

52

1) Linux
Linux AMI:
https://aws.amazon.com/amazon-linux-ami/
http://zfsonlinux.org/

53

2) OmniOS

54

3) FreeBSD
55

And now what?

It works the same as with any other hardware
platform:
1) Attach a few EBS volumes to your EC2 instance;
2) use them as you would use regular disks with ZFS
An example follows.

56

$ sudo modprobe zfs

[attach 3 EBS volumes to the instance]
$ sudo zpool create test raidz sdf sdg sdh
$ sudo zpool status
pool: test
state: ONLINE
scan: none requested
config:
NAME
STATE
READ WRITE CKSUM
test
ONLINE
0
0
0
raidz1-0 ONLINE
0
0
0
sdf
ONLINE
0
0
0
sdg
ONLINE
0
0
0
sdh
ONLINE
0
0
0
errors: No known data errors

57

$ mount
/dev/xvda1 on / type ext4 (rw,noatime)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
test on /test type zfs (rw,xattr)
$ df -h
Filesystem
Size Used Avail Use% Mounted on
/dev/xvda1
7.9G 1.6G 6.3G 20% /
tmpfs
15G
0
15G
0% /dev/shm
test
196G
0 196G
0% /test

58

What can I do with ZFS?

Many things! Heres one example:
Use an EBS Provisioned IOPS volumes as ZFS cache
and log (ZIL) devices to accelerate your zpools.
This works much like you would use SSDs in the real
world to accelerate ZFS reads and writes.
For instance, you could provision an EBS volume with
1000 IOPS and use it as a ZIL device for your zpool in
order to accelerate database writes.

59

Learn more
http://pthree.org/2012/04/17/install-zfs-on-debiangnulinux/

60

Extra stuff

09

Friends, nice people,

cool things, etc.

61

62

63

64

65

66

Thank
you!
67

things
you dont know about
Amazon Web Services
Simone Brunozzi
Sr. Technology Evangelist, AWS

@simon
68