Leveraging the cloud
Cloud computing providers such as Rackspace Cloud and Amazon EC2 provide on-demand computing power at the push of a button, a feature that has become immensely popular with developers and systems administrators alike. One of the most touted benefits of cloud computing is cost savings; however, these on-demand instances can become very expensive if they are left running. Often the capacity of time will be configured in order to handle large-scale events and then left online because of the time required to reconfigure the systems if they are needed again. As underutilized capacity ends up costing money rather than saving it, being able to reduce or expand the capacity quickly and easily will help you match your computing needs while saving both time and money.
This section specifically looks at two of the more popular cloud providers: Amazon EC2 and Rackspace Cloud; however, there are others, and the techniques described here will be broadly applicable to any other supported cloud provider.
Amazon EC2
Amazon EC2 is a very popular cloud-computing platform, and knife has support to manage EC2 instances from the command line through the knife-ec2 plugin. The following steps demonstrate how you can work with EC2:
As of Chef 0.10, the ec2 subcommands have been moved from being built in knife to an external gem, knife-ec2. In order to use EC2 commands, you will need to install the gem, which can be done via the following command:
gem install knife-ec2
This will install all of the gem dependencies that the EC2 plugin requires.
Tip
Some of the cloud provider plugins have conflicting dependencies, so it may be best to leverage a gem manager in order to isolate them. For example, using RVM or rbenv, you might create one Rubygem environment per provider so that you could switch back and forth with a simple command such as rvm gemset use chef-ec2.
In order to manage your EC2 hosts, you will need your EC2 key-pair properly registered with SSH and your AWS access keys set in your knife configuration file.
To do the first, make sure you have your EC2 SSH keys downloaded and registered with your SSH agent. One way to do this is to add the following to your SSH configuration file, typically, $HOME/.ssh/config:
Host *.amazonaws.com ForwardAgent yes CheckHostIP no StrictHostKeyChecking no UserKnownHostsFile=/dev/null IdentityFile ~/.ssh/ec2_keypair.pem
In order to configure your AWS keys, you will need to add some information to your knife.rb configuration file ($HOME/.chef/knife.rb):
knife[:aws_access_key_id] = "YOUR ACCESS KEY" knife[:aws_secret_access_key] = "SECRET KEY"
These keys tell knife which AWS credentials to use when making API calls to perform actions such as provision new hosts and terminate instances. Without this, knife will be unable to make API calls to EC2. With these required changes made, let's look at how to create a new EC2 instance with knife.
Initially, we will look at provisioning an instance using one of the Ubuntu AMIs. With knife, we can specify the AMI to use, the availability zone to target, and the size instance to be created. For example, to create an m1.large size in the us-east-1e availability zone with Ubuntu 12.04.3 LTS, we would need to use the AMI with ami-23447f4a as its identifier.
In order to determine the AMI ID, you will need to look it up at the following URL:
The progress of provisioning can be seen using the following command:
$ knife ec2 server create -I ami-23447f4a -f m1.large -Z us-east-1e -N <node name> -x ubuntu --sudo
The output from the previous command will show you the progress of the provisioning (this may take a minute or two, depending on the region, instance size, how long status checks take, and so on):
[user]% knife ec2 server create -I ami-23447f4a -f m1.large -S ec2-keypair -Z us-east-1e –N <node name> -x ubuntu --sudo Instance ID: i-0dfec92d Flavor: m1.large Image: ami-23447f4a Region: us-east-1 Availability Zone: us-east-1e Security Groups: default Tags: Name: i-0dfec92d SSH Key: ec2-keypair Waiting for instance................. Public DNS Name: ec2-54-80-59-97.compute-1.amazonaws.com Public IP Address: 54.80.59.97 Private DNS Name: ip-10-157-31-234.ec2.internal Private IP Address: 10.157.31.234
As you can see, knife will tell you the public IP and public DNS name of the new instance along with the instance ID, tags, and so forth. Once the instance is provisioned and is online, it will need to be bootstrapped. Remember that bootstrapping will install the Chef client and register the instance with the Chef service, which we can do in the same way we bootstrap any other host:
$ knife bootstrap <instance-public-ip-address> -N <node-name> -x ubuntu --sudo
As EC2 provisions each instance with an ubuntu user that has sudo privileges, we provide the bootstrap command with –x ubuntu and --sudo to ensure we have the required privileges to perform the bootstrapping. Additionally, as you more than likely do not want the AWS-provided DNS name as the node name, the Chef node name is set through the -N <node-name> command line flag. Once the bootstrap step is finished, assuming that there are no errors, verify that your newly provisioned host is listed in your chef service:
$ knife node list
The output will contain your newly bootstrapped node ID, as specified by you in the command line or the DNS name, if you don't specify a node name. You have now provisioned a new EC2 instance and registered it with your Chef service with only two commands!
Once you are done with testing, you may not want to leave the EC2 instance running, as it will incur costs if it remains idle. To ensure this doesn't happen, perform the following four steps:
- List your EC2 instances
- Delete the server from EC2
- Remove the server from Chef
- Verify that the instance no longer exists in Chef or EC2
To list our EC2 instances, use the server list subcommand of the ec2 command, which will list all of the EC2 instances in the specified region. If you do not specify a region, us-east-1 is the default region. The full command to list EC2 servers is as follows:
$ knife ec2 server list
As an example, executing this command after provisioning the first host will show a table of one instance as follows:
Instance ID Name Public IP Private IP i-0dfec92d i-0dfec92d 54.80.59.97 10.157.31.234
For most knife commands, you will need the instance ID so the previous table can be truncated to fit in print.
Tip
Listing EC2 nodes will result in a table that contains all the currently provisioned EC2 instances in the region by means of the EC2 API, which is separate from the Chef service API. This means you will get a list of all the instances in EC2 whether or not they are registered with Chef. The full table will contain most of the information you can see on the EC2 control panel, including the public and private IP, flavor, AMI, SSH key, and so on.
Deleting an instance is just as easy as creating or listing them. Here, the server delete subcommand is invoked with the instance identifier to be terminated. This will use the EC2 API to issue a terminate command—this is not reversible and so the command will prompt you to ensure that you really did want to delete the instance:
[user]% knife ec2 server delete i-0dfec92d Instance ID: i-0dfec92d Flavor: m1.large Image: ami-23447f4a Region: us-east-1 Availability Zone: us-east-1e Security Groups: default SSH Key: ec2-keypair Root Device Type: instance-store Public DNS Name: ec2-54-80-59-97.compute-1.amazonaws.com Public IP Address: 54.80.59.97 Private DNS Name: ip-10-157-31-234.ec2.internal Private IP Address: 10.157.31.234 Do you really want to delete this server? (Y/N) WARNING: Deleted server i-0dfec92d WARNING: Corresponding node and client for the i-0dfec92d server were not deleted and remain registered with the Chef Server
At this point, the EC2 instance is being terminated and removed from your account. However, it is not removed from the Chef service that needs to be done separately with the node delete command. Here, the Chef node name is specified, not the instance identifier:
$ knife node delete my-first-ec2-instance
Verify that the node was removed from Chef using node list:
$ knife node list
The output should show you that your EC2 instance is no longer registered with Chef.
Rackspace Cloud
Rackspace Cloud is another popular cloud-computing provider that is well supported by Chef. Similar to EC2, there is a knife plugin for Rackspace Cloud:
gem install knife-rackspace
In the same way that AWS requires a set of credentials to interact with the API to create and terminate instances, Rackspace Cloud has its own configuration. However, the Rackspace Cloud API is a little simpler; you will need to provide knife with your Rackspace Cloud's username and API key. For those who do not already have their API key, it can be found in your Rackspace Cloud control panel. The appropriate configuration to add to your knife.rb file is as follows:
knife[:rackspace_api_username] = "Your Rackspace API username" knife[:rackspace_api_key] = "Your Rackspace API Key"
This data can be hard coded into your configuration file, or since the knife configuration file is just Ruby, it can be generated by evaluating environment variables or looking at a local file. This is useful if you are submitting your knife.rb file into a source repository so that credentials are not leaked.
Rackspace Cloud server provisioning is just as straightforward as it is with EC2. There is some variation in the command-line options passed to knife because of the way Rackspace provides images for systems. Instead of using the instance size and an AMI, you can specify the flavor of the system to provision (the node's CPU, memory, and disk allocation) and the operating system to image the instance with. In order to determine what flavors are available, the knife-rackspace plugin provides the rackspace flavor list subcommand:
$ knife rackspace flavor list --rackspace-region=IAD
As it is possible that there are different capacities in different regions, it is a good idea to check what is available in the region where you want to provision a node. This will result in a list of flavors and their specifications; as of now, some of the current offerings in IAD are as follows:
ID Name VCPUs RAM Disk 2 512MB Standard Instance 1 512 20 GB 3 1GB Standard Instance 1 1024 40 GB 4 2GB Standard Instance 2 2048 80 GB performance1-1 1 GB Performance 1 1024 20 GB performance1-2 2 GB Performance 2 2048 40 GB performance2-120 120 GB Performance 32 122880 40 GB performance2-15 15 GB Performance 4 15360 40 GB
In addition to knowing which flavor to provision, you need an image identifier (similar to an AMI) to apply to the new host. Again, this list may vary with region and possibly change over time so there is a command, rackspace image list, to list the various images:
$ knife rackspace image list --rackspace-region=IAD
The output here is quite long, so it has been sampled to show enough to be useful:
ID Name ba293687-4af0-4ccb-99e5-097d83f72dfe Arch 2013.9 41e59c5f-530b-423c-86ec-13b23de49288 CentOS 6.5 (PVHVM) 857d7d36-34f3-409f-8435-693e8797be8b Debian 7 (Wheezy) 896caae3-82f1-4b03-beaa-75fbdde27969 Fedora 18 (Spherical Cow) fb624ffd-81c2-4217-8cd5-da32d32e85c4 FreeBSD 9.2 1705c794-5d7e-44d6-87da-596e3cf92144 Red Hat Enterprise Linux 6.5 df27d481-63a5-40ca-8920-3d132ed643d9 Ubuntu 13.10 d88188a5-1b02-4b37-8a91-7732e42348c1 Windows Server 2008 R2 SP1
As you can see, there are a number of Linux, BSD, and Windows distributions available to provision. In order to provision a new host, you will use the server create command, similar to the EC2 command. The following knife command will provision a 512 MB host with Ubuntu 13.10 in the IAD datacenter:
$ knife rackspace server create -I df27d481-63a5-40ca-8920-3d132ed643d9 -f 2 --rackspace-region=IAD
As soon as the API responds to the request to provision a new host, you will see the Rackspace metadata for the host, such as the instance ID, name, flavor, and image:
Instance ID: 993d369f-b877-4f0f-be4b-cfc45c240654 Name: rs-21230044929009695 Flavor: 512MB Standard Instance Image: Ubuntu 13.10 (Saucy Salamander)
Shortly after this—once the system has been provisioned, the network interfaces have been configured, and the root password has been set—the IP and root password will be displayed:
Public DNS Name: 162.209.104.248.rs-cloud.xip.io Public IP Address: 162.209.104.248 Private IP Address: 10.176.65.92 Password: yZ3D3Tck8uGm
After SSH becomes available, knife will initiate the process of bootstrapping the host. By default, knife will use the chef-full template, which will install Chef via the omnibus installer for the platform you are bootstrapping. This can be altered by providing knife with the –d command-line option. Assuming that the host is bootstrapped properly, the system data will be displayed once again for your information:
Instance ID: 993d369f-b877-4f0f-be4b-cfc45c240654 Host ID: c478865ebb70032120024a9a2c8c65b9bb0913087991d4bab5acde00 Name: rs-21230044929009695 Flavor: 512MB Standard Instance Image: Ubuntu 13.10 (Saucy Salamander) Public DNS Name: 162.209.104.248.rs-cloud.xip.io Public IP Address: 162.209.104.248 Private IP Address: 10.176.65.92 Password: yZ3D3Tck8uGm Environment: _default
Once the bootstrap step is finished, assuming that there are no errors, verify that your newly provisioned host is listed in your chef service:
$ knife node list
The output will contain your newly bootstrapped node ID as specified by you in the command line (via -N) or the name generated by Rackspace (in this example, it will be rs-21230044929009695). Congratulations! You have provisioned a new Rackspace instance with a single command.
Once you are done with testing, you may not want to leave the EC2 instance running, as it will incur costs if it remains idle. To ensure this doesn't happen, perform the following four steps:
- List your Rackspace servers.
- Delete the server from Rackspace.
- Remove the server from Chef.
- Verify that the instance no longer exists in Chef or Rackspace.
To list your Rackspace instances, use the server list subcommand of the rackspace command, which will list all of the Rackspace instances in the specified region. Similar to the output from the EC2 server list command, the output will look like the following:
$ knife rackspace server list --rackspace-region=IAD Instance ID Name 993d369f-b877-4f0f-be4b-cfc45c240654 rs-21230044929009695
You can delete an instance using a single command; the server delete subcommand is invoked with the Rackspace instance identifier to be terminated. Remember that this is not reversible, so the command will prompt you to ensure that you really do want to delete the instance:
$ knife rackspace server delete 993d369f-b877-4f0f-be4b-cfc45c240654 --rackspace-region=IAD Instance ID: 993d369f-b877-4f0f-be4b-cfc45c240654 Host ID: c478865ebb70032120024a9a2c8c65b9bb0913087991d4bab5acde00 Name: rs-21230044929009695 Flavor: 512MB Standard Instance Image: Ubuntu 13.10 (Saucy Salamander) Public IP Address: 162.209.104.248 Private IP Address: 10.176.65.92 Do you really want to delete this server? (Y/N) y WARNING: Deleted server 993d369f-b877-4f0f-be4b-cfc45c240654 WARNING: Corresponding node and client for the 993d369f-b877-4f0f-be4b-cfc45c240654 server were not deleted and remain registered with the Chef Server
At this point, the EC2 instance is being terminated and removed from your account. However, it is not removed from the Chef service; this needs to be done separately with the node delete command. Here, the Chef node name is specified, not the instance identifier:
$ knife node delete rs-21230044929009695 Verify that the node was removed from Chef with node list: $ knife node list
The output should show you that your recently created Rackspace instance is no longer registered with Chef.