Integration testing
With integration testing, your tests move into testing beyond your code. With ChefSpec, we looked at how to verify that a recipe was executing statements as expected. However, unit testing has its limits; integration testing completes the picture by testing cookbooks in conjunction with real hosts that run the desired operating system(s):
This means testing cookbooks from the outside rather than from the inside, where a unit test provides very narrow testing of code inside of cookbooks, integration tests perform deployment of your cookbooks in a test-specific environment and then execute probes to validate that the system is in the desired state. One of the most popular tools for this is Test Kitchen, which we will take a brief tour of.
Using Test Kitchen
Test Kitchen is an open source tool that helps to automate integration testing of Chef cookbooks. Where a ChefSpec test would validate that a file resource was called during an execution, a similarly tasked Test Kitchen test would spin up a new instance, execute your recipe(s), and then validate that the file was actually created on all platforms that you expect the recipe to work on. One of the great things about Test Kitchen is that it supports a number of different driver plugins to manage your target hosts, including Vagrant (the default), EC2, OpenStack, Docker, and Rackspace Cloud among others. This enables you to test your cookbooks not only on local virtual machines using Vagrant, but to verify that they work correctly on a cloud service as well. The ability to perform integration tests on different types of hosts brings your tests that much closer to matching a production environment, thereby increasing your confidence in the changes you are making.
Currently, Test Kitchen is distributed as a Ruby gem, so installation is quite straightforward:
gem install test-kitchen
By installing this gem, you are also installing a command-line tool, kitchen, which is used to interact with Test Kitchen. Similar to Vagrant, Bundler, and other tools, Test Kitchen uses a configuration file to store information about what to test, where to test it (on what virtual machines), and how to test it.
As mentioned before, Test Kitchen focuses on integration testing of your Chef components. This means that it needs to be able to execute your recipes on a host (or a set of hosts) and then invoke a set of tests to validate that the expected behavior occurred on the endhost(s). Test Kitchen is responsible for the following:
- Provisioning a clean host for testing
- Installing Chef onto the new host
- Executing our recipes on the host
- Validating the behavior of the recipes on the host
So, let's get started with some actual testing!
In order to demonstrate how to use Test Kitchen, we will need to write a cookbook that we can run on our test hosts and write tests to validate the behavior. In order to focus on how to use Test Kitchen, we will take a look at a very simple cookbook that just creates a file on the host filesystem so that it is guaranteed that it will work on both Ubuntu and CentOS platforms.
First, create a place to do your work:
mkdir -p testfile-cookbook/recipes
Then, add a very simple metadata.rb in your testfile-cookbook directory with the following contents inside:
name "testfile" version "1.0"
Once that is complete, add a default recipe, recipes/default.rb, with the sample recipe code as follows:
file "/tmp/myfile.txt" do content "My awesome file!" end
Now that we have a complete (albeit simple) cookbook, let's take a look at how to test it using Test Kitchen.
In order to start using Test Kitchen, you will need to prepare your test environment. Test Kitchen relies on a configuration file, .kitchen.yml, to tell it what to do. You can generate it by hand, or you could use the init command as part of the kitchen tool:
kitchen init
This will do a few things for you: first, it will create a .kitchen.yml file with some sane defaults in the current directory. Then, it will create a test/integration/default directory for your integration tests, and then it will install the Vagrant driver for Test Kitchen so that it can interact with Vagrant virtual machines (if it has not already been installed).
If you look at the .kitchen.yml file, you will see that the initial file contains the following YAML code:
---
driver:
name: vagrant
provisioner:
name: chef_solo
platforms:
- name: ubuntu-12.04
- name: centos-6.4
suites:
- name: default
run_list:
- recipe[testfile::default]
attributes:
This configuration file instructs Test Kitchen to use Vagrant to manage the target instances and to use Chef-solo for provisioning, and it should execute the default suite of tests on both Ubuntu 12.04 and CentOS 6.4. Of course, you can always modify or extend this list if you have other systems that you want to run tests on, but this is a reasonable default list for now.
Notice that we don't have any attributes specified as our default recipe and it does not use attributes. If you need to provide attributes to test recipes, this would be the place to do it, which is laid out as a dictionary in YAML. Each test suite has its own run list and defined attributes that allow you to check the behavior of a variety of configuration data and recipe combinations.
Testing a cookbook with Test Kitchen is outlined in the following three steps:
- Provisioning a host if needed.
- Converging the host so that it is up to date.
- Executing tests.
Let's take a look at how we will perform these with our simple cookbook to check that it works properly.
Before you can test, you will need to prepare the instances for testing—this is done using the kitchen create <instance name> command. Only you don't know what instance to bring up just yet. To get the list of instances that can be run, we will use the list subcommand:
[jewart]$ kitchen list Instance Driver Provisioner Last Action default-ubuntu-1204 Vagrant ChefSolo <Not Created> default-centos-64 Vagrant ChefSolo <Not Created>
You will see that this list is generated by a combination of the platforms and the suites listed in the .kitchen.yml file. If you were to define a new suite, named server, then your list would include two additional instances, server-ubuntu-1204 and server-centos-64.
Once you have seen this list, you can create an Ubuntu 12.04 instance with the following command:
kitchen create default-ubuntu-1204
This will use Vagrant and VirtualBox to provision a new headless Ubuntu 12.04 host and boot it up for you to start testing with. If you don't already have an Ubuntu 12.04 Vagrant image downloaded, then it will be downloaded for you automatically (this is a large image so it may take a while to complete this operation, depending on your connection speed, if you are following along).
This will look familiar to those that used Vagrant at the beginning of the chapter:
[jewart]% kitchen create default-ubuntu-1204 -----> Starting Kitchen (v1.2.1) -----> Creating <default-ubuntu-1204>... Provisioning happens... Finished creating <default-ubuntu-1204> (4m2.59s). -----> Kitchen is finished. (4m2.83s)
However, this only builds the virtual machine for our tests; it does not run our recipes or our tests in the newly constructed host. Before we move on to writing a test, let's take a look at how to run our recipe inside our instance.
The next step is to execute our run list ("converge" in Chef parlance) on the new instance. This is done with Test Kitchen's converge command and can be used for all or one specific instance. In order to converge our Ubuntu 12.04 instance, the following command is used:
kitchen converge default-ubuntu-1204
What this will do is transfer any required data files, install Chef as needed, and then execute the run list for the specified suite (default in this case) on the instance. Here is a sample run of converge (with some parts removed):
[jewart]% kitchen converge default-ubuntu-1204 -----> Starting Kitchen (v1.2.1) -----> Converging <default-ubuntu-1204>... Preparing files for transfer Preparing current project directory as a cookbook Removing non-cookbook files before transfer -----> Installing Chef Omnibus (true) Installation of Chef happens... Compiling Cookbooks... Converging 1 resources Recipe: testfile::default
And then you will see the output of a normal Chef run after that—at this point, the run list is complete and the instance has been converged to the latest state. Now that it's been converged, we will continue to write a very simple test to verify that our recipe did the right thing.
Tests are stored in a directory whose structure is similar to other Chef components—the directory we created previously with kitchen init, tests/integration/default, allows us to keep our integration tests separate from spec tests or other types of tests. The integration directory will contain one directory per suite so that test files are grouped together based on the particular component or aspect of your cookbook that is being tested. Additionally, depending on the type of the test framework being used, your tests will be contained in another child directory for the given suite. In this case, we will take a look at the BASH Automated Testing System (BATS), so our test will be placed in the tests/integration/default/bats/file_created.bats file and will look like the following code:
#!/usr/bin/env bats
@test "myfile.txt exists in /tmp" {
[ -f "/tmp/myfile.txt" ]
}
This allows us to use the simple -f BASH test (which returns true if the specified value exists) to guarantee that the file was created on the instance.
Next, we can run this test with kitchen verify default-ubuntu-1204 and see that the BATS plugin was installed and that our test was executed and passed:
[jewart]% kitchen verify default-ubuntu-1204 -----> Starting Kitchen (v1.2.1) -----> Setting up <default-ubuntu-1204>... Fetching: thor-0.19.0.gem (100%) Fetching: busser-0.6.2.gem (100%) Successfully installed thor-0.19.0 Successfully installed busser-0.6.2 2 gems installed -----> Setting up Busser Creating BUSSER_ROOT in /tmp/busser Creating busser binstub Plugin bats installed (version 0.2.0) -----> Running postinstall for bats plugin Installed Bats to /tmp/busser/vendor/bats/bin/bats Finished setting up <default-ubuntu-1204> (0m8.94s). -----> Verifying <default-ubuntu-1204>... Suite path directory /tmp/busser/suites does not exist, skipping. Uploading /tmp/busser/suites/bats/file_created.bats (mode=0644) -----> Running bats test suite √ myfile.txt exists in /tmp 1 test, 0 failures Finished verifying <default-ubuntu-1204> (0m0.67s). -----> Kitchen is finished. (0m9.86s)
To demonstrate what happens if the file does not exist, we can clone our test to create a second simple test that validates the existence of /tmp/myotherfile.txt, and run our verify command again without making a corresponding change to our recipe. The output from Test Kitchen will tell us that our test failed and why:
-----> Running bats test suite √ myfile.txt exists in /tmp x myotherfile.txt exists in /tmp (in test file /tmp/busser/suites/bats/other_file.bats, line 4) 2 tests, 1 failure Command [/tmp/busser/vendor/bats/bin/bats /tmp/busser/suites/bats] exit code was 1 >>>>>> Verify failed on instance <default-ubuntu-1204>.
Fortunately, the fine folks that created Test Kitchen realized that it would be tedious to run all three steps every time you wanted to run some tests. As a result, there is the kitchen test command that will provision an instance, execute the run list, verify the results, and then tear down the instance with only one command. In this case, you can replace them with the following single command:
kitchen test default-ubuntu-1204
This covers the basics of testing your cookbooks with Test Kitchen. There are other things that can be done with Test Kitchen, including using other testing mechanisms, testing cookbook dependencies, validating whether services are running, fully automating Test Kitchen as part of your release process, and plenty more. For more information, visit the project at http://kitchen.ci/.