Infrastructure Testing: Cucumber vs Rspec


As I wrote previously, I was inspired by this blog post on DNS testing with cucumber. I decided to try an experiment along the same lines. I compared cucumber and rspec for infrastructure testing. I liked cucumber’s literate programming style and wanted to see how it compared to rspec’s sparse style specifically for infrastructure testing. ‘Infrastructure testing’ is a term I have seen the serverspec people use and I think it describes the intent of this testing better than the term ‘integration testing’.

I developed four DNS tests:

  1. Test host name to IP address lookup (A records) for internal, static addresses
  2. Test IP address to host name reverse lookup (PTR records) for internal, static addresses
  3. Test that the IP address of host records created via DHCP and DDNS fell into the correct range
  4. Test that the servers can resolve an external host name

I have two DNS servers, a master and slave. Each of the four tests were run against both to verify that replication is taking place. Test number three also indirectly tests the DDNS configuration of my DHCP server. The test code is listed below.

After creating and running the tests with both frameworks I decided I preferred rspec. I did like the self documenting style of cucumber better but ultimately I preferred the flexibility of rspec and the fact that I can mix rspec and serverspec tests. Mixing these tests means that I can test the client and server perspectives in one test suite. That’s a powerful capability. If the serverspec team will add the ‘delegate’ feature I requested then I could test from the perspective of multiple clients on different subnets.

Framework Notes

The frameworks were similar in many ways. As you can see from my example below they use similar code and a similar test structure. cucumber is self documenting but with rspec you have to be careful to add documentation in the form of addition describe blocks. I found the default error reports in both to be clunky and very technical—both show a Ruby stack trace. There may be more non-technical user friendly report formats or libraries but I relied on the defaults. I like cucumber’s report the best since it was the most readable.

cumber does require that you follow it given-when-then format which isn’t too difficult be still felt artificial to me. That may be because I am more accustomed to unit test frameworks. In my view the ability of cucumber tests to communicate to non-technical users outweighs the rigidity of its format. The issue I had with cucumber’s format was repeating example cases. cucumber doesn’t allow Scenario Outlines to share examples. This means that any change has to be made in multiple places and that is an unwelcome attribute.

I thought a lot about using cucumber as an historical record of changes. After this experiment I’ve changed my mind. I now think it would be too cumbersome to use cucumber for this purpose. I have to go back to my idea of tracking the change ID using git. This means:

  1. enter the change in the change management system and assign it an ID
  2. make the change in the automated systems management tool (e.g., Ansible) store the change ID in git
  3. make the change in the infrastructure tests and store the change ID in git

I will experiment with this process also and will hopefully find a better way.

Here are some specific notes I made during the experiment:

cucumber

  • Didn’t like that I had to repeat the example tables. This makes it less maintainable.
  • Features are very readable.
  • It takes work to keep the feature and step files in sync. A small change in the wording of a feature means the step must change also. I spent a lot of time keeping them in sync.
  • Its ability to auto-import support code was nice.
  • Nice report printout during test runs.
  • The wording for each scenario element must be different otherwise cucumber cannot tell which belongs to which. This can be used to reduce duplication (see how I used the same Given code for all tests) or it can cause unexpected errors. This is the price for the loose structure of cucumber.
  • It does generate the step skeleton for you when it can’t find a match. That is a nice feature.
  • This was the first time I used it and I thought it had a comparatively easy learning curve. As with most open source projects its primary documentation is abysmal. However, there are many examples on the Internet to copy, one of which is cucumber’s own test set.

rspec

  • Also easy to wire the code and tests together using the spec_helper.rb file.
  • Error printout has a lot of opaque technical jargon such as ‘Run options: include {:focus=>true}’. I couldn’t find a way to suppress this.
  • I found I could easily mix plain rspec and serverspec code. This feature tilted the scale in favor of rspec.
  • I could eliminate duplication by putting my host databases in external csv files. This will make maintaining them easier.
  • The test structure in rspec is more flexible that in cucumber. It does not require the somewhat artificial when-then code separation.

Experiment Code

I will explain the code and steps of my experiment here. This will assume you are already familiar with cucumber, rspec, and Ruby. I also assume you are using a Unix workstation. The blog I mentioned above has an excellent cucumber tutorial which I highly recommend.

To get started you need Ruby version 1.8.7 (the ancient version I have on CentOS) or later, and ruby gems installed. Then install the following gems:

  • rspec
  • cucumber
  • ipaddress

Ruby should automatically load any other dependencies when you install these gems so I think they are all you need.

cucumber test

I used the following file structure:

dns.cucumber/
    features/
        dns_host_resolution.feature
        steps/
            dns_host_resolution_steps.rb
        support/
            dns.rb
            env.rb

The ‘/’ character above means a directory. All these files are shown below except for env.rb. To create env.rb just type touch dir/env.rb where dir is the full path to the file.

The 4 tests (called ‘scenarios’ by cucumber) are listed in the feature file below. It is very readable but note the repetition of the DNS server info and the examples.

Feature: DNS name resolution

  All the DNS servers in my network should return the correct hostname and IP address when queried. If the records are propagating correctly each host will return the same answer. Hosts entered through DHCP should also resolve correctly.

  Scenario Outline: All DNS servers return the correct IP addresses for a given hostname
    Given my DNS servers are:
       | 192.168.11.1   | Master |
       | 192.168.11.2   | Slave  |
    And my domain name is "example.com"
    When I query my DNS servers for host <hostname>
    Then I should receive the IP address <ip_address> from each DNS server

 Examples:
      | hostname                       | ip_address       |
      | "gateway.example.com"          | "192.168.12.1"   |
      | "mail.example.com"             | "192.168.12.3"   |

Scenario Outline: All DNS servers return the correct hostname for a given IP address
    Given my DNS servers are:
       | 192.168.11.1   | Master |
       | 192.168.11.2   | Slave  |
    And my domain name is "example.com"
    When I query my DNS servers for IP address <ip_address>
    Then I should receive the hostname <hostname> from each DNS server

 Examples:
      | hostname                       | ip_address       |
      | "gateway.example.com"          | "192.168.12.1"   |
      | "mail.example.com"             | "192.168.12.3"   |

Scenario Outline: All DNS servers return an IP for dynamic hostnames within the expected range
  Given my DNS servers are:
   | 192.168.11.1   | Master |
   | 192.168.11.2   | Slave  |
  And my domain name is "example.com"
  When I query my DNS servers for dynamic host <hostname>
  Then I should receive an IP address between <start_ip> and <end_ip> from each DNS server

    Examples:
      | hostname                      | start_ip            | end_ip              |
      | "ddns1.example.com"           | "192.168.12.100/24" | "192.168.12.200/24" |
      | "ddns2.example.com"           | "192.168.12.100/24" | "192.168.12.200/24" |

Scenario Outline: All DNS servers can resolve an external hostname
    Given my DNS servers are:
       | 192.168.11.1   | Master |
       | 192.168.11.2   | Slave  |
    And my domain name is "example.com"
    When I query my DNS servers for the external host <hostname>
    Then it should be resolvable by each DNS server

 Examples:
      | hostname         |
      | "www.google.com" |
      | "www.cnn.com"    |
      | "www.example.com"  |

The steps file is where the code for each scenario resides. The nice thing here is that I could re-use the same given code for each scenario. That reduced the duplication at the code level.

Given(/^my DNS servers are:$/) do |servers|
 @server_data_table = servers.raw
end

Given(/^my domain name is "(.*?)"$/) do |domain_name|
  @dns_servers ||= []
  @server_data_table.each do |row|
   @dns_servers.push( DNS.new(row[0], row[1], domain_name ) )
 end
end

# A records
When(/^I query my DNS servers for host "(.*?)"$/) do |hostname|
  @responses = Hash.new
  @dns_servers.each do |nameserver|
    @responses[ nameserver ] = nameserver.address( hostname )
  end
end

Then(/^I should receive the IP address "(.*?)" from each DNS server$/) do | expected_ip_address |
  @responses.each do| server, ip_address |
    ip_address.should eql(expected_ip_address), "Server #{server} returned #{ip_address} instead of #{expected_ip_address}"
  end
end

# PTR records
When(/^I query my DNS servers for IP address "(.*?)"$/) do |ip_address|
 @responses = Hash.new
 @dns_servers.each do |nameserver|
    @responses[ nameserver ] = nameserver.hostname( ip_address )
end
end

Then(/^I should receive the hostname "(.*?)" from each DNS server$/) do |expected_hostname|
  @responses.each do| server, hostname |
    hostname.should eql(expected_hostname), "Server #{server} returned #{hostname} instead of #{expected_hostname}"
  end
end

# dhcp ddns
When(/^I query my DNS servers for dynamic host "(.*?)"$/) do |hostname|
  @responses = Hash.new
  @hostname = hostname
  @dns_servers.each do |nameserver|
    @responses[ nameserver ] = nameserver.address( hostname )
  end
end

Then(/^I should receive an IP address between "(.*?)" and "(.*?)" from each DNS server$/) do |start_ip, end_ip|
  @responses.each do| server, ip_address |
    ip_address.should_not eql('Hostname not found'), "Server #{server} could not resolve #{@hostname}"
    server.host_in_range?( ip_address,start_ip,end_ip).should be_true, "Server #{server} returned IP address #{ip_address} outside of range."
  end
end

# External
When(/^I query my DNS servers for the external host "(.*?)"$/) do |hostname|
  @responses = Hash.new
  @hostname = hostname
  @dns_servers.each do |nameserver|
    @responses[ nameserver ] = nameserver.address( hostname )
  end
end

Then(/^it should be resolvable by each DNS server$/) do
  @responses.each do| server, ip_address |
    ip_address.should_not eql('Hostname not found'), "Server #{server} could not resolve #{@hostname}"
  end
end

To offer a simple interface for the tests I created a helper class called dns.rb. It provides a simple API to the Ruby resolv library. I used this class for the rspec experiment as well so I will only include the file it here.

require 'rubygems'
require 'resolv'
require 'ipaddress'

class DNS

  # ip_address = dotted notation as a string, e.g. '192.168.1.1'
  # role = 'Master' or 'Slave'
  # domain_name = the domain name as a string, e.g. 'sharknet.us'
  def initialize(ip_address, role, domain_name)
    @role = role
    @ip_address = ip_address
    @resolver = Resolv::DNS.new(
      :nameserver => [ip_address],
      :search => [domain_name],
      :ndots => 1)
    # @resolver.timeouts = 2 # Enable for Ruby 2.1 or newer. Fail if lookup takes longer than 2 seconds
  end

  # returns the ip address string
  def name
    @ip_address
  end

  # returns the role string
  def role
    @role
  end

  # returns the DNS server as a string in the form '192.168.1.1 [Master]'
  def to_s
    "#{name} [#{role}]"
  end

  # returns the ip address as a string or 'Hostname not found' if there was any error
  def address( hostname )
   begin
    @resolver.getaddress( hostname ).to_s
  rescue
   'Hostname not found'
 end
end

# returns the hostname as a string or 'IP address not found' if there was any error
def hostname( ip_address )
  begin
   @resolver.getname( ip_address ).to_s
 rescue
   'IP address not found'
 end
end

  # return true if ip is between (inclusive) ip_address_start and ip_address_end
  # ip_address_start and ip_address_end use CIDR notation; e.g., '192.168.1.1/24'
  # All the addresses must be passed as strings
  def host_in_range?( ip, ip_address_start, ip_address_end )
    return ip >= ip_address_start && ip <= ip_address_end
  end

end

Once you recreate this file structure then simply run the command cucumber features while in the dns.cucumber/ directory.

rspec

For rspec I used the following file structure:

dns.rspec/
    .rspec
    lib/
        dns.rb
    spec/
        dns_spec.rb
        spec_helper.rb
        records.csv
        dynamic_hosts.csv

You should first create the dns.rspec/ directory. Then from a command line in that directory type rspec --init. This will create a few files that rspec needs.

I changed the .rspec file that rspec generates with the init command to produce a more verbose report. Mine contains:

--color
--format doc

The dns.rb file in the lib/ directory is the same one I used for cucumber. Under spec/ I changed spec_helper.rb by adding the single line require dns at the top. Otherwise it is the same as init generated.

require 'dns'

# This file was generated by the `rspec --init` command. Conventionally, all
# specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`.
# Require this file using `require "spec_helper"` to ensure that it is only
# loaded once.
#
# See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration
RSpec.configure do |config|
  config.treat_symbols_as_metadata_keys_with_true_values = true
  config.run_all_when_everything_filtered = true
  config.filter_run :focus

  # Run specs in random order to surface order dependencies. If you find an
  # order dependency and want to debug it, you can fix the order by providing
  # the seed, which is printed after each run.
  #     --seed 1234
  config.order = 'random'
end

The dns_spec.rb file contains the actual test code. It is more compact than the cucumber step code and contains no duplication. To prevent it from being overly terse (which is worse than overly verbose) I tried to use descriptive it and describe text. The DNS server objects are instantiated once and then re-used by each test. I moved the host lists to external CSV files so that I could update them independently. My actual list contained all my A and PTR records. CSV files are easy to manage and I use them often.

require 'spec_helper.rb'
require 'csv'

describe "DNS Server Infrastructure" do
    describe "Record Lookup" do
        before(:all) do
            # Cache DNS servers
            @dns_servers ||= []
            @dns_servers.push( DNS.new('192.168.11.1', 'Master', 'example.com' ))
            @dns_servers.push( DNS.new('192.168.11.2', 'Slave', 'example.com' ))

            # Load DNS records from a CSV file
            @records = CSV.readlines('spec/records.csv')
        end

        it "Should return the correct IP address for static hostnames (A records)" do
            @dns_servers.each do |nameserver|
                @records.each do |record|
                    if record[2] == 'A'
                        ip = nameserver.address( record[0] )
                        expect(ip).to eql( record[1] ), "Server #{nameserver} returned #{ip} instead of #{record[1]} for #{record[0]}"
                    end
                end
            end
        end

        it "Should return the correct hostname for static IP addresses (PTR records)" do
            @dns_servers.each do |nameserver|
                @records.each do |record|
                    if record[2] == 'A'
                        host = nameserver.hostname( record[1] )
                        expect(host).to eql( record[0] ), "Server #{nameserver} returned #{host} instead of #{record[0]} for #{record[1]}"
                    end
                end
            end
        end

        it "Should return an IP within the DHCP scope range." do
            @dns_servers.each do |nameserver|
                CSV.open('spec/dynamic_hosts.csv', 'r') do |record|
                    ip = nameserver.address( record[0] )
                    expect(ip).to_not eql('Hostname not found'), "Server #{nameserver} could not resolve #{record[0]}"
                    expect( nameserver.host_in_range?( ip,record[1],record[2])).to be_true, "Server #{nameserver} returned IP address #{ip} for #{record[0]} outside of range."
                end
            end
        end

        it "Should resolve external host names." do
            external_hosts = [
                "www.google.com",
                "www.cnn.com",
                "tampa.voip.ms",
                "www.facebook.com"
            ]
            @dns_servers.each do |nameserver|
                external_hosts.each do |host|
                    ip = nameserver.address( host )
                    expect(ip).to_not eql('Hostname not found'), "Server #{nameserver} could not resolve #{host}"
                end
            end
        end

    end
end

The two CSV files are below. The static hosts are in records.csv. The DHCP managed hosts are in dynamic_hosts.csv. I included a record type in records.csv because I intend to add tests for MX, NS, CNAME, and other records later.

gateway.example.com,192.168.12.1,A
mail.example.com,192.168.12.3,A

The addresses in the dynamic_hosts.csv require a CIDR format netmask.

ddns1.example.com,192.168.12.100/24,192.168.12.200/24
ddns2.example.com,192.168.12.100/24,192.168.12.200/24

When you are done customizing these files for your environment, change to the dns.rspec/ directory in a terminal and run the command rspec spec/dns_spec.rb. If all your tests pass you will get a report that looks like this:

Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

DNS Server Infrastructure
  Record Lookup
    Should resolve external host names.
    Should return the correct hostname for static IP addresses (PTR records)
    Should return the correct IP address for static hostnames (A records)
    Should return an IP within the DHCP scope range.

Finished in 1.04 seconds
4 examples, 0 failures

Conclusion

Now that I have decided (for now at least) to use rspec for infrastructure testing my next step will be to integrate it with serverspec. I plan to expand these DNS tests to fully cover all DNS functions. I would like to have these client tests run from a variety of servers on different subnets. That may entail creating a custom servespec type which means using the Unix dig command instead of pure Ruby. More work to do!



Categories: DevOps, Testing

Tags: ,

Share Your Ideas

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: