Infrastructure Testing With Ansible and Serverspec: Part 2


This is part of my series on serverspec testing for Ansible. In this part of the guide I will explain how to add hosts to your tests using the properties.yml file and then I will show some example tests. In the next part I will show how to further automate these tests with a continuous integration (CI) server.

File Layout

I will expand the file structure of the first part to include two concrete roles: one for my OS (CentOS), and one for a core application, the BIND DNS server. The OS role tests basic configuration settings that should be standard across all your servers. I use CentOS but you could easily change this to another flavor of Unix and the tests will be the same. serverspec handles the mapping to the specific OS.

serverspec does not test Windows. I’m not sure what one would use for Windows since it lags in tool support. Perhaps there is a PowerShell based test framework out there? I haven’t needed one yet but I may look for one in the future.

In my experience so far, Ansible roles nearly always equate to an application rather than a true role. For example, I have one for BIND, one for postfix, one for nginx, etc. If one of my servers needs several applications then I just include all the relevant roles in my site.yml file. This makes my roles somewhat modular. My serverspec roles under spec/ are named after my Ansible roles.

This github project has a clever way of combining the Ansible and serverspec layouts. For now I prefer to keep them separate but you may prefer this style to mine so I wanted to provide the link.

The file layout for this part is:

lib/
    dns.rb
spec/
    centos/
        centos_spec.rb
    bind/
        dns_spec.rb
        records.csv
    spec_helper.rb
Rakefile
properties.yml
Gemfile
.rspec

Adding hosts

The serverspec site shows two ways to add hosts. I prefer the properties.yml file method. I hope to soon write a generator for this file and for the related Ansible files so I can use a single source database. I do not like having to maintain the same information in several places. That is just asking for bugs!

My properties file looks like this (abbreviated for space):

---

host1.sharknet.us:
  :roles:
    - centos
    - bind
  :networks:
    - device_id: 0
      mac_address: A6:A7:A3:7D:37:10
      ip_address: 192.168.1.5
  :default_gateway: 192.168.1.1

host2.sharknet.us:
  :roles:
    - centos
    - nginx
  :networks:
    - device_id: 0
      mac_address: A6:A7:A3:7D:37:01
      ip_address: 192.168.1.6
    - device_id: 1
      mac_address: A6:A7:A3:7B:37:01
      ip_address: 192.168.2.9
  :default_gateway: 192.168.1.1

host3.sharknet.us:
  :roles:
    - vm
  :vms:
    - host: host1
      uuid: da77acb9-6ee4-63d4-8482-123456789
      vcpus: 1
      ram: 524288000

    - host: host2
      uuid: 89b47b60-5754-ed41-faf9-123456689
      vcpus: 1
      ram: 1073741824

Each host in this file is a separate entry. The first two hosts in this file, host1 and host2, run the tests directly. That means that the severspec tests execute test commands on these machines via SSH. Well, mostly. Actually, I mix in some client side DNS testing in the BIND role but more on that later.

host3 is my Xen server host (a creaky old HP DL360 you might recall if you read my earlier posts). In order to test the VM parameters of the first two virtual hosts I have to run the command on the physical host. host is my Xen server pool master so it doesn’t matter if the VMs are actually running on that physical host. I imagine you can do something similar with VMWare or SmartOS.

This is the type of test where a delegate_to parameter would be useful in serverspec. With that feature I would put the VM attributes under the host itself but delegate it to the Xen pool master just as I do in Ansible.

Under each host you can set custom variables for use in your tests. Network settings (the arch-enemy of DevOps automation) are the ones I use most often. I use the Ansible host variable almost unchanged. Some of these variables are lists because a host can have more than one. I will explain how to reference list variables in the following test section. You can add as many custom variable as you like. The :roles: variable is, however, mandatory. You must at least include that one.

Example tests

Below are just two examples of many and are here to illustrate different testing methods. I highly recommend that you follow these steps in order:

  1. Write your serverspec tests (always, always, write your tests first!) clearly describing the state you want.
  2. Write the Ansible plays that will put your server in that state and make you serverspec tests pass.

There should be a one-to-one correspondence between your tests and your Ansible plays. Every change that Ansible makes must be tested. If you add a package, test it. If you add a user test it, and so on.

Operating System Tests

First here are some snippets from my centos_spec.rb file. Always name your tests with an _spec.

require 'spec_helper'

packages = [
  'libselinux-python',
  'mlocate',
  'vim-minimal',
  'mailx',
  'ntp',
  'git',
  'wget',
  'ferm']

  describe "CentOS Operating System Checks for #{ENV['TARGET_HOST']}" do

    packages.each do|p|
      describe package(p) do
        it { should be_installed }
      end
    end

    describe "Core services should be running" do
      describe service('sshd') do
        it { should be_enabled   }
        it { should be_running   }
      end

      describe port(22) do
        it { should be_listening }
      end

      describe service('iptables') do
        it { should be_enabled   }
        it { should be_running   }
      end

      describe service('postfix') do
        it { should be_enabled   }
        it { should be_running   }
      end
    end

    describe "The network should be configured properly" do

      property[:networks].each do |p|
        describe command("cat /sys/class/net/eth#{p['device_id']}/address") do
          its(:stdout) {should include "#{p['mac_address'].downcase}" }
        end

        describe interface("eth#{p['device_id']}") do
          #its(:speed) { should eq 1000 }
          it { should have_ipv4_address("#{p['ip_address']}/24") }
        end
      end

      describe host( ENV['TARGET_HOST'] ) do
        it { should be_resolvable.by('dns') }
        it { should be_reachable }
      end

      describe command('hostname') do
        its(:stdout) { should include ENV['TARGET_HOST'] }
      end

      describe default_gateway do
        its(:ipaddress) { should eq "#{property[:default_gateway]}" }
      end

      describe command('cat /etc/resolv.conf') do
        its(:stdout) {should include '192.168.1.5'}
      end

    end #Network
end

Most of these are directly from the serverspec guide so I don’t need to explain them. I will highlight a few areas where I have done something different. The packages variable at the top is an embedded list. These tests are Ruby code and so you can do anything a normal Ruby program would allow. I put the packages in a list to make them easier to manage. The packages.each code below it shows how to loop through them and test the presence of each package on the target server.

The target server’s hostname is identified using the global ENV['TARGET_HOST'] variable. You embed that in a Ruby string by adding #{}. That is called “variable interpolation” in case you are interested. All variables from the properties file (or global ones like ENV['TARGET_HOST']) must be interpolated using #{} when they are in a string (i.e., between “quotes”).

I recommend using lots of Describe strings in your tests. The standard serverspec test output is overly terse so add Describe statements to better document them.

The test that begins with property[:networks].each do |p| is where the network list variable from the properties file is used. Each Ethernet adapter (identified by device_id) is tested individually. You reference the variable in a list element using square brackets []. For example p['device_id'] is the Ethernet device number attribute from the properties.yml in the :networks list. p is the loop variable that stands for each element of the list as the test works through them.

Be sure to change the DNS server IP addresses to match your system in the resolv.conf tests.

BIND tests

My DNS test are more complex and require some support files. They are described below.

require 'spec_helper'
require 'csv'

###################################
# All the files are installed
###################################
describe "DNS Server Infrastructure" do

    describe "Check that all configuration files are present" do
        describe file('/etc/named.conf') do
          it { should be_file }
      end

      describe file('/etc/named.rfc1912.zones') do
          it { should be_file }
      end

      describe file('/var/named/sharknet.us.zone') do
          it { should be_file }
      end

      describe file('/var/named/db.1.168.192') do
          it { should be_file }
      end

  end

  describe "Test BIND services" do
    describe package('bind-chroot') do
      it { should be_installed }
  end

  describe service("named") do
      it { should be_enabled }
      it { should be_running }
  end

  describe port(53) do
      it { should be_listening.with('udp') }
  end

  describe port(53) do
      it { should be_listening.with('tcp') }
  end
end

describe "The BIND user 'named':" do
    it "should exist" do
        expect( user('named') ).to be
    end

    it "should belong to a group called 'named'" do
        expect( user('named') ).to belong_to_group 'named'
    end
end

###################################
# DNS file syntax check
###################################

describe "Validate configuration files" do

    describe command('named-checkconf /etc/named.conf') do
      it { should return_stdout '' }
    end

    describe command('named-checkconf -t /var/named/chroot /etc/named.conf') do
      it { should return_stdout '' }
    end

    describe command('named-checkzone sharknet.us /var/named/sharknet.us.zone') do
        its(:stdout) { should include 'OK' }
    end

    describe command("named-checkzone 1.168.192.in-addr.arpa /var/named/db.1.168.192") do
        its(:stdout) { should include 'OK' }
    end

end

describe "Record Lookup" do
    before(:all) do
        @domain_name = 'sharknet.us'
            # Cache DNS servers
            @dns_servers ||= []
            @dns_servers.push( DNS.new('192.168.1.5', 'Master', @domain_name ))
            # You can add others here

            # Load DNS records from a CSV file
            @records = CSV.readlines('spec/bind/records.csv')
        end

        it "Should return the correct IP address for static hostnames (A records)" do
            @dns_servers.each do |nameserver|
                @records.each do |record|
                    if record[2] == 'A'
                        expect(nameserver.is_host?(record[0],record[1])).to be_true , "Server #{nameserver} did not find IP address #{record[1]} for #{record[0]}"
                    end
                end
            end
        end

        it "Should return the correct hostname for static IP addresses (PTR records)" do
            @dns_servers.each do |nameserver|
                @records.each do |record|
                    if record[2] == 'A'
                        expect(nameserver.is_pointer?(record[1],record[0])).to be_true , "Server #{nameserver} did not find host name #{record[0]} for #{record[1]}"
                    end
                end
            end
        end

        it "Should resolve external host names." do
            external_hosts = [
                "www.google.com",
                "www.cnn.com",
                "www.facebook.com"
            ]
            @dns_servers.each do |nameserver|
                external_hosts.each do |host|
                    ip = nameserver.address( host )
                    expect(ip).to_not eql('Hostname not found'), "Server #{nameserver} could not resolve #{host}"
                end
            end
        end

        it "Should return the correct mail servers (MX records)" do
            @dns_servers.each do |nameserver|
                @records.each do |record|
                    if record[2] == 'MX'
                        exists = nameserver.is_mail_server?( @domain_name, record[0], 10 )
                        expect(exists).to be_true, "Server #{nameserver} did have an MX record for #{record[0]}"
                    end
                end
            end
        end

        it "Should return the correct nameservers (NS records)" do
            @dns_servers.each do |nameserver|
                @records.each do |record|
                    if record[2] == 'NS'
                        exists = nameserver.is_nameserver?( @domain_name, record[0] )
                        expect(exists).to be_true, "Server #{nameserver} did have an NS record for #{record[0]}"
                    end
                end
            end
        end

        it "Should return the correct aliases (CNAME records)" do
            @dns_servers.each do |nameserver|
                @records.each do |record|
                    if record[2] == 'CNAME'
                        exists = nameserver.is_alias?( record[0], record[1] )
                        expect(exists).to be_true, "Server #{nameserver} did have a CNAME record for #{record[0]} that aliased to #{record[1]}"
                    end
                end
            end
        end

    end
end

Keep in mind these might be somewhat CentOS specific, especially the way CentOS handles chrooted BIND.

This tests three things: 1) that the BIND server is properly configured, and 2) that clients can perform successful lookups, and 3) that the static records are correct. I hope what each of these tests do is obvious from the Describe statements that accompany them. If not just post a question in the comments and I will clarify. Here I will focus on the part that does the lookups.

The section called “Record Lookup” is rspec rather than serverspec code. This shows that you can easily mix the two. However, be aware that they run on different machines. serverspec code runs on the remote server. rspec code runs on your local workstation.

I keep my DNS records in separate file called records.csv. This is easier for me to maintain. The format is simple and shown below:

host1.example.com,192.168.1.1,A
host2.example.com,192.168.1.2,A
host3.example.com,192.168.1.5,NS
host4.example.com,192.168.1.6,MX
host5.example.com,example.github.io,CNAME

And finally I include the DNS helper code (dns.rb) you need for the rspec tests. This goes in the lib/ directory.

require 'rubygems'
require 'resolv'
require 'ipaddress'

class DNS

  # ip_address = dotted notation as a string, e.g. '192.168.1.1'
  # role = 'Master' or 'Slave'
  # domain_name = the domain name as a string, e.g. 'sharknet.us'
  def initialize(ip_address, role, domain_name)
    @role = role
    @ip_address = ip_address
    @resolver = Resolv::DNS.new(
      :nameserver => [ip_address],
      :search => [domain_name],
      :ndots => 1)
    # @resolver.timeouts = 2 # Enable for Ruby 2.1 or newer. Fail if lookup takes longer than 2 seconds
  end

  # returns the ip address string
  def name
    @ip_address
  end

  # returns the role string
  def role
    @role
  end

  # returns the DNS server as a string in the form '192.168.1.1 [Master]'
  def to_s
    "#{name} [#{role}]"
  end

  # returns the ip address as a string or 'Hostname not found' if there was any error
  def address( hostname )
   begin
    @resolver.getaddress( hostname ).to_s
  rescue
    'Hostname not found'
  end
end

# returns the hostname as a string or 'IP address not found' if there was any error
def hostname( ip_address )
  begin
   @resolver.getname( ip_address ).to_s
 rescue
   'IP address not found'
 end
end

  # return true if ip is between (inclusive) ip_address_start and ip_address_end
  # ip_address_start and ip_address_end use CIDR notation; e.g., '192.168.1.1/24'
  # All the addresses must be passed as strings
  def host_in_range?( ip, ip_address_start, ip_address_end )
    return ip >= ip_address_start && ip <= ip_address_end
  end

  def is_host?( hostname, ip_address )
    records = @resolver.getresources( hostname, Resolv::DNS::Resource::IN::A )
    records.each do |record|
      if record.address.to_s == ip_address
        return true
      end
    end
    return false
  end

  def is_pointer?( ip_address, hostname )
   if hostname( ip_address ) == hostname
    return true
  end
  return false
end

def is_mail_server?( domain_name, hostname, preference )
  mx_records = @resolver.getresources( domain_name, Resolv::DNS::Resource::IN::MX )
  mx_records.each do |record|
    if record.exchange.to_s == hostname && record.preference == preference
      return true
    end
  end
  return false
end

def is_nameserver?( domain_name, hostname )
  mx_records = @resolver.getresources( domain_name, Resolv::DNS::Resource::IN::NS )
  mx_records.each do |record|
    if record.name.to_s == hostname
      return true
    end
  end
  return false
end

def is_alias?( hostname, host_alias )
  mx_records = @resolver.getresources( hostname, Resolv::DNS::Resource::IN::CNAME )
  mx_records.each do |record|
    if record.name.to_s == host_alias
      return true
    end
  end
  return false
end

end

This code can handle the case of a multi-homed host; i.e., a host with multiple IP addresses.

Conclusion

This was a long post and I hope it helps you write your own tests. Using test has helped me enormously to debug my server configurations. My next post will explain how to automate these on check-in to git. Happy testing!



Categories: Testing

Tags: ,

Share Your Ideas

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: