Testing Automated Configuration Management Tools

I have written many times about the need to test a server configuration after making a change with a CM tool. What I have started to see is that this testing is as much about security and auditing as it is about validating the change. Here is an example to illustrate .

Policy Testing Example

Let’s say you want to ensure that the xinetd is not set to autostart on your Linux servers. This is your policy. First you need to run the command to make that change to put your servers in compliance with your policy. Then you need to test that the change has been made successfully. You also want to verify that the server remains in policy. These steps can be done with different tools.

Make the change with Ansible. Other CM tools can also do this.

- name: Ensure xinetd does not autostart
  command: chkconfig xinetd off

Once we run this command, all our servers should be in policy.  We need to verify the change on every server.

We can test it using serverspec with this simple bit of Ruby code:

describe service('xinted') do
  it { should_not be_enabled }

We can also test this using a security scanning application. For example, here is the equivalent test using Nessus.

  description:"Make sure that xinetd is disabled"

Both the serverspec test and the Nessus scan are external tests that only run when we manually launch them or run them on a schedule. We could instead use an IDS agent running on the server that automatically and repeatedly tests itself. For example, in OSSEC (an IDS) we could use a rule like this:

    <command>chkconfig | grep xinetd</command>

<rule id="99999" level="7">
    <match>ossec: output: 'chkconfig | grep xinetd'</match>
    <description>xinetd is enabled.</description>

If OSSEC detects a violation of our no xinetd policy it will notify us.


I find it interesting that three very different tools, severspec, OSSEC, and Nessus can be used to test the change made by the CM application. As I wrote about in my post on idempotency, I would like a CM application that allowed my to specify the state of my servers (my policy) and which would then make the changes needed to achieve that state, and continuously monitor them to make sure that they remain in that state.

In the mean time, there are some interesting options for infrastructure testing worth exploring. My point here is to show that there are many tools that can accomplish this, some of which you may not think of as infrastructure or integration testing tools.

Categories: DevOps, Testing

Tags: , , ,

2 replies

  1. ansible version:
    – name: Ensure xinetd does not autostart
    service: name=xinetd enabled=no


    • In my view, this would not fill the role of testing Ansible initiated changes. I favor independent validation of changes and a test-first approach. For example, a process more like this:

      1. Identify business need for a change.
      2. Have change approved by the CAB.
      3. Write test for the change using something like ServerSpec
      4. Document the change in your CMDB or other tool (maybe JIRA?)
      5. Write Ansible code to affect the change
      6. Test the change in the staging environment
      7. Push to production once QA has signed off

      Obviously all of the artifacts are tracked in version control. Ansible is code and I wouldn’t allow any code going into production that hasn’t been tested. Companies may have different Change Management and Release Management pipelines but I think this captures they key points.


Share Your Ideas

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: