How to Create Xen VMs Using Ansible


Introduction

This guide explains how to create and provision a Centos 6.5 VM on a XenServer 6.2 host using Ansible 1.4. The steps shown here fully automate the process. It is not simple but it does work reliably. The most complex part is bootstrapping the VM. Once Ansible can connect to the VM using its host name it becomes a straight-forward Ansible automation task.

I do not use libvirtdbecause of its poor documentation and because I do not want to install an additional service, libvirtd, on my Xen hosts. The Xen CLI works fine and can be called by Ansible via SSH. Perhaps someone will eventually write a Xen module for Ansible.

This process should scale up to as many VMs as you want, hundreds if your infrastructure can support it. I have only tested 10 at a time, however. Creating these VMs is a complex process. The steps I describe work well for my environment but may not work in yours.

I highly recommend that you test this in a Vagrant development environment first. You can create a plain CentOS server in Vagrant, install Xen on it, and then test my process using this virtual test environment.

Prerequisites

To run this playbook you will need the following supporting infrastructure already configured and running. This guide does not cover the basic installation and configuration of these components:

  1. A DHCP and DNS server. The DHCP server must be configured to send dynamic updates to the DNS server.
  2. A web server to server static files.
  3. A XenServer 6.2 pool.
  4. Ansible version 1.4 (or later) on a development workstation.

Ansible_Infrastructure

You will need a playbook directory structure like the one below. I will refer to these files throughout this guide. Directories are shown ending with a “/”. In this guide it does not matter what your top level playbook is called.

Playbook [your top level playbook directory]
  - site.yml [where hosts in the inventory file are mapped to roles. Give all VMs the roles: vm, centos, and then whatever specific roles you want them to fulfill]
  - hosts [The inventory file. This contains the list of your Ansible managed hosts. Your VM names must be in here.]
  - group_vars/
      - all [A file with variables that apply to all roles and hosts]
  - host_vars/
      - host1.devops.com [A file named after your VM FQDN. It contains VM specific settings. You need one for each VM.]
  - roles/
      - dhcp/
          - tasks/
              - main.yml [the DHCP server role configuration script]
          - templates/
              - dhcpd.j2 [the template to configure dhcpd.conf. It will add your VM host names and MAC addresses]
      - vm/
          - tasks/
              - create_vm.yml [the script for creating a CentOS VM on a Xen host]
              - main.yml [this script controls whether or not the VM is created]
      - centos/
          - tasks/
              - main.yml [A script to standardize your CentOS machines]

To bootstrap the VM, I use the DHCP server to map a MAC address to a host name. Ansible needs a host name as the target for its playbooks. You cannot use Ansible to set the VM’s initial host name. The VM install process goes: MAC Address –> Host name –> Ansible role. This links a virtual (or physical) server to an Ansible role. The files involved are:

  1. MAC Address, Hostname: dhcpd.conf on the DHCP server
  2. Hostname: Ansible inventory file. The file used to tell Ansible about your hosts and groups.
  3. Hostname, Role: Ansible main site.yml file. The top level site configuration file.

The web server will host several items. These are:

  1. The CentOS repository. Optional since you can also use the URL of a public one. I prefer a local copy because it is faster and it is controllable.
  2. The CentOS kickstart configuration file. This file automates the CentOS operating system install.
  3. The XenServer guest VM tools. Installing these tools gives XenServer finer grained control over your VM. Normally XenServer installs them by mounting them as a DVD. Putting them on the web server allows their installation to be automated.

I will explain how the prerequisites are configured in a subsequent section.

Planning

Before creating the VMs, I recommend you develop a plan. The first step is to define your environments (e.g. production) and the networking topology needed to support them. VMs can have their own virtual networks and still interface with external hosts. One way to define environments is to use separate inventory files. You can use one for production, one for acceptance testing, etc. Defining a network topology is a complex topic and is also very site specific. This guide does not use a complex network and VLAN configuration for the VMs but could easily be modified to do so.

The next step is to define what applications you want to install and divide them into Ansible roles. Roles are simply named server configurations. You can create roles using any scheme you want. I prefer to define them by major application. Roles are modular in that one server can have many roles. As long as you define your roles with no redundant scripts you should be fine.

Once the roles are named, I highly recommend that you create validation tests for them before you create your playbooks. This is part of the test driven development (TDD) process. Every time you script or code something you need to independently test it. You can use serverspec or another product to write these tests. They will “unit test” your playbooks.

These tests will also document your administrators’ knowledge of how to best configure an application. This is very important. By documenting it, that knowledge becomes a team resource. The DevOps approach works best when configuration expertise is written down and saved in a version control system. Good managers will takes steps to make sure that the administrators who do this do not feel threatened by this step. Writing good test scripts makes administrators more, not less valuable to the team.

The next step is to write the playbook scripts that go with each role. The Ansible web site has a playbook guide that I recommend you follow. Pay special attention to the directory structure. Remember to check them into version control when you are done.

Now you have to map hosts to roles. Decide how many VMs you want to implement the roles. I recommend you use a spreadsheet for this and the next few steps. You should also add this planning spreadsheet to version control. Next you have to add a MAC address for each host you have identified. Enter the roles and other VM specific parameters. These parameters are explained below. In a spreadsheet this would have the following columns:

  1. Host name
  2. MAC Address
  3. Role(s)
  4. VCPUs
  5. RAM
  6. Template UUID
  7. Storage UUID
  8. Network UUID
  9. Comments

If you have a lot of servers then you can write a simple script to generate the MAC addresses. For reference, look at the MAC address that XenServer generates for you and use that as your starting point. There are libraries available that can manipulate MAC address such as netaddr in Python. If you are an Excel guru you can generate them directly in your planning spreadsheet.

Here is a short Python script that will create MAC addresses for you from your starting point:

#! /usr/bin/python

from netaddr import *
from optparse import OptionParser

parser = OptionParser()
parser.add_option("-s", "--start-mac", dest="start_mac",
help="The starting MAC address, inclusive.", metavar="START-MAC")
parser.add_option("-n", "--number", dest="number", type="int",
help="The number of MAC addresses to generate.", metavar="NUMBER")
(options, args) = parser.parse_args()

# All uppercase Unix style mac address formatter
class mac_custom(mac_unix): pass
mac_custom.word_fmt = '%.2X'

mac = EUI(options.start_mac, dialect=mac_custom)

for _ in range(options.number):
print mac
mac = EUI( int(mac) + 1, dialect=mac_custom)

To run it first make it executable (chmod u+x mac.py) then execute it from the command line telling it what address to start from (-s) and how many to create (-n).

./mac.py -s "01:23:45:67:89:AB" -n 10

You will use this planning data in your Ansible configuration files as I will explain next.

Preparation

To run the VM playbook you will need to prepare the supporting infrastructure. You need a DHCP server, a web server, and a set of configuration files that capture your planning data.

The steps are as follows:

1. Configure the ISC DHCP Server. You can do this part manually but I recommend that you manage your DHCP server using Ansible. The goal of this step is to have the DHCP server assign your VM a host name based on its MAC address. The dhcpd.conf entry for each host looks like this:

host [hostname] {
   hardware ethernet [MAC];
   option host-name "[hostname]";
   ddns-hostname "[hostname]";
}

I realize the repeated host names look redundant. They probably are but this is the incantation that finally worked after much trial and error. Assuming your dhcpd.conf is managed by a jinja template you will need to add this bit of code to the end of the template:

{% for item in host_entries %}
host {{ item.hostname }} {
hardware ethernet {{ item.mac_address }};
option host-name "{{ item.hostname }}";
ddns-hostname "{{ item.hostname }}";
}
{% endfor %}

If you have a role called dhcp and the name of your playbook top-level folder is playbook then this code would go at the end in playbook/roles/dhcp/template/dhcpd.j2. I store the variables this uses in a file called playbook/group_vars/all. The YAML for this looks like:

host_entries:
- hostname: host1
mac_address: a6:a7:ad:02:ff:00

- hostname: host2
mac_address: a6:a7:ad:02:ff:01

This is a list of all your host names with their MAC addresses. It can be as long as you like. The Ansible script will generate one host line for each. Use the host names and MAC addresses from your planning spreadsheet.

2. Configure Ansible roles. The dhcp server will map MAC addresses to host names. Now you need to complete the mapping by including the mapping from host names to Ansible role. You have to change two Ansible files. The first is you inventory file. This is the file where you list you host names. It is the file you pass to Ansible after the -i option. The second file is your top-level site.yml file. Here you need to set the role for each host name you put in your inventory file.

I recommend that you create a role called vm for the VM creation playbook I will describe below. Then add an OS level role (for example, centos) to set OS level configurations. Then add specific roles on top of those such as web, dns, hadoop, etc. The roles should be modular.

3. Load configuration files on the web server. You will need at least two files to be accessible without password to your Xen hosts: the CentOS kickstart file, and the Xen guest tools. I also recommend a keeping a local CentOS repository. VM install go much faster when the files are local.

My XenServer specific kickstart file looks like this:

#platform=x86, AMD64, or Intel EM64T

# Install OS instead of upgrade
install
url --url="http://[host FQDN]/CentOS/6/os/x86_64/"

# Language settings
lang en_US.UTF-8
timezone  America/New_York
keyboard us

# Forces the text installer to be used (saves time)
text
skipx
firstboot --disable

# Security
rootpw --iscrypted [your root password]
auth  --useshadow  --passalgo=md5
firewall --enabled --ssh --service=ssh
selinux --enforcing

# Installation logging level
logging --level=info

# Network information
network  --bootproto=dhcp --device=eth0 --onboot=on

# System bootloader configuration
bootloader --location=mbr --driveorder=xvda --append="console=hvc0"
zerombr
clearpart --drives=xvda --all --initlabel

# Disk partitioning information
part /boot --bytes-per-inode=4096--ondisk=xvda --asprimary --fstype="ext2" --size=100
part swap  --bytes-per-inode=4096--ondisk=xvda --asprimary --fstype="swap" --size=4096
part /     --bytes-per-inode=4096--ondisk=xvda --asprimary --fstype="ext4" --size=20480 --grow

# Reboot after installation
reboot

%packages
@base
@server-policy
%end

%post

# Install xen tools automatically
cd /root
wget http://[local web server FQDN]/XenServer-6.2.0/tools/xen_tools.tar.gz
tar xvfz xen_tools.tar.gz
/root/Linux/install.sh -n
rm -rf Linux/
rm xen_tools.tar.gz

# Not sure if this is needed
reboot

At a minimum you must change this file in three places (everywhere there are [], delete them and add your text): a) change the url line to include a valid CentOS repository (test it in your browser to make sure it works), b) add your root password, and c) in the post install section enter the full host name and path to the location where you stored the Xen tools file. The root password should be encrypted in the kickstart file. You generate it using this command (from a Linux machine) openssl passwd -1. Enter your password when it asks. Then copy and paste the entire string it return into your kickstart file.

4. Put the Xen UUIDs in an Ansible variable. You need to pass these three UUIDs to the playbook:

  • The CentOS template UUID. On a Xen host command line type xe template-list. Find the UUID for CentOS 6 64bit.
  • The storage UUID. On a Xen host type xe sr-list and find the SR that you want serve as the VM’s primary storage. I use use a SAN for this hence it’s the same for all my VMs.
  • The network UUID. On a Xen host type xe network-list. Find the network you want to use for the VM. This is a complex topic and I am going to assume you know how to do this part already.

If you use a pool and SAN then the UUIDs for these three are the same for all VMs. If you don’t then you will have to put the UUIDs in the host-specific variables file. I put mine in the group_vars/all file which looks like this:

centos_template_uid: 8ad91af3-0075-f996-9c8b-d9301d4aa802
primary_sr_uid: 1b6766ac-9ed1-7300-aa11-2f2b7382ffe2
user_network_uid: 43f017ee-ba69-3fe9-078d-93cebd65a689

You can also put them in each host-specific variable file.

5. Set VM specific parameters. This is the last step. Having all the VM specific parameters from the planning spreadsheet in a single file would be easier than having them in multiple files. I tried several ideas but could not get Ansible to use a single YAML file.

First enter all the VM host names into your inventory file. I recommend putting them under a group called vm. Then add them to the site.yml file and assign the roles to each. Now create a separate YAML file for each VM under host_vars.

In this VM host variables file put at least:

hostname: host1
mac: a6:a7:ad:02:ff:00
ram: 1GiB
vcpus: 2

You may also need to add the UUIDs for the items mentioned above depending on your Xen configuration.

All the preparation is done. The next step is to run the playbook.

VM Provisioning

Two files are needed to create the VMs. The reason for two is so that the VM creation process can be skipped once the VM exists. This makes the playbook idempotent as Ansible people like to say. the first script creates the VM.

---

  - name: Prevent template from creating storage
    command: xe template-param-remove uuid={{ centos_template_uid }} param-name=other-config param-key=disks
    ignore_errors: yes
    delegate_to: [Xen host FQDN]

  - name: Create VM
    command: xe vm-install template={{ centos_template_uid }} new-name-label="{{ hostname }}" sr-uuid={{ primary_sr_uid }}
    register: vm
    delegate_to: [Xen host FQDN]

  - name: Set the repository location
    command: xe vm-param-set uuid={{ vm.stdout }} other-config:install-repository="http://starlight.sharknet.us/CentOS/6/os/x86_64/"
    delegate_to: [Xen host FQDN]

  - name: Set the location of the kickstart file
    command: xe vm-param-set uuid={{ vm.stdout }} PV-args="ks=http://[web host FQDN]/CentOS/kickstart/ks.cfg ksdevice=eth0"
    delegate_to: [Xen host FQDN]

  - name: Assign a network
    command: xe vif-create vm-uuid={{ vm.stdout }} network-uuid={{ user_network_uid }} mac={{ mac }} device=0
    delegate_to: [Xen host FQDN]

  - name: Allocate VM storage
    command: xe vdi-create name-label="{{ hostname }} storage" sr-uuid={{ primary_sr_uid }} type=system virtual-size=25GiB
    register: disk
    delegate_to: [Xen host FQDN]

  - name: Assign storage to VM
    command: xe vbd-create vdi-uuid={{ disk.stdout }} vm-uuid={{ vm.stdout }} type=Disk bootable=true device=0
    delegate_to: [Xen host FQDN]

  - name: Set the VM RAM limits
    command: xe vm-memory-limits-set vm={{ vm.stdout }} static-min={{ ram }} static-max={{ ram }} dynamic-min={{ ram }} dynamic-max={{ ram }}
    delegate_to: [Xen host FQDN]

  - name: Set the number of CPUs
    command: xe vm-param-set VCPUs-max={{ vcpus }} uuid={{ vm.stdout }}
    delegate_to: [Xen host FQDN]

  - name: Launch the VM
    command: xe vm-start uuid={{ vm.stdout }}
    delegate_to: [Xen host FQDN]

You must change this script in two ways. First enter the FQDN of your Xen host under delegate_to:. Any host in the pool will work. The second is to enter the FQDN of the web server storing your kickstart file. You may also need to change the path if you use a different one from mine.

The final file serves as the entry point for the VM role. Ansible will first run this file and only create the VM if it doesn’t exist. That’s all it does. If you created a separate CentOS role then it would run right after this VM role.

---

  - name: Check if VM exists
    command: xe vm-list name-label="{{ hostname }}"
    register: vm_found
    delegate_to: [Xen host FQDN]

  - include: create_vm.yml
    when: vm_found.stdout == ""

  - name: Wait for the Kickstart install to complete and the VM to reboot
    local_action: wait_for host={{ hostname }}.[your domain] port=22 delay=15 timeout=1200 state=started

In this file you must change the delegate host name again. You must also add your domain name. I recommend you experiment with the timeout parameters to suit your system. You may require a longer timeout or a shorter one.

Run the playbook from within the playbook directory like this:

ansible-playbook -i hosts site.yml -k

The -k option is needed for the root password which you created in your kickstart file. In your CentOS role you can setup key based SSH instead.

Conclusion

Automating this lengthy and complex process will save time and reduce errors. I hope this works well for you. Please offer feedback and improvements in the comments section below.



Categories: DevOps

Tags: , ,

Share Your Ideas

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: