As I experiment with a complex IT environment I’m starting to realize how similar IT is to software development. The more I work with it, the more I want to treat it exactly like a software development project. I see the need for a development environment and development process like the one I used as a software developer.
Software developers need an isolated, sandboxed environment to develop their code. The development process is often messy, breaks a lot of things, and is generally experimental in nature. You try different things and you learn as you go. Development requires a free and unrestricted environment to foster creativity. It must also be isolated from the production environment so that the experimentation stays local.
I think IT has the same need. For example, an administrator might want to experiment with certain email or DNS settings. Perhaps he has some ideas how to improve performance. Doing this on the production environment is obviously the wrong thing to do. A better answer is to use a tool like Vagrant and an automated system management tool like Ansible or Chef to spin up a private replica of the production environment on the administrator’s workstation. He can then experiment without fear of harming the production environment.
Then, like with software development, the changes are checked into configuration management, are checked by the continuous integration server, and are fully tested in the integration testing environment. After the change has passed all the tests (and I envisage a large number of test) then it is ready for transfer to the production environment per whatever change management process is in place.
The IT development environment is a simulation (an isolated replica) of the production environment. The integration testing environment is a higher fidelity simulation of the production environment. Virtualization makes this possible. All changes to the production environment including changes to devices such as switches, routers, SANs, firewall, etc. should should be tested via simulation using this process.
I suspect it is not yet possible to fully simulate all the devices on a modern network, but as the DevOps approach gains in popularity I hope that the vendors of these devices will start to recognize this need.