Kyle Brandt

System administration is often about seemingly mundane tasks, but each task can provide an interesting puzzle to solve and there are many ways to solve it. I want to share a typical task in depth. I don’t intend to present every step of the task, but I will examine some of the steps in detail. I will also show some specific command line examples and draw larger themes of system administration from those examples.

The task is to move about 20 of instances of various LAMP web applications from one cluster to another cluster. These applications are all different, but they are built on a common framework and by the same company. So there is coherency but also discrepancies to deal with. In addition, the servers are on the other side of the country so the task is done remotely.

Preparation
The first step is to brainstorm about the different aspects that must be considered before starting the task. Some of these are moving the data itself, symbolic links, storage and system requirements, DNS, variables in the code itself, permissions, coordinating with the testing team, communicating with your own team, documentation and diagrams, firewall rules, room for expandability, correct versions of applications (PHP, Apache, MySQL), backup of data, business requirements, and network layout.

After brainstorming the ideal result is that you anticipate issues which need to be resolved in preparation for the task. That way there will be less interruptions and you will have a smoother work flow. This is because many of the above items are dependent upon each other, and your work flow will be interrupted if you didn’t think of these steps ahead of time (this will probably happen more than enough even with brainstorming). Taking the time to brainstorm might also inspire ways to improve the structure of the environment, find ways to make things more modular (the Unix way), and more efficient.

Some of the ideas and issues that brainstorming led to for my example are:

• One of the servers had to be moved into the new cluster and put on the same subnet, this may seem obvious, but this is all being done remotely and my hosting provider can’t read my mind. Also deploying the applications and setting up things such as MySQL replication are dependent upon this.
• Firewall rules had to be altered to transfer the data to the new environment from the old servers.
• New testing domain names on our public and private DNS servers had to be set up so the applications could remain live while the new instances were tested. This is an example of business needs. Communicating with the business managers is essential so they can plan around the migration and you can plan around their needs.

The Migration

Step 1: Build the Environment:

One of the first steps was to deploy the service applications (i.e Apache, MySQL, etc) to the new environment. The choice was to compile these applications from source because customization beyond using a package (i.e. .deb or .rpm) was required.

A wise question for system administrators to ask themselves is: “Is there a chance I am going to have to do this or something similar again?” This question leads to doing things in a way that will make them easy to repeat in the future and, “working smarter, not harder.” So with compiling LAMP the smart thing to do (if time permits) is to design a custom makefile or script to compile all these applications. Even smarter might be to find someone else who has already done this, for example: http://www.apachetoolbox.com/ for installing Apache.

A simple bash script might have option parsing with each program as an argument. Then each option would download, extract, and then compile the program logging any build errors. You also would probably want to make functions that would be common to all the lamp applications, such as downloading and extracting. As you deploy the applications you can edit the script so it is ready for the next time you need to build a LAMP environment.

An unforeseen issue:
For ease of backup the choice was to put all applications (web applications and the service applications) under an /opt directory. You can specify a base path at compile time for the service applications. However, I did run into a problem with libraries when doing this. Shared libraries not being found is a common error. To resolve missing shared library errors:

• run ldd on the executable and it should tell you which libraries are missing. It will be something like library.so.1 => Not Found
• Then find the library directory for the application you just installed (somewhere under /opt in this case).
• Add the directory path to a text file in /etc/ld.so.conf.d, something like /etc/ld.so.conf.d/myapp.conf
• run ldconfig

To learn more about shared libraries and how it all works I recommend this article.

The last thing I would like to mention about setting up the environment is the use of symbolic links. It is often beneficial to make the root of a website a symbolic link. When updating the code you can set up a new virtual host and a new copy of the site for testing. Then when it is time to move to the updated version you can just change the symbolic link. This makes the structure more modular (again, the Unix way) and will allow for virtually no down time of the application if executed properly.

In the next part I will talk about moving the web applications.

Update: It has been a while since I actually did this task and since writing this I got side tracked with other projects. So I will probably not be writing any more parts unless I receive a specific request.