Kyle Brandt

Introduction
For a system administrator it is important to have an efficient and comfortable interface to all your servers. GNU Screen is an excellent utility to be able to have a single terminal connected to multiple servers that won’t disappear when you close the window. I have a set up that allows me to spawn gnome-terminal with different screen sessions for each location I administer in a different tab. Then each screen session has a named ‘tab’ that automatically logs into each server at that location. It ends up looking like this:

My main two recommendations for screen are to set up the meta character as back-tick ( ` ) and to give screen a ‘tab bar’. You can read how to do these two things here.

Setting up Screen
Once you have screen set up the way you like, you can the specify an additional screenrc file with the -c switch and your settings in the ~/.screenrc will still be used. The secondary screenrc is where you can list different server groups. This file will make it so there are named ‘tabs’ for each server, and each tab will log into the server you specify. Each line in the file should be something like ’screen -t myServer ssh myServer’, the first mySever is the name of the tab, and then ssh myServer is the command that will be run. To simplify doing this in the future, I made a little Perl script that reads a file that has one server name per line and prints the rc file to standard out.

#!/usr/bin/perl
#===============================================================================
#         FILE:  makeScreenRc.pl
#        USAGE:  ./makeScreenRc.pl
#       AUTHOR:  Kyle Brandt (kb), kyle@kbrandt.com
#===============================================================================

use strict;
use warnings;

print "zombie qr\n";
while (<>) {
        chomp;
        my $server = $_;
        print "screen -t $server ssh $server", "\n";
}

So if you called the above script with something like ‘perl makeScreenRc.pl myDmzList > myScreenDmzRc’ you can then use the created file with ’screen -R DMZ -c myScreenDmzRc’. The capital R switch looks for an existing detached session and will attempt to reattach it before creating a new one. This will be useful with gnome-terminal in case gnome-terminal crashes.

Setting Up Gnome-Terminal
The next step is to create a profile for each of the screen sessions. You can do this by going to File::New Profile and then create a profile with a relevant name for the screen session, i.e. ‘DMZ’ . After that, Go to the Title and Command tab, check ‘Run a custom Command instead of my shell’ and and edit the command to be something like ’screen -R DMZ -c myScreenDmzRc’. Then repeat this for each of the screen sessions you have set up. Then, you can run something like ‘gnome-terminal –tab-with-profile=DMZ –tab-with-profile=MyOffice’ where DMZ and MyOffice are the names of the gnome-terminal profiles you created. This automatically detaches itself from the controlling terminal, so if you close the terminal you launched this from, the new terminal will not close. Lastly, you can set up a shell alias to run the above command, so all you have to do to open up your command central is type something like ‘myservers’.

Update: David Harding pointed out in his comment to this post that the column utility does exactly this. Therefore, the following is really just an academic exercise.

In my last post I showed how to get columns outputted in the command line to line up using python. In this post I am going to show you how to do it with Bash scripts (I think you could also use this same method with python using calls to the shell). Instead of padding the columns with spaces as I did in my previous post, this time we use a tab character for the delimiter and manually set the tab stops in the terminal itself. Since this is the terminal, not the shell, this will work with other shells as well (such as my favorite interactive shell, Zsh).

There is example code at the bottom. Instead of creating functions as I did with my previous post I have kept this example pretty tedious (repetitive code etc) to lessen the levels of abstraction and make the example a little clearer.

The first part finds the max width of each column of a text file. This example has 4 columns and a while loop that splits them on the tab character by setting the IFS ( Input Field Separator ) variable to tab for the loop only. Each iteration of the while loop remembers the value of each column; it saves the value in the $max# variable if the length was larger then the previous iteration ( the variable substitution ${#variable} returns the length of the variable ).

The second part, after the while loop, finds where the tab stops should be placed. The setterm command with the -tabs switch sets tab stops at absolute positions up to 160 ( each argument specifies where the tab stop is relative to the start of the line, not relative to the previous tab stop ) . So for this example, the second tab stop position is found by adding the width of the first column to the width of second column — this gives us the position relative to the start of line. Lastly, after setting the tab stops the file is displayed on the terminal with cat.

A caveat is that it is hard to find out what to set $TERM to. On my machine, when I am in screen session $TERM is equal to ’screen’, but this doesn’t work with setterm, I have to set TERM to ‘linux’.

I hope this helps someone when creating their next command line utility that uses columns.

I created a solution for a program I am writing that makes columns line up when outputted to the command line. I am new to Python, and am hoping I might get some input on this topic.

In the text file that the program reads, the fields (columns) are delimited by ‘$’ and the records (lines) are delimited by newlines ‘\n’. So one line looks like: 1$foo$bar

I broke this task into two separate functions. The first function, opt_output, finds the max length of each column and returns a list such as [1, 23, 14] where 1 is the max width of the first column, 23 is the max length of the second column etc. This function takes a list object as an argument; that list is the file described above with one record as an object in the list. It then iterates over the list, splitting each record into a list on its delimiter. Then the function iterates over an individual record, saving the length of each item if the length is larger then the the length of the equivalent column in previous record. This is of course clearer in the code itself:

The Second function, print_line, takes one record as a list object (already split into items) and the value returned from the previous function. It then uses the values returned from opt_output and pads each item in the list with spaces. The number of spaces to pad it with is figured out by subtracting the length of the item from the max length of the column that was found by the previous function. Finally, it rejoins the list with the delimiter ‘$’ and then splits the list again using the padded spaces as the delimiter for each column:

When calling these functions I record the value of the first function in a variable so it does not have to iterate over the list over and over again. The print line function is called for each line in the file when outputting file to the screen. As long as there is a fixed width font everything lines up nicely in the terminal.

System administration is often about seemingly mundane tasks, but each task can provide an interesting puzzle to solve and there are many ways to solve it. I want to share a typical task in depth. I don’t intend to present every step of the task, but I will examine some of the steps in detail. I will also show some specific command line examples and draw larger themes of system administration from those examples.

The task is to move about 20 of instances of various LAMP web applications from one cluster to another cluster. These applications are all different, but they are built on a common framework and by the same company. So there is coherency but also discrepancies to deal with. In addition, the servers are on the other side of the country so the task is done remotely.

Preparation
The first step is to brainstorm about the different aspects that must be considered before starting the task. Some of these are moving the data itself, symbolic links, storage and system requirements, DNS, variables in the code itself, permissions, coordinating with the testing team, communicating with your own team, documentation and diagrams, firewall rules, room for expandability, correct versions of applications (PHP, Apache, MySQL), backup of data, business requirements, and network layout.

After brainstorming the ideal result is that you anticipate issues which need to be resolved in preparation for the task. That way there will be less interruptions and you will have a smoother work flow. This is because many of the above items are dependent upon each other, and your work flow will be interrupted if you didn’t think of these steps ahead of time (this will probably happen more than enough even with brainstorming). Taking the time to brainstorm might also inspire ways to improve the structure of the environment, find ways to make things more modular (the Unix way), and more efficient.

Some of the ideas and issues that brainstorming led to for my example are:

• One of the servers had to be moved into the new cluster and put on the same subnet, this may seem obvious, but this is all being done remotely and my hosting provider can’t read my mind. Also deploying the applications and setting up things such as MySQL replication are dependent upon this.
• Firewall rules had to be altered to transfer the data to the new environment from the old servers.
• New testing domain names on our public and private DNS servers had to be set up so the applications could remain live while the new instances were tested. This is an example of business needs. Communicating with the business managers is essential so they can plan around the migration and you can plan around their needs.

The Migration

Step 1: Build the Environment:

One of the first steps was to deploy the service applications (i.e Apache, MySQL, etc) to the new environment. The choice was to compile these applications from source because customization beyond using a package (i.e. .deb or .rpm) was required.

A wise question for system administrators to ask themselves is: “Is there a chance I am going to have to do this or something similar again?” This question leads to doing things in a way that will make them easy to repeat in the future and, “working smarter, not harder.” So with compiling LAMP the smart thing to do (if time permits) is to design a custom makefile or script to compile all these applications. Even smarter might be to find someone else who has already done this, for example: http://www.apachetoolbox.com/ for installing Apache.

A simple bash script might have option parsing with each program as an argument. Then each option would download, extract, and then compile the program logging any build errors. You also would probably want to make functions that would be common to all the lamp applications, such as downloading and extracting. As you deploy the applications you can edit the script so it is ready for the next time you need to build a LAMP environment.

An unforeseen issue:
For ease of backup the choice was to put all applications (web applications and the service applications) under an /opt directory. You can specify a base path at compile time for the service applications. However, I did run into a problem with libraries when doing this. Shared libraries not being found is a common error. To resolve missing shared library errors:

• run ldd on the executable and it should tell you which libraries are missing. It will be something like library.so.1 => Not Found
• Then find the library directory for the application you just installed (somewhere under /opt in this case).
• Add the directory path to a text file in /etc/ld.so.conf.d, something like /etc/ld.so.conf.d/myapp.conf
• run ldconfig

To learn more about shared libraries and how it all works I recommend this article.

The last thing I would like to mention about setting up the environment is the use of symbolic links. It is often beneficial to make the root of a website a symbolic link. When updating the code you can set up a new virtual host and a new copy of the site for testing. Then when it is time to move to the updated version you can just change the symbolic link. This makes the structure more modular (again, the Unix way) and will allow for virtually no down time of the application if executed properly.

In the next part I will talk about moving the web applications.

Update: It has been a while since I actually did this task and since writing this I got side tracked with other projects. So I will probably not be writing any more parts unless I receive a specific request.

Basics
Thunderbird messages can be composed from the command line, and the syntax is pretty basic (field=’value’). For Example:
admin@box:$ thunderbird –compose “to=’foo@bar.com,bar@foo.com’,subject=’nice tip’,attachment=’file:///home/admin/tip.txt’,body=’Check out this neat command line tip’”

Caveats:
Values don’t have to be in single quotes. However, if you have multiple values or characters that might be interpreted by Thunderbird, as actually being part of the command, you will get results you may not have intended. So, put the whole part after –compose in double quotes and use single quotes for the values. You also need to make sure that there are no spaces after or before the commas that separate the field value pairs (Basically anything outside of the single quotes). Lastly, this is not entirely from the command line. The Thunderbird compose email window will pop up with the fields populated, so you can edit them and then send it.

Tip:
This can be combined with command substitution. So if you want to email a file to someone with the contents of a file in the body of the message and not as an attachment you can do the following:
admin@box:$ thunderbird –compose “to=’foo@bar.com’,subject=’I sent this message using command substitution’,body=’$(cat file.txt)’”

You can now enjoy having some of the power of the command line in your email while still just using a simple gui application.

Reference:
http://www.mozilla.org/docs/command-line-args.html

Intro
This topic has been covered more than enough on the internet, so this goes a bit against my mission statement of ‘original articles.’ However, after a recent experience of being put on the spot to find some files on the Linux file system using various parameters, without having my beloved man pages, or being in front of a console–I have realized that I need a review. I have decided it won’t hurt to share my review with others.

Update: I have written a GUI in python that generates find commands called pyGnomeFind that one might find helpful.

Generate a Testing Directory with some random files
Some of these examples will actually make changes to files, I wanted a bunch of differently name files, that also hand commonalities. So I came up with:

for i in $(grep allow /etc/dictionaries-common/words | grep -v “‘”); do touch $i; done

What this does is search my dictionary (just a list of words, one per line) for any line that contains the string “allow”, but not an apostrophe. The command then creates an empty file for each match, and then each match has a filename of the matched string. On my system this creates 50 files. The techniques used are as follows: For Loop, Command Substitution (See previous Post, this time I used $( ) instead of back ticks), and Piping. The two commands used are grep which is for searching documents, and touch which creates an empty file with the given name (i.e. touch test creates a file called test).

The locate Command
This is the command I used most frequently in the past, it is very simple to use. It is database driven (meaning it search a database of files, not the file system itself) which makes it fast. The basic syntax is: locate filename This will find all files in the system either named “filename” or that contain the string “filename”. I don’t recommend using this command until you can use the find command in your sleep, or else you might limit yourself, and not remember the find syntax when you need to.

The find Command
The find command is far more powerful, and can find files based on such things as: filename, text contained with the file, modified time, accessed time, permissions, and file type. You can them perform all sorts of operations on the files found. Combined with regular expressions, this flexibility and power is a fine example of the power of the command line interface.

Abstract Usage (and some Vocabulary): find [path...] [expression]
Path is synonymous for directory (the path you take to the file). The default is the current directory, but I usually put a period, which represents the current directory to bash. The expression will consist of options, tests, and actions. Tests are essentially criteria, for example, time modified. The default action is to print what find found, but commands can be applied to what is found as well. The Expression can also have operators. Operators are things such as And, Or, Not–if no operators are specified the default is AND.

Usage by Examples:

• Filename: find / -name “*.jpg”
• The asterisk is a filename matching character which means zero or more of any character(s). This command will find all files with the .jpg extension (but not .JPG) in the root path and all its subdirectories. This is a different behavior than the asterisk metacharacter which is an element of regular expressions. This is worth explaining, there is shell globing and regular expressions. Some programs in GNU/Linux use regular expressions and some use shell globing (such as the find utility). The BASH interpreter itself uses globing, so if you don’t quote an asterisk the interpreter will send all the files in the current directory that match the pattern and send them to the program, not the *.jpg string itself (unless there are no .jpg files in the current directory, then BASH sends the string itself). So in this case find sees *.jpg not whatever .jpg files happen to be in the current dir. It is easy to make mistakes when writing scripts since utilities such as grep and sed use regular expressions.
• Case insensitive filename: find / -iname “*.jpg”
• Modified Time: find / -mtime 0
• Find files modified in the past 24 hours. The time switch (aka flag or option) works in 24 hour blocks moving backwards (since the computer doesn’t know if files will be modified in the future, yet), so 0 means 24 hours before now, 1 means 24-48, 2 means 48-72. You can use + to mean more than the block, or – to mean less than the block of time. More than starts at the upper limit of the block, and less then starts at the lower limit of the block. So 0 and -1 mean the same thing. Substitute -mmin for minutes.
• Access Time: find / -atime 0
• Same as above but access time, read above for explanation.
• Size: find / -size +1024k
• find all files more than 1M. Similar behavior to time, see above.
• Permissions: find / -perm +o=x
• Find all files that are executables by all, can use symbolic or octal modes for permissions, see chmod documentation for a further explanation of this.
• Using operators: find / -iname “*.jpg” ! -type 1
  
• find all .jpg files that are not symbolic links. ! is the NOT operator. type 1 are symbolic links.
  

Some Common Options for find:
-xdev: Don’t go to other filesystems
-maxdepth #: How many directories in depth, 1 = current directory only.
-prune: used by find / -path ‘/dev’ -prune o

The xargs Commmand: Combine Arguments
This is used in combination with find for efficiency, it is used to perform actions on the files that find returns, but instead of running the command each time for each file, it passes all the files to the command at once so the command only has to be run once. The common method is to pipe the results of find to xargs. Be careful, not all commands handle the input of of xargs by default properly.

Example 1: find ~/ -maxdepth 1 -name “*” | xargs grep -i “test”
This will search all files in your home directory and no subdirectories (maxdepth) for the case insensitive string “test”. Note: grep uses regular expressions so an asterisk here will have different behavior.

Example 2: find ~/ -maxdepth 3 -name “*.back” | xargs rm
This will find all files in the home directory out to 2 sub directories, with the extension of .back, and delete them.

That’s my thousand words on finding files.

Preparing the Spreadsheet:
Before doing any of these operations you are going to want to save them as TAB delimited files.

In order for your tabs to line up when being displayed in the terminal you may need to change your tab stop settings. In a regular terminal you would use:

Replace # with the new number of spaces a tab stop should be, the default is 8. setterm -tabs will display an output of what the tabs will look like. If you are using xterm or gnome-terminal add -term linux so the full command might look like:

I have also set my non-x console for a higher resolution, so I can get more of the sheet on the screen, this can be done by editing your kernel boot parameters in your grub or lilo conf. See: http://shrimpworks.za.net/2005/01/31/change-your-console-resolution-and-colour-depth/

Some Basic Operations:

DO NOT USE:

So to add a column or row numbers to your spreadsheet:

1. Sort all columns by a specific row:

2. Add the sum of all numbers in a column, in this case column 2 (thanks to http://www.pixelbeat.org/cmdline.html)

Explanation: cut -f2 = extract column 2 from the file. | = sends the output of that command to the input of the next command (pipe). (tr ‘\n’ +; echo 0) = translate new lines to the plus sign, the semi-colon tells bash that a new command is starting so echo 0 is separate from the tr command. Bash understands new lines and tabs etc as characters, in this case \n = new line. The ‘echo zero’ is because the last new line will be a plus sign, so bc needs something after it, 0 won’t effect the mathematical results. bc is the Basic Calculator, and will print the mathematical result of its input.

3. Get the mathematical mean of a column 2, out to 20 decimal places

4. How to reorder the columns. With the following the original Column 3 is now column 1, and the original column 1 is now column 2, etc.

Due to the complexity of awk I am not going to explain this command, awk is considered a programming language in itself, but I wanted to include this basic operation.

I hope this is a good starting point for those who want to know how to perform some text manipulation from the command line with files in a spreadsheet format.

I had a bunch of bin/cue disk image files that I needed to convert to .iso files, and the program bchunk didn’t seem to have any way to process multiple files that I could see. So I came up with a command that would do this for me, so if you need to do the same:

1. Install bchunk: In Ubuntu you can get it by: sudo apt-get install bchunk
2. Navigate to the directory that has your bin/cue file pairs that you want to convert.
3. You can convert them all to iso files with the following command, assuming all your bin/cue file pairs have the same root file name, and the extensions are all lowercase: for i in *.cue; do bchunk ${i/.cue}.bin ${i/.cue}.cue ${i/.cue}.iso; done
4. That should be it, hope this helps someone out. Anyone have a better way?

I will explain how this works for people trying to learn a little bit more about the command line. The bchunk command takes three arguments, and normally would look like “bchunk foo.bin foo.cue. foo.iso”. The command is a for loop, and everything between “do” and the the last semicolon gets repeated as a whole unit for as many times as there are cue files in the current directory (This would be the loop body). The variable i represents each .cue file for each time the loop body is run. “${i/.cue}” strips the file name of its extension (So really “${i/.cue}.cue” is redundant) and the extension that bchunk expects is added with what follows the curly braces.