Kyle Brandt

The Problem
I have some users who are on a different mail system but still part of my company. The problem was that the users without Exchange 2003 accounts were not showing up in the Global Address List (GAL).

Solution
The first step was to look at the LDAP filter that generates the GAL. This can be viewed by going into the ‘Exchange System Manager’ and then ‘Recipients::All Global Address Lists::Default Global Address List::General Tab’. The following is the filter on my Exchange system (which I am guessing is the default ):

(& (mailnickname=*) (| (&(objectCategory=person)(objectClass=user)(!(homeMDB=*))(!(msExchHomeServerName=*)))(&(objectCategory=person)(objectClass=user)(|(homeMDB=*)(msExchHomeServerName=*)))(&(objectCategory=person)(objectClass=contact))(objectCategory=group)(objectCategory=publicFolder)(objectCategory=msExchDynamicDistributionList) ))

Looking at this filter, and not being rain man, I couldn’t just glance at and figure out what it meant. The trick was to load it with VIM, because it will highlight matching parenthesis (When your cursor is over the opening parenthesis, the match closing parenthesis gets highlighted). I then I indented based on that which resulted in:

(&
   (mailnickname=*)
   (|
      (&
         (objectCategory=person)
         (objectClass=user)
         (!(homeMDB=*))
         (!(msExchHomeServerName=*))
      )
      (&
         (objectCategory=person)
         (objectClass=user)
         (|
            (homeMDB=*)
            (msExchHomeServerName=*))
      )
      (&
         (objectCategory=person)
         (objectClass=contact)
      )
      (objectCategory=group)
      (objectCategory=publicFolder)
      (objectCategory=msExchDynamicDistributionList)
   )
)

The next step was to look at the active directory key value pairs for for one of the users that wasn’t showing up in the GAL. I know of two ways to do this, one is to use adsiedit.msc for windows, or, if you want to be super cool, use ldapsearch in Linux. To use ldapsearch to look at the attributes for the object, you would use a command like: ldapsearch -w $PW -v -x -D "cn=Administrator,cn=Users,dc=myDomain,dc=com" "cn=Kyle Brandt" where ‘Kyle Brandt’ is the user you want to look at, Administrator is the user you use to authenticate with AD, PW is a shell environment variable with you password, and myDomain is your company’s AD domain name.
Once I saw that mailNickname was not set, and since the filter says ’show in GAL if mailNickname is set to something, OR if … lots of stuff …’ all I had to do was use adsiedit to set that attribute to something. To learn how to read and write these filters see this rfc or this msdn page. You can see if the change will effect the GAL book by clicking ‘Preview’ on the tab were the filter originally was. It will probably take a day or so (depending on your settings) before the change is actually made to the GAL.

Introduction
For a system administrator it is important to have an efficient and comfortable interface to all your servers. GNU Screen is an excellent utility to be able to have a single terminal connected to multiple servers that won’t disappear when you close the window. I have a set up that allows me to spawn gnome-terminal with different screen sessions for each location I administer in a different tab. Then each screen session has a named ‘tab’ that automatically logs into each server at that location. It ends up looking like this:

My main two recommendations for screen are to set up the meta character as back-tick ( ` ) and to give screen a ‘tab bar’. You can read how to do these two things here.

Setting up Screen
Once you have screen set up the way you like, you can the specify an additional screenrc file with the -c switch and your settings in the ~/.screenrc will still be used. The secondary screenrc is where you can list different server groups. This file will make it so there are named ‘tabs’ for each server, and each tab will log into the server you specify. Each line in the file should be something like ’screen -t myServer ssh myServer’, the first mySever is the name of the tab, and then ssh myServer is the command that will be run. To simplify doing this in the future, I made a little Perl script that reads a file that has one server name per line and prints the rc file to standard out.

#!/usr/bin/perl
#===============================================================================
#         FILE:  makeScreenRc.pl
#        USAGE:  ./makeScreenRc.pl
#       AUTHOR:  Kyle Brandt (kb), kyle@kbrandt.com
#===============================================================================

use strict;
use warnings;

print "zombie qr\n";
while (<>) {
        chomp;
        my $server = $_;
        print "screen -t $server ssh $server", "\n";
}

So if you called the above script with something like ‘perl makeScreenRc.pl myDmzList > myScreenDmzRc’ you can then use the created file with ’screen -R DMZ -c myScreenDmzRc’. The capital R switch looks for an existing detached session and will attempt to reattach it before creating a new one. This will be useful with gnome-terminal in case gnome-terminal crashes.

Setting Up Gnome-Terminal
The next step is to create a profile for each of the screen sessions. You can do this by going to File::New Profile and then create a profile with a relevant name for the screen session, i.e. ‘DMZ’ . After that, Go to the Title and Command tab, check ‘Run a custom Command instead of my shell’ and and edit the command to be something like ’screen -R DMZ -c myScreenDmzRc’. Then repeat this for each of the screen sessions you have set up. Then, you can run something like ‘gnome-terminal –tab-with-profile=DMZ –tab-with-profile=MyOffice’ where DMZ and MyOffice are the names of the gnome-terminal profiles you created. This automatically detaches itself from the controlling terminal, so if you close the terminal you launched this from, the new terminal will not close. Lastly, you can set up a shell alias to run the above command, so all you have to do to open up your command central is type something like ‘myservers’.

Introduction:
A common practice in networks is to have at least two Exchange servers, a front-end server in the demilitarized zone (DMZ, perimeter network in Microsoft terminology) and a back-end server in the company’s main network.  The front-end server hosts the public SMTP server and web mail.  The back-end server hosts the actual mail (information stores).  There are two firewalls, one between your DMZ and your own network, and the second is between the DMZ and the Internet.

Although the firewall rules are well documented on the web, I ran into a couple of blocks when moving the Exchange front-end server into the DMZ.  I hope in sharing what I encountered helps others find doing the same thing a little less frustrating. My instructions will be specific to my experience in doing this, and is certainly not the only way to do it.

Planning the Firewall Rules:
As I mentioned in previously there are two firewalls in this setup. One is between the Internet and the DMZ and the other is between the DMZ and the internal network. In this example both the firewalls are just the Access Control Lists ( ACLs ) on the Internet interface and the DMZ interface of a Cisco router. The the front-end server is also behind Network Address Translation ( NAT ), so the server gets assigned a local IP.

The logical layout is how to think about the set up.  Firewall 1 is actually an inbound ACL on the Gig 0/0 interface and Firewall 2 is and inbound ACL on the Gig 0/1 interface.  In this usage of inbound, ‘in’ means heading into the interface ( see the actual layout ) and not ‘inbound’ in the sense of inbound traffic from the Internet. Firewall 1 is the firewall between the front-end server and the Internet. Firewall 2 is the firewall between the front-end exchange server and the primary domain controller as well as the back-end Exchange server. The rules on Firewall 2 assume you have bound remote procedure call ( RPC ) to specific port, which I will talk about more in the section about configuring the domain controller. The rules for Firewall 1 use the public Internet IP because that is in front of the NAT, in my example the public IP is 74.125.45.100 and it gets translated to 172.16.1.2 .

Gig 0/0 ACL in ( Firewall 1 ):
permit tcp any host 74.125.45.100 eq 25
permit tcp any host 74.125.45.100 eq 80
permit tcp any host 74.125.45.100 eq 110
permit tcp any host 74.125.45.100 eq 143
permit tcp any host 74.125.45.100 eq 443
Gig 0/1 ACL in ( Firewall 2 ):
permit tcp host 172.16.1.2 host 10.10.1.51 eq 25
permit tcp host 172.16.1.2 host 10.10.1.51 eq 80
permit tcp host 172.16.1.2 host 10.10.1.51 eq 143
permit tcp host 172.16.1.2 host 10.10.1.51 eq 110
permit tcp host 172.16.1.2 host 10.10.1.51 eq 691
permit tcp host 172.16.1.2 host 10.10.1.50 eq 88
permit udp host 172.16.1.2 host 10.10.1.50 eq 88
permit tcp host 172.16.1.2 host 10.10.1.50 eq 389
permit udp host 172.16.1.2 host 10.10.1.50 eq 389
permit tcp host 172.16.1.2 host 10.10.1.50 eq 3268
permit tcp host 172.16.1.2 host 10.10.1.50 eq 53
permit udp host 172.16.1.2 host 10.10.1.50 eq 53
permit tcp host 172.16.1.2 host 10.10.1.50 eq 135
permit tcp host 172.16.1.2 host 10.10.1.50 eq 12125


Active Directory Changes:
There were three changes I had to make in Active Directory. The first was to change the DNS entries for the mail servers. The second, which took me sometime to figure out, was to add the subnet to ‘Active Directory Sites and Services’. I had to do this because this was the first Windows server being deployed in the DMZ. To do this go to Control Panel :: Administrative Tools :: Active Directory Sites and Services :: Right-Click Subnets :: New Subnet and then for our example add ‘172.16.1.1/24′ and assign to whatever site it it is in. You may also need to give it time to replicate. The third change is to bind RPC replication to a specific port so you don’t have to open the firewall between the front-end and back-end servers as much. I chose to bind it to port 12125, if you don’t do this you will have open up TCP ports 1024 and above on the firewall. To bind RPC you need two registry entires, see: http://support.microsoft.com/kb/224196

Changes on the Front End Mail Server:
Besides Assigning the new IP to the front end exchange server, you might need to bind each service to this new IP. This wasn’t apparent to me because I could still connect to the SMTP and 443 ports but I got disconnected immediately. This indicates that even though the service is set not to be listening on that IP, it does listen and just disconnects if the first packets of the session are not destined for the IP the service is bound to. I can’t image how someone at Microsoft thought that this would be a good way to design the network stack on their operating system but that seems to be the way it works. To adjust what IP webmail (ports 80 and 443) are bound to, go to: Control Panel :: Administrative Tools :: Internet Information Services ( IIS ) Manager :: Local Computer :: Web Sites :: Right Click which ever web sites is for web mail :: Web Site Tab and then change the IP to whatever the new IP is for your front-end server, in our example, 172.16.1.2. Then to change the SMTP port and other service are bound to, go to: Exchange System Manager :: Administrative Groups :: Your Administrative Group ( i.e. First Administrative Group ) :: Servers :: The Front-End Server :: Protocols and then expanded each protocol, right click the virtual server, and change the IP in the General Tab.

Changes on the Back End Mail Server:
When I did this I did not have to make any changes to the back end server, but you can check the references section for things you may need to do. Also, this is windows, so you should probably reboot the server for good measure, and maybe several times.

Conclusion:
Although your experience may be different I hope this helps people trying to do a similar thing.

References:
http://technet.microsoft.com/en-us/library/aa997436(EXCHG.65).aspx
http://windowsitpro.com/article/articleid/46571/what-ports-do-you-need-to-open-to-allow-communication-between-a-microsoft-exchange-server-back-end-server-and-an-exchange-front-end-server.html

System administration is often about seemingly mundane tasks, but each task can provide an interesting puzzle to solve and there are many ways to solve it. I want to share a typical task in depth. I don’t intend to present every step of the task, but I will examine some of the steps in detail. I will also show some specific command line examples and draw larger themes of system administration from those examples.

The task is to move about 20 of instances of various LAMP web applications from one cluster to another cluster. These applications are all different, but they are built on a common framework and by the same company. So there is coherency but also discrepancies to deal with. In addition, the servers are on the other side of the country so the task is done remotely.

Preparation
The first step is to brainstorm about the different aspects that must be considered before starting the task. Some of these are moving the data itself, symbolic links, storage and system requirements, DNS, variables in the code itself, permissions, coordinating with the testing team, communicating with your own team, documentation and diagrams, firewall rules, room for expandability, correct versions of applications (PHP, Apache, MySQL), backup of data, business requirements, and network layout.

After brainstorming the ideal result is that you anticipate issues which need to be resolved in preparation for the task. That way there will be less interruptions and you will have a smoother work flow. This is because many of the above items are dependent upon each other, and your work flow will be interrupted if you didn’t think of these steps ahead of time (this will probably happen more than enough even with brainstorming). Taking the time to brainstorm might also inspire ways to improve the structure of the environment, find ways to make things more modular (the Unix way), and more efficient.

Some of the ideas and issues that brainstorming led to for my example are:

• One of the servers had to be moved into the new cluster and put on the same subnet, this may seem obvious, but this is all being done remotely and my hosting provider can’t read my mind. Also deploying the applications and setting up things such as MySQL replication are dependent upon this.
• Firewall rules had to be altered to transfer the data to the new environment from the old servers.
• New testing domain names on our public and private DNS servers had to be set up so the applications could remain live while the new instances were tested. This is an example of business needs. Communicating with the business managers is essential so they can plan around the migration and you can plan around their needs.

The Migration

Step 1: Build the Environment:

One of the first steps was to deploy the service applications (i.e Apache, MySQL, etc) to the new environment. The choice was to compile these applications from source because customization beyond using a package (i.e. .deb or .rpm) was required.

A wise question for system administrators to ask themselves is: “Is there a chance I am going to have to do this or something similar again?” This question leads to doing things in a way that will make them easy to repeat in the future and, “working smarter, not harder.” So with compiling LAMP the smart thing to do (if time permits) is to design a custom makefile or script to compile all these applications. Even smarter might be to find someone else who has already done this, for example: http://www.apachetoolbox.com/ for installing Apache.

A simple bash script might have option parsing with each program as an argument. Then each option would download, extract, and then compile the program logging any build errors. You also would probably want to make functions that would be common to all the lamp applications, such as downloading and extracting. As you deploy the applications you can edit the script so it is ready for the next time you need to build a LAMP environment.

An unforeseen issue:
For ease of backup the choice was to put all applications (web applications and the service applications) under an /opt directory. You can specify a base path at compile time for the service applications. However, I did run into a problem with libraries when doing this. Shared libraries not being found is a common error. To resolve missing shared library errors:

• run ldd on the executable and it should tell you which libraries are missing. It will be something like library.so.1 => Not Found
• Then find the library directory for the application you just installed (somewhere under /opt in this case).
• Add the directory path to a text file in /etc/ld.so.conf.d, something like /etc/ld.so.conf.d/myapp.conf
• run ldconfig

To learn more about shared libraries and how it all works I recommend this article.

The last thing I would like to mention about setting up the environment is the use of symbolic links. It is often beneficial to make the root of a website a symbolic link. When updating the code you can set up a new virtual host and a new copy of the site for testing. Then when it is time to move to the updated version you can just change the symbolic link. This makes the structure more modular (again, the Unix way) and will allow for virtually no down time of the application if executed properly.

In the next part I will talk about moving the web applications.

Update: It has been a while since I actually did this task and since writing this I got side tracked with other projects. So I will probably not be writing any more parts unless I receive a specific request.