Kyle Brandt

Introduction:

From what I have learned there are at least two typical scenarios for WAN level redundancy with BGP that you would use in a co-location environment. The first is when you get an AS from ARIN, and advertise your IP block to two different providers so you become an edge (aka border) router on the Internet receiving full BGP Internet routing tables. The other scenario is when you get a private AS from your hosting provider, and the hosting provider is the actual edge router that is multi-homed to different ISPs. This lab is about the second scenario.

A Brief Introduction to BGP:
BGP stands for Border Gateway Protocol. If there is a key concept to understand, it is that with BGP the goal is no longer to build the routing table based on each single IP network and hop (at least with eBGP), but rather between Autonomous Systems (AS). An autonomous system is a network, or a group of networks, under the control of a single organization. An example of an autonomous system might be an entire ISP, or the northeast region of an ISP, and each AS on the Internet is given a number called an ASN. So the Border in BGP is the border between these autonomous systems (Between ISPs for example). So with BGP, routing tables are built based on the number of AS hops (or in real life, many other factors), not IP hops so BGP in effect provides a macro view of the entire network (ie the Internet) leaving out the details of all the hops within an AS. The routers that live on these borders are called border or edge routers, and they speak BGP to each other.

What I just described is external BGP (eBGP), there is also internal BGP (iBGP) which is used to communicate between routers within an AS. Routers that speak BGP to each other are neighbors, and these neighbors are manually entered on each router. If the neighbors have the same AS number (ASN) then it is iBGP, if they are different, then it is eBGP.

The way that Autonomous Systems and IP networks work together is that an AS has a summary of all the networks under it’s control. Border routers announce the networks in their AS to their neighbors and then those neighbors pass it on so it reaches every border router on the Internet. As the routing information passes through a neighbor to another neighbor , the neighbor appends its AS number to the route in the AS_PATH attribute. This resulting list of AS numbers ends up being a possible route that router can use to build its routing table.  It is important to understand that the the BGP table and the routing table are different, the BGP table is used to build the routing table. This attribute also prevent routing loops because by not allowing information to enter into the BGP tables that has its own AS in the AS_PATH. So BGP does need to know the next hop IP address, but only the next hop ip for the next hop AS.

Two last things I should mention are that BGP routes have attributes attached to them, and that a router attached to only one AS is called a stub. Example attributes are AS Path, Next Hop, and Local Preference.

Exploring BGP:
If you like to get to ‘touch’ the things that you learn, there are a couple of things I recommend with BGP. With traceroute, you can display the ASN numbers (at least with the one that comes with Ubuntu) by using the -A switch. So you can see where there border routers are and how the packets travel with an ASN. Some major ISPs also operator “Looking glass routers” that you can telnet into and use some show commands. For example:

$ telnet route-server.ip.att.net
...
User Access Verification
Username: rviews
route-server>show ip bgp 24.61.0.0/15
BGP routing table entry for 24.60.0.0/15, version 5701270
Paths: (18 available, best #14, table Default-IP-Routing-Table)
Not advertised to any peer
7018 7922 7015, (received & used)
12.123.25.245 from 12.123.25.245 (12.123.25.245)
Origin IGP, localpref 100, valid, external
Community: 7018:5000 7018:34011

You can find a list of these routers that you can log into here.

Introduction to HSRP:
HSRP stands for Hot Standby Router Protocol and it is designed to provide LAN redundancy by providing a backup of the default gateway that each machine on the LAN has set. The typical setup is that you have two routers and each has an interface on the LAN.  On each of these interfaces, they get an ip that is on the lan, but they get an additional virtual IP that will be the gateway for machines on the LAN.  This IP is held by one of the routers at a time, the routers send messages to each other, and if one of the routers goes down and that router had the gateway IP, the other router will pickup the gateway IP.

The Dynamips/Dynagen Lab:

You know the cliche about pictures, so let me start with this diagram:

So in this lab, Provider1 and Provider2 are the two routers by the same provider.  Mine1 and Mine2 are the routers that a client would own.  The two Autonomous Systems are 1500 and 64512.  64512 falls into the range of Private ASNs, so true edge routers don’t worry about them, they know how to get to 1500, so it is loosely analogous to NAT. There were two main objectives to this lab. My first objective was to make it so that any one router or switch in the lab can fail, and traffic would still continue to route.  The second objective was to make it so Mine1 is always preferred as long is it is up and running.  The configuration on the provider routers are probably an over simplification of what is going on, but the configuration is functional for the purpose of this lab. The final thing I need to point out before going into the configuration was that the switches and the bonded nics are not actually part of the lab because they are beyond the capability of dynamips.

The Router and Lab Configurations:
I am going to assume that you already know how to use Dyanagen and Dynamips and how to assign ip addresses to interfaces.  If you don’t know how to use Dynagen you can go through a tutorial here.  I have posted the Dynagen configuration for this lab here , and the complete configurations for all the routers in the lab are posted here.

The BGP Configuration:

Neighbor Relationships:

The first step after creating the lab and setting up the ip addresses is to set up the neighbor relationships.  These are created by manually entering in each side of a neighbor relationship on the routers. There are 4 of these neighbor relationships, two iBGP sessions with one between Mine1 and Mine2 and the other between Provider1 and Provider2, and two eBGP sessions with one between Mine1 and Provider1, and the other between Mine2 and Provider2. So For example, to configure the iBGP session between Mine1 and Mine2 we do the following:

Mine1(config)#router bgp 64512
Mine1(config-router)#neighbor 192.168.8.2 remote-as 64512

Mine2(config)#router bgp 64512
Mine2(config-router)#neighbor 192.168.8.1 remote-as 64512

Notice that on each router the bgp AS number, 64512, is the same as the configured router, so this is iBGP. When we set up these neighbor relationships, the ip of the other side is specified. Also, 64512 falls in the range of 64512-65534 which means it is a private autonomous system number. In order to verify the relashionship is established do the following:

Mine1#show bgp sum
BGP router identifier 192.168.8.1, local AS number 64512
BGP table version is 4, main routing table version 4
2 network entries using 234 bytes of memory
3 path entries using 156 bytes of memory
5/2 BGP path/bestpath attribute entries using 620 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1034 total bytes of memory
BGP activity 2/0 prefixes, 5/2 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
172.16.8.1      4  1500    2775    2777        4    0    0 1d22h           1
192.168.8.2     4 64512    2800    2798        4    0    0 1d22h           1

The last line should an up time for the 192.168.8.2 neighbor relationship (In this I had already set up the eBGP neighbor relationship as well). BGP isn’t the fastest routing protocol, so if it isn’t up you may want to wait about 30 seconds. Also, if you haven’t edited the logging at all, you should see a message printed to the console when it is established. If it still isn’t working, double check your ASN and IP numbers, that you can ping the neighbor ip from both routers, and also that you are not filtering TCP port 179 with an ACL because this is what BGP uses.

Setting up a eBGP session is essentially the same:

Mine1(config)#router bgp 64512
Mine1(config-router)#neighbor 172.16.8.1 remote-as 1500

Provider1(config)#router bgp 1500
Provider1(config-router)#neighbor 172.16.8.2 remote-as  64512

This time and the neighbor remote AS is different, so it is eBGP.

Injecting Routes:

The main point of BGP is to advertise your IP range to the Internet, to do this you inject the routes into BGP. In this lab the Provider has given us a range of 12.12.12.0/27. This range doesn’t appear on any of the router interfaces because all of those IP addresses are NAT’d to the 192.168.1.1/30 network. Since 192.168.1.1/30 is accessible from both routers we want 12.12.12.0/27 to be available from both routers, so we announce the route from both Mine1 and Mine2. The BGP configuration to announce a route is straight forward:

Mine1(config)#router bgp 64512
Mine1(config-router)#network 12.12.12.0 mask 255.255.255.224

The one caveat is that a route will not be announced unless the it is in the routing table and since we don’t have a directly connected route because no interface has that route we need to add a null route:

Mine1(config)#ip route 12.12.12.0 255.255.255.224 Null0

When traffic destined for the 12.12.12.0 arrives from the Provider network, it doesn’t go to NULL because NAT happens before routing (There might be a better explanation for this however, this is just how I think of it, either way, it works). After doing this, you can verify that the provider network sees the network with the following:

Provider1#show ip bgp
BGP table version is 2, local router ID is 172.16.8.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network          Next Hop            Metric LocPrf Weight Path
*> 12.12.12.0/27    172.16.8.2               0             0 64512 i

This same configuration is then repeated on Mine2 changing the neighbor accordingly.

Achieving WAN Redundancy:

The steps we have taken so far fail to provide redundancy for the WAN. An example of why this is can be seen by shutting down Mine 1. Before we do this, let us look at the BGP table and the routing table on Provider 1.

Provider1#show ip bgp
...
Network          Next Hop            Metric LocPrf Weight Path
* i12.12.12.0/27    172.16.8.6               0    100      0 64512 64512 i
*>                  172.16.8.2               0             0 64512 i
Provider1#show ip route
....
172.16.0.0/30 is subnetted, 1 subnets
C       172.16.8.0 is directly connected, FastEthernet0/0
10.0.0.0/30 is subnetted, 1 subnets
C       10.0.0.0 is directly connected, FastEthernet1/0
12.0.0.0/27 is subnetted, 1 subnets
B       12.12.12.0 [20/0] via 172.16.8.2, 1d23h

With BGP, only one route to a destination will be entered into the routing table from the BGP table.  The current route to 12.12.12.0/27 is via 172.16.8.2. If we look at the second route in BGP it is via 172.16.8.6.  However there is a problem with this, if you look at the routing table there is no route to 172.16.8.6 in the routing table.  So lets look at the tables after we shutdown Mine 1.

Provider1#
*Mar  3 00:22:42.223: %BGP-5-ADJCHANGE: neighbor 172.16.8.2 Down BGP Notification sent
*Mar  3 00:22:42.223: %BGP-3-NOTIFICATION: sent to ...
... neighbor 172.16.8.2 4/0 (hold time expired) 0 bytes
Provider1#show ip bgp
...
   Network          Next Hop            Metric LocPrf Weight Path
* i12.12.12.0/27    172.16.8.6               0    100      0 64512 64512 i
Provider1#show ip route
...
Gateway of last resort is not set
     172.16.0.0/30 is subnetted, 1 subnets
C       172.16.8.0 is directly connected, FastEthernet0/0
     10.0.0.0/30 is subnetted, 1 subnets
C       10.0.0.0 is directly connected, FastEthernet1/0

There is now no route on Provider 1 to the 12.12.12.0 network even though there is an entry in the BGP table. The reason for this is that routes will not go from the BGP table into the routing table if the next hop can not be reached. The solution to this problem is to use the next-hop-self command. What this does is change the next-hop attribute to be the router that shares this information. To set this up on the provider side we do the following:

Provider1(config)#router bgp 1500
Provider1(config-router)#neighbor 10.0.0.2 next-hop-self

Provider2(config)#router bgp 1500
Provider2(config-router)#neighbor 10.0.0.1 next-hop-self

Then if you clear the iBGP session with you can see that the next hop has changed on Provider 1:

Provider1#clear bgp all 1500
Provider1#show ip bgp
...
   Network          Next Hop            Metric LocPrf Weight Path
*>i12.12.12.0/27    10.0.0.2                 0    100      0 64512 64512 i

Because the next hop is now reachable, there is now a route in route table for Provider 1 for the 12.12.12.0/27 network. The same thing should be done for the iBGP session between Mine1 and Mine2 as well. This problem could also be solved by using an Interior Routing Protocol as well. At this point if you are following along you can start Mine 1 again.

Configuring the Default Gateway:

Since the Mine 1 and Mine 2 routers are not getting full Internet tables from the provider, they will need a default gateway. BGP can supply a default gateway to its neighbors. To configure this is easy, you just use the default-originate command on each of the Provider routers.

Provider1(config)#router bgp 1500
Provider1(config-router)#neighbor 172.16.8.2 default-originate

Provider2(config)#router bgp 1500
Provider2(config-router)#neighbor 172.16.8.6 default-originate

Setting the Preferred Route:

The other objective I stated is that I want Mine 1 to be the preferred router as long as it is up and running. To accomplish this I use two tools, local-preference to set the preferred egress path from network, and AS_PATH appending to tell AS 1500 the preferred ingress route to use.

A higher local preference means the path is more favorable. For this portion, I have Mine 1 tell Mine 2 that Mine 1 is best way to go. This is done with a route map:

Mine1(config)#route-map LOCALPREF permit 10
Mine1(config-route-map)#set local-preference 200
Mine1(config-route-map)#exit
Mine1(config)#router bgp 64512
Mine1(config-router)#neighbor 172.16.8.1 route-map LOCALPREF in

This sets it so Mine1 will mark any routes it has learned about with a higher local preference when sending them to Mine 2:

Mine2#show ip bgp
...
   Network          Next Hop            Metric LocPrf Weight Path
*>i0.0.0.0          192.168.8.1              0    200      0 1500 i
*                   172.16.8.5               0             0 1500 i
* i12.12.12.0/27    192.168.8.1              0    100      0 i
*>                  0.0.0.0                  0         32768 i

So now if any egress traffic ends up Mine 2 that would be use the default route, it will go through Mine 1.

To tell the Provider routers that Mine 1 is preferred, we have Mine 2 prepend the AS number again so routes coming from it have a longer AS_PATH and therefore will be less preferred. Again this is done with a route map:

Mine2(config)#route-map AS_PATH_APPEND permit 10
Mine2(config-route-map)#set as-path prepend 64512
Mine2(config-route-map)#exit
Mine2(config)#router bgp 64512
Mine2(config-router)# neighbor 172.16.8.5 route-map AS_PATH_APPEND out

Now on Provider 2, the routes that Mine 2 announced will seem longer than the route to Mine 1 from Provider 2 so the route to 12.12.12.0/27 via Provider 1 is preferred:

Provider2#show ip bgp
...
   Network          Next Hop            Metric LocPrf Weight Path
*>i12.12.12.0/27    10.0.0.1                 0    100      0 64512 i
*                   172.16.8.6               0             0 64512 64512 i

To get a feel for all of this, I recommend taking down the routes and/or interfaces, and then running traceroutes to see how all the traffic then goes once the BGP updates have been received by neighboring routers (BGP isn’t that fast, so wait for the log messages to show up).

The HSRP Configuration:

A basic HSRP set up is actually pretty simple. Each LAN interface gets its own IP and they share a Virtual IP that computers on the LAN use as their gateway. The virtual IP is held by one router at a time only as long as both routers can still send broadcasts to each other over the LAN. In this lab, the virtual gateway IP address is 192.168.1.1:

Mine1(config)#int FastEthernet2/0
Mine1(config-if)#ip address 192.168.1.2 255.255.255.0
Mine1(config-if)#standby ip 192.168.1.1
Mine1(config-if)#standby priority 110
Mine1(config-if)#standby preempt delay reload 20
Mine1(config-if)#standby track FastEthernet0/0

Mine2(config)#int F2/0
Mine2(config-if)#ip address 192.168.1.3 255.255.255.0
Mine2(config-if)#standby ip 192.168.1.1
Mine2(config-if)#standby priority 109
Mine2(config-if)#standby track FastEthernet0/0

So each interfaces does get it’s own IP, but the virtual IP is shared by both. The track command makes it so that if the WAN side of either of the Mine routers goes down the gateway will follow. I have set it up so this doesn’t need to happen because traffic can enter Mine 1 on the LAN side, and leave Mine 2 on the WAN side, but I figure it makes more sense to have them stay together. The preempt command makes it so that if Mine 1 has failed, but comes back up, it will become primary again. The higher priority also makes sure that it is primary.

Conclusion:

The parts that are left out of the lab are a server with bonded nics, and the switches. The switches should be connected together with a couple of connections and be running some version of Spanning Tree Protocol (STP). The simplest set up for the bonded NICs if you are using Linux would be Active-Backup mode as it doesn’t require special configuration on the switches. I included the basic NAT configuration in the complete router configurations, however, stateful NAT would be an option for more rapid fail over because both routers would share the NAT table. With this setup there is no longer a single point of failure on the network at the device, port, or cable level. A router, switch, or network card can fail and the network should recover on its own.

I remotely administer Cisco routers, which can be a bit stressful since I need make sure not lock my self out by do something like removing a NAT command, editing ACL, or screwing up a VPN tunnel. A Cisco support person pointed out to me that I can use the reload command to schedule reboots, so if I lock myself out, I just have to wait until that timer runs out. If I don’t need it, I can cancel the scheduled reload. So, before you start running, copy the running configuration if you are happy with it using ‘copy run start’. Then you can use this trick as follows:

Lab1#copy run start
Destination filename [startup-config]?
Building configuration...
[OK]
Lab1#reload in 8
Reload scheduled in 8 minutes by console
Reload reason: Reload Command
Proceed with reload? [confirm]y
Lab1#
Lab1#show reload
Reload scheduled in 7 minutes by console
Reload reason: Reload Command
Lab1#reload in 12
Reload scheduled in 12 minutes by console
Reload reason: Reload Command
Proceed with reload? [confirm]y
Lab1#show reload
Reload scheduled in 11 minutes by console
Reload reason: Reload Command
Lab1#reload cancel
***
*** --- SHUTDOWN ABORTED ---
***
Lab1#show reload
No reload is scheduled.
Lab1#

Notice in line 13 you can reset or bump up the timer. And you can always use ’show reload’ to see how much time you have. Of course, now you have to be careful not to let the timer run out or forget to remove it.

A Intermittent Problem:
I wrote a Perl script for Nagios that would figure out the bandwidth of an interface by parsing TX (transmit) and RX (receive) bytes from /proc/net/dev. The proc file system is a virtual file system that provides the ability to view various kernel statistics as well as modify some kernel parameters. My script parses the file twice at a specified interval, and then subtracts the old value from the new value to return bytes per second. I realized that this wasn’t the most accurate method, but it was good enough for my purposes and I didn’t have to install snmp. Also, the larger the interval, the smaller the error would generally be assuming light load.

The problem was that this script would fail every so often with ‘Not numeric subtraction’. So I started saving snapshots of /proc/net/dev and noticed that the script would fail after when the values were around 4 billion something. This I knew to be in the neighborhood of 2^32 (The max of a positive only 32-bit integer value). To confirm my thoughts that this was the max value for this counter, I decided to have a poke around the kernel source code.

Into the Kernel:
I didn’t know where to look in the source for this, but /proc/net/dev has the string ‘Inter-|’ which I figured would be a unique enough string to give me a place to start. Sure enough, a recursive grep for this string returned only 3 lines of code. The function I wanted was dev_seq_printf_stats in dev/core/dev.c:

static void dev_seq_printf_stats(struct seq_file *seq, struct net_device *dev)
{
        struct net_device_stats *stats = dev->get_stats(dev);
        seq_printf(seq, "%6s:%8lu %7lu %4lu %4lu %4lu %5lu %10lu %9lu "
                   "%8lu %7lu %4lu %4lu %4lu %5lu %7lu %10lu\n",
                   dev->name, stats->rx_bytes, stats->rx_packets,
                   stats->rx_errors,
                   ///.....

Looking at the printf specifiers for this they were %ul — unsigned long integer, which on my system was indeed a max of 4294967295 ( 32^2 – 1). I wanted to be extra sure, so I traced the net_device_stats struct to include/linux/netdevice.h and confirmed that the net_device_stats->rx_bytes member was in fact an unsigned long integer. So now I knew the error happened when the counter maxed out and then reset to zero, but why a non-numeric subtraction error?

Problem Found:
%8lu as a ANSI C standard library printf specifier defaults to 8 characters wide, and also defaults to right justify since there is no hyphen flag. To find out if the kernel did the same I traced seq_printf to lib/vsprintf.c and saw that the Linux kernel version formatted this in the same way. When the bytes value was less than 8 characters long, there was leading white space that threw off my parser. All I needed was to add the extra line at line 9 to eliminate any leading whitespace:

sub parseBandwidth {
    my $interface = shift;
    my @ifconfigOutput = @_;
    foreach my $line (@ifconfigOutput) {
        if ( $line =~ /:/ ) {
            my @interfaceLine = split( /:/, $line);
            if ($interfaceLine[0] =~ /$interface/) {
                # Next line is to sanitize leading whitespace
                $interfaceLine[1] =~ s/^\s+//;
                my @interfaceStats = split( /\s+/, $interfaceLine[1] );
                print( LOG "DEBUG I have parsed out: @interfaceStats\n") if $debug;
                return @interfaceStats;
            }
        }
    }
}

Introduction:
I just got a new G1 Android phone, and since it runs Linux I just had to get the Bash shell running on it, the built in shell would just not do.  I do need my tab completion after all. Cross compilation is the process of compiling software on one platform that is meant to run on another.  With the following an ARM executable is compiled on a x86 Linux machine.

Requirements (Not sure if all this is needed, but it is what I used):

• Cupcake 1.5 JS Build with root access: http://androidandme.com/2009/05/guides/beginners-guide-for-rooting-your-android-g1-to-install-cupcake/
• ARM Toolkit (The Cross Compiler): http://www.codesourcery.com/gnu_toolchains/arm/download.html
• Android SDK installed on Linux: http://developer.android.com/sdk/1.5_r2/index.htm
• A G1
• The source code for Bash 4.0: ftp://ftp.cwru.edu/pub/bash/bash-4.0.tar.gz

Step 1: Connect your pc to your phone with the SDK
You first have to be able to connect to your computer with adp which is included with the sdk.  To do this with Ubuntu Januty Jackalope you first need to create a /etc/udev/rules.d/51-android.rules file with the following contents:

SUBSYSTEM=="usb", SYSFS{idVendor}=="0bb4", MODE="0666"

After this run the following to restart udev: ’sudo /etc/init.d/udev reload’. Lastly, on your phone make sure Settings :: Applications :: Development :: USB Debugging is enabled and the plug in your phone.  When you run ‘./adp devices’ you should see a device listed.

Step 2: Build the Bash Shell
After installing ARM toolkit in /home/kbrandt/bin/arm-toolkit (used for this example) set the following environment variables in your shell.

CC='/home/kbrandt/bin/arm-toolkit/bin/arm-none-linux-gnueabi-gcc'
PATH="$PATH:$HOME/bin/"

Then cd to the directory where you extracted the Bash source and run the following:

./configure --prefix=/opt/arm_bash/ -host=arm-linux --enable-static-link --without-bash-malloc

Assuming that worked, edit the ‘Makefile’ file and change ‘CFLAGS = -g -O2′ to ‘CFLAGS = -g -O2 -static’ and then run ‘make’. If this works, it should create a bash executable in the current directory. You can verify that this has been compiled for the ARM architecture with ‘file bash’. This should return:

bash: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, for GNU/Linux 2.6.14, not stripped

Step 3: Copy the File to your android.
From the host computer in the tools directory of the SDK run ‘./adb push ~/src/bash-4.0/bash /data/’ to copy the executable to the phone. If you try to copy it to your sdcard, make sure the sdcard is not mounted with the noexec mount option as this disables the executable permission bit.

Step 4: Run, Enjoy, and Find Bugs.
You can now connect to your phone with ‘./adp shell’ and cd to the data directory and run ‘./bash’ and you should get a bash prompt.  You might need to ‘chmod 555 bash’ if you get permission denied.

References:
http://jiggawatt.org/badc0de/android/index.html

The Problem
I have some users who are on a different mail system but still part of my company. The problem was that the users without Exchange 2003 accounts were not showing up in the Global Address List (GAL).

Solution
The first step was to look at the LDAP filter that generates the GAL. This can be viewed by going into the ‘Exchange System Manager’ and then ‘Recipients::All Global Address Lists::Default Global Address List::General Tab’. The following is the filter on my Exchange system (which I am guessing is the default ):

(& (mailnickname=*) (| (&(objectCategory=person)(objectClass=user)(!(homeMDB=*))(!(msExchHomeServerName=*)))(&(objectCategory=person)(objectClass=user)(|(homeMDB=*)(msExchHomeServerName=*)))(&(objectCategory=person)(objectClass=contact))(objectCategory=group)(objectCategory=publicFolder)(objectCategory=msExchDynamicDistributionList) ))

Looking at this filter, and not being rain man, I couldn’t just glance at and figure out what it meant. The trick was to load it with VIM, because it will highlight matching parenthesis (When your cursor is over the opening parenthesis, the match closing parenthesis gets highlighted). I then I indented based on that which resulted in:

(&
   (mailnickname=*)
   (|
      (&
         (objectCategory=person)
         (objectClass=user)
         (!(homeMDB=*))
         (!(msExchHomeServerName=*))
      )
      (&
         (objectCategory=person)
         (objectClass=user)
         (|
            (homeMDB=*)
            (msExchHomeServerName=*))
      )
      (&
         (objectCategory=person)
         (objectClass=contact)
      )
      (objectCategory=group)
      (objectCategory=publicFolder)
      (objectCategory=msExchDynamicDistributionList)
   )
)

The next step was to look at the active directory key value pairs for for one of the users that wasn’t showing up in the GAL. I know of two ways to do this, one is to use adsiedit.msc for windows, or, if you want to be super cool, use ldapsearch in Linux. To use ldapsearch to look at the attributes for the object, you would use a command like: ldapsearch -w $PW -v -x -D "cn=Administrator,cn=Users,dc=myDomain,dc=com" "cn=Kyle Brandt" where ‘Kyle Brandt’ is the user you want to look at, Administrator is the user you use to authenticate with AD, PW is a shell environment variable with you password, and myDomain is your company’s AD domain name.
Once I saw that mailNickname was not set, and since the filter says ’show in GAL if mailNickname is set to something, OR if … lots of stuff …’ all I had to do was use adsiedit to set that attribute to something. To learn how to read and write these filters see this rfc or this msdn page. You can see if the change will effect the GAL book by clicking ‘Preview’ on the tab were the filter originally was. It will probably take a day or so (depending on your settings) before the change is actually made to the GAL.

Introduction
For a system administrator it is important to have an efficient and comfortable interface to all your servers. GNU Screen is an excellent utility to be able to have a single terminal connected to multiple servers that won’t disappear when you close the window. I have a set up that allows me to spawn gnome-terminal with different screen sessions for each location I administer in a different tab. Then each screen session has a named ‘tab’ that automatically logs into each server at that location. It ends up looking like this:

My main two recommendations for screen are to set up the meta character as back-tick ( ` ) and to give screen a ‘tab bar’. You can read how to do these two things here.

Setting up Screen
Once you have screen set up the way you like, you can the specify an additional screenrc file with the -c switch and your settings in the ~/.screenrc will still be used. The secondary screenrc is where you can list different server groups. This file will make it so there are named ‘tabs’ for each server, and each tab will log into the server you specify. Each line in the file should be something like ’screen -t myServer ssh myServer’, the first mySever is the name of the tab, and then ssh myServer is the command that will be run. To simplify doing this in the future, I made a little Perl script that reads a file that has one server name per line and prints the rc file to standard out.

#!/usr/bin/perl
#===============================================================================
#         FILE:  makeScreenRc.pl
#        USAGE:  ./makeScreenRc.pl
#       AUTHOR:  Kyle Brandt (kb), kyle@kbrandt.com
#===============================================================================

use strict;
use warnings;

print "zombie qr\n";
while (<>) {
        chomp;
        my $server = $_;
        print "screen -t $server ssh $server", "\n";
}

So if you called the above script with something like ‘perl makeScreenRc.pl myDmzList > myScreenDmzRc’ you can then use the created file with ’screen -R DMZ -c myScreenDmzRc’. The capital R switch looks for an existing detached session and will attempt to reattach it before creating a new one. This will be useful with gnome-terminal in case gnome-terminal crashes.

Setting Up Gnome-Terminal
The next step is to create a profile for each of the screen sessions. You can do this by going to File::New Profile and then create a profile with a relevant name for the screen session, i.e. ‘DMZ’ . After that, Go to the Title and Command tab, check ‘Run a custom Command instead of my shell’ and and edit the command to be something like ’screen -R DMZ -c myScreenDmzRc’. Then repeat this for each of the screen sessions you have set up. Then, you can run something like ‘gnome-terminal –tab-with-profile=DMZ –tab-with-profile=MyOffice’ where DMZ and MyOffice are the names of the gnome-terminal profiles you created. This automatically detaches itself from the controlling terminal, so if you close the terminal you launched this from, the new terminal will not close. Lastly, you can set up a shell alias to run the above command, so all you have to do to open up your command central is type something like ‘myservers’.

The Module
This new module I have written provides an object oriented interface to TVRage’s XML service which allows you to get episode and other information for television shows. It is written very similarly to my previous module, WebService::UPS, and also uses XML::Simple and Mouse. You can get the module from CPAN here. You can also install it with ’sudo cpan -i WebService::TVRage’.

Example:

use WebService::TVRage::EpisodeListRequest;
use WebService::TVRage::ShowSearchRequest;

my $searchReq =   WebService::TVRage::ShowSearchRequest->new();
my $searchResults = $searchReq->search('Heroes');
my $heroFromSearch = $searchResults->getShow('Heroes');
print $heroFromSearch->getLink(), "\n";
print $heroFromSearch->getCountry(), "\n";
print $heroFromSearch->getStatus(), "\n";
my $heroes =  WebService::TVRage::EpisodeListRequest->new( 'episodeID' => $heroFromSearch->getShowID() );
my $episodeList = $heroes->getEpisodeList();
print $episodeList->getNumSeasons(), "\n";
my $episode = $episodeList->getEpisode(1,3);
print $episode->getTitle(), "\n";
print $episode->getAirDate(), "\n";
foreach my $showtitle ($searchResults->getTitleList()) {
   my $show = $searchResults->getShow($showtitle);
   print $show->getLink();
}

Introduction:
O’Reilly’s Mastering Algorithms with Perl is written for programmers who are already quite familiar with Perl. I thought it might help myself and maybe others to walk through the code for selection sort that is on Page 120 because the code isn’t the clearest Perl. My analysis is not meant to explain selection sort because the book does that. Rather, it is to explain the Perl code.

The Code:

#!/usr/bin/perl 

use strict;
use warnings;

sub selection_sort {
    my $array = shift;

    my $i;      # The starting index of a minimum-finding scan.
    my $j;      # The running index of a minimum-finding scan.

    for ( $i = 0; $i < $#$array ; $i++ ) {
        my $m = $i;             # The index of the minimum element.
        my $x = $array->[ $m ]; # The minimum value.

        for ( $j = $i + 1; $j < @$array; $j++ ) {
        ( $m, $x ) = ( $j, $array->[ $j ] ) # Update minimum.
            if $array->[ $j ] < $x;
        }

        # Swap if needed.
        @$array[ $m, $i ] = @$array[ $i, $m ] unless $m == $i;
    }
}

my @array = (1, 7, 2, 8, 2, 5, 20);
selection_sort(\@array);
print "@array\n";

The only change I made to the code from the example is to change lt to < on line 18 so the comparison is numerical and not by ascii value.

Analysis:
Line 27 passes a reference to the @array array to the selection_sort subroutine. This makes it so the array is changed in place and a copy of the array does not have to be made.

Line 7 makes the variable $array , which is local to the function selection_sort, a reference to the array in line 26. ’shift’ removes the first argument to the function from the @_ array. The @_ is the default variable within a subroutine so it can be omitted. The line could have been written as $array = shift(@_); .

Line 12 uses a c-style for loop (Please forgive the messed up syntax highlighting). In Perl for and foreach do the same thing. However, a for or foreach loop is context sensitive depending on what comes after the for/foreach keyword. So Perl programmers use for when they are writing a c-style loop. The c-style loop is used for array indexing here. The first statement, $i = 0; initializes the index variable. The second statement, $i < $#$array; is the exit condition as in a while loop. The third statement $i++ increments by one each loop. In the second statement, $#$array, means ‘the index of the last element of the array that the reference $array points to’. Since it is less than instead of less than or equal to, and it is the last index of the array, not the number of items in the array, the loop will stop before the last element of the array.

Line 14 is again using references, so $array->[ $m ] returns the value of the index $m in array that $array points to.

Line 16, like 12, uses the c-style loop. This time the exit condition (the second statement in the loop header) is $j < @$array . This dereferences the array that $array points to, and evaluates in scalar context which returns the number of items in the array. So this also could have been written as $j <= $#$array .

Line 17 and 18 are like a standard if statement but it is written backwards. This is known as postfix syntax, a trailing conditional, or a statement-modifying if. Note, this is written as one line — there is no semi-colon.

Line 22 also uses the trailing conditional, this time it is ‘unless’. This line uses array slices to swap the items. The slice [ $m, $i ] selects two items, the item at index $m and index $i . It does not select a range like the python array[0:2] . In Perl you use the range operator .. instead of : . And again @$array deferences the array, you will often see this written as @{$array} .

Conclusion
I hope this helps someone else who is reading this book, thanks to everyone in #perl on irc.freenode.com for the help.

The Module:
I have made a Perl object oriented module for tracking UPS shipments. To use this module you will need to get a developer key for the UPS online tools here. This module makes a XML request to the online tools, and then parses the response using XML::Simple. The module has methods to get specific information such as recent activity. You can read the full module documentation as well as download the module at CPAN’s site here.

Example:

    my $Package = WebService::UPS::TrackRequest->new;
    $Package->Username('kbrandt');
    $Package->Password('topsecrent');
    $Package->License('8C3D7EE8FZZZZZ4');
    $Package->TrackingNumber('1ZA45Y5111111111');
    print $Package->Username();
    my $trackedPackage = $Package->requestTrack();
    print $trackedPackage->getActivityList();

Installation:
You can install this module with cpan. In Linux the command is ‘cpan -i WebService::UPS::TrackRequest’ . The required prerequisite modules are: Mouse, LWP::UserAgent , HTTP::Request::Common , XML::Simple , and Data::Dumper .

Introduction:
I think of the American government as a democratic republic. The government is run by a small group of people, a republic, that is elected by the public to represent them, a democracy. Congress, and the bills they pass, should have oversight from the people. Although the bills are made available to the public, the size makes them somewhat inaccessible, and in my opinion, the media fails at providing enough detailed information on the content of these bills.

My goal was to take the 2009 stimulus bill and try to parse out some information about where the money is going in this massive bill. Although my parser is incomplete, it was still able to parse out information that I could not find by searching for it on the Recovery Website. So I consider it useful.

Parsing the Bill:
My first step was to get the bill into something I could parse. When I started this, I wasn’t aware of THOMAS, so I got the pdf from the recovery.org website and then converted it. I did this using pdftotext, and I then converted it to ascii to make it easier to work with the funny start and end quotes (also called left handed and right handed quotes):

pdftotext -enc UTF-8 -eol unix recoveryAct.pdf recoveryAct2.txt
cat recoveryAct2.txt | uni2ascii -e > recoveryAct3.txt

The parsing is done with regular expressions. After looking over the file, I decided the best way in was to parse the file line by line. I did however want to look ahead at certain points, so I read the file into memory to make this simple. This is known as slurping. The first two regular expressions capture the page number and the section of the document. These are important because all this script really does is make a sort of index, so I can find things that might be of interest.

The group of four regular expressions starting at line 50 do the bulk of the work. These parse the dollar amount after variations of a frequent phrase in the bill. The phrase goes something like ‘For an additional amount for … $50,000,000′. I put the regular expressions in the order of what I think will provide me the most accurate match first since ‘or’ is a short-circuit operator. I also do this on a larger scale with the if and elsif blocks.

As an example, I will explain the first regular expression at line 49. (I am going to explain the meta characters, not all the backtracking and how the Perl regular expression engine handles this, I would recommend “Pro Perl Parsing” or that you search the Perl Journal to learn about that). ‘For’ simply matches the word ‘For’ with any case variation (for example for or fOr), the /i modifier at the end makes the whole expression case insensitive. For ‘.*?’. the period means ‘any character’, the asterisk means match ‘any character’ zero or more times. The question mark makes the asterisk non-greedy. Since zero or more of any character would also ‘consume’ the word ‘additional’, making the asterisk non-greedy makes it so it will stop consuming characters if it finds the word ‘additional’. With \`\`(.*?)\’\’ I am capturing what appears between the ‘funny quotes’ I mentioned previously, this is the ascii interpretation of these funny quotes. The parenthesis around (.*?) capture what is between the funny quotes, which is what the money following is for as far as my parser is concerned. (\$[0-9,]*) captures any sort of dollar amount by looking for any combination of number and commas after a dollar sign. Lastly, the /g makes it so this will work if the pattern happens multiple times on the same line. I then print out the captured information with the page number.

Starting at line 60 I made it so it can read ahead a few lines in case my regular expression is interrupted by a new line. I did this by keeping track of which line I am at (which is the same as the index of the array), and then having a nested loop which reads ahead a few lines but without interrupting the flow of the main parsing loop.

The last sections, starting at line 84, I used to print out more basic matches and to look at them so I could see what dollar amounts were not captured, and use that information to improve my parser.

You can get the output of the script here. It has lines to show to start of each section of the bill, and then lines for the amount of money, what the parser thinks it is for, and the page of the bill that the line refers to. The page is important, because the script doesn’t understand that the following amount might not be the total amount of money, and might get confused elsewhere as well.

My Next Steps:
The next steps I would like to take are to start looking at the Lingua modules and look into incorporating natural language processing. It also might be helpful if I capture the html versions from THOMAS, as this will allow me to already have the sections parsable with HTML::TreeBuilder.

Conclusion:
I think congress should develop, and actually start using, XML markup for their bills. This will allow people to develop proper parsers that could retrieve the information, and display it visual formats so people could have a better handle on the where the money is going. Our country now has a CTO, Vivek Kundra, and I think he should lead the government to provide more open and accessible information.

#!/usr/bin/perl
#===============================================================================
#
#         FILE:  parseBill.pl
#
#        USAGE:  ./parseBill.pl
#       AUTHOR:  Kyle Brandt (kb), www.kbrandt.com
#      COMPANY:  Boston, MA
#      VERSION:  1.0
#      CREATED:  03/19/2009 10:16:54 AM
#===============================================================================

use strict;
use warnings;
use Roman;

#Globals
my $printit = 1;
my $delim = "\t";
my $page = 1;
my $titleSection;
my $resolution = 1;
my $total = 0;
my %causeMoney;
my %notParsed;
my $romanRegex = '';
foreach my $number (1..20) {
	$romanRegex .= Roman($number);
	unless ($number == 20) {
		$romanRegex .= '|';
	}
}

my @Bill = <>;
my $index = 0;
foreach (@Bill) {
	#Get Page Number
	if (/H\. R\. 1.*?([0-9]{1,3})/) {
		#print $1, "\n";
		$page = $1;
	}
	if ( m/TITLE ($romanRegex)-[A-Z ]*/) {
		$titleSection = $&;
		print $titleSection, $delim, $page, "\n" if $printit;
	}
	#For additional is a common phrase, this gets the dollar amount after it and what it is for
	my @amounts;
	if (
	   ( @amounts = /For.*?additional.*?\`\`(.*?)\'\'.*?(\$[0-9,]*)/gi) or
	   ( @amounts = /For.*?additional.*?for(.*?)(\$[0-9,]*)/gi) or
	   ( @amounts = /For an amount for \`\`(.*?)\'\'.*?(\$[0-9,]*)/gi) or
	   ( @amounts = /For necessary expenses for(.*?)(\$[0-9,]*)/gi)
	   ) {
		my $whatfor;
		my $amount;
		while (@amounts) {
			$whatfor = shift @amounts;
			$amount = shift @amounts;
            $amount =~ tr/,$//d;
            print $amount, $delim, $whatfor, $delim, $page, "\n" if $printit;
            $causeMoney{$whatfor . ':' . $page} = $amount;
            $total += $amount;
    	}
	}
	#Maybe if we read ahead a few lines, we will find what we are looking for
	elsif ( @amounts = /For.*?additional.*?\`\`(.*?)\'\'/) {
		AMOUNT:
		while (@amounts) {
			my $whatfor = shift @amounts;
			if (length($whatfor) > 40) {
				next AMOUNT;
			}
				if ( $index < ($#Bill - 6 )) {
					for my $line (($index + 1) ... ($index + 6)) {
					if (my $amount = $Bill[$line] =~ /\$[0-9,]*/) {
            			$amount =~ tr/,$//d;
            			print $amount, $delim, $whatfor, $delim, $page, "\n" if $printit;
						$causeMoney{$whatfor . ':' . $page} = $&;
					}
				}
			}
		}
	}
	#Like above, but can't figureout what it is for
	elsif ( my @unknownAmounts = /For.*?additional.*?(\$[0-9,]*)/gi) {
		for my $unknownAmount (@unknownAmounts) {
			$unknownAmount =~ tr/,$//d;
			$causeMoney{'UnknownAtPage' . $page} = $unknownAmount;
			$total += $unknownAmount;
		}
	}
	#All money, that doesn't fit into the above, could be a portion of what is above.
	elsif ( my @dontKnow = /\$[0-9,]*/gi ) {
		for my $money ( @dontKnow  ) {
			$money =~ tr/,$//d;
			if ( $money >= 1000000 ) {
			$notParsed{$money} = $page;
			}
		}
	}
	$index += 1;
}