Linux & Things

Resize a Lot of Pictures

2012-01-28T18:59:00.001-05:00

The church web site I run has a box the displays random pictures from our photo album. The formatting of the page is easiest if these pictures are no wider than 530 pixels and no higher than 340 pixels. Of course most pictures are taken with a camera these days are a couple of thousand pixels wide, so each picture needs to be resized. You can do that by hand, of course, but I found it easiest to write a script. If you're running Linux, this uses the ImageMagick package, particularly the identify and convert programs. On a Mac, it uses sips, which has been available at least since 10.3. I suppose it could be modified for Windows or run under Cygwin, but I don't use Windows enough to make it worth my time.

Save the file as websize, make it executable, and save it to a directory in your path. Then run it with the command

websize [list of image files]

This will take a file named, say, picture1.jpg, and create a new file named picture1_web.jpg that fits into the box defined by MAXWIDTH and MAXHEIGHT. If the picture already fits into the box it will duplicate the file.

#! /bin/bash

# Resizes a picture or pictures to make sure it fits into the UPB
# frontpage box of MAXWIDTH pixels wide by MAXHEIGHT pixels tall,
# keeping the same format (PNG, JPEG, BMP, etc.)

# usage

# websize [pictures]

# If a picture is named "picturename.ext", the shrunken picture
#  will be named "picturename_web.ext".

# If a picture already fits into the website, it will simply be
#  copied with its new name.  This makes it easier to find the
#  pictures that should be posted on the web.

# Define the maximum dimensions of the picture

let MAXWIDTH=530
let MAXHEIGHT=340

# For Linux machines we need "convert" from the ImageMagick package.
#  We'll assume it's in the standard location:

# For Macs the standard program is sips, at least since 10.3

if [ -f /usr/bin/sips ]
then
    OS="Mac"
elif [ -f /usr/bin/convert ]
then
    OS="Linux"
else
    echo "Cannot find a program to do the file conversion"
    exit 1
fi

for PICTURE
do

if [ "$OS" = "Linux" ]
then

#   identify is also part of the ImageMagick package

    let WIDTH=`identify $PICTURE | sed "s/x/ /" | awk '{print $3}'`
else
    let WIDTH=`sips -g pixelWidth $PICTURE | tail -1 | awk '{print $2}'`
fi

# Need new picture name:

HEADER=${PICTURE%.*}
EXT=${PICTURE##*.}
NEWPICT=${HEADER}_web.$EXT

echo Converting $PICTURE to $NEWPICT

# Is the picture too wide?

if (( $WIDTH > $MAXWIDTH ))
then
    if [ "$OS" = "Linux" ]
    then
 convert -resize ${MAXWIDTH}x $PICTURE $NEWPICT
    else
 sips --resampleWidth $MAXWIDTH $PICTURE --out $NEWPICT
    fi
else
    cp $PICTURE $NEWPICT
fi

# Note that the new picture may still be too large:

if [ "$OS" = "Linux" ]
then
    let HEIGHT=`identify $NEWPICT | sed "s/x/ /" | awk '{print $4}'`
else
    let HEIGHT=`sips -g pixelHeight $NEWPICT | tail -1 | awk '{print $2}'`
fi

if (( $HEIGHT > $MAXHEIGHT ))
then
#   echo Resizing height of $PICTURE

    if [ "$OS" = "Linux" ]
    then
 convert -resize x$MAXHEIGHT $NEWPICT 1_${NEWPICT}
    else
 sips --resampleHeight $MAXHEIGHT $NEWPICT --out 1_$NEWPICT
    fi

    mv 1_${NEWPICT} $NEWPICT

fi

done

The obvious modification is to allow MAXWIDTH and MAXHEIGHT to be read from the command line, maybe using -w and -h flags. That's for another day.

Joining a Group

2012-01-07T15:54:00.000-05:00

In an effort to make Ubuntu less scary to users (i.e., more Mac/Windows like), Ubuntu 11.10 removed some graphical system administration utilities that were previously available. One of those was a called Users and Groups, which let you see groups and make changes to group memberships.

The new Ubuntu would rather you just forget groups, so it includes no graphical tools for handling them. Fortunately it's still Linux, so you can fix it up yourself. Liberian Geek has details, but the first hint (using groupmod) didn't work for me, so here's my modified version. In this example, I want to add myself to the group vboxusers, which should allow me to get USB access for my Virtualbox installation. More on that some other time. For now, return with us to those thrilling days of yesteryear, before the GUI:

Look at the file /etc/group. Specifically, we want to see who belongs to the group vboxusers:
```
$ grep vboxusers /etc/group
vboxusers:x:125:
```
Nobody, as we expected.
Now to follow the Geek and add myself:
```
sudo usermod -a -G vboxusers rchawk
```

Finally, confirm the addition

$ grep vboxusers /etc/group
vboxusers:x:125:rcjhawk

You'll need to log out and log back in before the system will recognize the new group setting.

Installation Notes and Rants for Ubuntu 11.10

2011-12-29T18:05:00.008-05:00

A few weeks ago I updated Hal to Ubuntu 11.10 (Oneiric Ocelot). This was relatively easy because Hal was already at 11.04, allowing me to do a distribution update from the update manager. Our other Linux box, harlie, was running Ubuntu 10.10, so that option wasn't available unless I updated first to 11.04 and then to 11.10. Instead, I downloaded 11.10 to a CD, and did an install more or less for scratch.

Now, mind you, the Ocelot has what are supposedly a lot of nifty features, including the not-quite-Mac-like-enough-to-be-sued (you might want to ask Google about that) Unity desktop interface. And, if you don't like Unity, you can get Gnome 3! I didn't really want any of that. What I wanted was a machine that had all the modern software but looked and acted exactly like the old Ubuntu. I didn't quite get that, but I came pretty close.

BUT: This release of Ubuntu sets computing back at least 10 years. In the effort to make everything super friendly, they have dumbed down the user interface so much that you MUST use the command line to make even the simplest changes: For example, up through 11.04 you could easily change theme colors through a GUI widget. This is an appropriate use of the GUI, you can see what the current colors are, select the colors you want, and if they aren't right you can fix them right then and there. But now, guess what? THE ONLY WAY TO CHANGE THEME COLORS IS TO EDIT A FRAKIN' TEXT FILE. AS THE SUPERUSER! (I only shout because I'm mad.) This is (pick one):

Dumb
Dumber
So stupidly idiotic that one expects that there must be a malevolent being doing this, someone who wants everyone to throw up their hands in exasperation and switch to OS X. Or worse, Windows 8.

And this kind of idiocy exists all over the place: Things that were easy to do before are now hidden. Usually you have to edit a text file that hasn't appreciably changed since 1999. OK, I can do that, but until last month I didn't have to. Hence my frustration.

The worst of it is, this isn't just Ubuntu, a lot of the changes are driven by Gnome 3. It's enough to drive one back to the ugliness that is FVWM, where at least you know you have to edit menus going in.

But not just yet. Click below to see how I overcame the obstacles, at least so far.

I didn't do this all alone, there is a world of information out there on web. I've linked to specific bits of information as they occur, but you might also want to take a look at some of the big overviews:

BACK UP EVERYTHING IN /home, AT LEAST
Run $ cat /etc/passwd
Note the output associated with individual user names, particularly the user number. You'll need this later.
Especially note who is user 1000, as that should be the initial user for this system.
Get the Ubuntu CD We're using the 32-bit version for Harlie, as he's a little older than Hal. Burn CD.
Boot up with your CD.
Try out the Live CD first. This is a good time to see if your wireless connection will work out of the box. I found that my Broadcom wireless card wouldn't — not surprising, it never had before. So I pulled out an old USB wireless dongle, which worked perfectly, again as before. We'll get the Broadcom working later. You'll have to log into your wireless network, of course, click on the standard fan-shaped icon at the top right of your screen to set it up. If you don't have a working wireless connection, you'll have to dig out an Ethernet cable. Hope it's long enough.
Select Start Installation. It will ask if you want to download updates during the install, and if you want the Fluendo MP3 plugin. Say yes to the first, your choice on the second.
Under Installation Type: Harlie was running 10.10, with separate partitions for the system (/) and our data (/home). Ubuntu was smart enough to figure this out and offered to install 11.10 in the place of 10.10, and not to touch /home. This is the one major advance, IMHO, over previous installations. The alternative way to do this at this point is to choose the Something else option and map the appropriate partitions yourself. In either case, remember to back up any files you want to keep before proceeding. If this is the first time you are installing Ubuntu on this machine, you can let Ubuntu install everything in one big partition, or select Something else and set up the partitions the way you want them.
Worrisome: the install didn't recognize my swap partition, which was already set up. It did let me go fix the partition table to tell it where the swap partition was. Looking at the partition table it seems it has mapped everything else correctly, we'll see.
Set time zone appropriately. Choices seem to be New York; New York, New York; NY,NY; and NYC. I can only assume you get a different choice if you live in, say, Seattle.
English (US) keyboard. That's QWERTY, of course.
Set machine name, initial account. Use the account name associated with "1000" if you want to set the machine up like it was before.
The everybody here is converting from Windows fallacy is revealed. The install program asks it you want to import all the accounts from the Windows installation on this computer. It cares not about the already set up Linux accounts, which I will apparently have to rescue myself.
Installation commences. Wait until it finishes. Have a coffee, read a blog. You do have another computer handy, right?
Reboot.
Select Ubuntu 2D as window manager and log in. If you try straight Ubuntu, indicating 3D, you may get a your graphics card isn't up to snuff error.
Pause to admire the wonderfulness of the Unity interface. Try not to barf.
Something will start to wiggle on the Dock. It's called the update manager. Click it and do it. Restart if (when) required.
Click the top box on the Dash, and type terminal into the search-bar. Drag the terminal icon onto the dock. You'll thank yourself later.
Open a terminal. Type
$ sudo apt-get install synaptic
(If you prefer Ubuntu Software Center, ignore this step. But you don't, right?)
From the same terminal, run
$ sudo synaptic
Install a bunch of stuff. You can do this from the command line, or with synaptic. In order of importance (to me):
- gnome — an almost reasonable window manager
- gnome-tweak-tool — Because it isn't all that reasonable
- tcsh — My favorite shell. I use bash is for scripts, but I learned Unix with csh.
- build-essential — because you want to be able to install your own programs
- emacs and emacs-goodies-el — because you want to use the One True Editor to write your programs
- gfortran — You mean there's another programming language?
- chromium-browser — if you prefer Chrome as your browser.
- ubuntu-restricted-extras — because a computer isn't just for programming
- vlc — Watch just about anything, in any format.
- texlive and texlive-bibtex-extra — Make your musings look professional.
- backintime-gnome (or backintime-kde, if that's how you roll) — because at some time you will delete a file you really wanted to save
- k3b — You're going to want to burn CDs and DVDs. This program I understand.
Click Apply.
At some point a box may open up saying Configuring gdm. Set the default display manager to gdm or lightdm, your choice. Wanting to keep everything the same as before, I chose gdm. You'll also have to accept the MS-Fonts license, if you want the Fonts.
Wait until Synaptic (or apt-get, if you're on the the command line) is done, then close synaptic (if open).
Set up backups for your machine. I use a permanently mounted USB disk and the aforementioned backintime software. I also have extensive setup notes.
If you want this machine to talk to other boxes on your network, install openssh-client and openssh-server. You will also have to set up /etc/hosts.allow and /etc/hosts.deny to secure your connection. I wrote up some notes about that. You can also install sshfs for a quick-and-dirty, but secure, way of mounting a remote computer's disk on your machine.
If you want to watch DVDs, head over to 15 Things I Did After Installing New Ubuntu 11.10 Oneiric Ocelot and look for Enable Full DVD Playback(Dual Layer DVD Support). Remember, you didn't hear it here.
Click on System Settings in the Dock, then click on User Accounts. Set up other accounts now. If you mounted your old /home, you can match users to the appropriate home directories. You used to be able to do this right from this menu. Now you can't. Ubuntu will now cheerfully assign the first user account number 1000, the second 1001, etc. The only way to get around this is to give yourself superuser privileges and edit the /etc/passwd file. If you don't know how to do this, your probably don't want to do it.
Log out, or restart if you're told it's necessary.
Log back in, but before you do, click on the box that says Ubuntu 2D and change it to GNOME Classic (No effects). Note that Classic is something of a misnomer.
You'll now have something that approximates the Gnome 2 desktop. In order to make it more like Gnome, under Applications, click on Other, and then Advanced Settings. (I have another rant ready about the moronic organization of the menus. Maybe later.) This is the gnome-tweak-tool by another name. Under the Desktop listing in the tool, I set Have file manager handle the desktop and Show mounted volumes on the desktop to On, and everything else Off. YMMV. Under Theme, see if you can find something you can live with.
Gnome Classic still comes with a top and bottom panel. In Real Gnome 2 you could manipulate the panel by right-clicking on it. In Faux 2 you have to press Alt while you right click. I mean, what would happen if a naive Windows user accidentally right clicked on the top panel and had to make (gasp) a choice? And, yes, you really do have to reconstruct your panels.
Now let's get the real wireless card working. This computer has a Broadcom BCM4318 wireless card dating from about 2009. You can read more about installing Broadcom drivers at WifiDocsDriverbcm43xx. For me, it was easy. All I had to do was run
```
$ sudo apt-get install b43-fwcutter firmware-b43-installer
```
and the card automagically connected. At this point I disconnected the USB wireless dongle and everything works file. (This is another substantial improvement over 10.10, the last time I had to figure out how to get the wireless card to connect.)
Set up the printer. We have a Canon printer connected to a USB port on Harlie. When I turned on the printer Ubuntu started the setup, identified the printer as an iP4000, and then refused to work. I had to go to Applications/System Settings/Printers, uninstall the printer, and then add it back. Why? Who knows. No, he probably doesn't either.
Now to set up the printer to be shared on the network. Easy? It used to be, but it's not anymore. See Ubuntu 11.10 Shared Network. It wasn't obvious even then, but here's what you do:
- Open a browser and go to http://localhost:631
- Click the Administration tab.
- Click Share printers connected to this system
- Click Change Settingsxc
Now go back to the remote computer. Again browse to http://localhost:631. Click on Administration, then on Add Printer. Your network printer should show up under Discovered Network Printers. Click Continue and proceed with the install like it's 1999.
Add in the other users to the computer, if any, remembering to give them the IDs found in step 2. As noted, this is difficult to do. If you previously had users Alice (1000), Bob (1001), and Clarence (1002) as users, then you'll have to reinstall them in that order or you may find Bob has access to Clarence's files and vice versa. See the above entry about editing /etc/passwd. You may also want to set up backintime for each of these accounts.

That's it for now. You'll note we have nearly 40 bullets to go through, many with sub-bullets to go along. This, my friends, is the state of Ubuntu today.

Random Wallpaper Switcher for GNOME 3

2011-12-03T22:54:00.001-05:00

Some things bore me — KU losing in March every year, the Yankees winning more than one pennant a decade, Republican Presidential Debates, etc..

One of my major points of boredom is a static background on my computer screen. Having the same picture up constantly just irks.

Apple solved this long ago, with Mac OS X you can pick a directory and have the pictures selected randomly from that directory. Linux has been a little slow to pick this up. A random wallpaper selector has never shipped with any window manager I'm aware of, leaving the field to third-parties and home-grown hacking.

Way back in the day I wrote setbg, a clunky Perl script which randomly changed the X11 background. That worked for window mangers such as FVWM, but it failed for GNOME, which has its own wallpaper protocol.

For GNOME 2 I used Scott Balneaves's Gnome background switcher, aka background.py, a Python script which let you specify a directory and a switching time. Every X seconds you got a new picture for wallpaper.

GNOME 3 does wallpaper a different way (Surprise!). Fortunately I stumbled on a one-liner which changes the background to a random picture. If you want to do it that way, you just enter

gsettings set org.gnome.desktop.background picture-uri file://$(find DIR -type f | shuf -n1`)

where DIR is the path to your picture directory.

OK, I can work with that. I decided to write a script, based on the one-liner, which combined the best features of setbg and background.py. What I want is a script that:

Displays a random picture as wallpaper (see the gsettings one-liner, above).
Changes pictures at random times. Say someplace between every 10 minutes and every 20 minutes, but not always every 10 minutes and not always every 20 minutes. setbg did this, but background.py did not. If I knew more Python I would have fixed that, but I never had time.
Can be loaded as a Startup Application (now conveniently located in Applications/Other on your Ubuntu menu).
Works on any Linux computer running GNOME 3. Basically requires a common scripting language. setbg used Perl, background.py Python. For kicks, I decided to do this one in bash.
Only selects pictures to display. This was a little tricky. I don't know of a native GNOME command that says this is a picture or that is not a picture. I cheated on the previous requirement a bit and used the identify command from the ImageMagick package. No defense, except that it's available in every Linux distribution I know of.
Shuts off when you log off, so that there aren't multiple copies running if you log back in. background.py did this. setbg relied on an FVWM command to explicitly kill it. I fudge this one a bit. Basically, my script checks every minute or so to see if its parent (usually gnome-session) is still running. If it's not, the script dies.

As I said, I wrote the script in bash. It's not really a long script, but it comes in at 173 lines because I commented everything. You can download my Gnome 3 Random Wallpaper Switcher script from my software page. It's not licensed, since it's all based on someone else's one-liner. But if you make changes to the script, or find a bug, please let me know about it.

To run the script, put it in your path and type

gnome3_random_wallpaper T1 T2 DIR

Where T1 and T2 are times in seconds, and you want the wallpaper to change at some time T where T1 < T < T2. DIR is the full path to a directory that contains pictures. If it contains some other files, that's OK, the script will skip over them. You and put the command in your startup applications.

I've only tried this with Ubuntu 11.10, as that's the only GNOME 3 desktop I have at the moment, but I don't see why it wouldn't work elsewhere.

Download the official RCJHawk Gnome 3 Random Wallpaper Switcher

I For One, Welcome Our New Canonical Overlords

2011-11-25T19:07:00.001-05:00

I've just updated Hal to Ubuntu 11.10, Oneiric Ocelot.

I don't regret it — quite. It is however, going to take a long time to get it to where I want it to be, e.g., something that looks like my old Gnome 2 desktop.

Why not just change to a distribution that still uses Gnome 2? Because Gnome 2 is going away. openSUSE 12.1 is out now, and it uses Gnome 3. Fedora uses Gnome 3. You see where this is going.

So this post, and probably many more, will be rough notes on how to make Hal's desktop look like it did before. I'm not going to go cursing Mark Shuttleworth. (I've learned from the last ten years of cursing Dan Snyder — it just doesn't work.) Ubuntu is, for now, a useable distribution, and I can, eventually, tweak everything to get to where I want to go. That's the beauty of Linux.

Besides, it makes for a lot more blog posts.

Hal, I should point out, is just fine. He can still do fancy LAPW electronic structure calculations faster than the Cray XMP of my youth, and he still reads my email and browses the web with aplomb. He can still develop code with Emacs. He just doesn't look very good right now, but we're dealing with that.

Eventually I'm going to update the spouse's computer, Harlie. When I do that I hope to do one of those step-by-step guides, which will be more coherent. But for now, think of this as a set of Google Notebook entries. Oh, wait, Google pulled the plug on that and is now moving all my files to Google Docs.

Oh, well, let's just get on with it.

Oneiric comes with the brand-spanking new Unity desktop, which, as I said once before, is so like Mac OS X that it comes with a lawyer to deflect lawsuits from Cupertino.

I was wrong — it's worse than Mac OS.

Like OS X, Oneiric comes with a dock, or maybe it's called a Launcher. On the Mac the dock can be moved, it can change size, it can automatically hide if you don't want it on the screen, and it's quite easy to drag applications over to it. In Oneiric the dock is fixed on the left, you can't change its size, it only hides itself when a window pushes it away. You can drag applications onto the dock, but it seems harder, somehow, than on the Mac — that's my subjective point of view, of course.

So I quickly changed over to Gnome 3.2. Well, not quickly, because Gnome isn't installed in Oneiric. And since my 11.04 distribution used Gnome 2, the upgrade process ripped out all of that software as well. So one must install all things Gnome. I used to do this with Synaptic, but, guess what, Synaptic isn't installed either.

So first open up a terminal window. Can't find a terminal? Click on the Dash Home icon at the top of the dock, click More Apps, click to show all of the installed apps, scroll down and find the terminal, and click it. (First you might want to drag the icon over to the dock, for later use.)

With the terminal finally open, you can install Synaptic by typing

sudo apt-get install synaptic

You could then find synaptic in the Dash Home menu, but it's easier to just say:

sudo /usr/sbin/synaptic

and go from there.

Once in synaptic, search for Gnome and install it. You'll probably want to install some Gnome themes, etc. While you're there, also install the Gnome Tweak Tool as well.

Then log out. When the login screen appears, click on the little gear by your name and select Gnome Classic (no effects) or whatever other version of Gnome you desire.

It's OK to throw up your hands and go off to figure out Unity at this point. Really.

Because when you log in, you'll see something that looks only vaguely like your old Gnome desktop, sort of like the difference between Darrel Hammond and Bill Clinton. There's a menu on the top, but it only says Applications and Places. And the clock seems to be stuck to the middle of the upper panel, neither movable nor removable. Also the Theme will be hideous. You can fix that, sort of. Click Applications/Other/Advanced Settings, (that's the Gnome Tweak Tool), click Theme in the box that pops up, and play with the Window Theme and the GTK+ Theme until you get something you like. One of my missions is to figure out how to edit the theme colors, to create what used to be called a Custom Theme. No luck, so far, but if I find it I'll let you know.

You should have two toolbars, one at the top and one at the bottom of your window. You add things to these bars, change their properties, etc., as before, but instead of just right-clicking on the bar, you have to hold down the Alt key and right-click. That's easy enough.

You'll also have to search for some of your options. The old Gnome menu used to have a Preference section and an Administrative section. Those functions are now hidden in the Applications menu, under Other and System Settings, apparently with neither rhyme nor reason. Play around with that until you're comfortable in finding things.

Finally, for now, go to Things to Tweak After Installing Oneiric Ocelot 11.10, and see what's useful there. And don't forget 10 things to do after installing Ubuntu 11.10.

More as I find more stuff.

Recording with PulseAudio

2011-09-10T11:47:00.000-04:00

Sometimes I find that I want to record the sound coming out of Hal's speakers. Say that my automatic recording of Prairie Home Companion goes south because an earthquake/hurricane/remnants of a tropical storm knocked out my power. It would be nice to be able to go the Prairie Home web site, click on the archive for the show, play the streaming download and record it.

I've done this kind of thing for years, mostly without the earthquakes. Occasionally there have been problems, but it's always possible to find a solution. After which I write it down so that I'll remember how to fix things.

Except sometimes my notes aren't helpful, because somebody decided it was a good idea to change things. Take a look at the above link. To be able to record sound, I'm suppose to open Gnome's Sound preferences, click on Recording, and click on Capture. Let's look at gnome-volume-control as it's shipped with Ubuntu 11.04:

Looks nothing like what was described above, does it?

So what happened? Beats me. All I know is that someone changed things, and that the changes have something to do with PulseAudio, Ubuntu's choice for sound.

Fortunately, there is a Ubuntu Wiki article that solved things for me. Of course, the information you want is buried way down at the bottom of the article, so let me summarize it here, down at the bottom of this article. (Yes, I admit to being long-winded.)

First, you need to set up the PulseAudio volume control. For reasons known only to Mark Shuttleworth, this isn't installed by default in 11.04, and goes by the perfectly obvious name of pavucontrol. So install it first,

sudo apt-get install pavucontrol

and open it, either by typing pavucontrol from the command line or clicking "PulseAudio Volume Control" under Applications=>Sound & Video. You'll get something like this:

Not very helpful, is it? The problem is that there is no application open trying to record sound. So let's open up Gnome Sound Recorder:

Start something playing, and press Record. PulseAudio will now look something like this:

The key to making things work is the line that says:

Sound Recorder: Record Stream from: …

You may have several options, but what you want is Monitor of Internal Audio Analog Stereo. When this is selected Gnome Sound Recorder will record what's coming out of the speakers, and will look something like this:

You can see it is recording as the brown (surprise!) bar at the bottom moves in time with the music's volume.

We'll leave the rest of this, e.g., recording directly to .ogg or .mp3, converting from .flac to .ogg, using Audacity instead of Sound Recorder, etc., as an exercise for the reader. Just remember, to record sound with Ubuntu 11.04, you need to use Pulse Audio Volume Control to tell whichever application you are using to record to monitor the internal audio.

Until someone decides it's time for a change.

Transparent PNG (or GIF) From the Command Line

2011-08-14T14:55:00.000-04:00

Given the popularity of my post about making transparent PNG images with the Gimp, I should share a recent discovery: If you know the exact color you want to make transparent, you can do it from the command line.

What you need is the ImageMagick package, picture manipulation software that's available for all major Linux distributions, Windows, and Mac OS X (through MacPorts). If none of those work for you, you can also compile the source code.

Specifically, you need the convert command line tool. The syntax is:
convert -transparent color original_picture.png picture_with_transparent_background.png
and you can covert from png to gif, or visa versa, with the same command.

An example: here's the good ol' pengjay, with a horrid green (#00FF00) background:

Since we know the background color is #00FF00, we can pipe it through convert:

convert -transparent #00FF00 pengjay_green.png pengjay_transparent.png

and we get this:

Note that if you don't know the color, you're better off using the GIMP.

Can I Go Home Now?

2011-08-06T15:33:00.000-04:00

We've all been there. The home team drops seven runs in an early inning. By the seventh inning stretch they've shown no sign of an offense, and are still six runs down. It's hot, muggy, Washington night, there are thunderstorms brewing over the horizon, and your wife just phoned that she heard on WTOP that Rt. 50 to Annapolis was closing for repair work at 10 p.m.

Question: If I go home now, am I likely to miss anything? Aside from heat stroke and road rage?

Answer: Probably not. If the visiting team is ahead by six runs after the top of the seventh, historically the chance that the home team will pull out a victory is 1.6%, or about 60 to 1 against. Unless you're of the extreme optimist persuasion I'd suggest going home.

I got to thinking about this a few years ago, when Bill James published an article in Slate on when a college basketball game is really over. Based on his observations, he was able to come up with an algorithm which predicts when a team has a safe lead, based on the lead, time left in the game, and who has the ball.

In baseball we can do the same kind of thing, except that there are only a finite number of logical stopping points (the end of a half-inning) and leads (the largest of which was less than 30 runs). Plus, we have a line score for just about every major league baseball game played since 1900, so we have a lot of data. This means that we don't need no stinkin' algorithm, we can give you the history probability that any given lead was overcome.

I didn't go all the way back to 1900. I stopped at 1948, because that was the data that was available using Retrosheet's Play-by-Play Files. With the Chadwick Software Tools we can go through all the games in the database and see how many times that, say, the visiting team was ahead by five runs after then end of the first, and how often the home won in that situation. Do that for all possible combinations of leads and innings and we get the table below. (You may have to widen your browser window to see everything.)

	Visitors Lead										Tie	Home Lead
Inn.	10+	9	8	7	6	5	4	3	2	1	0	1	2	3	4	5	6	7	8	9	10+
T 1	.077	.000	.040	.041	.083	.157	.187	.304	.378	.486	.591
B 1	.000	.000	.063	.056	.055	.132	.148	.236	.313	.416	.533	.643	.740	.819	.893	.912	.933	.960	1.00	.944	1.00
T 2	.000	.027	.067	.057	.080	.169	.180	.262	.345	.452	.580	.692	.781	.846	.925	.929	.969	.974	1.00	.941	1.00
B 2	.000	.016	.043	.032	.055	.134	.127	.218	.295	.397	.532	.656	.756	.829	.895	.920	.956	.966	.957	.976	1.00
T 3	.034	.032	.019	.043	.075	.133	.166	.250	.339	.456	.582	.705	.801	.863	.927	.939	.972	.984	.980	1.00	1.00
B 3	.015	.024	.016	.031	.047	.095	.122	.200	.278	.394	.526	.655	.766	.840	.909	.930	.958	.977	.986	.989	.990
T 4	.012	.021	.028	.033	.071	.096	.145	.225	.314	.446	.589	.719	.820	.886	.933	.963	.971	.988	.990	.994	.988
B 4	.004	.011	.022	.028	.039	.062	.094	.164	.252	.369	.526	.677	.788	.866	.917	.949	.969	.986	.990	.992	.996
T 5	.004	.009	.020	.031	.045	.078	.114	.199	.290	.422	.591	.741	.843	.904	.945	.967	.982	.995	.995	.991	1.00
B 5	.002	.005	.009	.019	.026	.049	.077	.139	.224	.344	.523	.692	.808	.887	.936	.962	.981	.989	.995	.993	.999
T 6	.001	.005	.011	.019	.031	.055	.091	.161	.268	.406	.600	.771	.873	.931	.964	.975	.994	.995	.999	.998	.998
B 6	.001	.000	.005	.009	.014	.030	.049	.097	.185	.305	.520	.725	.848	.917	.956	.972	.989	.994	.998	.999	.998
T 7	.001	.001	.004	.010	.016	.038	.061	.118	.217	.357	.610	.819	.912	.958	.977	.991	.995	.998	.999	1.00	.999
B 7	.000	.002	.002	.002	.007	.015	.033	.059	.130	.245	.523	.772	.894	.949	.973	.990	.995	.998	.999	.999	.999
T 8	.000	.001	.003	.004	.007	.017	.041	.074	.153	.296	.634	.890	.956	.985	.992	.997	.999	1.00	.999	1.00	1.00
B 8	.000	.000	.000	.001	.002	.006	.013	.027	.067	.150	.519	.865	.944	.981	.990	.997	.999	1.00	.999	1.00	1.00
T 9+	.000	.000	.000	.002	.003	.007	.014	.033	.071	.157	.615	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
B 9+	.000	.000	.000	.000	.000	.000	.000	.000	.000	.000	.524	1.00	1.00	1.00	1.00

So what is all of this?

The left hand column represents the situation at the end of either the top (visitor's) half of an inning, or the bottom (home team's) half, so T 7 describes the situation just before the Cubs let that day's designated karaoke singer ruin Take Me Out to the Ballgame. Since all innings after the ninth are played under the same conditions — the team ahead after the bottom of the inning wins, and if it's tied we try, try again — I combined all the data from those games into the labels T 9+ and B 9+.
The other columns represent the lead by either the home team (on the right) or the visiting team (left) at the end of the half-inning. Ties are right in the middle.
The decimal fraction in each block indicates the home team's chance of winning in that situation. So, for example, if you've just sat through an agonizing top of the third, when Yankees have scored a bunch of runs and lead by 7, we look at row T 3, column Visitors 7, and see that the Orioles have a 0.043 (4.3%) chance of winning the game — no, that's not true. Over the course of the last 60 years, 4.3% of all major league home teams down by 7 going in to the bottom of the third have come back to win. These are the Orioles, however, so they have about a 0.001% chance of coming back.
I put all leads of ten or more runs in the 10 + categories. The software I wrote to write the table is easy to modify to list bigger leads, if you like.. I stopped at 10 because that will more or less fit on a standard blog page.
The blank spaces are impossible situations. The home team can't score before it gets up to bat, so the right-hand side of the T 1 row can never be reached. And the home team doesn't need the bottom of the ninth or later inning unless it was tied or behind after the top of the inning. In that case they can never win the game by more than four runs.
Indeed, I debated about putting in the B 9+ column, since it's a trivial case, but I wanted to highlight the fact that the home team still has a big advantage if the score is tied in late or extra innings. I've discussed this elsewhere.
Finally, I haven't told you about the statistical significance of these results, i.e, the standard deviation or the sample size. Suffice it to say that there are more than enough events here for leads of nine runs or less. When we get to the 10+ category, especially in the early innings, there just aren't that many games. If you want to see the raw numbers (home wins/visitor wins/total games) drop me a line and I'll send them to you.
Really last finally: I made a slight modification to the Chadwick source code to make it easier to parse the runs scored per inning. I also wrote a Perl script and some Fortran code to parse the output from Chadwick and write the HTML for the table above. If you're interested in the codes, drop me a line. If there's enough interest I'll make all the code available on my website.

Almost Spot On

2011-07-31T13:19:00.000-04:00

I'm currently listening to The Essential Kansas, the 70's boy band (I jest) from Topeka. For some reason, I've never bought a Kansas album, and I don't plan to in the near future. Yet I'm listening completely legally, and not paying a dime.

That's because Spotify has finally made it from Europe to the U.S. I read about it in a New York Times article (I just access the free stuff there, too) the other day and decided to try it out. As far as I can tell, the deal is:

With a free-as-in-beer account, you can stream any song in Spotify's vast library (and it's vast, if somewhat uneven).
For now, you can listen to an unlimited amount of music, only interrupted by two minutes of commercials any hour.
After six months, your free account is limited to ten hours of music per month. I read that in the NYT article, good luck in finding it in Spotify's account description.
Of course, there are other plans that let you stream unlimited amounts of music, without ads, for a price.
You have to use Spotify's music player.

That last part was almost a deal breaker. (Can you break a free deal?) There is no generally available Linux client. There is an alpha version of a Linux player, but it doesn't seem to have been worked on for a year or so, and it can only be used with the paid accounts, because we haven't found a reliable way to display ads yet.

However, the Windows client runs under Wine, and Spotify gives detailed instructions on how to get it started. You can't play your local MP3 files through Spotify (not sure I'd want to), but the stuff that comes over the web sounds good — according to the NYT article, it's in 160-kbps Ogg Vorbis format.

All right, how is it? Funny you should ask.

The sound quality is good enough for my speakers-in-the-monitor setup. Beyond that I couldn't say. Audio purists will probably find some fault, but the real purists are listening to vinyl anyway. The playlist is extensive, but not complete. OK, I didn't expect the Beatles here, but the Eagles are mostly missing. You do get what seems to be complete coverage of the Rolling Stones, Simon & Garfunkel, Creedence Clearwater Revival, Bob Marley, Peter, Paul & Mary, the Mommas and the Papas, the aforementioned Kansas, Roger Miller, Glen Campbell, Johnny Cash, a limited amount of Dylan, and even Hugh Laurie reading Three Men in a Boat. They have what seems to be the complete Jimmie Rodgers the Elder, but I can't find anybody's version of Jimmie Rodgers the Younger's classic ballad It's Over. This would frustrate me if I was actually paying for this stuff, but since it's all free I really can't complain. It seems as though they're working on it.

Oh yes, one more thing: if you want to sign up for the free account, you need an invite. Apparently paid-for account holders can give you an invite, or you can go to Spotify.com and ask for one. It took about thirty seconds for me to get an email after I asked, but YMMV.

All in all, it seems to be a pretty good service, at least for now. If you put a map of the continental U.S. on a dartboard so that Kansas is the bull's eye, Spotify's dart hits at about Oklahoma City. Add the Eagles, more Dylan, and the Beatles (Ha!), and they can hit Holyrood.

When He's Right, He's Right

2011-07-17T19:00:00.000-04:00

Penguin Pete. Opinionated and passionate about it.

And boy, when he's right, he's right.

Baseball After the All Star Break

2011-07-11T22:05:00.000-04:00

Some years ago, I did a predictive study on how Major League Baseball teams would rank at the end of the season, based on their records at the All Star Break and the Pythagorean projection of future wins, based on the runs scored and allowed by each team.

It didn't work all that well. In particular, I predicted that Boston would win the AL East pennant, and Washington would be the NL Wild Card. That sorta didn't happen.

Nevertheless, I'll try again. Here's the table, based on the MLB standings at the All Star Break. The method is the same as last time, so you can read all about it there.

American League

East	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB

New York Yankees	53	35	0.602	2	1	455	334	0.637	74	47.14	26.86	100.14	61.86	0.618	0.00
Boston	55	35	0.611	1	0	482	371	0.617	72	44.42	27.58	99.42	62.58	0.614	0.73
Tampa Bay	49	41	0.544	3	6	380	343	0.546	72	39.35	32.65	88.35	73.65	0.545	11.80
Toronto	45	47	0.489	4	11	426	416	0.511	70	35.76	34.24	80.76	81.24	0.498	19.39
Baltimore	36	52	0.409	5	18	355	454	0.390	74	28.85	45.15	64.85	97.15	0.400	35.29

Central	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB

Cleveland	47	42	0.528	2	0.5	386	382	0.505	73	36.85	36.15	83.85	78.15	0.518	0.00
Detroit	49	43	0.533	1	0	413	421	0.491	70	34.39	35.61	83.39	78.61	0.515	0.46
Chicago White Sox	44	48	0.478	3	5	366	383	0.479	70	33.55	36.45	77.55	84.45	0.479	6.29
Minnesota	41	48	0.461	4	6.5	347	414	0.420	73	30.69	42.31	71.69	90.31	0.443	12.16
Kansas City	37	54	0.407	5	11.5	402	449	0.450	71	31.94	39.06	68.94	93.06	0.426	14.91

West	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB

Texas	51	41	0.554	1	0	457	404	0.556	70	38.91	31.09	89.91	72.09	0.555	0.00
Los Angeles Angels	50	42	0.543	2	1	355	330	0.533	70	37.32	32.68	87.32	74.68	0.539	2.59
Seattle	43	48	0.473	3	7.5	301	319	0.474	71	33.63	37.37	76.63	85.37	0.473	13.28
Oakland	39	53	0.424	4	12	315	339	0.467	70	32.66	37.34	71.66	90.34	0.442	18.24

National League

East	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB

Philadelphia	57	34	0.626	1	0	384	295	0.618	71	43.86	27.14	100.86	61.14	0.623	0.00
Atlanta	54	38	0.587	2	3.5	365	312	0.571	70	39.96	30.04	93.96	68.04	0.580	6.89
New York Mets	46	45	0.505	3	11	399	388	0.513	71	36.40	34.60	82.40	79.60	0.509	18.46
Washington	46	46	0.500	4	11.5	352	354	0.497	70	34.82	35.18	80.82	81.18	0.499	20.04
Florida	43	48	0.473	5	14	352	396	0.447	71	31.71	39.29	74.71	87.29	0.461	26.15

Central	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB

St. Louis	49	43	0.533	2	0	433	407	0.528	70	36.97	33.03	85.97	76.03	0.531	0.00
Milwaukee	49	43	0.533	1	0	405	406	0.499	70	34.92	35.08	83.92	78.08	0.518	2.05
Pittsburgh	47	43	0.522	3	1	354	346	0.510	72	36.75	35.25	83.75	78.25	0.517	2.22
Cincinnati	45	47	0.489	4	4	437	408	0.531	70	37.18	32.82	82.18	79.82	0.507	3.79
Chicago Cubs	37	55	0.402	5	12	375	459	0.409	70	28.63	41.37	65.63	96.37	0.405	20.34
Houston	30	62	0.326	6	19	358	464	0.384	70	26.89	43.11	56.89	105.11	0.351	29.08

West	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB

San Francisco	52	40	0.565	1	0	332	322	0.514	70	35.97	34.03	87.97	74.03	0.543	0.00
Arizona	49	43	0.533	2	3	416	407	0.510	70	35.70	34.30	84.70	77.30	0.523	3.28
Colorado	43	48	0.473	3	8.5	395	407	0.486	71	34.53	36.47	77.53	84.47	0.479	10.44
Los Angeles Dodgers	41	51	0.446	4	11	340	373	0.458	70	32.06	37.94	73.06	88.94	0.451	14.92
San Diego	40	52	0.435	5	12	304	338	0.452	70	31.63	38.37	71.63	90.37	0.442	16.34

Abbreviations:

W: Current team wins
L: Current team loses
PCT: Winning rate
Place: Current place in standings
GB: Games Behind
RS: Total Runs scored by team
RA: Total Runs allowed by team
Pyth: Pythagorean expected win rate. Following MLB, I used an exponent of 1.82 rather than the original James value of 2. It doesn't make a lot of difference, and didn't change the order.
GL: Games left in season for the team
PW: Projected wins in remainder of season, assuming they win at the Pythagorean rate
PL: Projected Pythagorean loses
TW: Total wins, current + projected Pythagorean
TL: Total loses
PCT: Projected final winning ratio
GB: Projected final games behind

OK, not a lot of changes going on. Despite an anemic offense, San Francisco's fantastic pitching will keep them in first in the NL West. Philadelphia will win the NL East going away, even though Atlanta wins the NL Wild Card. Texas will hang on in the AL West.

There are a few predicted swaps, highlighted in yellow: St. Louis will pull ahead of Milwaukee. And Cleveland will (yawn) edge out Detroit. Surprisingly, the only changes occur at the top, which probably says something about competitive balance in MLB.

And, finally, Red Sox will be the AL Wild Card. Which means …

Frak

The Home Team Wins Most Ties (In Baseball, Anyway)

2011-07-02T21:37:00.000-04:00

I've been playing around with Retrosheet to see how often a baseball team wins a game if it's, say, five runs ahead at the end of the fourth inning. I plan to get that up sometime during this long weekend, but while doing the study I found another interesting result.

Retrosheet's Play-by-Play files, along with the cwevent program from Chadwick, let you extract all sorts of information from almost every MLB game played between 1950-2010. From that data I extracted every game that was tied at the end of a half-inning, and figured out who eventually won. Then I counted up the number of times the home team won for each half-inning. The results are shown below:

Click on graph to see a larger figure

The black diamond represents the situation at the start of the game, the red diamonds the situation where the game is tied in the middle of the inning, and the blue diamonds when it's tied at the end of an inning. We'll get to the error bars in a minute.

So what is all of this? Well, at the beginning of a game the score is obviously tied, so that should be part of the study. So if we look at all 115,748 games in the database, we find:

The Home Team won 62,418 games,
the Visiting Team won 53,192 games, and
there were 138 games that were tied when the game was called.

If we throw out the ties, then the Home Team won 53.990% of the games that went to a decision. That's the black diamond at the far left of the graph. The 54% win rate is baseball's version of Home Field Advantage, and it has been very constant:

Click on graph to see a larger figure

I performed the same calculation for all the games which were tied at the end of a half inning. For example, if the game is tied at the end of the fifth, the home team has a 52.0% chance of eventually winning the game. Tied after the top of the sixth? It's up to 60%.

So what do the error bars represent? Basically they give you an idea of the number of games in the sample. Suppose that in a given game the home team wins with probability p. Then in an N game sample the probability that the home team wins n games follows the binomial distribution, e.g.

             N!        n      N-n
P(N,n) = -----------  p  (1-p)
          n! (N-n)!

If we look at a large number of N-game samples, then we'll find that on average the home team will win N p games, which makes sense. The standard deviation will be [N p (1-p)]^½. Since the graph normalized everything by the number of games played, the error bars are the standard deviation divided by N, or [p (1-p)/N]^½. Since most values of p are between 0.5 and 0.7, wider error bars basically tell you that fewer games have gotten to that point. And when the error bars get really wide, as they do after the fourteenth inning or so, it says there aren't enough statistics available to give you meaningful information.

What does it all mean, you ask? Well, first it says that the home team advantage is real. Why there is a home field advantage is another question, and there is not enough information here to answer that question.

Then there's the observation that the home team has a larger advantage if the game is tied in the middle of the inning than it does if the game is tied at the end of an inning. That's just common sense. In the middle of the fourth inning, the visiting team has five more innings at the plate. The home team has six — five for sure, and one more if they need it. This isn't the home team bats last advantage, it's the home team gets one more at-bat than the visitors advantage, not the same thing.

Next, we see that if the game is tied at the end of an inning, the home team's advantage decreases slightly, so that at the end of eight innings it's only 51.94%. Presumably that's because the home team does have some advantage in being at home, but as the game progresses they have less and less chance to use that advantage. After the fifth inning the home team's advantage oscillates around 52%, down from the 54% advantage they had at the start of the game.

Indeed, the fact that the blue dots go down from innings 1-4 suggests that the home team bats last advantage isn't worth a whole lot. If it was, you'd expect the advantage to be greater in tie-game situations as the game wears on, because that last at-bat becomes a larger and larger proportion of what's left of the game.

Finally, there is that dip in the red diamonds between the first and second inning. If the game is tied going into the bottom of the first, meaning that the visiting team didn't score, then the home team will win 59.15% of the time, with a standard deviation of 0.17%. If the game is tied going into the bottom of the second, however, then the home team only has a 58.01% chance of winning, with σ = 0.22%. That's a five-σ change in the probability, which I would think is statistically significant. The win rate is pretty much constant in the third inning ( 58.16% ± 0.27%) and then starts going up, as you'd expect, since the home team has proportionately more at-bats than the visitors at the middle of an inning.

Why is this so? I have no idea. All I can say is that if your a visiting baseball team, it's better to be tied with the home team in the middle of the second inning than it is to be tied before the home team comes to bat. And you better be ahead by the middle of the third.

The information used here was obtained free of charge from and is copyrighted by Retrosheet. Interested parties may contact Retrosheet at www.retrosheet.org.

Near Miss

2011-06-25T12:34:00.000-04:00

You may have heard that Asteroid 2011 MD will pass by Monday morning. And by pass by, we mean within 8,000 miles.

That's pretty close. How close? Well, have a look at these animations:

Asteroid 2011 MD Flyby

An Update on the Average Major League Hitter

2011-06-12T22:26:00.001-04:00

This is the promised update to my table of pretty good averages for everyday baseball players at the major league level. As before, I consider an everyday ball player to be one who was eligible for a league batting title in a given year, which from 1996-2010 means that he made at least 502 plate appearances in a given year.

The raw data for this study was taken from the mySQL database thoughtfully provided by the folks at Baseball-DataBank.org. As outlined in yesterday's post, I used mySQL to pull out the appropriate data, this time using the command:

select b.yearID as Year, m.nameLast as Last, m.nameFirst as First,
b.teamID as TEAM, b.G, b.AB+b.BB+b.HBP+b.SH+b.SF as PA, b.AB, b.R, b.H, b.2B,
b.3B, b.HR, b.RBI, b.SB, b.CS, b.BB, b.SO, b.IBB, b.HBP, b.SH, b.SF, b.GIDP
from Batting b inner join Master m
where b.playerID=m.playerID and b.yearID>1995 and
b.AB+b.BB+b.HBP+b.SH+b.SF > 502
order by b.yearID ASC, m.nameLast, m.nameFirst;

to get a list of all players from 1996 on who were eligible for a batting title. I then used mysql-query-browser to export all of the data into a spreadsheet, and there computed all of the averages and standard deviations. All the calculations are the same as in my original post, except:

I added the batting data for the 2009 and 2010 seasons.
My original post fraked up David Smyth's Base Runs statistic. I used the right formula (the second one on the page), but miscalculated total bases by forgetting that doubles, triples, and home runs are already counted as hits. So the numbers found in my earlier study are too high.
I dropped 1995 from the study this time because only 144 games were scheduled for each team, so the batting eligibility criterion was 144 × 3.1 = 447 plate appearances. Just lazy.

Here's how it works. For, say, Batting Average in 1996, I computed the batting average for each of the 137 players eligible for batting awards that year. I then computed the average of that average, which is the entry in the BA column for 1996. I also computed the standard deviation of those averages for the year, and put that number in the σ column. I did this for each year and each statistic listed. Finally, I found the average of all the batting averages over the 2168 player-years, and put that under All Years, along with its standard deviation. So, to use another example, we can see that from 1996-2010 the average everyday major league batter hit someplace between 11 and 31 home runs in a year. Obviously there were some notable (and ignoble) players who hit more than that, and many who hit less. But in general, if you can hit 20 home runs in a year off big-league pitching, this table tells you that you've probably got a job. I hear it pays pretty well.

The first table lists the traditional stats: Batting Average, Home Runs, and Runs Batted In (RBI).

Year	Players Eligible	Games		Batting Average		Home Runs		Runs Batted In
		Avg.	σ	BA	σ	HR	σ	RBI	σ
1996	137	147.336	10.285	0.288	0.027	21.358	12.814	84.993	29.394
1997	133	147.286	9.864	0.283	0.029	19.970	11.331	80.962	26.357
1998	148	148.574	10.217	0.287	0.026	20.926	13.012	82.500	28.442
1999	152	145.875	11.647	0.290	0.026	22.020	12.613	85.401	28.495
2000	150	146.553	10.817	0.288	0.030	22.007	11.733	85.700	27.728
2001	146	147.034	10.800	0.282	0.028	21.623	13.400	81.890	29.106
2002	144	147.417	9.630	0.278	0.027	20.750	11.188	78.819	24.090
2003	146	146.610	11.739	0.281	0.027	19.863	11.056	78.863	24.173
2004	154	145.877	10.302	0.284	0.026	20.786	11.012	79.117	23.576
2005	143	147.441	10.615	0.280	0.022	19.965	10.937	78.042	23.308
2006	146	147.116	10.302	0.286	0.024	20.459	12.181	80.952	25.700
2007	156	146.686	10.568	0.283	0.028	19.128	9.921	79.763	23.370
2008	137	146.657	10.834	0.280	0.025	19.620	9.758	78.029	22.900
2009	140	145.821	11.482	0.282	0.026	19.929	10.809	78.129	22.584
2010	136	147.213	10.235	0.274	0.027	18.728	10.276	75.125	22.502
All Years	2168	146.890	10.635	0.283	0.027	20.491	11.531	80.595	25.671

The next table has the modern statistics: On Base Percentage (OBP), Slugging Average (SLG), and what's now a standard statistic for overall batting prowess, On Base Plus Slugging (OPS = OBP + SLG). Looks like if you can keep your OPS above about 0.750 won't have to worry about the kid's college tuition:

Year	Players Eligible	On Base %		Slugging		OPB + SLG
		OBP	σ	SLG	σ	OPS	σ
1996	137	0.361	0.041	0.472	0.085	0.833	0.117
1997	133	0.354	0.041	0.460	0.075	0.815	0.108
1998	148	0.355	0.036	0.467	0.081	0.822	0.108
1999	152	0.364	0.039	0.479	0.077	0.843	0.107
2000	150	0.362	0.041	0.479	0.085	0.841	0.119
2001	146	0.353	0.040	0.470	0.092	0.823	0.127
2002	144	0.351	0.044	0.461	0.080	0.812	0.118
2003	146	0.351	0.039	0.461	0.075	0.812	0.108
2004	154	0.355	0.039	0.468	0.073	0.822	0.104
2005	143	0.348	0.034	0.458	0.069	0.806	0.095
2006	146	0.355	0.035	0.469	0.073	0.824	0.100
2007	156	0.354	0.036	0.458	0.066	0.812	0.095
2008	137	0.350	0.033	0.457	0.065	0.807	0.089
2009	140	0.353	0.032	0.461	0.066	0.814	0.090
2010	136	0.343	0.032	0.444	0.070	0.787	0.095
All Years	2168	0.354	0.038	0.464	0.076	0.819	0.106

And finally, this table has two other batting ability figures of merit. Bill Jame's Runs Created, using the technical version and David Smyth's Base Runs, using the version that includes stolen base and grounded into double play data. For each of these I also included a per Game category, where a batter is considered to have played a game every time he makes twenty-four (24) outs. Outs are defined as

Outs = At Bats - Hits + Caught Stealing + Sacrifice Hits + Sacrifice Flies + Times Grounded into Double Plays

even though Smyth's formula does not include Sacrifice data.

Year	Players Eligible	Runs Created		Runs Created/Game		Base Runs		Base Runs/Game
		RC	σ	RC/G	σ	BsR	σ	BsR/G	σ
1996	137	96.600	28.370	5.617	1.738	89.402	24.627	5.195	1.494
1997	133	91.108	26.341	5.330	1.624	84.905	22.644	4.962	1.385
1998	148	94.738	26.595	5.431	1.576	88.110	23.177	5.048	1.354
1999	152	97.993	26.840	5.749	1.619	91.017	23.332	5.337	1.389
2000	150	97.624	30.106	5.740	1.905	90.290	25.426	5.302	1.596
2001	146	93.714	32.514	5.468	2.006	86.611	27.214	5.048	1.656
2002	144	90.261	27.370	5.294	1.970	84.007	23.490	4.920	1.672
2003	146	90.263	26.200	5.291	1.663	83.832	22.653	4.908	1.406
2004	154	92.899	25.243	5.440	1.805	86.339	21.569	5.050	1.521
2005	143	89.551	23.903	5.160	1.389	83.377	20.729	4.803	1.198
2006	146	93.534	24.146	5.420	1.446	86.664	20.849	5.018	1.231
2007	156	91.098	24.147	5.292	1.387	84.725	20.931	4.920	1.191
2008	137	89.927	21.584	5.197	1.315	83.891	18.794	4.846	1.133
2009	140	90.753	22.379	5.299	1.308	84.578	19.630	4.935	1.131
2010	136	85.296	21.911	4.894	1.296	79.946	19.000	4.584	1.114
All Years	2168	92.424	26.167	5.379	1.632	85.902	22.523	4.996	1.387

So there you have it. From now on, if someone says that a player has an OBP/SLG/OPS of 0.350/0.425/0.775, or some other string of statistics, you can use these tables to see how that fits in with the average (i.e. really, really good, compared to you or me) every-day Major League batter.

Take Me Out to the SQualL Game

2011-06-11T19:06:00.003-04:00

As some of you know, I have a moderate interest in Baseball and Baseball statistics. However, I'm not a database programmer, and the little bit of DB manipulation I've looked at has left me hopelessly confused.

Several years ago I did buy a copy of Baseball Hacks, but I never really got started with it. Last weekend I had some time on my hands and started playing around with it. Turns out you actually have to try to do something with a program in order to learn it (who'da thunk?). And it also turns out the Baseball Hacks, even though it's an O'Reilly book, is pretty oriented to Windows/Microsoft Access, although it does have substantial hints for Linux and Mac users, not to mention an online collection of scripts from the book (ZIP file), along with some other stuff.

So what should we do? As a first shot, how about updating my table of averages for several modern baseball statistics to include 2009 and 2010? I did the previous tables by doing some judicious editing of the Batting.csv file in Sean Lahman's baseball database, but the folks at Baseball-Databank.org have all of the data packaged neatly into a mySQL database (ZIP file), so we'll use that.

So as a start, we're going to

Install appropriate parts of the mySQL database program in Ubuntu,
Set it up to read the database file,
Find the eligible batters for 2009 and 2010,
Get their batting data into a spreadsheet,
Find the appropriate averages, runs created, etc., and
add the results to the appropriate tables.

That should be enough for one day. It's going to be a long journey, though, so when you've got some time join us after the break.

Ready? OK, let's go.

Installing mySQL under Ubuntu

and, probably, any Linux system, with the usual modifications for different distributions. For what it's worth, I set up a virtual machine which didn't previously have mySQL installed so that we can see how everything works on a clean system.

This will install everything we're going to use today, and then some:

$ sudo apt-get install mysql-server mysql-query-browser

During the installation process you'll see a screen pop up that looks like this:

This is the password to the mySQL server. As it says, you may be able to get away with no password, but it's probably a good idea to have one, especially if you share your machine with others. Given this password, you can change content and access to any database that's installed on you system. We'll make the password here gBrett.

After installation, the mysqld daemon will always be running on your system, though it may spend a lot of time sleeping:

$ ps xa | grep mysqld
25154 ? Ssl 0:00 /usr/sbin/mysqld

Probably better check to see if it works:

$ mysql -u root -p
Enter password: gBrett
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 123
Server version: 5.1.54-1ubuntu4 (Ubuntu)

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> quit

Set up the Database

Now download the Baseball-Databank mySQL database (ZIP file) and unzip into a convenient directory:

$ mkdir Documents/baseballDB
$ cd Documents/baseballDB
$ wget http://www.baseball-databank.org/files/BDB-sql-2011-03-28.sql.zip
$ unzip BDB-sql-2011-03-28.sql.zip

Now we want to set up access to the database for a user. You can use your current username for this, but you could create another user name if you wished. Here, we'll use our own user name, and we'll follow Baseball Hacks and call the database bbdatabank:

$ mysql -p -u root
Enter password: gBrett
mysql> GRANT ALL ON bbdatabank.* TO 'dave'@'localhost' IDENTIFIED BY 'cRojas';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE DATABASE bbdatabank;
Query OK, 1 row affected (0.00 sec)

mysql> quit
Bye

(Note the semicolons at the end of the line, which tells mySQL you're finished with the command. If you don't enter a semicolon, nothing much will happen until you do.)

This command allows dave to log into mySQL and play with the database called bbdatabank. Of course, there's nothing in bbdatabank as yet, but now dave can fix that:

$ mysql -u dave -p -s bbdatabank < BDB-sql-2011-03-28.sql
Enter password: cRojas

This will take a bit of time, but eventually the DB will load. To see if it's correct, do something like this:

$ mysql -u dave -p -s bbdatabank
Enter password: cRojas
mysql> show tables;
Tables_in_bbdatabank
AllstarFull
Appearances
AwardsManagers
AwardsPlayers
AwardsShareManagers
AwardsSharePlayers
Batting
BattingPost
Fielding
FieldingOF
FieldingPost
HallOfFame
Managers
ManagersHalf
Master
Pitching
PitchingPost
Salaries
Schools
SchoolsPlayers
SeriesPost
Teams
TeamsFranchises
TeamsHalf
xref_stats
mysql> quit

That lists all of the various tables in the database. One's really important, that's the one called Master, and if you look into the BDB-sql-2011-03-28.sql near the top you'll find something like this:

LOCK TABLES `Master` WRITE; /*!40000 ALTER TABLE `Master` DISABLE KEYS */; INSERT INTO `Master` VALUES (1,'aaronha01','','aaronha01h',1934,2,5,'USA','AL','Mobile',NULL,NULL,NULL,NULL,NULL,NULL,'Hank','Aaron',NULL,'Henry Louis','Hammer,Hammerin\' Hank,Bad Henry',180,72.0,'R','R','1954-04-13','1976-10-03','','aaronha01','aaronha01','aaroh101','aaronha01','aaronha01')

which tells you that somebody named Hank Aaron, born in 1934 in Mobile, Alabama, will be referred to as aaronha01 in other parts of the database. You can probably figure out most of the rest of the things here. This is the key to the entire database. In the Batting table, for example, there will be twenty-two references to aaronha01, one for each year he played in the majors.

OK, we've got all of that. What do we do with it? First, as a warm-up, let's identify the players in MLB since 2000 who came to the plate more than 740 times (just to fit the list onto one screen) in a year, and see which one of them had the best batting average. This is a modification of Baseball Hacks Hack 18, at the bottom of page 95.

$ mysql -u dave -p -s bbdatabank
Enter password: cRojas
mysql> select b.H/b.AB as AVG,
    -> b.AB+b.BB+b.HBP+b.SH+b.SF as PA,
    -> m.nameLast, m.nameFirst, b.teamID, b.yearID 
    -> from Batting b inner join Master m
    -> where b.playerID=m.playerID
    -> and
    -> b.yearID>1999
    -> and b.AB+b.BB+b.HBP+b.SH+b.SF > 735
    -> order by AVG DESC;
AVG PA nameLast nameFirst teamID yearID
0.3722 762 Suzuki Ichiro SEA 2004
0.3510 736 Suzuki Ichiro SEA 2007
0.3497 738 Suzuki Ichiro SEA 2001
0.3260 748 Pierre Juan FLO 2004
0.3223 752 Suzuki Ichiro SEA 2006
0.3140 748 Young Michael TEX 2006
0.3130 739 Young Michael TEX 2004
0.3105 749 Suzuki Ichiro SEA 2008
0.3089 752 Jeter Derek NYA 2005
0.3085 739 Utley Chase PHI 2006
0.3054 746 Pierre Juan FLO 2003
0.3034 739 Suzuki Ichiro SEA 2005
0.3003 741 Soriano Alfonso NYA 2002
0.2997 736 Furcal Rafael LAN 2006
0.2965 763 Reyes Jose NYN 2008
0.2961 778 Rollins Jimmy PHI 2007
0.2918 750 Pierre Juan CHN 2006
0.2901 751 Sizemore Grady CLE 2006
0.2805 765 Reyes Jose NYN 2007
0.2772 758 Rollins Jimmy PHI 2006
0.2771 748 Sizemore Grady CLE 2007
0.2700 739 Jeter Derek NYA 2010
0.2688 754 Weeks Rickie MIL 2010
0.2681 745 Sizemore Grady CLE 2008

(Note that I added a lot of carriage returns to input, just to make it visible. I could have run it all together:

select b.H/b.AB as AVG, b.AB+b.BB+b.HBP+b.SH+b.SF as PA, m.nameLast, m.nameFirst, b.teamID, b.yearID from Batting b inner join Master m where b.playerID=m.playerID and b.yearID>1999 and b.AB+b.BB+b.HBP+b.SH+b.SF > 735 order by AVG DESC;

I'd recommend keeping the commands you're trying in a text file, and then cutting and pasting into the database program.)

Let's analyze that command line by line and see what it does:

# Set up a variable called AVG, which measures hits (H) divided by At Bats (AB), and print it out:
select b.H/b.AB as AVG,
# Set up another variable called PA, for plate appearances (every time the player came to the plate) and print it out:
-> b.AB+b.BB+b.HBP+b.SH+b.SF as PA,
# This line print the Players name (Last name first), the team he played for, and the year:
-> m.nameLast, m.nameFirst, b.teamID, b.yearID
# This identifies the tables. A prefix of "b." means the data comes from the Batting table, while "m." comes from the Master table:
-> from Batting b inner join Master m
# This makes sure that we're looking at the same player in the Master table as in the Batting table:
-> where b.playerID=m.playerID
# But wait, there's more:
-> and
# We only want to look at years after 1999:
-> b.yearID>1999
# and we only want to see batters who came to the plate a large number of times:
-> and b.AB+b.BB+b.HBP+b.SH+b.SF > 735
# Finally, we want this ordered by average, starting from the top and going down:
-> order by AVG DESC;

Find Eligible Batters for 2009 and 2010

This is pretty easy, right? To be eligible for a batting title, Rule 10.22(a) says that you must come to the plate at least 3.1 times for every scheduled game your team plays. Since every team is scheduled for 162 games per year, that works out to 502.2 plate appearances. So let's work it out for 2009:

$ mysql -u dave -p -s bbdatabank
Enter password: gBrett
mysql> select b.AB+b.BB+b.HBP+b.SH+b.SF as PA,
    -> m.nameLast, m.nameFirst, b.teamID, b.yearID
    -> from Batting b inner join Master m
    -> where b.playerID=m.playerID and b.yearID=2009
    -> and b.AB+b.BB+b.HBP+b.SH+b.SF>502
    -> order by PA DESC;

Whoa! That will list 140 names, headed by Jimmy Rollins with 734 PA and ending with Grady Sizemore and Martin Prado, who came up 503 times each. Unless you have a really big monitor, or you like to cut and paste from your terminal buffer, you'll need a better interface.

OpenOffice.org and LibreOffice have Base components, and with sufficient work I'm told you can get them to interact with mySQL. Unfortunately, despite pursuing various tutorials I couldn't get that to work. Fortunately, mySQL has its own graphical interface, mysql-query-browser. Basically it the mySQL commands above and puts them in table form on your screen. We downloaded it as part of the mySQL installation process, so let's try it on this problem:

mysql-query-browser &

This launches a graphical interface. The first time through you'll have to fill in the boxes, after that it will remember everything except the password (which is cRojas):

Once you're logged in, click the tab that says Result Set 1 or something like that. You'll get a screen that looks like this:

The top box on the left gives you the details of the database. For example, if you click on Batting, you'll see categories such as playerID, teamID, H, etc. These are categories you can search or print out with the database commands. Of course, if you don't know that GIDP means Grounded Into Double Play then you are going to be mystified, but presumably the database creator (which may be you) will have all of these things identified somewhere.

The top box is the query box. Enter the entire command as we did before, including the ;, then click the Execute icon to the right:

select b.AB+b.BB+b.HBP+b.SH+b.SF as PA,
m.nameLast, m.nameFirst, b.teamID, b.yearID
from Batting b inner join Master m
where b.playerID=m.playerID and b.yearID=2009
and b.AB+b.BB+b.HBP+b.SH+b.SF>502
order by PA DESC;

This gives you a spreadsheet like format

which is scrollable and much easier to use. You can also export it into a spreadsheet or table. Click on File => Export Resultsheet and you can save this table as a set of Comma Separated Values, or into HTML, XML, or Excel. Why Excel? Beats me. I'd hope for something open source, but I guess .xls is the default spreadsheet these days. Fortunately all of the graphical open source spreadsheets I know can work with these spreadsheets.

Enter Eligible Players batting data into the spreadsheet:

Simple enough right? We just need to dump everything for both years into a spreadsheet. Try this in the query box:

select b.yearID, m.nameLast, m.nameFirst,
b.teamID, b.G, b.AB+b.BB+b.HBP+b.SH+b.SF as PA, b.AB, b.R, b.H, b.2B,
b.3B, b.HR, b.RBI, b.SB, b.CS, b.BB, b.IBB, b.HBP, b.SH, b.SF, b.GIDP
from Batting b inner join Master m
where b.playerID=m.playerID and b.yearID>2008 and
b.AB+b.BB+b.HBP+b.SH+b.SF > 502
order by b.yearID ASC, m.nameLast, m.nameFirst;

Note that the sort is on year, last name, and first name. We want 2009 first in the sort, so we go in ascending order (ASC) rather than descending (DESC). However, ASC is the default, so we don't really need it, and I left it off the first and last name search.

Export this to a spreadsheet I'll call batting0910.xls and open it up with your favorite spreadsheet program. You should get something that looks like the picture below, except that I did a bit of editing to make it look cleaner. It's rather small here, so right-click and chose view image in another tab so see it more clearly.

We'll be using total bases at least twice, so let's make a column for it.

In cell V1 add the label TB
In cell V2 enter the string =I2+J2+2*K2+3*L2 (remember that hits (H) includes doubles, triples and home runs)

Your spreadsheet should look something like this:

Now let's fill in some other columns. Most of these will be fractions, so format them for as many decimal places as you like. Following long-time baseball convention I'm going to print out everything to three places.

Column	Category	Symbol	Row 2 Formula
`W`	Outs	OUTS	=G2-H2+O2+S2+T2+U2
`X`	Batting Average	BA	=I2/G2
`Y`	On-Base Pct.	OBP	=(I2+P2+R2)/F2
`Z`	Slugging Pct.	SLG	=V2/G2
`AA`	On-Base + Slugging	OPS	=X2+Y2
`AB`	Runs Created	RC	=(I2+P2-O2+R2-U2)(V2+0.26(P2-Q2+R2)+0.52*(S2+T2+N2))/F2
`AB`	Runs Created / Game	RC/G	=24*AB2/W2
`AC`	Base Runs	BsR	=(I2+P2-L2)(1.02(1.4V2-0.6I2-3L2+0.1P2))/(1.02(1.4V2-0.6I2-3L2+0.1*P2)+G2-H2+L2)
`AD`	Base Runs/Game	BsR/G	=24*AC2/W2

I'm not really sure about the last one: the original Wikipedia article for Baseruns has lost its formulas, I can't find other web page that mentions 1.02 or 1.4, and in my original calculations I seem to have miscalculated total bases. So take that with a grain of salt.

Once you have all of that, you can find average and standard deviations of any category you like, by year or over the range of years you selected. I'm going to recreate the entire table sometime this week, but for now, here are the results for 2006-10, including results from my original tables. For now I'll leave out the standard deviations.

Year	# Players	AVG	HR	RBI	OBP	SLG	OPS	RC	BsR
2006	146	0.286	20.46	80.95	0.355	0.469	0.824	113.09	96.13
2007	156	0.283	19.13	79.76	0.354	0.458	0.812	110.98	93.58
2008	137	0.280	19.62	78.03	0.350	0.457	0.807	108.99	92.95
2009	140	0.282	19.93	78.13	0.353	0.461	0.814	90.75	58.35
2010	136	0.273	18.73	75.12	0.343	0.444	0.787	85.30	55.42

OK, problems: I'm not sure if the falloff in Runs Created is really that dramatic, or if I just made a mistake in the previous calculations. The falloff is certainly consistent with, if more dramatic than, the falloff in the other categories, and fits with the cough end of the steroid era. The falloff in Base Runs is undoubtedly due to errors in my previous calculations. So for now we'll stop here, and I'll give you a fully redone table in a week or so.

Sunday, June 12, 2011: I updated all of the tables mentioned here, including corrected Base Runs and the years 2009 and 2010. All of that data is available here.

At least we've covered the basics of running mySQL and doing some analysis. I'll try to cover some more of this, along with other tricks and tips from Baseball Hacks in the near future. If you've got anything you'd like to look at, leave a comment.

The Natty Narwhal Upgrade

2011-05-30T11:12:00.001-04:00

Like Travis McGee, this week I'm taking an installment of my retirement. Unlike McGee, my retirement seems to consist of fixing up computers.

So today, Ladies and Gentlemen, Boys and Girls, Desktops and Netbooks, we're going to talk about upgrading Hal here to Ubuntu 11.04.

Yawn. This shouldn't really be that big a deal, should it? You just hit the update button, or type do-release-upgrade at the command line, right?

Well, it's not always that simple. A friend of mine did that, and he seemed to have a lot of problems. I'm not sure why, as I never got to examine his system before he reinstalled 10.10. Maybe it was the change from the Xorg window system and Gnome desktop to Wayland and the Unity desktop. Or not. Maybe it was just the way the wind was blowing that day. But I decided not to take the chance on the upgrade, but rather install 11.04 from the ground up.

I didn't have to be pushed hard, because I was leaning toward a clean install anyway. When I bought this version of Hal I was in a hurry to get Linux set up, but wanted to keep Windows 7 around. As a result, I let Ubuntu do the repartitioning of the disk, and ended up with the entire Linux system in one giant partition.

This is less than optimal. (Thanks, Dave. Don't mention it, Hal.) Ideally one wants /home and possibly /usr/local and /opt in partitions separate from the root operating system. That way when you upgrade the OS, you don't lose your previous data. (That doesn't mean you shouldn't back up that data before upgrading. There's a word for people who don't. The polite form of it is idiot)

But this requires repartitioning Hal's disk. Fortunately, all of Hal's memories reside in the extended partition, so I don't have to fiddle with the Windows partition. After about 30 secons of thought, I decided to repartition Hal's big Linux partition into 15GB for /, the root system, another 15GB for /usr/local, /opt, and /scratch (more on how to do this later), and leave the rest for /home. That's probably too much for /, and maybe not enough for /scratch, depending on what I right there, but that's how I'm going to do it this go-round. Ideally gparted would leave the data on /home as is, but that takes forever because of all the data that needs to be moved around, so we're just going to repartition everything and hope the backup holds.

Preparation

First, get the software we'll need. I downloaded and burned CDs for the 64-bit versions of:

Ubuntu 11.04
openSUSE 11.4
Fedora 15
and, last but probably most importantly,
Gparted Live CD version

OK, Gparted is obvious: that's what's going to do the disk repartitioning. But why the alternative distributions?

Because you never know. The largely hypothetical long-time reader will remember that I used to use Fedora, but that I switched to Ubuntu when the installation of Fedora 5 failed miserably, and Ubuntu was there, ready and waiting. So having alternative Linux CDs on hand seems to be a really good idea. Besides, we bought 100 CDs maybe five years ago and still have a bunch left. They just aren't used much anymore. (That's right, rcjhawk is still a trailing indicator for device popularity.)

OK, we've got hopefully every bit of software that we need, so let's go.

Find out what's on your machine

I never remember which packages are installed from one upgrade to the next. So let's make a list:

dpkg --get-selections > installed.txt

Edit installed.txt, leaving out any packages you think will be installed by the system (e.g., kernel, window manager, etc.). If you can't remember what a package does, delete it too. If you need it, it will eventually show up as a dependency to a package you want, or you can reinstall with apt-get or synaptic. The important thing to do here is to remove cruft from the system. For example, I have the bsd-games package on here. I never play those games, so why leave them in place?

It would have been nice to make a copy of /etc/fstab, but some dingbat forgot to do that (more on this later).

Back it all up

Well, not everything. No need to back up the kernel, or Emacs, or latex, or any of that. The distribution will provide. No, I want to back up my data (basically $HOME), the Intel Fortran compiler, a royal pain to install, and my renegade version of SoX. So do the following (/backup is my ext4 formatted USB disk):

$ cd
# -rpv == recurse directories,
#  preserve permissions and timestamps,
#  and speak up about it verbosely.
$ cp -rpv dave /backup
# Assuming you're named dave, of course
$ cd /usr
$ cp -rpv local /backup
$ cd /
$ cp -rpv opt /backup

To be safe, I also had backintime, my backup manager, do an additional labeled backup, which should keep it on the disk forever and a day, or at least until the end of 2012.

Repartition

The next step is the scariest one. I've taken what was an 800+GB partition and broken it up into three parts: 15GB for /, 15GB for what will be /usr/local, /opt and /scratch, and the remainder for /home. It's scary because this wipes out the primary data, so you are now relying on the kindness of backups. It's theoretically possible to gave gparted to keep all the data in what's going to be the /home partition, but that requires moving a lot of stuff around, at least the way I tried it, and an estimated 18 hours to get it done. So I just said a prayer, bit my lip, and reformatted. Fortunately it all went well.

Installation

Not much to say here. I told Ubuntu that I wanted to do an install with a custom installation, had it mount everything where I wanted it, and let it go. The usual waiting around occurred.

Copying Files

This was rather easy: turn on the backup USB disk drive, and copy the files back. I copied back configuration files I knew I wanted to keep, e.g. things like .emacs and the .devilspie directory, but I'm letting Ubuntu set up the desktop. This means I'll have to play with preferences later, but it should get eliminate any conflicts between old and new configuration parameters.

Mounting the Backup Drive

I could just plug the backup drive in and access it at /media/really_long_hexadecimal_string, but what I really want to do is have the backup drive mounted to /backup. I've covered this before, but as I mentioned above, I forgot the UUID of the drive. You get that with the command:

blkid

which gives you all the UUID of all connected disks.

Restoring `/usr/local` and `/opt`

This is awkward. Add-on programs that you compile yourself generally go into /usr/local. Other programs, such as the Intel Fortran compiler and Google's Picasa, end up in /opt. I want both to stick around between upgrades, so they need to be out of the / partition. But I don't want two extra partitions. So here's what I did:

Create a partition /usr/local
Create a directory /usr/local/opt
Copy all of the /usr/local files where they belong.
Copy all of the /opt files to /usr/local/opt
Then, as root, run the command
ln -s /usr/local/opt /opt
If you want a /scratch directory, do the same thing, but don't forget to make the original world writable:
```
# mkdir /usr/local/scratch
# chmod 777 /usr/local/scratch
# ln -s /usr/local/scratch /scratch
```

Restoring your add-on packages.

Now we want to get back all of our old packages, the ones that aren't installed by default, at least so far as Ubuntu will let us. Remember that file installed.txt I mentioned you should create and edit? Here's how we'll use it:

Select System => Administration => Update Manager
Click on Settings in the lower left corner
Under Ubuntu Software make sure all of the sources you want enabled are. If you don't want proprietary drives, etc., turn them off. Do the same under Other Software
Open a terminal window and go the the directory where you have that installed.txt

Run the following commands:

sudo apt-get update
sudo apt-get install `awk '{print $1}' installed.txt | xargs`

You'll probably get some error messages, but they're pretty clear about what you need to fix. Edit the installed.txt file to match.
This will take a while, as all the packages have to be downloaded from the net.

What Works

Pretty much everything. I selected Gnome-classic for my desktop, and after a little fiddling to with System => Preferences => Startup Applications got the system to look pretty much as it did before.

As for third party software, the Intel Fortran compiler still works fine. I was able to reinstall Picasa from Google's supplied 64-bit .deb.

Wayland seems to work just like Xorg, at least from my point of view. It pops up windows in the same way as before, and I can run, say
ssh -X majel
to get me to another machine, and then run
firefox
on majel, and the window pops up here on Hal. So no major problems for me there.

What doesn't work

Google Earth. Maybe this is a 64-bit problem, I don't know. I tried using Google's .deb file, and the official Ubuntu method. Neither worked.

By default Ubuntu installs the 3-D version of Unity, which requires a pretty good graphics accelerator. Hal lives in 2-d, so that wouldn't run. (Funny, the Unity desktop shows up when you run the live CD.) I installed the 2-D version and that works. Which brings me to

What Sucks

Unity. From my limited experience (about two minutes, after which I ran away screaming), it's an overblown and somewhat hideous version and of the Mac desktop. But don't mind me, I was the last person on Earth to use fvwm. I suppose if I played around awhile I could make Unity behave the way I wanted it to, but why bother, since Ubuntu still supplies the classic Gnome desktop. Mind you, I've seen Gnome 3, and I don't like that, either. (The word you're looking for is Luddite.)

Summary

All in all, a successful update, as long as I stay way from Unity. I have one more computer that needs updating to 11.4, I think I'll just try that as a distribution update. If it doesn't work, I can always do a full install.

More Later

Troubles will surface, they always do. When they do, I'll write about them here.

Compression

2011-05-21T17:37:00.000-04:00

The other day an email arrived from sourceforge which mentioned that they were hosting the file compression program 7-zip. Now I had used 7-zip under windows, as an all purpose archiving tool, mostly for reading zip files. I'd never thought of it in a Linux context. But both openSuSE and Ubuntu have the command-line version available, and what more do you need?

In OpenSuSE, the RPM is called, simply enough, 7z. In Ubuntu it's a bit more complicated. There's p7zip, which provides the bare-bones standalone version of the compression program, 7zr, and a wrapper p7zip, which makes 7zr work like gzip. For the full-blown 7z program, you want the package p7zip-full, which includes 7z. While you're at it you might want to get the p7zip-rar package, which lets you decompress RAR files.

The reason you want 7z and not just 7zr is that the smaller program only compresses to the 7z format, but, as it says in the blurb,

not only does [7z] handle 7z but also ZIP, Zip64, CAB, RAR, ARJ, GZIP, BZIP2, TAR, CPIO, RPM, ISO and DEB archives.

And by handle, they mean read and write to these formats (except RAR). So I could create a zip archive with 7z, or a gzip/bzip2 compressed tarball. And guess what?

7z compression is 30-50% better than ZIP compression.

Is it? Well, let's find out.

Test 1

Presented for your consideration: an uncompressed tarball:

$ ls -l Ru.tar
-rw-r--r-- 1 dave dave 26716160 2011-05-21 15:41 Ru.tar

This is IO from the elk FP-LAPW code, so it has a lot of repetition of text, and what for our purposes are a bunch of random numbers. First we'll try compressing it with programs using their native formats. I'll try for maximum compression in all cases. Note that zip and 7z will create separate archives, while gzip and bzip2 compress the file in place.

Program	Command	File Size	Ratio
zip	zip -9 Ru Ru.tar	3899523	0.146
gzip	gzip -9 Ru.tar	3899386	0.146
bzip2	bzip2 -9 Ru.tar	2992422	0.112
7z	7zr a -mx=9 Ru.7z Ru.tar	2242708	0.084

Pretty good, huh? As advertised, 7z is about 40% better than zip/gzip, and 25% better than bzip2. But wait, there's more. Not every computer is going to have 7z available, so you may want to compress files using a more established protocol. 7z can do that, too, which is why we wanted it, not just 7zr:

Format	Command	File Size	Ratio
zip	7z a -mx=9 -tzip Ru.zip Ru.tar	3287420	0.123
gzip	7z a -mx=9 -tgzip Ru.tar.gz Ru.tar	3287335	0.123
bzip2	7z a -mx=9 -tbzip2 Ru.tar.bz2 Ru.tar	2989193	0.112

So 7z compresses to zip/gzip better than the native programs do it themselves. It doesn't really outperform bzip2 here, though. The only disadvantage compared to gzip or bzip2 is that it doesn't compress the files in place, unless you go through a script such as the one in /usr/bin/p7zip.

Test 2

Pi to 4 million Decimals has, duh, π to, actually, 4,194,034 places. The file pi.tar.gz has it in ascii, with a bit of header information. If we uncompress that file, it comes in at 4362370 bytes. Since the digits of π don't repeat, it's hard for a compression program to find blocks of bytes to compress. The following table lists the compressions achieved by our test programs, in whatever formats they can use. (See the above tables for the appropriate commands.) Let's see how everybody does:

Program	Protocol	File Size	Ratio
None	None	4362370	1.000
zip	zip	2041130	0.468
gzip	gzip	2040997	0.468
bzip2	bzip2	1863892	0.427
7z	zip	1983378	0.455
7z	gzip	1983297	0.455
7z	bzip2	1860047	0.426
7z	7z	1884999	0.432

Here, bzip2 is competitive with 7z. Oddly, though, you should use 7z to do the bzip2 compression. Weird, huh? But still, 7z is pretty good.

Wrapping Up

So what's not to like?

For one thing, 7z is slow. If you just want to quickly compress a file, go ahead and use gzip, or bzip2. There is a price to pay for better compression.

Then, too, there's a warning on the 7z man page:

DO NOT USE the 7-zip format for backup purpose on Linux/Unix because 7-zip does not store the owner/group of the file.

you can get around this by piping tar into 7z:

tar cf - directory_to_be_archived | 7z a -si directory.tar.7z

which creates the analog of a gzip/bzip2 tarball.

And finally, the native 7z format isn't standard, yet, so it's not going to be available everywhere, and might even vanish. But 7z and its compression algorithm LZMA are open source, so they are likely to stay around for awhile. A few years ago bzip2 wasn't standard, and once upon a time neither was gzip. It's probably safe to compress your files to 7z format, but if you want to be safe, use 7z to compress to gzip or bzip2 format.

Why I Hate March

2011-03-18T22:06:00.002-04:00

I just spent the last three hours not watching the Kansas-Boston University first round NCAA game. In case you didn't join me in not watching, KU spent the first 25 or so minutes wondering why the BU players weren't genuflecting. Then they concentrated for a few minutes, and the game was over. Because of course, you know, when #1 meets #16, #16 has never won, right?

Trust me. Some day it will happen. And which team will that be? We'll, let's look at the past for some clues, starting here:

2010: #1 Seed, lost in second round
2007: #1 Seed, lost in Sweet 16
1998: #1 Seed, lost in second round
1997: #1 Seed, lost in Sweet 16
1995: #1 Seed, lost in Sweet 16
1992: #1 Seed, lost in second round

Not to mention Bradley, or Bucknell, or the suffering I did during the Ted Owens years.

It's enough to make one start a website, except a) someone beat me too it, and b) Roy was as bad, or worse, because he didn't win a Championship until he clicked his heels three times and went home.

But mark my words, the #1 Seed in NCAA history to lose in the first round will have a six-letter name on the front of the jersey, and have two Crimson and Blue mascots that look something like this.

Superbowl Commercials

2011-02-20T19:05:00.000-05:00

There weren't many that were any good this year. I'll show you my favorite, below, but I found that a lot of people didn't get the reference. So let's go back to September, 1979, when this famous commercial aired, starting Pittsburgh's Mean Joe Green:

And now that we've set the stage, here's my favorite commercial from this year's Superbowl:

And one more reference to TV gone by:

FUSE on the Mac

2011-01-26T19:30:00.000-05:00

Awhile ago I wrote about how one can mount remote computer filesystems using ssh. That was for Linux. You can do it with Mac OS X as well, but FUSE (File System in UserSpace) isn't installed by default, as it is in modern Linux systems.

So first you need to get FUSE installed. The package I use is called, duh, MacFUSE. It installs like a standard Mac package: Download the disk image, click on it, and go.

Next is sshfs itself. There seem to be many versions, but Google's seems to work well. Download the appropriate version for your OS (the Leopard version works on Snow Leopard), rename it sshfs, make it executable, and put it in your path. From then on it works pretty much like in Linux, with minor differences. Assuming you have a directory ~/hal on your Mac, you can look at the files on your Linux machine hal with the command

sshfs rchawk@hal:/home/rcjhawk ~/hal

So a file named /home/rcjhawk/Documents/testthis.doc on hal will appear as ~/hal/Documents/testthis.doc on your Mac.

And it's simple to unmount it:

umount ~/hal

A More Perfect World

2011-01-05T13:02:00.002-05:00

Me, to.

Wikipedia's List of Common Misconceptions.

Happy New Year

2010-12-30T20:19:00.000-05:00

This blog started just over seven years ago, and sometime tonight or tomorrow it will get its forty thousandth recorded hit. Thanks to all of you for listening.

Thanks to a Penguin Pete rant (see Sitemaps). I've discovered Google's webmaster tools and Analytics. My New Year's Blogging Resolution is to learn how to use these next year.^† If I come up with any neat tricks I'll let you know.

In the meantime, Happy New Year, everyone.

^† My primary resolution is to still have a job next year.

Should We Call It gnarcissism?

2010-12-27T17:13:00.000-05:00

Linux Journal, or, more properly, the janitor at Linux Journal since everyone else is off this week, is running a pole asking if you Google® yourself.

The answer I'd choose isn't there:

No, because there are at least a half-dozen people^* with the same name who are at least as famous as I am, and that doesn't count the 1,300 or so in China.^†

^* Yes, I know, but only for the purpose of this post.^§

^†Working from the old if you're one in a million line.

^§That's my story, and I'm sticking to it.

The Christmas Toy

2010-12-25T21:20:00.000-05:00

So here I am, Christmas Evening, waiting for BBC America to broadcast the Dr. Who Christmas special and playing with my new toy: a Logitech Harmony 650 Universal remote. I bought the thing at Radio Shack because: a) It was cheap ($69.95), and b) I'm sick and tired of having to manipulate three remotes to change from FIOS to BluRay to USB/Harddisk when watching shows on the bigscreen TV.

The neat thing about the Harmony series of remotes is that you program it on-line. The bad part is that there isn't a native Linux program to do it. There is something called Concordance which is supposed to work, but I'll try that out when I'm prepared to trash the remote if the software fraks out, OK? It's not worth the struggle (forgive me, Saint Richard).

So we boot Hal up in Windows 7 mode, and I'm blogging what's going on using a laptop.

First: gather up the model numbers of the gadgets to be used. That's fairly easy. I've got a Samsung LCD 46 inch TV, and the model numbers on the side. A Samsung BluRay BD-P360. The hardest part is the FIOS DVR. It's by Motorola, and looking at the pictures on the FIOS site it appears to be a 6416. We'll see.
Launch IE8 on Hal (NNNNOOOOO!!!!!!!). Go to http:www.logitech.com/myharmony and create an account. Click on My Account. No registration box. Click on the safest looking link, Go to My Stuff There's a login screen and a registration option there. For some idiot reason they want a daytime phone number. Tempted to give them the one for our security, but refrain.
Click on Product Support. Says it will detect USB corded product. The unit comes with a USB 2 cable, but of course the connector on the remote side isn't standard USB2. Annoying, especially if I loose the cord, but I sort of forfeited the moral outrage option when I booted into Windows. So plug it in, let Windows find the driver software, and click the Detect your devices button. Of course Logitech has an IE add-on that has to be installed. It seems to work. It says I've got something called an N/N: N-10003 Harmony 650 remote. Well, I know the Harmony and 650 part are right.
It's not particularly obvious what to do next. Click Download Software and get version 7.7.0-WIN-x86 version of LogitechHarmonyRemote, I guess. Install that. You know, I'm kind of glad I'm doing all of this on the Windows side. Installation involves the usual number of clicks to accept the license. Eventually an Icon shows up on my desktop and the thing launches. First thing it wants to know is if it should check for updates. Surprisingly, there are none.
Log in. You'd think the account I just set up with Harmony would work, but no. So set up another account here. It's another complete registration, folks.
Now things start. It finds my remote (again). Keep hitting Next until we get to the Add Device page. Adding devices seems straightforward, so long as you have the right device numbers. Fortunately they give you examples of how they want the information entered in.
Once I finally got the right models in, the software found my devices. The only question I had to ask is if I wanted to turn the cable box off when it wasn't in use. I said yes. We'll see if that's a good idea or not. (I worry what's going to happen when I switch devices. Is the TV going to be out of sync with the cable box?)
Specify that FIOS uses HDMI1 on the TV, and the BluRay uses HDMI2.
And somehow we lost the signal, so I HAVE TO LOG IN AGAIN???? And it lost the connection to the remote. But now it has, indeed, started downloading the settings to the remote.
Looks done. I'm told to disconnect the remote and go try it out.
Preliminary tests done. It mostly works. There are several glitches, though:
1. Both the TV remote and the FIOS remote have options for resizing the picture on the screen. It's especially handy when you are watching a standard definition channel, but they're broadcasting something in letterbox format (as in the above-mentioned Dr. Who special). Resize the picture, and it fits the whole screen. Not high-def, but in this case bigger is better. The 650 doesn't have such a button.
2. The FIOS remote has a standalone DVR button, which bypasses the main menu and takes you to the stuff you use most. The 650 doesn't, you have to go through the main Menu (though the Menu, Info, and Guide buttons work exactly the way they should, as does the Favorites option on the 650's screen).
3. The worst problem: If I want to stream Netflix using the BluRay, I press a button labeled B on the BluRay remote. One of four buttons, Red/A, Green/B, Yellow/C, Green/D. The 650 has unlabeled Red/Blue/Yellow/Green buttons, but they don't do anything to the BluRay. Since I stream Netflix as much as watch DVDs, if not more, this is a serious difficulty, and a deal-breaker.
Back to Win-Hal. Click the I need to make some adjustments box. Log in again. At least this time it remembered my password.
There's a Customize Buttons section for the remotes. For the BluRay, set up Blue to be Netflix, Yellow Pandora, and Green for YouTube. A/Red on the BluRay is Blockbuster, which we don't use. For the FIOS box, set A on the screen (not to be confused with A/Red below) to Aspect, which should set the screen size properly. While we're at it, set B up there to Close Caption, something we use a lot since we're old. Do that for the BluRay as well. I may have to tweak that later. Can't find a DVR button option, but I may have a way around that. Let me test out the new setup and get back to you in a minute.
OK, those changes work. Nice touch: the buttons attached to the screen are labeled A/B/C..., but you can change those to Aspect/Closed Caption/On Demand, etc.
Now let's tackle that DVR button problem. On the setup screen for the FIOS remote (labeled PVR in 650's language) there is a button labeled Learn Command. Click that, and go down until you find Learn a New Command. Type DVR Menu in the box and click the Learn a New Command button. Put the front of the FIOS remote to the back of the 650, press the DVR button on the FIOS remote. Wait. Go back to the Additional Buttons setup screen, and assign the button you want the new DVR Menu command.

And, for now, that's enough. If I encounter more difficulties I'll let you know. In particular, we'll see if the family can use it easily or not.

Preliminary bottom line: This isn't all that easy to program, especially if you have an exotic system. But it does seem to be straightforward to program, and you can add special keys as needed. We'll see how it all works in practice.

Mounting Remote Computer Files with FUSE

2010-11-28T22:06:00.000-05:00

I'm sitting here, upstairs, typing on my laptop. There's a file I want on my office machine downstairs. So I should go downstairs and look for the file, right?

Wrong. The TV up here is on NFL Red Zone ( (sigh) Red Zone). I'm addicted, OK? It's hard enough to break away for a bathroom trip.

Several options exist, of course. I could ssh to the downstairs machine, search for the file, and use scp to copy it upstairs. But the file is a picture, and I while I remember what it looks like, I don't remember what name I used for it. So ideally I'd like to use an image viewer, say gthumb or F-Spot to look through the directory and find the file. But I can't, because this computer doesn't mount the downstairs computer's disk.

Ah, but it can. The classic way to do this is to use the Network File System protocol, aka NFS. But that requires work to set up. Besides, some days I might want to search my computer from work, or from a hotel. It's tricky to get NFS to work that way.

Fear not, there is a solution. It uses File System in Userspace (FUSE), a complicated name for using ssh and scp to fool my laptop into thinking that the downstairs computer's files are on the laptop's local drive.

In Ubuntu the way to do this is to install a package called sshfs. Then you just follow the directions on the linked page. Let's say I want to be able to access all the files in /home/rcjhawk on hal, my downstairs computer, from my laptop, majel:

First make sure I have both hal and majel set up to use ssh communications. If I'm away from home, I'll also need to know hal's ip address, and make sure I can ssh to hal from outside my router.
Next, install sshfs on majel. Just use
$ sudo apt-get install sshfs
from a terminal window on majel. This doesn't need to be done on hal. Hal's just going to see a series of ssh requests.
Add myself to the FUSE group on majel:
$ sudo gpasswd -a $USER fuse
from the same command window.
Log off my account on majel, log back on. Yes, it's annoying, but you're not officially added to the FUSE group until you do.
Make a directory on majel. Doesn't matter what the name is:
$ mkdir ~/hal
Now we're going to mount all of hal's directory tree starting from /home/rcjhawk in majel's ~/hal directory:
$ sshfs -o idmap=user rcjhawk@hal:/home/rcjhawk ~/hal
If this works, then running the command
$ ls ~/hal
from a terminal window on majel lists hal's directory tree.
To unmount hal, just run:
$ fusermount -u ~/hal

To make things easier on myself, I created two files. The first, mthal, contains the mounting command, and the second, umthal, the unmount command. Make these executable, put them in your path, and it's easy to mount and unmount.

A caveat: I was getting an error remote host has disconnected. That occurred because my .cshrc file, which used a variable that's undefined when you are running ssh commands. You can find out how to diagnose that here.

Does it work? We'll, I started typing this on the laptop, and finished it up on the big machine downstairs. So, yes, it does work.

There are also versions of FUSE for BSD, OpenSolaris, and Mac OS X. In fact, I first learned about this trick using my Mac at work. With FUSE you can mount just about anything anywhere. There's even GmailFS, which transforms your Gmail account into backup storage.

Finding Your Header Files — A Better Way

2010-11-27T10:25:00.001-05:00

I should really read my own posts. Especially since that's the original purpose of this blog.

The other day I was looking through old posts with the Ubuntu label, and found this comparison of dpkg/apt-get and rpm/yum commands. The relevant one is

dpkg -l '*'

which lists all of the packages available in your repository, installed or not.

Duh

This makes my header file lister a lot simpler to write, towit:

#! /bin/bash

# First make sure you've got all your repositories updated:
sudo apt-get update

# Now search through every repository database, looking for lines that
#  start with "Package", and end with "-dev".
# Pull out the package name, look through each package, and print out
#  the header files:  those that end in "-h"

# No matter how it wraps on your screen, this next line starts with
# "for" and ends with "awk '{print $2}'`"

for hfile in `dpkg -l '*' | grep dev | awk '{print $2}'`
do
# echo $hfile
apt-file list $hfile | grep "\.h$"
done

It's still going to take a long time to go through every package, but it's a lot neater now.

Look Familiar?

2010-11-25T13:20:00.000-05:00

I saw this watching the Star Trek reboot on the bigscreen with our Epix OnDemand account. It's during the Old Spock/New Kirk mind-meld, at the part of the story where Spock is transported back in time. The blue box is suggestive, don't you think?

Since Epix lets you stream its movies to a Linux box, unlike Netflix, it was relatively easy to find the shot and save it. No, the resolution's not that great, but, frankly, this movie suffers even on a 48" high-def screen. To do it justice you really need to watch it in a theater.

Undercutting Yourself

2010-11-20T18:19:00.000-05:00

Maybe I just don't understand…

The other day Borders sent me an email which gave me ten dollars in Borders' Bucks. A great promotion, to be sure. And I had a book in mind I wanted to buy, but not spend a lot of money on: Jim Butcher's collection of Harry Dresden short stories, Side Jobs.

First check it out online. We have:

List Price	$25.95
Online Discount	-$11.22
Borders' Bucks	-$10.00
Shipping	$3.98
MD Tax	$0.29

Total	$9.00

Not bad, but the shipping price sucks, it will take days to get here, and anyway I want to get out of the house for awhile, so let's go to a nearby Borders and buy it there.

I get to the Borders, and find Side Jobs. The sticker on it says 20% off. Do a little bit of math: 80% of $25.95 = $20.76 - $10 Borders' Bucks = $10.76, + 6% tax ( $0.65) = $11.41. A bit of a rip, compared to online, but I'm here, I've already spent a buck in gas, and I'll walk out with the book.

Except: I then notice that the sticker also says Borders Rewards Plus Members. I'm just an ordinary Borders Rewards member. I checked with one of the sales people, and no, I'm not entitled to the discount. That makes the cost $16.91, Borders' Bucks and tax included. I made a point of telling the guy his store's website was undercutting the store by selling the book for $5 less, including shipping, put the book down, and walked out.

I suspect some marketing guru has it all figured out, but to me this just seems plain old-fashioned stupid. If Borders gave me the online price at the store, or even got close to it, I would have gotten the book then and there. Since I was in the store, I probably would have had a coffee as well. Maybe a cookie. I would have looked at a Linux magazine while I was sipping my coffee, and possibly would have purchase that, too. I would most likely have spent the $5 in the store, maybe more, and I wouldn't have been pissed.

Instead, I went home and ordered the Dresden book online. Even with the $2 gas^† I just wasted I still saved $3 over buying it in the store. And, instead of waiting for more coupons/bucks to come my way, I went over to Amazon and ordered two books I would have otherwise eventually gotten at Borders (along with more coffee, cookies, magazines, etc.): Lois McMaster Bujold's newest Vorkosigan novel, Cryoburn, and Terry Pratchett's new Tiffany Aching/Discworld novel, I Shall Wear Midnight. Add in a book Child_II wanted, we made the supersaver price, and got free shipping.

In summary: by charging full price for a book in the store, and a deep discount online, Borders lost the sale of at least two books, two-three cups of coffee, a couple of cookies, and maybe a magazine.

It looks like Borders would rather you didn't come to their store. OK, stores cost money to keep up. Employees have to be paid, not to mention heating, lights, insurance, and all of that. But if I come into the store, I'm going to browse. I'll look around, and might well find something else I'd like to buy. Online, I'm going to get the books I want, hit the send button, and go on with my life.

Like I said, some genius with an MBA probably has figured out that this approach makes more sense for Borders, but how that could be is a mystery to me.

^† The trip wasn't wasted, I drove a mile to the local wine emporium, tasted a couple of wines, and bought a bottle.

Unintended Consequences

2010-11-11T19:19:00.000-05:00

Article 1, Section 2, of the United States Constitution says, in part, The Number of Representatives shall not exceed one for every thirty Thousand, but each State shall have at Least one Representative. The very first amendment proposed (you can look it up) tried to change this:

After the first enumeration required by the first article of the Constitution, there shall be one Representative for every thirty thousand, until the number shall amount to one hundred, after which the proportion shall be so regulated by Congress, that there shall be not less than one hundred Representatives, nor less than one Representative for every forty thousand persons, until the number of Representatives shall amount to two hundred; after which the proportion shall be so regulated by Congress, that there shall not be less than two hundred Representatives, nor more than one Representative for every fifty thousand persons.

but this was never ratified. So we can be pretty sure that the Founders of this Republic wanted us to have one Representative for every 30,000 Americans.

Currently, however, we have 435 Congressmen, thanks to something called Public Law 62-5. I guess this was supposed to keep the number of Congressmen down to a reasonable number. Otherwise, with the current U.S. population around 300,000,000, we'd have 10,000 Congressmen running around. Think of it! Mark Russell once said something on like every night 500,000 people in the District of Columbia go to bed without a Congressman. But if we had 10,000 Congressmen, you give yours (DC residents excepted) a booty call just about every night.

The current limit of 435 Congressmen also gives rise to some startling inequities. For example, according to the 2000 Census, the total population of the United States was 281,421,906. If we exclude the District of Columbia (population 572,059) since it has no Congressional representation, then there are 280,849,847 Americans and 435 representatives, so each member of Congress should represent 645,632 people. However, Wyoming (493,782) and Montana (902,195) each have only one representative. To put it another way, Cynthia Lummis only needs to attract half as many votes as Denny Rehberg to get elected, but each gets one vote in the House. So by moving from Wyola, Montana to Parkman, Wyoming (16 miles) you can effectively double the strength of your vote for Congress.

There is a lot more information about the inequities of the current system, and the sorry history of Amendment the First, at thirty-thousand.org. I'd recommend downloading their pamphlet, Taking Back Our Republic, for more information.

One thing they didn't cover, though: the effect of all of these changes on the Electoral College. Each state gets N+2 electoral votes, where N is the number of representatives, plus the two members of the Senate allowed each state. That +2, along with Public Law 62-5, weights the Electoral College heavily toward the smaller states. Since D.C. does have 3 votes in the Electoral College, each elector theoretically represents 281,421,906/538 = 523,098 voters. However, each state has a minimum of three electoral votes. So in Wyoming each elector represents 164,594 people. In Montana, it's 300,732 people, still substantially below the national average. It's likely (I haven't checked in detail) that the state with the most people/elector is California, where there are 33,871,648 people and 55 electors, or 615,848 people/elector. Put it this way: an elector in California represents four times as many people as one in Wyoming, but each has the same say in who's elected President.

Now, as anyone can tell you, most of the smaller states, population-wise, are in the west, and most of those states vote Republican, while many of the larger states tend to vote Democratic. Another wrinkle in the system is that most states (Maine and Nebraska excepted) cast their electoral votes in a block.

Couple these two facts with the 30,000 people/representative rule and you get some interesting results. Very interesting, in some cases.

In particular, let's consider the 2000 Presidential Election. Very close, as we all recall. Al Gore got 50,999,897 votes, while George Bush got 50,456,002, but because most states cast their electoral votes as a block, Bush won in the Electoral College, where it counts, 271-266.

What if we'd apportioned Congress the way the Founders intended? And kept the current winner-take-all (mostly) electoral system? The table below shows the results. I constructed it using the following rules:

Winner take all in each state, as now. Maine (42 votes under these rules) went to Gore, and Nebraska (54) went to Bush. Changing all of Maine's votes to Bush or all of Nebraska's to Gore isn't going to change the final result, so it doesn't matter if several congressional districts shifted from Red to Blue or vice versa.
To get the number of representatives, I took the states' populations from the 1990 census, which governed apportionment at the time, divided by 30,000, and rounded down. Rounding to the nearest integer, or rounding up, can't change the electoral vote by more than 51, and so won't affect the result, either.
If the District of Columbia was a state, it would have 22 electoral votes in this scheme. However, by the Twenty-Third Amendment, D.C. gets 17 electoral votes, equal to the state with the fewest, Wyoming. In the real 2000 election one D.C. elector abstained. Even assuming all 17 of the new electors abstained it wouldn't change the bottom line here, so I assume that they all voted — for Gore, naturally.

The result: Since each state has more or less the same representation per capita in the electoral college, and since most of the larger states went for Gore, he wins the election, 4,314–4,047.

Think about it. Al Gore, Forty-Third President of the United States. Somehow, I don't think that's what those currently saying we should go back to the Principles of the Founders had in mind.

State	Population	Congress	Electors	Bush	Gore	Bush	Gore
				Popular	Popular	Electoral	Electoral
Alabama	4040587	134	136	941173	692611	136	0
Alaska	550043	18	20	167398	79004	20	0
Arizona	3665228	122	124	781652	685341	124	0
Arkansas	2350725	78	80	472940	422768	80	0
California	29760021	992	994	4567429	5861203	0	994
Colorado	3294394	109	111	883748	738227	111	0
Connecticut	3287116	109	111	561094	816015	0	111
Delaware	666168	22	24	137288	180068	0	24
DC	606900	0	17	18073	171923	0	17
Florida	12937926	431	433	2912790	2912253	433	0
Georgia	6478216	215	217	1419720	1116230	217	0
Hawaii	1108229	36	38	137845	205286	0	38
Idaho	1006749	33	35	336937	138637	35	0
Illinois	11430602	381	383	2019421	2589026	0	383
Indiana	5544159	184	186	1245836	901980	186	0
Iowa	2776755	92	94	634373	638517	0	94
Kansas	2477574	82	84	622332	399276	84	0
Kentucky	3685296	122	124	872492	638898	124	0
Louisiana	4219973	140	142	927871	792344	142	0
Maine	1227928	40	42	286616	319951	0	42
Maryland	4781468	159	161	813797	1140782	0	161
Massachusetts	6016425	200	202	878502	1616487	0	202
Michigan	9295297	309	311	1953139	2170418	0	311
Minnesota	4375099	145	147	1109659	1168266	0	147
Mississippi	2573216	85	87	572844	404614	87	0
Missouri	5117073	170	172	1189924	1111138	172	0
Montana	799065	26	28	240178	137126	28	0
Nebraska	1578385	52	54	433862	231780	54	0
Nevada	1201833	40	42	301575	279978	42	0
New Hampshire	1109252	36	38	273559	266348	38	0
New Jersey	7730188	257	259	1284173	1788850	0	259
New Mexico	1515069	50	52	286417	286783	0	52
New York	17990455	599	601	2403374	4107697	0	601
North Carolina	6628637	220	222	1631163	1257692	222	0
North Dakota	638800	21	23	174852	95284	23	0
Ohio	10847115	361	363	2350363	2183628	363	0
Oklahoma	3145585	104	106	744337	474276	106	0
Oregon	2842321	94	96	713577	720342	0	96
Pennsylvania	11881643	396	398	2281127	2485967	0	398
Rhode Island	1003464	33	35	130555	249508	0	35
South Carolina	3486703	116	118	785937	565561	118	0
South Dakota	696004	23	25	190700	118804	25	0
Tennessee	4877185	162	164	1061949	981720	164	0
Texas	16986510	566	568	3799639	2433746	568	0
Utah	1722850	57	59	515096	203053	59	0
Vermont	562758	18	20	119775	149022	0	20
Virginia	6187358	206	208	1437490	1217290	208	0
Washington	4866692	162	164	1108864	1247652	0	164
West Virginia	1793477	59	61	336475	295497	61	0
Wisconsin	4891769	163	165	1237279	1242987	0	165
Wyoming	453588	15	17	147947	60481	17	0
Totals			8361	50455156	50992335	4047	4314

Sorry about the lack of commas in the table above. I used a very primitive spreadsheet to generate the data.

1990 Census data is from http://lwd.dol.state.nj.us/labor/lpa/census/1990/poptrd1.htm. Presidential vote totals are from http://www.presidency.ucsb.edu/showelection.php?year=2000.

Find Every Header File in Your Ubuntu Repositories

2010-11-08T20:33:00.000-05:00

Last night I was following a thread on Penguin Pete's blog where Pete uses Ubuntu's dropping the Dillo browser from the distribution as a launching pad for a discussion about — heck, I can't describe what it's about, read it yourself.

Anyway, in the course of reading the discussion — if that's the word, Pete gets worked up about certain things — I was possessed by the moral imperative to install Dillo from source.

This would seem to be straightforward. You go to the Dillo home page, click Download, and follow the instructions.

Except (there's always an except) — the first install you have to make is the FLTK2 (Fast Light Toolkit) library.

Now Ubuntu offers FLTK1.1, which might well work, but hey, we need the latest and the greatest, right? So I downloaded the tarball, followed the instructions, and the make command crashed and burned with:

fatal error: X11/extensions/XInput.h

I looked on the web, found that XInput.h is in the package libxi-dev, installed that, reran make,

fatal error: GL/glu.h

and so on. It took about an hour to find every single header file needed. And that was just to install FLTK2, after which I still had to install Dillo.

So what's going on here? Well, first we have to know what a header file is. Basically, it's a file that defines a bunch of variables and statements that can be used by multiple routines. See Wikipedia for a better explanation. In particular, header files tell a program's source code what variables are used in a particular library. Thus the header XInput.h defines variables available in some X11 libraries.

OK, suppose I'm compiling a program and I find that I need a header file GrantsTomb/Buried.h. All I have to do is find the package that has that header, install it with apt-get, and I'm good to go. Right?

Well, without prior knowledge you're pretty much doomed here. I was able to find XInput.h and glu.h by web search, but who's going to tell you where is Buried.h in GrantsTomb?

Yet there has to be a solution, because:

I know that when I use synaptic or aptitude to install packages I can see every package that is, or can be, installed on my system from my selected repositories.
After searching around a bit, I find that apt and its children list the available files in /var/lib/apt/lists, each repository has its own file, and each file name is on a line starting with Package.
I know about the program apt-file, in particular that the command
apt-file list foobar
will give me a list of all of the files in the foobar package.
Header files are traditionally located in packages ending with the string -dev.

Given all of that, I should be able to write a script which will list every header file available in every Repository I've accessed. I can then save it in a file, and the next time a program says I need bojangles/tambourine.h I can just search the list, find that the header file is in the NittyGritty-dev package, load that up, and I'm ready to compile:

First, install apt-file, if you don't have it already:
sudo apt-get install apt-file

Then run this script:

#! /bin/bash

# First make sure you've got all your repositories updated:
sudo apt-get update

# Next sync apt-file's database.  Note that if you do this without
#  the sudo you'll write the data into your home directory, which is
#  probably OK

sudo apt-file update

# Now search through every repository database, looking for lines that
#  start with "Package", and end with "-dev".
# Pull out the package name, look through each package, and print out
#  the header files:  those that end in "-h"

#! /bin/bash

# No matter how it wraps on your screen, this next line starts with
# "for" and ends with "xargs`"

for hfile in `find /var/lib/apt/lists -type f -exec grep '^Package: ' {} ';' | awk '{print $2}' | grep -e '-dev' | xargs`
do
apt-file list $hfile | grep "\.h$"
done

Be warned, this takes a long time, since every call to apt-file goes out over the network. On my system, with FIOS broadband, it took over an hour. So you'll only want to run this file once in a while, probably only when you update your repository list. You may also get an error message like
grep: /var/lib/apt/lists/lock: Permission denied
Don't worry about that.

This script should work pretty well for Debian. There are also equivalent commands for RPM-based distributions. And, of course, you can modify it to list any subset of files in any subset of repositories you want. I'll leave all of that as an exercise for the reader.

Nature Publishes Science Fiction

2010-10-28T20:10:00.000-04:00

As in The Greatest Science Fiction Story Ever Told, by Eric James Stone.

Nature 467 , 1146 (28 October 2010) doi:10.1038/4671146a

An Epix Decision

2010-10-23T15:13:00.000-04:00

We've had Verizon FIOS for nearly three years and have been pretty happy with it. So last week we finally decided to drop our final link to twentieth-century audio-visual technology, the copper-wire phone line, and rely on the FIOS line and cell phones.

Since this meant we were going to be using FIOS for a while, I decided to call up Verizon and see what kind of a deal they could give me if we locked in for a couple of more years. Primarily, I was looking to save money and add Showtime, so that we could watch Dexter, Weeds, and Nurse Jackie without waiting for the DVDs to get to Netflix. (Yes, these are all shows about, shall we say, socially challenged individuals. Deal with it.)

Well, I didn't save any money — well, I will if we drop HBO, but otherwise not. But such a deal!. In addition to Showtime, we get all of the HD sports channels turned on, the regional Fox sports channels, and some other HD movie channels.

In particular, we now have something called EpixHD. This is a joint venture by Paramount, MGM, and Lionsgate to make a few more dollars off of their movies before Netflix buys about a million DVDs and starts sending them out essentially for free.

So here's the deal. Epix streams movies such as Star Trek, Iron Man, etc., starting about 90 days before Netflix gets streaming rights. In addition, Epix streams to a Linux computer, unlike Netflix.

Furthermore, after a little bit of searching I found that HBO also streams its shows, even to a Linux box, although I had to do the sign-up from Windows.

And, finally, even Showtime is coming around to streaming its shows, although right now it's apparently only available to Comcast customers through something called Fancast.

So here's the dilemma: I can watch movies and shows from HBO, Showtime, and Epix using Verizon's OnDemand service. I can also stream HBO and Epix to my Linux, Mac, or Windows boxes. So do I really need to keep Netflix?

Look, I love the fact that Netflix lets me watch ancient episodes of Dr. Who and things like Doctor Horrible's Sing-Along Blog, but I really don't watch it all that much. It's useful that I can get a copy of KU's 1988 NCAA championship DVD when I want it, but, really, how often is that? Is it worth another $10/month?

And, really, I'm upset with Netflix, which won't consider streaming to Linux. They say it's a DRM issue, but if the studios that make up Epix can solve that to their satisfaction, why can't Netflix?

For now, I guess, we'll keep Netflix, but we'll see for how long.

Remote Login to the Command Line

2010-10-02T13:51:00.000-04:00

The other day I was away from home when my wife called up and asked me where I'd put the papers I'd promised to print out.

I let out a very soft damn, because of course I'd forgotten to print out anything.

But all was not lost. I simply logged onto my home computer, found the appropriate file, and printed it, on my home printer. She was suitably impressed.

This, of course, is the whole point of a long-ago post, where we figured out how to determine a router's external IP address. What we want to do today is to use that information to log onto one of your home computers from afar. We'll assume you have figured out your current dynamic IP address, perhaps by looking at the bottom of your gmail page, or using the software I assembled in the previous post.

So let's say that you've found that your router currently has the dynamic IP address aaa.bbb.ccc.ddd . Then, in principle, all you have to do to log onto your account at home would be to open up a terminal and type

ssh rcjhawk@aaa.bbb.ccc.ddd

where you'd replace rcjhawk by your own account, of course. Then, if you have ssh set up properly, you'll be logged to a CLI on your home machine.

In the usual home setup, with many computers connected to a router with only one external IP address, this will never work out of the box. First, the router doesn't (usually) do ssh on its own, and if it did there wouldn't be any useful information for you there, unless you wanted to ~~trash~~ fix your own network. Second, the router doesn't know which machine you want to talk to. And third, even if it did, that machine might not be accepting ssh requests (and won't, if you've followed my previous recommendation).

So what do we need to do? Let's see: First, we need to tell the router which computer gets the ssh session. Then, we need to set that computer up to receive ssh commands. Finally, if we want to be able to contact multiple computers, all behind the same router, we have to figure out how to do that, as well.

Setting up the router

First things first: how does the router pick a computer for ssh? Well, though Wikipedia tries to explain it in technical detail, basically your computer connections to the outside world are controlled by a set of 65536 (or 2¹⁶) ports. If you like, you can consider each port as a connection in an old-style telephone switchboard. When you call up a specific service on a computer, you ask to be hooked up to a specific phone number.

Some of these phone numbers, or ports, are customarily assigned to specific services. For example, port 80 is usually reserved for a web server. It doesn't have to be that way, though, you could start your web server running on port 31526, if you liked. The problem is that everyone's browser would be looking for your server on port 80 of your computer, and you wouldn't get many hits. (If that's what you desire, go for it.)

ssh, it turns out, is typically assigned port 22. So when I run

ssh rcjhawk@aaa.bbb.ccc.ddd

the computer I'm on asks the operator (my home router), to connect it to port 22 so that it can talk to whoever is there. If that contact speaks ssh, then we can communicate. If, however, the contact is speaking telnet, or ftp, we're going to have problems.

The router, as we said above, knows nothing about ssh, or at least doesn't want to talk to the unwashed masses of the Internet. It has to pass the call on to a computer that I've designated to receive ssh requests. This works by what is known as Port Forwarding. This tells that router that whenever someone comes calling at port 22 it is to pass that signal along to a specific computer.

How do you do port forwarding? Well, that depends on your router, your firewall, if any, and maybe on your modem. I can't give general directions, though PortForward.com might be able to help. There is also a YouTube instructional video, and you ought to be able to find specific instructions for your own router/firewall.

Since I'm on Verizon FIOS here at the house, I have a Verizon MI424WR modem/router. There is an old set of instructions for port forwarding online, but the software seems to have changed. That did provide a starting point, however, and I've figured how how to set up forwarding on my current setup:

Point your Browser at http://192.168.1.1/
Click Firewall at the top of the page.
Click Port Forwarding on the left.
You see a line that says Create new port forwarding rule: and below that a box that says IP Address forward to or select from menu. Click on that, and then click on the machine you want to pipe your ssh requests to.
Click on the box labeled Application to forward … and select SSH. When things settle down, click apply.

Setting up your home machine

This is fairly straightforward. Let's assume that we've instructed the router to forward port 22 to hal, here. On hal, we've followed all the instructions to get the ssh daemon running, so that we know what to do with incoming ssh requests.

Currently, however, we have the line
ALL: ALL
in our /etc/hosts.deny file, and
ALL: 192.168.1.0/255.255.255.0
in /etc/hosts.allow, which means that only computers on the local network can ssh to hal. One way to fix this would be to add
sshd: ALL
to /etc/hosts.allow. This allows hal to receive ssh requests from any machine, anywhere. Another way to do it would be to specify those external machines that can access hal by host. For example, if I worked for Google, the line might read:
sshd: 72.14.0.0/255.255.0.0
or some-such, letting me log in from any computer with an ip number 72.14.xxx.yyy.

Once we have your home computer's /etc/hosts.allow file properly set up, then running
ssh youraccount@aaa.bbb.ccc.ddd
from an allowed machine, where youraccount is the account name on your home machine, and aaa.bbb.ccc.ddd is your router's current IP address, should get you in, at least after you enter the appropriate password.

Other machines on your network

The problem with all of that is that I can only access hal by this method. Suppose I want to access another computer, fred, instead. I could, of course, get into my router and change port 22 forwarding from hal to fred, but that's difficult to do from the road, unless I allowed my router to be administered remotely.

However, the port/service assignment is more of what you'd call a guideline. We can, in fact, use any port in a storm for ssh services. In the example above the Verizon router had presets for ssh using port 22, but we could have used any other port instead.

Again, the exact way of doing this will vary from router to router, but to set up forwarding of an arbitrary port to a machine with the Verizon router,

Point your Browser at http://192.168.1.1/
Click Firewall at the top of the page.
Click Port Forwarding on the left.
Under IP Address … click on your target machine.
Under Application click Custom Ports. When things settle down you'll have a bunch of boxes to click.
Under Protocol select TCP.
Under Source Ports select Any.
Under Destination Ports select Specify, and then put in your selected port. In this case we'll use 1022. We don't have to use 1022, but it's simple to remember, once we know that ssh ordinarily uses port 22.
Click on Apply and you're done here.

Now go back to your target machine. Open up a terminal window. Then,

Run
sudo vi /etc/hosts.allow
and edit the file to allow sshd access from your favorite machines, just as you did with the first machine.
Run
sudo vi /etc/ssh/sshd_config
Look for a line that says Port 22
Below that line add the line
Port 1022
or whatever other number you've chosen.
Stop and restart the ssh daemon by running the command:
sudo /etc/init.d/ssh restart
Now go over to your remote machine. Run
ssh youraccount@aaa.bbb.ccc.ddd -p 1022
if 1022 is the port you've chosen. You should be logged on to the second machine in your network.

That concludes the tutorial on accessing your home machines' command lines. Ubuntu also has a remote desktop feature, which I've yet to use. When I get around to trying it out we'll put that online here as well.

Bum-Bum-Bum-Bum-BUM-Bum

2010-09-25T15:52:00.002-04:00

I liked Boomer better than I like Kona.

Which is a short way of saying that if you're watching the re-make Hawaii Five-0 in order to see more of Grace Park, you're better off getting all the Battlestar Galactica disks from Netflix and watching them. Boomer/Sharon/Athena/Eight was interesting and sexy. Kona looks like a school girl. A painfully thin school girl.

To put it mildly, I'm unimpressed by the whole Five-0 experience. At the start, they do keep the same theme song, how could they not, but they make the images flash way too fast, and cut it short as well.

Then there's McGarrett. Not my idea of a NCIS/Seal/Cop. Well, maybe a Don Johnson-style cop, since he always needs a shave. But Alex O'Loughlin is no Jack Lord. Or Mark Harmon, for that matter.

I did like Danno. The first thing I thought was He reminds me of James Caan. Then I look up Scott Cann, and find that he's James Caan's son. This is one improvement over the old show, the old Danno was a wimp.

And I love the makeup work they did on Jean Smart, who plays the governor. She's my age, but they managed to make her look old.

OK, this is a meander, and not a review, but lets see — what was good and bad about the episode itself:

Good: There's an underlying mystery here, something only hinted at in the first episode. Sort of like in last year's The Good Wife, where we learn that there's some deep dark secret in Chicago politics (who knew?)

Bad: Jean Smart's roll is apparently going to be along the lines of:

McGarrett: You gave me unlimited authority!

Governor: And you exceeded it!

Implausible: McGarrett and Danno attack what is identified as a Chinese cargo ship. While there, they are attacked by a variety of bad guys, including the episode's main villain. They win easily, of course. In the entire scene we see not one person of Asian extraction.

I'll probably watch a few more episodes, as time permits, either on Hulu or with FIOS On Demand, but this isn't going to be one of those shows that's automatically DVR'd.

Ubuntu and Flash

2010-09-24T23:44:00.009-04:00

I've been running the 64-bit version of Ubuntu 10.04 for about five months, and have had few problems. Most of them you've heard about. Lately, though, I've had a major problem — Ubuntu's unbranded version of Google Chrome, aka chromium-browser, wouldn't run Flash.

If I tried to visit a flash site, I'd get something like this:

Annoying.

The apparent cause is a corrupted profile. Exactly what's corrupted, I couldn't say. Basically, to get Flash running again you need to wipe out all traces of chromium and start over. I did this, but I wanted to at least save my bookmarks. Here's how. Open a terminal and type in everything to the left of the # signs:

cd # Go to your home directory
mv .config/chromium chromium_old # save your old configuration, at least for now
chromium-browser # start the browser up.  Exit immediately.
cp chromium_old/Default/Bookmarks* .config/chromium/Default # restore you old bookmarks
chromium-browser # surf normally.

Of course, you've lost all your saved passwords, surfing history, etc. Think of it as a way of weeding out the chaff. A project, for someone who has the interest and time, would be to go through each file in the old profile and determine which one got corrupted. If you do that, feel free to let us know what you find.

Back to Blogging

2010-09-18T12:12:00.000-04:00

Well, not really, just another placeholder/bookmark entry. It was a busy summer, and now I'm coming up on Fiscal New Year's Day.

From our Department of Procrastination: I owe y'all a follow-up to Finding Your External IP Address, where I show you how to use the thing, i.e., log onto any computer in your house from anyplace else in the Universe. This is half-written, so maybe in a week or two it will actually get posted.

Book Reviews: I really want to mention the greatest history book ever written, Lies My Teacher Told Me by James Loewen. I lost the first edition long ago, but a second, substantially rewritten, edition is now out. Find it, buy it, read it. You won't agree with everything, but you'll agree that it's relevant.

Another one: Human Smoke, by Nicholson Baker. It's just a collection of quotations and stories from the end of World War I to the end of 1941. No one, and I mean no one, comes off well here. None of your heroes, and certainly none of your villains. You'll have to read it slowly, because it's just too damn depressing to read all at once. But read it. It covers that part of history that Loewen complains is left out of history books.

Tips and tricks: A recent update to my Chromium browser (Google Chrome, if you're using a branded version) left me without a bookmark toolbar. Annoying. Reading through this entire comment, and noting that I don't seem to have an option for this under the wrench, I found that Ctrl-Shift-B toggles the toolbar.

We'll get back to real live articles Real Soon Now. I hope.

Virtual Windows

2010-09-05T19:09:00.000-04:00

Or virtual anything else: I've been playing around with the open source edition of VirtualBox, a utility from Oracle (boo!) that lets you run another operating system in a Window in your current operating system.

Any host operating system: Linux, Windows, OS X. Any target OS: Linux, Windows, OS X — well, maybe you'd better ask Steve about that. Tell him I said it was OK.

The open source version of VirtualBox is included in the Ubuntu repositories, so install it with:

sudo apt-get install virtualbox-ose

On your Gnome menu you'll find and entry for VirtualBox OSE under Applications => Accessories

Click New and follow the directions. I was able to install Windows XP without the usual hassles. VB provided all the needed drivers.

OK, one problem. The virtual machine wouldn't boot. The problem turned out to be that VB was looking for a floppy disk drive, which Hal doesn't have. I had to click Settings => System, and then unclick the Floppy Drive label under Boot Order. After that, everything worked fine.

I was even able to stream Netflix movies in a browser under Windows, once I installed Silverlight.

Useful for the very few times I need Windows.

Updating the un-updatable

2010-07-29T18:31:00.000-04:00

In my post on the relative easy of updating Linux I used Ubuntu's graphical Update Manager to do the work.

Of course, in Linux/Unix you can always do this from the command line. In Debian based systems the relative commands are:

sudo apt-get update
sudo apt-get upgrade

The first line compares the packages available online with those installed on your machine, and the second line upgrades the packages that aren't current. Easy, huh?

Except that sometimes you get messages like this:

The following packages have been kept back:
  linux-generic linux-headers-generic linux-image-generic

This is more or less a are you sure you want to do this now? message. You'll note that the kernel is involved here, so you're going to have to reboot the machine once the packages are installed.

All right, I'll reboot, so what do I have to do? Well, it turns out the the packages can be explicitly installed:

sudo apt-get install linux-generic linux-headers-generic linux-image-generic

This will take care of everything. Just remember to reboot the computer to get the new kernel up and running.

This Week's Sign of the Apocalypse

2010-07-26T19:51:00.000-04:00

Since time began, classes at the University of Kansas have ended with the sound of a steam whistle which tooted for five seconds at 50 minutes past the hour.

As a budget cutting measure, the steam whistle, Tootie-Toot (never heard that name when I was there), now only sounds for 2.5 seconds.

The 2.5 second version is at the end of this clip:

The Update Log

2010-07-24T14:48:00.000-04:00

I've spent most of July out of town/state/country, so I haven't been very diligent about updating all the computers. Today, though, I took a stab at it:

I. Windows Vista

We have a Windows Vista desktop. It's slated to be upgraded to Linux after Child₂ moves files off of it to the new Macbook. For now, though, it runs several programs that said child uses, e.g. Photoshop, and has a full set of browsers (IE, Firefox, Chrome, Safari) that I use to judge how new web pages look to the masses.

OK, here's everything I needed to do to update that machine:

Update Virus Definitions: We have McAfee installed, primarily because we bought this machine before I knew about Microsoft's Windows Security Essentials. It's not smart enough to know that it should autoupdate virus definitions if the computer's been off a long time, it just gives you a Dire Warning of Impending Doom.
Update Firefox
Update Thunderbird
Update Google Chrome
Update Safari: This one popped up on its own, offering updates to both Safari and iTunes.
Check Flash Version: Up to date
Perform Microsoft Updates: At least these were done in the background, all I had to do was reboot.
Reboot: Note that three (3) programs asked for permission to reboot: Windows itself, Apple's Safari/iTunes, and McAfee
Run Virus Scan: otherwise it wouldn't do it until late tonight.

II. Mac OS X

This is my work laptop.

Run Software Update
Run Microsoft Autoupdate: to check if any Office components needed to be updated.
Update Firefox
Update Thunderbird
Update Google Chrome
Check Flash Version
Check for updates to Aqua Emacs
Run sudo port upgrade: to check for updates to Macports programs.
Virus check: I'm required to run a virus checker by the powers that be, never mind that the next Mac virus in the wild will be the first I'm aware of. Fortunately, on the Mac McAfee is smart enough to do all its work in the background, without the warnings.
Reboot to take care of the system updates

III. Linux machines

System > Administration > Update Manager > Check > Install Updates: then wait a bit
Reboot: to activate the new kernel

Now. Will some one please remind me how much easier it is to run Windows or Mac OS X as opposed to Linux? Because sometimes I forget.

Why People Don't Read History

2010-07-21T18:55:00.000-04:00

It's just not believable.

Stolen from a comment in Contrary Brin.

I've been on vacation. More actual Linux stuff this weekend, I hope.

Finding Your External IP Address

2010-06-26T13:05:00.005-04:00

For various reasons you might want to find your public IP address, that is, the 32-bit number, expressed in pseudo-decimal form as something like 72.14.204.99, that tells the world how to reach your computer.

For most businesses and government offices this is trivial: your computer is assigned a static IP address, one that never changes with time. For home users, though, it's a little more complex.

Start with the IP address itself. The largest possible IP address is 255.255.255.255. That means that you can have no more than 4,294,967,296 unique IP addresses, and some of those are reserved for local use, as we'll see. We're coming up on 7 billion people in the world. Not everyone has a computer (yet, and remember iPhones, iPads, Kindle, and all 'Droids count), but many people have more than one (see iPhones, iPads, …). So there aren't enough IP addresses to go around. Eventually, everyone will switch over to the IPv6 address scheme, which will give us 340,282,366,920,938,463,463,374,607,431,768,211,456 (2¹²⁸, or 3.4×10³⁸) possible addresses, which is probably enough for every non-hydrogen nucleus in the observable universe to have its own address.

But we're not there yet, so something has to be done to shrink the address space. Your local ISP solves this problem by 1) charging you extra to get a static IP address; 2) giving you only one IP address per gateway (your cable, FIOS, DSL, or dial-up modem, or some other router or hub that connects you to the internet); and 3) taking your IP address away from you when you're not using it.

That last one is called dynamic IP addressing, and it's part of what we're going to talk about today. If you have a dynamic IP address, then when you turn your gateway connection (usually some type of modem or router) on, your ISP assigns you an IP address from its bank of addresses. As long as your connection is turned on, you'll have an IP address, but your ISP can change it at any time. It will almost certainly change it if you turn off your connection and turn it on again. So if you know your IP address right now, there's no reason to expect it will be the same tomorrow, or even five minutes from now — though to be fair, Verizon FIOS has kept our house on the same IP address for weeks.

And there is worse to come: Suppose your ISP has given you the address 244.117.16.25. The computer you're reading this on doesn't have a clue as to what that address is. Go back to 2), above. Your house has only one IP address, and it belongs to your gateway. Every other computer in the house belongs to your home network. When you want to download your email, your computer politely requests the gateway to go fetch it:

Please, sir, may I have my email?

Oh, all right. Just this once. But don't tell any of the other computers what we're doing.

Thank you, sir! I'll be eternally grateful

You'd better be, or no Internet for you.

Let's see how this works. ping is a command that will poke a stick at a computer and get information about it. Let's try it with a well known site:

ping google.com
PING google.com (72.14.204.99) 56(84) bytes of data.
64 bytes from iad04s01-in-f99.1e100.net (72.14.204.99): icmp_seq=1 ttl=53 time=21.8 ms
64 bytes from iad04s01-in-f99.1e100.net (72.14.204.99): icmp_seq=2 ttl=53 time=20.2 ms
64 bytes from iad04s01-in-f99.1e100.net (72.14.204.99): icmp_seq=3 ttl=53 time=23.4 ms
^C

(you'll need to hit ctrl-C to stop this)

So we see that google.com has an IP address of 72.14.204.99 — that's actually one of many, but don't worry about that. Now let's try it with a local machine:

ping george
PING george.home (192.168.1.2) 56(84) bytes of data.
64 bytes from george.home (192.168.1.2): icmp_seq=1 ttl=64 time=9.77 ms
64 bytes from george.home (192.168.1.2): icmp_seq=2 ttl=64 time=2.70 ms
64 bytes from george.home (192.168.1.2): icmp_seq=3 ttl=64 time=0.961 ms
^C

So far has the home network is concerned, george lives at 192.168.1.2, which is one of those reserved addresses we mentioned above. The range 192.168.0.0 to 192.168.255.255 is reserved for internal networks. As we read this, there are no doubt hundreds of thousands of computers with the local IP address of 192.168.1.2. Every one of them, however, has a different public IP address.

Great. But sometimes I need to know my external IP address, and it's apparent that Hal, here, hasn't got a clue as to what it is. How can I find it?

The easiest way is to go to another website. Every website you connect with gets your public IP address automatically, unless you use some kind of anonymous service. Some websites are set up to tell you what your IP address is. One example is What Is My IP?. They even provide a way to get your IP address with a simple one-line terminal command:

wget www.whatismyip.com/automation/n09230945.asp -O - -q ; echo

If you read your Gmail via Thunderbird, Evolution, Outlook, etc., and keep your web client on all day, gmail will tell you the address of your computer. Just log on to your gmail account and look for the line that says Last account activity: …. The IP address there is the last computer to download email. If you're using more than one computer from different locations, say you left your office machine on over the weekend and now you're using a home computer, click on the Details link and you'll find all of the activity for this account.

There's a company DynDNS, which will not only find your IP address, but give you a domain name that constantly updates to your current numerical address. Of course, they do this by having your computer constantly tell them the public IP address.

And, of course, your gateway knows its public IP address. Just log onto it through your browser (something like http://192.168.0.1, http://192.168.1.1, or http://192.168.100.1, depending on your system), and your IP address will be up there someplace.

But all of these systems have flaws: Say you need your IP address in a script. You can get it via your browser from your gmail or your gateway machine, but it's hard to parse a browser page inside a script. The What's My IP? one-liner works in a script, but what if their site goes down, or, worse, they go out of business? Same for DynDNS. We want a solution that works any time your gateway is on and your computer has an internet connect, and we want to get it from the command line.

Any other way just ain't elegant.

If you search the Internet you find surprisingly few ways (i.e., none) to do this. Partly I think it's because there isn't a consistent way to find your gateway from your current machine. For example, in the days when we used Comcast, we had a cable modem as the gateway. On the house side of the modem we connected a Linksys router, which was the hub of our home network. All the computers in the house were connected through the router to the modem — a two step-process. Now we have Verizon FIOS, with a Actiontec MI424WR router that serves as gateway/modem and router — a one-step process.

By stumbling around, and with a little help from nixCraft, I was able to come up with a solution that works for me. It's a bash script. Let me put it here (finally, I hear you say), and we'll dissect it below:

#! /bin/bash

# Determine the public IP address for your DHCP system by bootstrapping

#  calls to your gateway/router

# Requires "route," available in Ubuntu package net-tools, and
# nslookup, available in Ubuntu package dnstools

# Disclaimers:
# 1) This works with Verizon FIOS using the Actiontec MI424WR Router
#    If you have another system YMMV.
#
# 2) And, as you'll no doubt note, this is set up for a Debian system,
#    specifically Ubuntu 10.04 .  Package names and installation
#    procedures for other systems WILL vary.
#

###
# Start Here:
###

#
# Get Router internal Gateway IP
# Information found on
# http://www.cyberciti.biz/faq/how-to-find-gateway-ip-address/
# If route is not installed, run
# sudo apt-get install net-tools
# and make sure /sbin is in your $PATH
# Note "U" = up, and "G" = gateway

internalip=`route -n | grep UG | awk '{print $2}'`

echo Your Gateway\'s Internal IP Address is: $internalip

# Now we need to get the name of the router.  If you don't
#  have nslookup installed, run
# sudo apt-get install dnsutils

echo -n "Your Gateways's name is: "

routername=`nslookup $internalip | grep "name =" | awk '{print $4}'`

# On the Actiontec, at least, the name has an annoying "." at the end.
# Remove it:

routername=${routername%\.}

echo $routername

echo -n "Your Public IP Address is: "

nslookup $routername | grep Address | tail -1 | awk '{print $2}'

OK, what's going on? First of all script only works if you have the route and nslookup commands installed on your system. If they aren't there, in Ubuntu, and presumably and Debian system, you can install them using

sudo apt-get install net-tools dnsutils

The first call, route -n, will look at the Kernel's IP routing table:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     0.0.0.0         255.255.255.0   U     1      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth0
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 eth0

which on this machine tells us that we have several local connections. The U under Flags means that the connection is up, and G means it's the gateway. From this we see that the numerical address of the gateway is 192.168.1.1. Let's see what nslookup can tell us about that. Note that the script has to parse all of the output to get this far.

$ nslookup 192.168.1.1
Server:  192.168.1.1
Address: 192.168.1.1#53

1.1.168.192.in-addr.arpa name = Wireless_Broadband_Router.home.

Here's something new, our gateway's got a name. Wireless_Broadband_Router.home. Again, the script has to do a little work to tease that name out of nslookup's output, but we'll just do it by hand here:

$ nslookup Wireless_Broadband_Router.home
Server:  192.168.1.1
Address: 192.168.1.1#53

Name: Wireless_Broadband_Router.home
Address: 192.168.1.1
Name: Wireless_Broadband_Router.home
Address: xxx.yyy.zzz.aaa

The first address, 192.168.1.1, is just the router's internal IP address. The last address (Of course I'm not giving you my IP address. I'm not daft, you know.) is the one we want. xxx.yyy.zzz.aaa is the external address of this machine.

Of course there are many, many, problems with this script.

So far it only works with the Verzion/Actiontec setup. A friend, running Ubuntu, wasn't able to pick out his gateway machine with the route -n command.
Some machines can have multiple gateways.
The Mac's route command doesn't work in the same way as under Linux, so you can't parse out the gateway, even if you have none.
You might be able to tease the gateway IP address out of the /etc/resolv.conf file. I don't know what happens if you have multiple gateways here. Under Ubuntu, the one numerical gateway is 192.168.1.1, on the Apple it gives me 192.168.1.1 and xxx.yyy.zzz.aaa, my public IP address.

If you try the script, leave a comment below. Let me know your internet setup (gateway/router), if it works, and any tweaks you did to make it work. Maybe we can come up with a universal script.

Now, by all that's geekly, why did we do all of this? That's the subject of an upcoming post, boys and girls …

Smart ssh tricks

2010-06-19T13:36:00.001-04:00

I want my computers to talk to one another.

More specifically, I want to be able to remotely log in to one of my computers from another one, and I want to be able to easily copy files from one computer to another. The easiest, most secure, way to do that these days is to use the Secure Shell protocol, aka ssh.

Examples: I have an account on a computer called grumpy (sleepy and doc were already taken). To log onto grumpy from hal, I open up a terminal window and type:

$ ssh rcjhawk@grumpy

The program then logs me onto grumpy, where I can access the system just as I can for any desktop machine.

Or, say, I have a file at /home/rcjhawk/Documents/strasburg_pictures.ppt on grumpy. To copy it over to Hal, I use the secure copy program, scp:

$ scp rcjhawk@grumpy:Documents/strasburg_pictures.ppt .

which, obviously, works pretty much like the regular copy program except that I have to specify the machine and account name right up front.

And, finally, I can send files the other way:

$ scp The_Hard_Times_of_Zach_Greinke.txt rcjhawk@grumpy:Documents

which sends that regrettable file into the Documents folder on grumpy.

Except:

You will probably find that this doesn't work out of the box in your Ubuntu setup. You'll get a message that looks something like this:

ssh: connect to host grumpy port 22: Connection refused

That's because Ubuntu ships ssh in two parts:

openssh-client: this lets you use ssh and scp from your machine to log onto another machine. This is installed with the standard Ubuntu installation, and is all you need if, for example, you want to log onto your work computer from home.
openssh-server, on the other hand, allows other machines to log onto your personal desktop machine. This is not installed by default in Ubuntu because it's an obvious security hole unless you know what you're doing. If you do know, or assume that you will learn, install the package with

$ sudo apt-get install openssh-client

Once you do that, sshd, the daemon that controls remote access, will automatically start up, and will start up every time you reboot the machine. Then you're good to go.

In the example above, grumpy needs to have openssh-server installed and running, but hal does not.

Note for Mac Users: You get the equivalent of openssh-server by going to System Preferences => Sharing and clicking Remote Login.

Security

ssh encrypts the transmissions between machines, but it also has other security features. The first time you try to log onto a machine, for example, you'll get a message that looks something like this:

$ ssh rcjhawk@grumpy
The authenticity of host 'grumpy (192.168.54.42)' can't be established.
RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
Are you sure you want to continue connecting (yes/no)?

If you answer yes, then you'll get the message:

Warning: Permanently added 'grumpy,192.168.54.42' (RSA) to the list of known hosts.

After that you'll be able to log onto grumpy without the warning message. If you want to start over, i.e. not recognize any computer as being previously visited, erase the file $HOME/.ssh/known_hosts

You can also limit access to the machine by editing your /etc/hosts.deny and /etc/hosts.allow files. As I've noted before, if you don't want ever Tom, Dick, and Susie to have a chance to access your machine, your /etc/hosts.deny file should read:

ALL: ALL

after which only addresses listed in your /etc/hosts.allow file will have access to the machine. In my /etc/hosts.allow file I currently have

ALL: 192.168.1.0/255.255.255.0 ,

which allows all machines on our local network to access grumpy.

OK, everything is good, right? Well, not quite. As things are set up right now, every time you ssh or scp to grumpy you must enter a password. This isn't a big annoyance with ssh, because presumably you will keep the remote login running for a relatively long time. However, it is an annoyance with scp, if you want to copy a large number of files from different places, requiring a large number of scp commands.

The solution is to use ssh-keygen to generate a a key. Here's how it works:

Start the program:

$ssh-keygen -t dsa Generating public/private dsa key pair. Enter file in which to save the key (/home/rcjhawk/.ssh/id_dsa):

We'll use the default directory location, so just hit return here.

Enter passphrase (empty for no passphrase):

Frankly, I don't know anyone who does use a passphrase. It does protect your key against someone who's gotten access to your account, but at that point you're probably hosed anyway. So for now we'll just hit return.

Enter same passphrase again: Your identification has been saved in /home/rcjhawk/.ssh/id_dsa. Your public key has been saved in /home/rcjhawk/.ssh/id_dsa.pub.

Now let's take a look at the .ssh directory:

$ ls -la $HOME.ssh
total 16
drwx------  2 rcjhawk rcjhawk  4096 2010-06-19 13:04 .
drwxr-xr-x 34 rcjhawk rcjhawk  4096 2010-06-19 13:01 ..
-rw-------  1 rcjhawk rcjhawk   672 2010-06-19 13:04 id_dsa
-rw-r--r--  1 rcjhawk rcjhawk   599 2010-06-19 13:04 id_dsa.pub

Here you'll notice two files. One, id_dsa, is your private key. The second, id_dsa.pub, is your public key. Key the private key private. Don't let anyone have access to it. Don't copy it over into the .ssh folder of another account, or to another computer. You'll note that both the .ssh directory and id_dsa are set so you are the only person who can read those files. Keep it that way.

The public key, now, that's another matter. If you place id_dsa.pub into the file $HOME/.ssh/authorized_keys on another computer, you'll be able to access that computer from your home computer. Here's one way to to it (I'm leaving out all the password prompts):

$ ssh rcjhawk@grumpy
Welcome to grumpy!
rcjhawk:~ $ cd .ssh
rcjhawk:~/.ssh $ scp rcjhawk@hal:.ssh/id_dsa.pub tempid
rcjhawk:~/.ssh $ cat tempid >> authorized_keys
rcjhawk:~/.ssh $ rm tempid
rcjhawk:~/.ssh $ exit

What did we do? We logged into grumpy, copied hal's public key over to grumpy's .ssh directory, and added it to the end of the authorized_keys file. I renamed the file during the copy so as not to write over grumpy's own id_dsa.pub file, if one was there. Notice that scp was able to get that file even though Hal's .ssh directory is copy protected, because you gave the system the right password.

The next time you try to ssh or scp from hal to grumpy, ssh will find the public key on grumpy, compare it to the private key on hal, and conclude that you are, indeed, allowed to log in or transfer files.

The discerning reader will note that I'm only applying this to my local network, which all sits behind a router and firewall. How, you may ask, can I use ssh to log onto my local machine from outside my home?

Well, I wish I could tell you, but at the moment I can't. It's something to be researched in the future. If you're reading this, and you know how to do it, post a link to the procedure in the comments.

A Sign From Above

2010-06-11T20:33:00.000-04:00

Tonight coming home from work I learned that there was a backup on US-50 East between I-97 and the Severn River Bridge in Annapolis.

I knew this, because I was given a Sign. This sign, as a matter of fact, which today had an actual message.
The sign in question is about 15 miles from the start of the backup. Not to worry, I had plenty of time to read it because I was in a five-mile backup caused by people slowing down to read the sign.

Obscene comments about Maryland drivers omitted.

Stupid Bash Tricks

2010-06-04T21:27:00.000-04:00

I grew up scripting with the C shell (csh), and I'm still more comfortable with it than any other shells. However, for really complicated scripts I find that BASH (the Bourne-again shell) is more flexible. The problem is that I don't really know how to use it all that well.

I could buy a book, I suppose, but that costs money. And anyway, much of what I want is already on the web. For example, we have

I've used all of these, but the only way to really remember something is to build an example, and refer back to it as needed. So here, presented for your consideration, are a couple of examples.

The first one just counts. It's basically here to show how to do math with variables in BASH:

#! /bin/bash

typeset -i count

count=$1
echo $count

while (( $count < $2 ))
do
 count=$count+1
 echo $count
done

Actually, that one is adapted from the Korn shell (ksh), but it works in bash. A purer bash form would be

#! /bin/bash

let count=$1
echo $count

while (( $count < $2 ))
do
 let count=$count+1
 echo $count
done

Either way, if we call the file count, then it counts from the first argument up to the second:

./count 5 10
5
6
7
8
9
10

On to another program. I often have programs that recompute a certain quantity repeatedly, say the total energy in a Density Functional Theory run. For example,

$ grep "TOTAL ENERGY=" INFO
                          TOTAL ENERGY=      -551.5802151120
                          TOTAL ENERGY=      -551.5798689965
                          TOTAL ENERGY=      -551.5810124810
                          TOTAL ENERGY=      -551.5809217653
                          TOTAL ENERGY=      -551.5796453493
                          TOTAL ENERGY=      -551.5797608209
                          TOTAL ENERGY=      -551.5806860076
                          TOTAL ENERGY=      -551.5804690663
                          TOTAL ENERGY=      -551.5793575483
                          TOTAL ENERGY=      -551.5794583810

where I only want the last line. For a single file, I could get that with

$ grep "TOTAL ENERGY=" INFO | tail -1
                          TOTAL ENERGY=      -551.5794583810

but I want to look at multiple files, make a note of the file I'm looking at, and extract the last entry. So I wrote this little script, which I call lastgrep. Among other things, it shows you how to use the shift command to delete elements from the calling string.

#! /bin/bash

# usage

# lastgrep string  file1 file2 file3 file4

# finds the last occurrence of string in each of the files
#  listed in the argument list.  If string contains white space
#  it should be quoted

string=$1

# Pop the first argument (string) off the stack.  The
#  remaining arguments should be files

shift

for file in $@
do
    echo -n $file

#    Put quotes around $string to keep white space in place

    grep "$string" $file | tail -1
done

So if I have a set of files INFO in directories a100, a200, a300, I might do something like this:

$ lastgrep "TOTAL ENERGY=" a*/INFO
a100/INFO  TOTAL ENERGY=      -551.7810244761
a200/INFO  TOTAL ENERGY=      -551.6774784178
a300/INFO  TOTAL ENERGY=      -551.6192819148

What this doesn't have is a way of passing arguments to grep. That's for another time.

CUPS and a Wireless Printer

2010-06-01T22:55:00.000-04:00

Eric Raymond once wrote a famous essay about the difficulty of using Linux's Common UNIX Printing System (CUPS) to set up a printer over a network. Thus, when we got Child_II's new HP C4780 Wireless Printer/Scanner I was less than optimistic about getting it Hal to access it. I mean, we're talking about a wireless printer which is talking to a Verizon FIOS wireless router, and through that to a Linux computer? No way.

I should note that I wouldn't go out and buy an inkjet printer on its own, anymore, particularly an HP. But we just got Child_II a MacBook Pro (new college uses Macs, and Mac Photoshop, for art projects), and they threw in an iPod Touch (which I get to glance at longingly) and the HP (which it's my duty to set up, operate, and maintain). So we got the thing, we might as well use it. Anyway, having a wireless or networked printer for a MacBook is an essential, given that the thing only has two USB and one Firewire port.

Setting up the HP required that I use the Mac or a PC, connected by USB to the printer, to enter the network login information. Given the connection between open source software and printers this left me a little chagrined, but I'm not RMS so I let it slide. Anyway, I didn't expect the thing to work with Linux, remember?

It took about a half-hour to install the software and drivers onto a Mac, and set up the printer. We tested the printer and scanner via wireless, and it all worked fine, though the scanner was pretty slow, much slower than my USB Canon. It all worked, though.

Then it took another half hour or so to install everything on a Windows PC. Again, it all worked flawlessly.

OK, let's try our luck. I left the printer on, went downstairs to Hal, clicked on System => Administration => Printing and watched the CUPS interface come up. Hit Add, Printer, Find Network Printer. Blinked. There it was, an HP C4700 type printer. Unlike the Canon, all the drivers were already available, I didn't have to play guess-the-correct-version. Within five minutes of clicking that Printing button I was printing a test page from Hal back to the printer.

Now all is not sweetness and light. While CUPS had no problem finding the printer, SANE, the corresponding program for scanners, couldn't find the HP. Not surprising, I guess, but it would have been nice if everything worked out. Of course I'm not particularly sure what good a remote scanner does anyone, especially if, like, me, they use it primarily to scan old pictures onto a disk.

But to summarize, things have much improved since the days of ESR's essay.

And I've got to look into getting a networked color Laser printer.

Backing Up Your Blog

2010-05-31T10:26:00.001-04:00

After the discussion about creating disk backups I realized that I haven't been terribly diligent about backing up this blog. After all, what if Google should fail? Thousands of words of my deathless prose might be lost. (I know I've written some unforgettable things. Just last week I got three emails telling me how my posts had affected them. Only two mentioned nightmares.)

So how do you back up a Blogger blog?

One way to do it is to download the monthly archive HTML. The advantage to this is that everything in in HTML, and easy to read. But you have to remember to do it. I did, for a while, but haven't done it for some years. Sloth, I know.

But there is a way to download the whole blog:

Log onto your Blogger account.
Go to the Dashboard, if you're not there already.
Click Settings.
Click Export Blog.
Click the orange button.
A big file, ending in .xml, will appear in your download folder.

Note that the xml file is hard to read, but if you're moving your blog from one account to another it's the easy way to go. Also note that this does not save pictures or comments, those linked (the HTML archives don't save the pictures, either). So you'll have to find that stuff yourself.

This doesn't save your blogger template, either. To do that, go back to the dashboard, click Layout, then Edit HTML, and Download Full Template. Important if you have witty sayings embedded in you blog header.

Also note that either the HTML or XML backups are useful for searching your blog, because that Google search box in the upper left doesn't always find everything you've written.

Return With Us Now to the Thrilling Days of Yesteryear

2010-05-30T16:06:00.001-04:00

I'm not very good at doing disk backups.

OK, let's be honest: I'm terrible at doing disk backups. Oh, back in the day, every once in a while I'd copy as much of the disk as I could onto CDs or DVDs. But that takes a lot of time, and, as disks become larger while DVDs don't, requires a lot of picking and choosing to figure out what goes where.

To be fair, I haven't needed to. In all the years I've run computers here at home, I've never had a disk crash that lost data, even after our most serious disk mishap. So not having a backup hasn't been that big a problem. (Insert your favorite ominous foreboding music here.) Anyway, backups are a pain in the neck.

My view of backups has changed, though, primarily because they're so much easier now. Easiest, of course, is Apple's Time Machine. Plug a second disk (USB, Firewire, internal, if you've got such a thing) into your Mac, let it format to Apple's HFS format (or not), and start Time Machine. Tell it which directories you want to back up, and it will do that, every few minutes. Then you can actually scroll back through time (Tardis sounds not included) and pick out a file you might have deleted a year ago — assuming you were running Time Machine back then. It's an extremely neat and easy-to-use utility, and takes the pain out of doing backups, as long as you remember to leave the external disk plugged in, if that's what you've got.

Then, last week, Office Depot ran a sale where you could get a Terabyte Verbatim USB disk for $79.99 + tax. I bought three, one for Hal, one each for the college students.

Now the disk comes with Nero backup software for Windows (taking care of student I), and Apple's got Time Machine (taking care of student II). But what about poor Hal, stuck in Linux?

I called up the Synaptic package manager and typed backup in the search box. That brought up, among other things, a package called backintime, which is a front-end which uses the utilities cron (to schedule backups), diff (to find which files are changed), and rsync (to copy files that have changed to the backup disk). Perfect. I installed the program — actually both backintime-common, the guts, and backintime-gnome, the graphical front end, then went to get the disk.

Problem: Verbatim formatted the disk to the lowest common denominator, FAT32. This is not suitable for low-volume backups, because it does not allow multiple hard links to the same file. What backintime does, you see, is create a copy of your chosen disk directories every time you schedule a backup. The trick is that unchanged files are hard-linked from one backup to the next, taking a minimum of space. Only the changed files are stored in multiple copies. Since the FAT format doesn't do multiple hard links, it can't do the trick, and so we need to reformat the disk to a better file system.

So I installed gparted from Synaptic, launched it (using sudo gparted), and formatted the Verbatim disk to ext3 format.

Second problem: If you just plug a USB disk into a Ubuntu box, it auto-mounts, but is usually only accessible to the user who did the mounting (to be fair, I haven't tried this with a USB ext3 disk). That's unacceptable in a backup system that needs to be accessible to everyone. So we have to set up /etc/fstab. To reflect this. Ideally, we'd add a line which looks something like this:

/dev/sdd /backup ext3 errors=remount-ro 0 2

(Note added after the fact: The last entry on this line should indeed be a 2, not a 1 as previously listed. This this tells fsck to check this file system after any file system with a 1 in the last column. The only file system with 1 there should be the boot partition. Everything else can be a 2, unless you don't want fsck to look at the file, in which case you put in a 0. Got that?)

create a directory entry /backup, and then doing

mount /backup

would load up the disk drive.

However, since you usually have multiple USB ports on your system, there's no guarantee that this disk is going to always get /dev/sdd as its port. The solution is to use the ext3's UUID, which was created when gparted did its thing. You find the UUID with the command:

sudo blkid
[sudo] password for hal:************
/dev/sda1: LABEL="PQSERVICE" UUID="0A72323D72322DB7" TYPE="ntfs" 
/dev/sda2: LABEL="SYSTEM RESERVED" UUID="CCA811C2A811AC48" TYPE="ntfs" 
/dev/sda3: LABEL="Gateway" UUID="7E143E7F143E3A8B" TYPE="ntfs" 
/dev/sda5: UUID="2eec3fff-73d2-419c-8a3c-b92733d46da2" TYPE="swap" 
/dev/sda6: UUID="f3781533-d891-46e1-b9f0-839e8a538d35" TYPE="ext4" 
/dev/sdb1: LABEL="Verbatim" UUID="9deee42d-539b-45c5-8683-e95f889c1792" TYPE="ext3"

Verbtim is how I labeled the disk partition of the USB drive, so that's the UUID we want. Fill out the line in /etc/fstab as:

UUID=9deee42d-539b-45c5-8683-e95f889c1792 /backup ext3 errors=remount-ro 0 1

and we're ready to go. Note that this will automount so that everyone can read it every time the disk drive is plugged in and turned on.

That settled, setting up backintime is easy. You can specify the directories you want to back up, and the file types (say, those ending in ~ or .o) and directories you want to exclude. You can then select a backup time interval: minutes, hours, days, weeks ... Once you have a set of backups, you can use the graphical interface to look through your files of any snapshot, and restore a file that was deleted or moved on your main system, or copy it to another location if you like. It's very nice and easy to use.

You can see your backup schedule using crontab:

crontab -l
@daily nice -n 19 /usr/bin/backintime --backup-job >/dev/null 2>&1

Note the @daily there. That means the backup will start every night at midnight, and it's the only choice backintime gives you if you select a daily backup. This is unsatisfactory for me, as someone is frequently using the computer on and after midnight. So I used crontab -e to edit this entry, changing it to
0 3 * * * nice -n 19 /usr/bin/backintime --backup-job >/dev/null 2>&1
which launches the backup at 3am. backintime accepts this, but note that every time you change your setup it will reset the time to @daily. Just watch it.

Anyway, now I have a disk backup. And since I've never had a major loss of disk data from one disk, having two disks copying the data should mean that it can never, ever, ever happen, right? (Insert even more ominous and foreboding musing here.)

Rob Pegoraro Talks About Ubuntu

2010-05-29T20:08:00.000-04:00

The Washington Post's tech columnist reviews Ubuntu 10.04 in his Sunday column and adds more comments in his blog.

I love the first line:

No Windows viruses. Free. Any questions?

hosts.deny

2010-05-28T16:19:00.000-04:00

Ubuntu, and most Linuxes, by default come with an empty /etc/hosts.deny file. I'd argue that by default it really should look like this:

$ cat /etc/hosts.deny ALL: ALL

Because: If you don't know what how the /etc/hosts system works, you shouldn't allow others to access your machine. If you do know how it works, then you know enough to edit the files to do what you want. Comments?

Ubuntu 10.04: Things I Forgot 2010-05-24T20:23:00.002-04:00 Things I forgot when I was listing my useful programs from Ubuntu 10.04: The gimp: Manipulate images just about anyway you want, without paying Photoshop prices. For example, you can make transparent PNG images. xsane: Scan documents (scanner not included). chromium: the unbranded version of Google Chrome. This is my current browser of choice, and I forgot to list it before because it was the very first thing I installed. Note that it's available in the repository. If you want the branded version, you can get a .deb file directly from Google's site. For some reason, Ubuntu only distributes the unbranded version of chromium, but it offers, by default, the branded Firefox browser and an unbranded version, which it calls abrowser. I'm sure there are more to come, I'll either add them here or start another post. Stay tuned. Ubuntu 10.04: Update Notification Icon 2010-05-23T09:10:00.000-04:00 One thing that doesn't seem to be present in Ubuntu 10.04 is the little notification icon that shows up on your panel when the update manager has found programs to install. If there is some notification I haven't seen it. Apparently this has been going on since 9.04, but for that version of Ubuntu I was running IceWM rather than Gnome as my window manager and so missed it. But Watching the Net didn't, and shows the way to restore the Update Icon by turning off the update notifier, whatever that is. Apparently once the new improved update notifier (which I've never seen in action) is turned off the old style notifier kicks in. This is in the province of gconf, that all-seeing, all-knowing application that tells other programs what your likes, wants, and needs are — sort of like Facebook. So to turn off the new-improved version and go back to the thing that works, open a terminal window and type (or cut and paste): gconftool -s --type bool /apps/update-notifier/auto_launch false Alternatively, launch gconf-editor, click on the + sign next to apps, scroll down to find update-notifier, click on that, and then uncheck the box that says auto_launch. Either way, you're done, and the old icon will come up the next time you have an update available. Ubuntu 10.04: Installing the 64-bit Intel Fortran Compiler 2010-05-22T13:54:00.000-04:00 OK, let's get take care of this right up front: Warning: This page discusses the installation of free-as-in-beer, but not free-as-in-speech, software. If you want to keep such software off your system, then don't install this compiler. Also note that the license is pretty restrictive, i.e. Non-commercial means you are not getting compensated in any form for the products and services you develop using these Intel® Software Development Products. However, the installation process should work for the commercial version of this compiler. I first learned to program in Fortran. I still have my original WATFOR/WATFIV textbook, though I later graduated to Fortran 77 and occasionally write some Fortran 90. I still do a lot of things in Fortran 77, just because I'm comfortable writing code starting in column 7. Now there is an open-source Fortran 77/90/95 compiler, gfortran. It's reasonably fast, portable, nice — did I mention free? But it has a few quirks. For example, one of my old codes has a subroutine second which measures elapsed time. gfortran assumes that I'm calling the intrinsic function second, and bombs the compile because I don't have a proper calling statement. So I have to go in and declare external second in every subroutine that calls my second. But Intel makes its Linux Fortran compiler free-as-in-beer for non-commercial development. And the 64-bit version of the code includes Intel's optimized Math Kernel Library (MKL). It's a fairly simple registration, and you don't sign over much of your soul. However, in Ubuntu it's a fairly difficult installation. What's more, it changes every time you install a new version of the compiler. The installation steps for the current compiler and Ubuntu are straightforward, but there's a lot of fine print, and the 64 bit version has a couple of quirks in the install. So let's go over it step by step. Note that I'm only referring to the 64-bit install in all of this. If you're running in 32-bit mode, your mileage will vary. Register. It's easy. You'll get a link to the download page an a serial number. Save the email, you'll need it. Download the appropriate tar file. I used the Genuine Intel® 64 bit version. There's another one called IA64, but I didn't try that. Now set up your machine to compile code. Run: sudo apt-get install build-essential gcc g++ rpm ia32-libs and either sudo apt-get install sun-java6-jre which requires you to accept Sun's Java license, or sudo apt-get install openjdk-6-jre-headless. Now you need a package called libstdc++5 which isn't included in Ubuntu. Worse, if you're on a 64 bit machine, you need to install the 32-bit version as well. So click on these the 64-bit code, libstdc++5_3.3.6-18_amd64.deb and the 32-bit code, libstdc++5_3.3.6-18_i386.deb Install the 64-bit version: $ sudo dpkg -i libstdc++5_3.3.6-18_amd64.deb DO NOT INSTALL THE 32-BIT VERSION. Instead: $ dpkg --extract libstdc++5_3.3.6-18_i386.deb ./ $ cd usr/lib $ sudo -i # cp libstdc++.so.5.0.7 /usr/lib32 # cd /usr/lib32 # ln -s libstdc++.so.5.0.7 libstdc++.so.5 # exit Now we can do the install. Go back and find your Fortran tarball $ tar xvzf l_cprof_p_11.1.072_intel64.tgz $ cd l_cprof_p_11.1.072_intel64 $ sudo -i # ./install.sh Take all the defaults, and enter your registration number when asked. Note that you'll get some error statements to the effect that you're running this on an unsupported system (Ubuntu) and the option 1. Skip missing optional pre-requisites [default] take that. Your compiler is now installed in /opt/intel/Compiler/11.1/072/ (the last number will vary), but you're not quite finished. If you use the bash shell, add the line source /opt/intel/Compiler/11.1/072/bin/ifortvars.sh intel64 to your ~/.bashrc file. If you use the tcsh shell, add the line source /opt/intel/Compiler/11.1/072/bin/ifortvars.csh intel64 to your ~/.cshrc file. Now you're done. Look at man ifort to see how to use the thing. My preliminary test indicates that the Intel compiler is about 10% faster than gfortran for a fairly CPU intensive calculation, without the MKL. Not all that much faster, really. If you're not doing a lot of serious computations, installing the Intel compiler might not be worth the trouble. Ubuntu 10.04: Remove the Chat and Mail Icons but keep the Volume Control 2010-05-20T22:19:00.000-04:00 I don't chat. I can find my email on my own, thanks. But I do like the volume control up on the panel. Thanks to Short IP Recipes for the fix: Remove the indicators from your system: sudo apt-get remove indicator-me indicator-messages Log out, log back in, and voilà! Ubuntu 10.04: Getting Started 2010-05-18T21:38:00.000-04:00 I thought about putting in installation instructions for 10.04, but someone beat me to it, did a better job, and added a lot more information: Getting Started with Ubuntu 10.04 Downloads a PDF manual. Disclaimer: I haven't read it all the way through, it may be chock full of errors. But it's cheaper than going out and buying Ubuntu for Dummies, and covers the current version. The “Right” way to move buttons in Ubuntu's Gnome 2010-05-17T20:00:00.000-04:00 Apparently using gconf-editor or directly editing the file $HOME/.gconf/apps/metacity/general/%gconf.xml to move the control buttons back where they belong is not the Correct Way to Do It and Will Cause Problems in the Future. Or so it says here (see #3). Despite there always being More than one way to do things^†, we here at Linux & Things are always willing to show you The One True Way. So here we go. Apparently, in the New Ubuntu Gnome, only the Ambiance, Radiance and Dust themes have buttons on the left. All the others still have buttons on the right. So if you want a Right-Wing Ambiance, say, you pick a Right-Wing Theme and edit it to Look Like Ambiance. Got that? That's right. To make Ambiance with buttons on the right, edit, say, the New Wave theme and make it look like Ambiance: Click System > Preferences > Appearance Select New Wave and click Customize Click the tabs Controls and Window Border and select Ambiance for each. Click Icons and select Ubuntu-Mono-Dark Close But don't forget The Other Way to Do It, for use when Ubuntu arbitrarily moves New Wave's buttons to the left. Ubuntu 10.04: The Good Stuff 2010-05-16T16:00:00.001-04:00 Well, boys and girls, today I was going to talk about installing the Intel Fortran compiler and MKL libraries in Ubuntu 10.04. Unfortunately, the tar files Intel sent me are coming up with archiving errors, so I'll have to wait the track all of that stuff down until later in the week. So today, let's talk about the good stuff, that is, things you can add to Ubuntu to make your computational life easier, or just more fun. Start with a little list. Not mine, at first, it belongs to the indexer, who gives us his to-do list after installing 10.04. He gives you categories of program, the Windows equivalent, and the possible Ubuntu variants. There's also a rather complete source list that will allow you to download all of those files. And then there's my list, which is a little more eclectic. All of these programs can be installed using the indexer's source list and the usual sudo apt-get install xxxx from a terminal window command line, where xxxx is the package name. Note that the links go to the program's home page, where I could or cared to find it: alien: Converts other Linux package installers, particularly RPMs, to DEB format. audacity: Record and edit audio files. build-essential: All the tools you need to compile source code and build DEB packages celestia: Planetarium program and space-travel simulator celestia-gnome: Front end of Celestia devilspie: Lets you tell programs where to open their windows on the Gnome desktop emacs23: The One True Text Editor emacs23-el: Extras for Emacs emacs-goodies-el: And even more Emacs stuff extremetuxracer: Get the penguin down the hill, while he collects fish. ffmpeg: audio and video recordings ftncheck: This is a Fortran lint checker, looking for errors in your Fortran code. gfortran: Back aways I said that build-essential had all the tools you needed to compile programs. I lied. If you write in Fortran, you need a compiler. This is the free one. googleearth: Google's mapping program. Note that this is technically non-free, you'll have to accept Google's license to use it. googleearth-data: For use with Google Earth gnumeric: For most applications I like this spreadsheet better than the one in OpenOffice.org gnuplot: Scientific plotting program. It's script-driven, so you can edit it on the fly. gthumb: Image/Picture viewer. Will import pictures from your camera. gv: Postscript/PDF file viewer. This is really old, folks, but sometimes it works where a more modern program won't. id3v2: Label MP3 file tags from the command line. k3b: Write CDs/DVDs kid3: Graphical MP3 tag editor kdegames: The KDE game package lame: Front end for writing MP3 files. See my SoX entry latex2rtf: Supposedly converts LaTeX files to Microsoft's doc-lite format, aka rtf. I've used this program maybe twice in ten years, but, hey, you never know. libk3b6-extracodecs: Addins to allow k3b to handle MP3 files lynx: Text-based web browser. The original version was written at KU mplayer: Music/video player pdksh: Public-domain Korn shell. Pretty much like bash. I have dome scripts that still use ksh, so I keep this installed. penguin-command: Clone of the Missle Command arcade game for Linux sc: A text-based spreadsheet. I keep it around because I can edit the files with a script. sox: See my Sox Post spider: My favorite double-deck solitare game sunbird: Stand-alone calendar program. If you want your calendar integrated with Thunderbird, see the Lightning plugin ssh: secure shell. Log onto other computers with encryption tcsh: The TENEX C Shell. An extension of the cshell texlive: The default TeX/LaTeX package for Ubuntu texlive-bibtex-extra: Puts your bibliography into a form LaTeX can use texlive-latex-extra: More LaTeX stuff texlive-publishers: Style files for various scientific journals texlive-science: And still more thunderbird: Mozilla's email program. I much prefer this to the default Evolution, especially since I can pass a users files back and forth between Linux and Windows machines ubuntu-restricted-extras: Fun stuff for playing with forbidden files (MP3s, DVDs, etc.) vim: Text-based full-screen file editor. This and Emacs are the first full-screen editors most early techies used. vlc: Watch just about any kind of video xmms2: Yet another music player xpdf: The original PDF file viewer for X windows Fixing SoX — Again 2010-05-15T15:30:00.003-04:00 June 15, 2011: In Ubuntu 11.04, you can add MP3 support to the Ubuntu supplied version of SoX with the libsox-fmt-mp3 package, so all of this is now unnecessary. For as long as I remember in my Linux career I've been using Sound eXchange (SoX) to convert audio files from one format to another — in particular, converting to MP3 format so that I could play them on my audio player. As I've noted before, this isn't an easy task. Distributions are understandably reluctant to deal with the licensing and patent issues encumbering MP3 files and so let you figure it out for yourself. The following seems to be the easiest way to do it. I got the basic process from Inverted Egg, but I'm not going to convert put everything into a .deb file, since it will just get rewritten with the next update. Here we go: Download the SoX source tarball. It doesn't matter where you put it, we'll assume it goes to /home/yourname/Downloads. Open a terminal window and become root: $ sudo -i You'll get prompted for the password. Go to a useful directory to unload the source: # cd /usr/local/src Unpack the tarball. Note that the version name will occasionally change: # tar xvzf /home/yourname/Downloads/sox-14.3.1.tar.gz Load Ubuntu's version of SoX into the system, if it's not there already, along with the lame library and header files, which are responsible for mp3 processing: # apt-get install sox liblame0 liblame-dev Find all the packages that Ubuntu's SoX needed to compile, and install those you haven't installed already: # apt-get source sox That's why you needed to install Ubuntu's SoX in the first place. Now do the usual installation stuff: # cd sox-14.3.1 # ./configure # make # make install Your new version of SoX is in /usr/local/bin/sox. Make sure it works, i.e. it converts files to mp3, and then delete Ubuntu's version of SoX: # apt-get purge sox And you're done. Of course, every once in a while you're going to go back and look for updates, right? Today's Ubuntu Annoyance: The Buttons are on the Left 2010-05-14T07:53:00.000-04:00 In 10.04 using the Gnome desktop, the buttons to minimize, maximize, and close a window moved from the right side of the screen to the left. Being right-handed, it feels unnatural to have the buttons on the left, so I want to move them back. Trevor has the solution: Open gconf-editor, either from the command line or by pressing Alt-F2 and typing it in the box. Click on the + sign next to Apps. Scroll down on the left-hand-side until you find metacity. Click on the + beside it. Click on the folder labeled general. On the right-hand-side, find the label button_layout. Double-click it. In the box that appears, type :minimize,maximize,close. Order is important. The colon divides the left-hand side from the right. So if you wanted the close button on the right and the rest on the left, you'd type minimize,maximize:close. Click OK and you're done. If you want to edit things by hand, it looks as though the appropriate place is the file $HOME/.gconf/apps/metacity/general/%gconf.xml, which should look something like this: <?xml version="1.0"?> <gconf> <entry name="button_layout" mtime="1273837457" type="string"> <stringvalue>:minimize,maximize,close</stringvalue> </entry> </gconf> Although there may be other entries between the gconf lines. Ubuntu 10.04: gnome-screensaver and switch user 2010-05-13T22:33:00.000-04:00 Bugs. Why does it always have to be bugs? Anyway, with the advent of New Hal, my wife and I use the same computer. She doesn't do much more than check email, check Snopes to refute emails from the clueless, and make plane and hotel reservations. Things like that, she's not a heavy computer user. It's not likely that she's going to be running a 16 atom Quantum Espresso simulation in the background, unlike some people I know. So it's OK for both of us to use the same box for most things. The idea way to do this is with the Switch User option, which works really well once you know each other's passwords and know about Ctrl-Alt-F7/Ctrl-Alt-F8. (Don't try this at home unless you have a really strong marriage.) So we both stay logged on, and use Switch User to get back and forth between our accounts. Except that when you're both logged on all the time, you don't want the monitor running all the while. It wastes power, and it's annoying to walk in to the room and see the thing lit up in the middle of the night. The default option is to use gnome-screensaver, which seems to be installed automatically with 10.04 and launches when you boot. Just set the options the way you want it and you're good to go. Except — of course there's an except, why do you think I'd be writing otherwise — there's a bug. Multiple reported bugs, actually. There's 478253, 555870, 546578, 561538, and, lest we forget, 535467. All of them report more or less the same thing: when you use Switch User, sometimes the screen goes black. A workaround is to run xgamma -gamma 1.0 when the black screen occurs. This is rather difficult because the screen is, in fact, black, and unless you're very lucky you won't be at the right spot on the screen. So, following a suggestion in the comments to 561538, I ripped gnome-screensaver off the computer and installed xscreensaver. Set it up on both accounts, then went out to work on the lawn. When I came back, GDM was cheerfully restarting itself every 2-3 seconds, complete with endless be-bop sound every time. I suspect that this is because the two instances of xscreensaver were competing with each other. I'm not sure that's the case, but I worked under that assumption and killed the screen saver in my account. That fixed the problem, so I've left things at that. Now I just remember to switch the screen to my wife's account when I'm done on the computer.

Ubuntu 10.04: Fixing mtrr errors 2010-05-11T23:21:00.004-04:00 Ah, so many things, so little time to write about them. I'm going to handle my Ubuntu 10.04 programs piecemeal, and in the order that I have time to write about them. So this one isn't the biggest problem, but I've got all the data available here, and it's a quick fix, so here goes: I was looking at /var/log/messages to try to figure out why my wife's account logged itself off last night (never did find out), when I saw the message MTRR allocation failed. Graphics performance may suffer. Googled it, found this fix, which is to add the kernel boot options: enable_mtrr_cleanup mtrr_spare_reg_nr=1 to the grub command line. Uh, OK, how do I do that? In the Good Old Days there was a file, /boot/grub/menu.lst, which had all the information you needed to get your computer started, including booting up and loading the kernel with the desired options. The problem was that this got rewritten every time you did a kernel update. There were some sections of the file that were supposed to be saved, but that required the update program to remember to save them. Just to make sure the programmer didn't forget those kind of things, all of the user-specified options were moved to the file /etc/default/grub, and menu.lst has become grub.cfg, which is generated from /etc/default/grub by running update-grub. Got that? Uh, so you just ... Oh, never mind. Here's the fix for the mtrr problem:

Run $ sudo vi /etc/default/grub to edit the grub configuration file.
Look for the line
GRUB_CMDLINE_LINUX=""
and change it to
GRUB_CMDLINE_LINUX="enable_mtrr_cleanup mtrr_spare_reg_nr=1"
Save and exit
Run
$ sudo update-grub
Check /boot/grub/grub.cfg to be sure the changes are in there
Reboot
Check /var/log/messages to be sure that the mtrr message has disappeared.

And, of course, I noticed a fantastic improvement in graphic performance &mdash which means none, actually. But hey! I got rid of the error message.

It's Here!

2010-05-09T09:37:00.000-04:00

It's a computer! Have a Cigar.

OK, it's not really that big a deal, but it was a long wait. A month ago I ordered a Gateway SX2840-01 desktop through J&R. After being on backorder for a month, it got here on Monday.

Turns out it's better than advertised. Not only does it have the glowing reviews mentioned in the first post, but Consumer Reports named the top compact desktop, though they said not to buy it because of technical support issues — of course I'm not going to get Linux technical support from HP, either, so that wasn't an issue. The computer also came with 6 GB of memory, more than the 4 originally offered. The only thing it didn't have was a 1 TB disk. The disk actually comes out as 931 GB, but you knew that was going to happen, didn't you?

Of course, it came with Windows 7. I turned it on, booted up 7, said Ooh, shiny!, then went and downloaded the 64 bit version of Ubuntu 10.04. This installed easily, including all the disk partitioning — I left 7 on there, with 100GB of disk, because you never know. I played with the system for a week, and then spent yesterday consolidating all of the files from my old computer and my wife's computer (aka Hal and Hal's formerly evil twin) to the new box. I did have a few minor, and one not-so-minor, problems with the install, which I'll tell you about in the next few days. But, suffice it to say, after a week of using this machine I'm very happy with it.

Neat LaTeX Tool

2010-05-08T21:24:00.000-04:00

For those of you that don't know, LaTeX is the way to write scientific papers. It's set up to produce nice equations, put figures and tables more or less where you want them, collect references in the proper order (with the help of BibTeX), and generally make you look like you know what you're talking about. Publishers can even set up their own style sheets and macros (see the American Physical Society's REVTeX) so that your article looks right for the journal, and it's a minimal amount of effort to translate your work to the format of another journal.

That said, in scientific articles you often need weird characters, e.g. √, &Delta, ∇, etc. Since LaTeX's source is ASCII, you need to know that √ = \sqrt or \textsurd, Δ = \Delta, and ∇ = \nabla. Not to mention hundreds of other characters.

You'll eventually remember that α = \alpha, but will you ever learn that ⊃ = \supset? I won't. You could spend all your time looking up these symbols in tables (as I just did for the HTML), or you can go to the Detexify² – LaTeX symbol classifier. Just draw in the symbol you want with a mouse, and you get a set of suggested LaTeX commands.

LaTeX and HTML handle formatting non-ASCII characters in similar ways. It would sure be helpful if someone would adapt this to work for HTML characters as well.

The Shortest Doctor Who Episode Ever

2010-05-03T10:56:00.000-04:00

Outside a small house in London. The Tardis appears. Wilfred Mott comes out of the house, just as the Doctor comes out of the Tardis.

Wilfred: Doctor? What are you doing here? Donna might see you.

This is an automatic reaction upon seeing the Tardis. As he pauses, Wilfred notices something has changed.

Wilfred: You've changed again, haven't you.

The Doctor: Hello, Wilfred. Yes, I've changed. Some unpleasantness with — well, with something unpleasant.

Wilfred:But why are you here? If Donna sees you, even in this – this – body, she'll start remembering. And then, you said …

The Doctor: Wilfred! I've found a way to cure her!

Wilfred: Are you sure?

The Doctor: Yes, but I have to take her away …

As they are talking, Donna Noble comes out of the front door.

Donna: Gramps? Who's this, then?

Donna starts to look at the Doctor.

Donna: Look, you, whatever it is you're selling, we don't …

Before Donna can get a good look at the Tardis, the Doctor pulls out his screwdriver. There is a faint whining noise, audible only to those under 50. Donna collapses.

The Doctor: Sonic sedative. Come on, help me get her into the Tardis.

The Doctor and Wilfred carry Donna into the Tardis. Wilfred exits, the Doctor shuts the door, and the Tardis vanishes with the usual noises.

The Tardis reappears in what is obviously a library. OK, it's not just a library, it's the Library. It's dark, probably damp, and the books seem to be moving slightly. Maybe it's just the arrival of the Tardis. Or maybe not. Books left open seem to be turning pages themselves. We see the Tardis from the point-of-view of the Librarian. The Doctor emerges from the Tardis, and approaches the desk.

The Doctor: Here we are then. Did you find the book?

The Librarian: Ook!

The Doctor: Good then. Here's what else I need …

As the two talk, the scene shifts back inside the Tardis, where Donna is waking up. She looks around, pulls herself up, and starts out the door. She sees the Doctor and the Librarian at the desk. She still doesn't remember anything, but that will come.

Donna: Hey!

Both turn to look at her. We shift to a point of view behind Donna.

Donna: What's this. Who are you? Why have you grabbed me and took me to this — this — place? And what's with the monkey?

With that, the Librarian explodes. All we see is a large, dark fist striking the camera. There's lots of fur. Sounds of a scuffle. Lots and lots of Ooks. Then black.

The Doctor: What a waste of a perfectly good regeneration.

The End

Thanks to the Doctor Who Wiki and the Discworld & Terry Pratchett Wiki

Rubbing It In

2010-04-10T11:17:00.000-04:00

Before every game in the Major Leagues, workers unwrap 100-200 brand-new, shiny baseballs, each bearing Bud Selig's signature.

You won't believe what they do to them.

A New Toy (Coming Soon)

2010-04-03T14:07:00.000-04:00

The hypothetical long-time reader of this blog will recall that when I started it I had a Dell Dimension 2350, aka Hal, running Fedora Core Linux, and another 2350, Hal's Evil Twin, running Windows XP.

Six and a half years later, the only change is that I've converted the evil twin to Linux, and switched to Ubuntu. But I still have the same machines, with some minor upgrades. I'm cheap, what can I say?

The status quo is about to change. Last week Office Depot had an HP Pavilion on sale at a really good price, about $440. 4GB memory, 640GB disk, dual core processor, etc. I was set buy it, until I read the CNET Review (on Best Buy's product page for the HP), which told me that Gateway had a much better computer, the SX2840-01. Dual core, 4 GB, 1 TB disk, HDMI output, etc. About $100 more, but it looks to be worth the price.

Gateway doesn't seem to sell retail anymore. They directed me to JSR, which had the computers in stock when I first looked but quickly ran out, probably because Best Buy had undercut its own sales by posting an honest review. The promise to get in more, though, so I put in my order and am now waiting expectantly. Hopefully the new two will get here about the same time as Ubuntu 10.04.

Well, It's Over

2010-03-27T19:54:00.000-04:00

and there are no Kansas teams left playing basketball this year.

Some thoughts:

K-State's loss to Butler was, in large part, due to the fatigue left over from the Xavier game. Just this morning, the Washington Post ran a story of the great KSU conditioning program.
At least the left side of everyone's bracket is blown up as badly like mine. I doubt there were more than 100 people, all from Indianapolis, who had Butler in the Final Four.
And, as will be noted 5,000,000 times between now and next Saturday, the Butler gym was the scene of Milan's historic defeat of Muncie Central in 1954.

Total World Domination

2010-03-27T11:36:00.000-04:00

Just a thought: most companies try to protect their brand names to keep them from becoming generic. But if "google" is ruled generic, how does Google loose? I mean, if you're going to google something, where are you going to start? Bing?

Damn

2010-03-20T23:15:00.000-04:00

Well, at least K-State is still in it.

You have no idea how painful that is to write.

One Down

2010-03-19T12:24:00.000-04:00

Phew!.

All year, every game, KU has always played down to to the opposition. And it's going to come back to bite them before we're done, certain prognosticators notwithstanding.

It will be a morality play, just like Memphis can't hit free throws turned out to be in 2008.

My bracket, and my heart, say KU wins it all (it's sort of mandatory for KU alums), but if I had to bet, my wallet would say it ends next week.

And I'd never be happier to lose a bet.

Who Woulda Thunk?

2010-03-13T20:07:00.001-05:00

Bobby Knight and I go way back — I was at Indiana University from 1973-79, including the great undefeated 1975-6 team featuring Scott May and Quinn Buckner. Over the years I've had the chance to notice that when he's coaching he's not particularly stable.

However, over the last year Knight and Brent Musburger telecast Big XII basketball games, mostly those involving KU and/or K-State. I've also heard him on the Mike & Mike radio show on my drive to work.

When he's not coaching, Bobby Knight is smart, witty, thoughtful, and doesn't just talk to fill the air. It's amazing.

Of course, I have no way of knowing what he's like off-camera, but it's a considerable improvement over the old Knight.

Maybe he should have started out as an announcer.

City of Big Shoulders Cries Over a Little Snow

2010-02-10T11:23:00.002-05:00

Yes, folks, Chicago has just broken a 102 year old record for snow in a single day.
12.6 inches
OMG!
Here in DC, we call that Saturday.

What Google Knows About You

2010-02-09T18:56:00.000-05:00

No, this isn't the revelation of any big conspiracy. It's just to point out (i.e., make myself a bookmark) that the page

http://www.google.com/dashboard/

lets you get information about most of your Google accounts, including your privacy settings.

Skip This Post

2010-02-05T10:49:00.000-05:00

L&T's thanks go to the Washington Post's Rob Pegoraro for this link to a site which addresses one of the Web's biggest problems:

The pointless opening page.

Parsing the Google Calendar XML Feed to Create New Calendars and RSS Feeds

2010-01-24T14:53:00.014-05:00

We use Google's Calendar as part of our church web page. It's easy to keep up to date and we don't have to install any special calendar software on our server. It even has an extremely sophisticated XML feed. OK, the feed is considerably shorter, if you tweak the call a bit, using Google's published parameters. You can use this feed in an RSS aggregator, informing your readers about what's happening in the near future. But this might not be exactly what you want.

For instance: It's helpful to have a web-page listing of upcoming events, or a customized RSS feed.

This turns out to be relatively easy to do, if you know the trick, with a bit of PHP code. Let me say right off the top that I didn't come up with the PHP code myself, I got it from P.J. Cabrera, and it's available under what looks to be a pretty open license.

So I used Cabrera's script as the starting point to construct to scripts:

First, an HTML web page, google_calendar_xml_parse_to_html.html, with an embedded PHP script which calls a Google calendar XML feed and parses it to form a web page that looks something like this.
Second, a PHP script, google_calendar_xml_parse_to_rss.php, which parses the Google calendar XML feed to produce an RSS feed that looks like this.

The scripts are fully, possibly painfully, annotated, so you can see exactly what I did at every step. I'm not going to discuss them here, then, but you can add comments below, and I'll address them there.

Now you'd think I could just provide a link to each of the two annotated pages, and you'd get the annotated pages. This turns out not to be the case. The server will execute the PHP code on each page, and then ship XML. Don't believe me? Load up the web page and/or the RSS Feed and look at the source code. You won't find the PHP code, just perfectly formed XHTML or XML.

So I've put both scripts in this

tar file, parse_google_calendar.tar

(Sorry, my free web page server prohibits direct links to non-html pages, and doesn't allow ZIP files.)

Download the file, and you can extract both the html page, to create a calendar listing, and the php page, to create an RSS feed.

A few things to note:

Don't forget to change the web addresses, as indicated. You'll also need your Google Calendar address.
Note that the two files are linked, that is, the HTML page has a link to the PHP page, telling the world that the PHP is an RSS feed, and the PHP page links back to the HTML page, so that the reader can find your web site. You don't have to put in those links, of course.
Of course, this only works if your server can handle PHP, but that doesn't seem to be all that much of a problem.
Once the code is uploaded, you can check to be that the HTML page validates using the W3C validation service.
Similarly, you can see if the PHP script creates a valid XML page by using the W3C Feed validation service.

Put all comments below.

You know, 1963-1972 was a decade, too

2010-01-09T10:34:00.000-05:00

Today's rant is brought to you by the Washington Post's Weekly Free for All. Now FfA actually has many intelligent and thoughtful comments. I read it every week. But FfA has a dark side, as seen at the bottom of the page, where we have yet another writer bemoaning the Post's referring to the ten-year span just ended as the decade. (It's right above the letter suggesting that the Post is trying to destroy the English language by quoting Gilbert Arenas verbatim.)

OK, people, listen up: we're going to set up some rules here:

From now on, there will be two types of Millennia, Centuries, and Decades: the Pedantic and the common.

In the Pedantic system, all counting begins at midnight between 1 BC and 1 AD.^† Thus a Millennium takes in the years 1-1000, 1001-2000, 2001-3000, … ; Centuries are 1901-2000, 2001-2100, … ; Decades are 1951-1960, 1961-1970, … These names are capitalized.

In the common (or vernacular) system, millennia, century, and decade (all lower-case) are defined by the similarity of the digits forming the years. Thus all the years in a millennium have the same digits except for the last three, in a century all the numbers are the same except the last two, and in a decade only the last digit changes.

Let's show a few examples, OK?

Name of era	Years
First Millennium AD	1-1000 AD
First millennium AD	1-999 AD
Twentieth Century AD	1901-2000 AD
Twentieth century AD	1900-1999 AD
First Decade, Twenty-First Century AD	2001-2010 AD
First decade, Twenty-First century AD	2000-2009 AD
Eleventh Millennium BC	11,000-10,001 BC
Eleventh millennium BC	10,999-10,000 BC

(Yes, the First millennium, AD or BC, has only 999 years. Deal with it.)

If all this bothers you, remember your Sesame Street:

One of these things is not like the other, one of these things is not the same:

2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010

Try explaining that all these years are alike to Big Bird.

^†People giving me a hard time about using BC/AD here will be required to listen to the complete collection of Brit Hume's sermons. Contact Eldrick Woods for more information.

The Big Snow of 2009

2009-12-20T18:43:00.000-05:00

You may have heard that the upper East Coast had a rather large storm yesterday. It wasn't a blizzard, because the winds never picked up as had been expected, but there were record amounts of snow for December. As best I can judge, given that the large number of trees in our yard disrupt the snowfall, we got 19“.

I've been playing around with the Linux version of Picasa, Google's photo organizing software, and with Picasa Web Albums. I don't claim to be a gifted photographer, but here's a few pictures of the storm and its aftermath, all taken from our front door.

2009-12-19T10:57:00.000-05:00

We're in the middle of what promises to be the biggest snow storm Maryland has seen in years. It will most likely set the December record for snowfall. We're scheduled to get something between 16 and 24 inches of snow, depending on how long the nor'easter hangs off the coast. Since it's a nor-easter, that means when the blizzard hits, an hour or two from now, the snow will be blowing east to west, a foreign notion for a Kansas boy like me.

The picture above was taken about 10:45 am. I'd estimate 10-12 inches of snow on the ground.

RSS Feed Template

2009-10-21T15:34:00.006-04:00

When we designed the new version of our church home page, we constructed a page where we collect sermons. It turns out that this page, and the associated sermons, gets hit a lot. I've been told that the hits match the biblical texts specified by the liturgical calendar for upcoming Sundays, which suggests that most of the traffic is pastors looking for sermon ideas.

Nevertheless, it's useful for a congregation to put its sermons on the web, because, let's face it, the sermon is the biggest chunk of time in a service, and is most likely to give you an idea of the church's, or at least the pastor's, theology. So we want to publicize the sermons as widely as possible.

One way to do this is to construct an RSS Feed. Once you have one, people can link to it as, say, a Live Bookmark, or with a feed aggregator. You can also export the feed to other sites. For example, we've set up a church newsletter blog, and you'll see the titles of the last few sermons on the right-hand side.

Setting up an RSS feed isn't all that hard. I was able to do it using the prescription in Elizabeth Castro's HTML, XHTML & CSS, with a few modifications needed to get the thing to properly validate. I'm putting a template for the feed reader here, as much for my use as anyone else's:


<?xml version="1.0" encoding="UTF-8"?>

<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">

<channel>

<title>Feed Title</title>

<!-- Link to feed homepage: -->

<link>http://www.5553456.com/index.html</link>

<!-- Address of this RSS file: -->

<atom:link href="http://www.5553456.com/rssfeed.xml" rel="self" type="application/rss+xml" />

<description>This is where you put a description of your feed.</description>

<!-- It's in English (or whatever) -->

<language>en</language>

<!-- Email address -->

<webMaster>webmaster@5553456.com</webMaster>

<!-- Copyright notice -->

<copyright>Copyright 2009 by Joe Shlabotnik. All Rights Reserved.</copyright>

<!-- Categories for this feed -->

<category>Dummy 1</category>

<category>Dummy 2</category>

<!-- Publication date of last blog entry -->

<!-- Note the syntax on all dates -->

<pubDate>Sun, 11 Oct 2009 18:00:00 EDT</pubDate>

<!-- When you actually built the thing -->

<lastBuildDate>Mon, 12 Oct 2009 12:40:00 EDT</lastBuildDate>

<!-- How long until the next update (minutes) -->

<ttl>1440</ttl>

<!-- Individual Entries.  Repeat as Necessary -->

<item>

<!-- Address of article: -->

  <link>http://www.5553456.com/article.html</link>

  <title>Article Title</title>
   
  <description>
    Description of the article, as shown in an RSS reader.  Note that this can be the entire article, if you're so inclined.
  </description>

<!-- Permanent address of article.  In most cases this will be the same as the link above -->

  <guid isPermaLink="true">http://www.5553456.com/article.html</guid>

<!-- Publication date of this article -->

  <pubDate>Sun, 11 Oct 2009 18:00:00 EDT</pubDate>
 
<!-- finish it up -->

</item>

<!-- And after all the items, close out the file -->

</channel>

</rss>

Fill in the blanks, add a new <item> </item> for each entry in your feed, and you're ready to go.

Well, not quite. You'll note that if you go to the sermon page, there's a little icon on the address bar that indicates a live bookmark/RSS feed is available for that page. To get Firefox/IE/etc. to do this, you need to put a line such as:

<link rel="alternate"
type="application/rss+xml"
title="Our RSS Feed"
href="http://www.5553456.com/rssfeed.xml" />

in your HTML file. Then the browser will know that a feed is available.