The Natty Narwhal Upgrade

Like Travis McGee, this week I'm taking an installment of my retirement. Unlike McGee, my retirement seems to consist of fixing up computers.

So today, Ladies and Gentlemen, Boys and Girls, Desktops and Netbooks, we're going to talk about upgrading Hal here to Ubuntu 11.04.

Yawn. This shouldn't really be that big a deal, should it? You just hit the update button, or type do-release-upgrade at the command line, right?

Well, it's not always that simple. A friend of mine did that, and he seemed to have a lot of problems. I'm not sure why, as I never got to examine his system before he reinstalled 10.10. Maybe it was the change from the Xorg window system and Gnome desktop to Wayland and the Unity desktop. Or not. Maybe it was just the way the wind was blowing that day. But I decided not to take the chance on the upgrade, but rather install 11.04 from the ground up.

I didn't have to be pushed hard, because I was leaning toward a clean install anyway. When I bought this version of Hal I was in a hurry to get Linux set up, but wanted to keep Windows 7 around. As a result, I let Ubuntu do the repartitioning of the disk, and ended up with the entire Linux system in one giant partition.

This is less than optimal. (Thanks, Dave. Don't mention it, Hal.) Ideally one wants /home and possibly /usr/local and /opt in partitions separate from the root operating system. That way when you upgrade the OS, you don't lose your previous data. (That doesn't mean you shouldn't back up that data before upgrading. There's a word for people who don't. The polite form of it is idiot)

But this requires repartitioning Hal's disk. Fortunately, all of Hal's memories reside in the extended partition, so I don't have to fiddle with the Windows partition. After about 30 secons of thought, I decided to repartition Hal's big Linux partition into 15GB for /, the root system, another 15GB for /usr/local, /opt, and /scratch (more on how to do this later), and leave the rest for /home. That's probably too much for /, and maybe not enough for /scratch, depending on what I right there, but that's how I'm going to do it this go-round. Ideally gparted would leave the data on /home as is, but that takes forever because of all the data that needs to be moved around, so we're just going to repartition everything and hope the backup holds.

Preparation

First, get the software we'll need. I downloaded and burned CDs for the 64-bit versions of:

• Ubuntu 11.04
• openSUSE 11.4
• Fedora 15
  and, last but probably most importantly,
• Gparted Live CD version

OK, Gparted is obvious: that's what's going to do the disk repartitioning. But why the alternative distributions?

Because you never know. The largely hypothetical long-time reader will remember that I used to use Fedora, but that I switched to Ubuntu when the installation of Fedora 5 failed miserably, and Ubuntu was there, ready and waiting. So having alternative Linux CDs on hand seems to be a really good idea. Besides, we bought 100 CDs maybe five years ago and still have a bunch left. They just aren't used much anymore. (That's right, rcjhawk is still a trailing indicator for device popularity.)

OK, we've got hopefully every bit of software that we need, so let's go.

Find out what's on your machine

I never remember which packages are installed from one upgrade to the next. So let's make a list:

dpkg --get-selections > installed.txt

Edit installed.txt, leaving out any packages you think will be installed by the system (e.g., kernel, window manager, etc.). If you can't remember what a package does, delete it too. If you need it, it will eventually show up as a dependency to a package you want, or you can reinstall with apt-get or synaptic. The important thing to do here is to remove cruft from the system. For example, I have the bsd-games package on here. I never play those games, so why leave them in place?

It would have been nice to make a copy of /etc/fstab, but some dingbat forgot to do that (more on this later).

Back it all up

Well, not everything. No need to back up the kernel, or Emacs, or latex, or any of that. The distribution will provide. No, I want to back up my data (basically $HOME), the Intel Fortran compiler, a royal pain to install, and my renegade version of SoX. So do the following (/backup is my ext4 formatted USB disk):

$ cd
# -rpv == recurse directories,
#  preserve permissions and timestamps,
#  and speak up about it verbosely.
$ cp -rpv dave /backup
# Assuming you're named dave, of course
$ cd /usr
$ cp -rpv local /backup
$ cd /
$ cp -rpv opt /backup

To be safe, I also had backintime, my backup manager, do an additional labeled backup, which should keep it on the disk forever and a day, or at least until the end of 2012.

Repartition

The next step is the scariest one. I've taken what was an 800+GB partition and broken it up into three parts: 15GB for /, 15GB for what will be /usr/local, /opt and /scratch, and the remainder for /home. It's scary because this wipes out the primary data, so you are now relying on the kindness of backups. It's theoretically possible to gave gparted to keep all the data in what's going to be the /home partition, but that requires moving a lot of stuff around, at least the way I tried it, and an estimated 18 hours to get it done. So I just said a prayer, bit my lip, and reformatted. Fortunately it all went well.

Installation

Not much to say here. I told Ubuntu that I wanted to do an install with a custom installation, had it mount everything where I wanted it, and let it go. The usual waiting around occurred.

Copying Files

This was rather easy: turn on the backup USB disk drive, and copy the files back. I copied back configuration files I knew I wanted to keep, e.g. things like .emacs and the .devilspie directory, but I'm letting Ubuntu set up the desktop. This means I'll have to play with preferences later, but it should get eliminate any conflicts between old and new configuration parameters.

Mounting the Backup Drive

I could just plug the backup drive in and access it at /media/really_long_hexadecimal_string, but what I really want to do is have the backup drive mounted to /backup. I've covered this before, but as I mentioned above, I forgot the UUID of the drive. You get that with the command:

blkid

which gives you all the UUID of all connected disks.

Restoring /usr/local and /opt

This is awkward. Add-on programs that you compile yourself generally go into /usr/local. Other programs, such as the Intel Fortran compiler and Google's Picasa, end up in /opt. I want both to stick around between upgrades, so they need to be out of the / partition. But I don't want two extra partitions. So here's what I did:

• Create a partition /usr/local
• Create a directory /usr/local/opt
• Copy all of the /usr/local files where they belong.
• Copy all of the /opt files to /usr/local/opt
• Then, as root, run the command
  ln -s /usr/local/opt /opt
• If you want a /scratch directory, do the same thing, but don't forget to make the original world writable:

  # mkdir /usr/local/scratch
  # chmod 777 /usr/local/scratch
  # ln -s /usr/local/scratch /scratch
  

Restoring your add-on packages.

Now we want to get back all of our old packages, the ones that aren't installed by default, at least so far as Ubuntu will let us. Remember that file installed.txt I mentioned you should create and edit? Here's how we'll use it:

• Select System => Administration => Update Manager
• Click on Settings in the lower left corner
• Under Ubuntu Software make sure all of the sources you want enabled are. If you don't want proprietary drives, etc., turn them off. Do the same under Other Software
• Open a terminal window and go the the directory where you have that installed.txt
• Run the following commands:

  sudo apt-get update
  sudo apt-get install `awk '{print $1}' installed.txt | xargs`
  

• You'll probably get some error messages, but they're pretty clear about what you need to fix. Edit the installed.txt file to match.
• This will take a while, as all the packages have to be downloaded from the net.

What Works

Pretty much everything. I selected Gnome-classic for my desktop, and after a little fiddling to with System => Preferences => Startup Applications got the system to look pretty much as it did before.

As for third party software, the Intel Fortran compiler still works fine. I was able to reinstall Picasa from Google's supplied 64-bit .deb.

Wayland seems to work just like Xorg, at least from my point of view. It pops up windows in the same way as before, and I can run, say
ssh -X majel
to get me to another machine, and then run
firefox
on majel, and the window pops up here on Hal. So no major problems for me there.

What doesn't work

Google Earth. Maybe this is a 64-bit problem, I don't know. I tried using Google's .deb file, and the official Ubuntu method. Neither worked.

By default Ubuntu installs the 3-D version of Unity, which requires a pretty good graphics accelerator. Hal lives in 2-d, so that wouldn't run. (Funny, the Unity desktop shows up when you run the live CD.) I installed the 2-D version and that works. Which brings me to

What Sucks

Unity. From my limited experience (about two minutes, after which I ran away screaming), it's an overblown and somewhat hideous version and of the Mac desktop. But don't mind me, I was the last person on Earth to use fvwm. I suppose if I played around awhile I could make Unity behave the way I wanted it to, but why bother, since Ubuntu still supplies the classic Gnome desktop. Mind you, I've seen Gnome 3, and I don't like that, either. (The word you're looking for is Luddite.)

Summary

All in all, a successful update, as long as I stay way from Unity. I have one more computer that needs updating to 11.4, I think I'll just try that as a distribution update. If it doesn't work, I can always do a full install.

More Later

Troubles will surface, they always do. When they do, I'll write about them here.

Compression

The other day an email arrived from sourceforge which mentioned that they were hosting the file compression program 7-zip. Now I had used 7-zip under windows, as an all purpose archiving tool, mostly for reading zip files. I'd never thought of it in a Linux context. But both openSuSE and Ubuntu have the command-line version available, and what more do you need?

In OpenSuSE, the RPM is called, simply enough, 7z. In Ubuntu it's a bit more complicated. There's p7zip, which provides the bare-bones standalone version of the compression program, 7zr, and a wrapper p7zip, which makes 7zr work like gzip. For the full-blown 7z program, you want the package p7zip-full, which includes 7z. While you're at it you might want to get the p7zip-rar package, which lets you decompress RAR files.

The reason you want 7z and not just 7zr is that the smaller program only compresses to the 7z format, but, as it says in the blurb,

not only does [7z] handle 7z but also ZIP, Zip64, CAB, RAR, ARJ, GZIP, BZIP2, TAR, CPIO, RPM, ISO and DEB archives.

And by handle, they mean read and write to these formats (except RAR). So I could create a zip archive with 7z, or a gzip/bzip2 compressed tarball. And guess what?

7z compression is 30-50% better than ZIP compression.

Is it? Well, let's find out.

Test 1

Presented for your consideration: an uncompressed tarball:

$ ls -l Ru.tar
-rw-r--r-- 1 dave dave 26716160 2011-05-21 15:41 Ru.tar

This is IO from the elk FP-LAPW code, so it has a lot of repetition of text, and what for our purposes are a bunch of random numbers. First we'll try compressing it with programs using their native formats. I'll try for maximum compression in all cases. Note that zip and 7z will create separate archives, while gzip and bzip2 compress the file in place.

Program	Command	File Size	Ratio
zip	zip -9 Ru Ru.tar	3899523	0.146
gzip	gzip -9 Ru.tar	3899386	0.146
bzip2	bzip2 -9 Ru.tar	2992422	0.112
7z	7zr a -mx=9 Ru.7z Ru.tar	2242708	0.084

Pretty good, huh? As advertised, 7z is about 40% better than zip/gzip, and 25% better than bzip2. But wait, there's more. Not every computer is going to have 7z available, so you may want to compress files using a more established protocol. 7z can do that, too, which is why we wanted it, not just 7zr:

Format	Command	File Size	Ratio
zip	7z a -mx=9 -tzip Ru.zip Ru.tar	3287420	0.123
gzip	7z a -mx=9 -tgzip Ru.tar.gz Ru.tar	3287335	0.123
bzip2	7z a -mx=9 -tbzip2 Ru.tar.bz2 Ru.tar	2989193	0.112

So 7z compresses to zip/gzip better than the native programs do it themselves. It doesn't really outperform bzip2 here, though. The only disadvantage compared to gzip or bzip2 is that it doesn't compress the files in place, unless you go through a script such as the one in /usr/bin/p7zip.

Test 2

Pi to 4 million Decimals has, duh, π to, actually, 4,194,034 places. The file pi.tar.gz has it in ascii, with a bit of header information. If we uncompress that file, it comes in at 4362370 bytes. Since the digits of π don't repeat, it's hard for a compression program to find blocks of bytes to compress. The following table lists the compressions achieved by our test programs, in whatever formats they can use. (See the above tables for the appropriate commands.) Let's see how everybody does:

Program	Protocol	File Size	Ratio
None	None	4362370	1.000
zip	zip	2041130	0.468
gzip	gzip	2040997	0.468
bzip2	bzip2	1863892	0.427
7z	zip	1983378	0.455
7z	gzip	1983297	0.455
7z	bzip2	1860047	0.426
7z	7z	1884999	0.432

Here, bzip2 is competitive with 7z. Oddly, though, you should use 7z to do the bzip2 compression. Weird, huh? But still, 7z is pretty good.

Wrapping Up

So what's not to like?

For one thing, 7z is slow. If you just want to quickly compress a file, go ahead and use gzip, or bzip2. There is a price to pay for better compression.

Then, too, there's a warning on the 7z man page:

DO NOT USE the 7-zip format for backup purpose on Linux/Unix because 7-zip does not store the owner/group of the file.

you can get around this by piping tar into 7z:

tar cf - directory_to_be_archived | 7z a -si directory.tar.7z

which creates the analog of a gzip/bzip2 tarball.

And finally, the native 7z format isn't standard, yet, so it's not going to be available everywhere, and might even vanish. But 7z and its compression algorithm LZMA are open source, so they are likely to stay around for awhile. A few years ago bzip2 wasn't standard, and once upon a time neither was gzip. It's probably safe to compress your files to 7z format, but if you want to be safe, use 7z to compress to gzip or bzip2 format.

Why I Hate March

I just spent the last three hours not watching the Kansas-Boston University first round NCAA game. In case you didn't join me in not watching, KU spent the first 25 or so minutes wondering why the BU players weren't genuflecting. Then they concentrated for a few minutes, and the game was over. Because of course, you know, when #1 meets #16, #16 has never won, right?

Trust me. Some day it will happen. And which team will that be? We'll, let's look at the past for some clues, starting here:

• 2010: #1 Seed, lost in second round
• 2007: #1 Seed, lost in Sweet 16
• 1998: #1 Seed, lost in second round
• 1997: #1 Seed, lost in Sweet 16
• 1995: #1 Seed, lost in Sweet 16
• 1992: #1 Seed, lost in second round

Not to mention Bradley, or Bucknell, or the suffering I did during the Ted Owens years.

It's enough to make one start a website, except a) someone beat me too it, and b) Roy was as bad, or worse, because he didn't win a Championship until he clicked his heels three times and went home.

But mark my words, the #1 Seed in NCAA history to lose in the first round will have a six-letter name on the front of the jersey, and have two Crimson and Blue mascots that look something like this.

Superbowl Commercials

There weren't many that were any good this year. I'll show you my favorite, below, but I found that a lot of people didn't get the reference. So let's go back to September, 1979, when this famous commercial aired, starting Pittsburgh's Mean Joe Green:

And now that we've set the stage, here's my favorite commercial from this year's Superbowl:

And one more reference to TV gone by:

FUSE on the Mac

Awhile ago I wrote about how one can mount remote computer filesystems using ssh. That was for Linux. You can do it with Mac OS X as well, but FUSE (File System in UserSpace) isn't installed by default, as it is in modern Linux systems.

So first you need to get FUSE installed. The package I use is called, duh, MacFUSE. It installs like a standard Mac package: Download the disk image, click on it, and go.

Next is sshfs itself. There seem to be many versions, but Google's seems to work well. Download the appropriate version for your OS (the Leopard version works on Snow Leopard), rename it sshfs, make it executable, and put it in your path. From then on it works pretty much like in Linux, with minor differences. Assuming you have a directory ~/hal on your Mac, you can look at the files on your Linux machine hal with the command

sshfs rchawk@hal:/home/rcjhawk ~/hal

So a file named /home/rcjhawk/Documents/testthis.doc on hal will appear as ~/hal/Documents/testthis.doc on your Mac.

And it's simple to unmount it:

umount ~/hal

A More Perfect World

Me, to.

Wikipedia's List of Common Misconceptions.

Happy New Year

This blog started just over seven years ago, and sometime tonight or tomorrow it will get its forty thousandth recorded hit. Thanks to all of you for listening.

Thanks to a Penguin Pete rant (see Sitemaps). I've discovered Google's webmaster tools and Analytics. My New Year's Blogging Resolution is to learn how to use these next year.^† If I come up with any neat tricks I'll let you know.

In the meantime, Happy New Year, everyone.

^† My primary resolution is to still have a job next year.

Should We Call It gnarcissism?

Linux Journal, or, more properly, the janitor at Linux Journal since everyone else is off this week, is running a pole asking if you Google® yourself.

The answer I'd choose isn't there:

No, because there are at least a half-dozen people^* with the same name who are at least as famous as I am, and that doesn't count the 1,300 or so in China.^†

^* Yes, I know, but only for the purpose of this post.^§

^†Working from the old if you're one in a million line.

^§That's my story, and I'm sticking to it.

The Christmas Toy

So here I am, Christmas Evening, waiting for BBC America to broadcast the Dr. Who Christmas special and playing with my new toy: a Logitech Harmony 650 Universal remote. I bought the thing at Radio Shack because: a) It was cheap ($69.95), and b) I'm sick and tired of having to manipulate three remotes to change from FIOS to BluRay to USB/Harddisk when watching shows on the bigscreen TV.

The neat thing about the Harmony series of remotes is that you program it on-line. The bad part is that there isn't a native Linux program to do it. There is something called Concordance which is supposed to work, but I'll try that out when I'm prepared to trash the remote if the software fraks out, OK? It's not worth the struggle (forgive me, Saint Richard).

So we boot Hal up in Windows 7 mode, and I'm blogging what's going on using a laptop.

• First: gather up the model numbers of the gadgets to be used. That's fairly easy. I've got a Samsung LCD 46 inch TV, and the model numbers on the side. A Samsung BluRay BD-P360. The hardest part is the FIOS DVR. It's by Motorola, and looking at the pictures on the FIOS site it appears to be a 6416. We'll see.
• Launch IE8 on Hal (NNNNOOOOO!!!!!!!). Go to http:www.logitech.com/myharmony and create an account. Click on My Account. No registration box. Click on the safest looking link, Go to My Stuff There's a login screen and a registration option there. For some idiot reason they want a daytime phone number. Tempted to give them the one for our security, but refrain.
• Click on Product Support. Says it will detect USB corded product. The unit comes with a USB 2 cable, but of course the connector on the remote side isn't standard USB2. Annoying, especially if I loose the cord, but I sort of forfeited the moral outrage option when I booted into Windows. So plug it in, let Windows find the driver software, and click the Detect your devices button. Of course Logitech has an IE add-on that has to be installed. It seems to work. It says I've got something called an N/N: N-10003 Harmony 650 remote. Well, I know the Harmony and 650 part are right.
• It's not particularly obvious what to do next. Click Download Software and get version 7.7.0-WIN-x86 version of LogitechHarmonyRemote, I guess. Install that. You know, I'm kind of glad I'm doing all of this on the Windows side. Installation involves the usual number of clicks to accept the license. Eventually an Icon shows up on my desktop and the thing launches. First thing it wants to know is if it should check for updates. Surprisingly, there are none.
• Log in. You'd think the account I just set up with Harmony would work, but no. So set up another account here. It's another complete registration, folks.
• Now things start. It finds my remote (again). Keep hitting Next until we get to the Add Device page. Adding devices seems straightforward, so long as you have the right device numbers. Fortunately they give you examples of how they want the information entered in.
• Once I finally got the right models in, the software found my devices. The only question I had to ask is if I wanted to turn the cable box off when it wasn't in use. I said yes. We'll see if that's a good idea or not. (I worry what's going to happen when I switch devices. Is the TV going to be out of sync with the cable box?)
• Specify that FIOS uses HDMI1 on the TV, and the BluRay uses HDMI2.
• And somehow we lost the signal, so I HAVE TO LOG IN AGAIN???? And it lost the connection to the remote. But now it has, indeed, started downloading the settings to the remote.
• Looks done. I'm told to disconnect the remote and go try it out.
• Preliminary tests done. It mostly works. There are several glitches, though:
  1. Both the TV remote and the FIOS remote have options for resizing the picture on the screen. It's especially handy when you are watching a standard definition channel, but they're broadcasting something in letterbox format (as in the above-mentioned Dr. Who special). Resize the picture, and it fits the whole screen. Not high-def, but in this case bigger is better. The 650 doesn't have such a button.
  2. The FIOS remote has a standalone DVR button, which bypasses the main menu and takes you to the stuff you use most. The 650 doesn't, you have to go through the main Menu (though the Menu, Info, and Guide buttons work exactly the way they should, as does the Favorites option on the 650's screen).
  3. The worst problem: If I want to stream Netflix using the BluRay, I press a button labeled B on the BluRay remote. One of four buttons, Red/A, Green/B, Yellow/C, Green/D. The 650 has unlabeled Red/Blue/Yellow/Green buttons, but they don't do anything to the BluRay. Since I stream Netflix as much as watch DVDs, if not more, this is a serious difficulty, and a deal-breaker.
• Back to Win-Hal. Click the I need to make some adjustments box. Log in again. At least this time it remembered my password.
• There's a Customize Buttons section for the remotes. For the BluRay, set up Blue to be Netflix, Yellow Pandora, and Green for YouTube. A/Red on the BluRay is Blockbuster, which we don't use. For the FIOS box, set A on the screen (not to be confused with A/Red below) to Aspect, which should set the screen size properly. While we're at it, set B up there to Close Caption, something we use a lot since we're old. Do that for the BluRay as well. I may have to tweak that later. Can't find a DVR button option, but I may have a way around that. Let me test out the new setup and get back to you in a minute.
• OK, those changes work. Nice touch: the buttons attached to the screen are labeled A/B/C..., but you can change those to Aspect/Closed Caption/On Demand, etc.
• Now let's tackle that DVR button problem. On the setup screen for the FIOS remote (labeled PVR in 650's language) there is a button labeled Learn Command. Click that, and go down until you find Learn a New Command. Type DVR Menu in the box and click the Learn a New Command button. Put the front of the FIOS remote to the back of the 650, press the DVR button on the FIOS remote. Wait. Go back to the Additional Buttons setup screen, and assign the button you want the new DVR Menu command.

And, for now, that's enough. If I encounter more difficulties I'll let you know. In particular, we'll see if the family can use it easily or not.

Preliminary bottom line: This isn't all that easy to program, especially if you have an exotic system. But it does seem to be straightforward to program, and you can add special keys as needed. We'll see how it all works in practice.

Mounting Remote Computer Files with FUSE

I'm sitting here, upstairs, typing on my laptop. There's a file I want on my office machine downstairs. So I should go downstairs and look for the file, right?

Wrong. The TV up here is on NFL Red Zone ( (sigh) Red Zone). I'm addicted, OK? It's hard enough to break away for a bathroom trip.

Several options exist, of course. I could ssh to the downstairs machine, search for the file, and use scp to copy it upstairs. But the file is a picture, and I while I remember what it looks like, I don't remember what name I used for it. So ideally I'd like to use an image viewer, say gthumb or F-Spot to look through the directory and find the file. But I can't, because this computer doesn't mount the downstairs computer's disk.

Ah, but it can. The classic way to do this is to use the Network File System protocol, aka NFS. But that requires work to set up. Besides, some days I might want to search my computer from work, or from a hotel. It's tricky to get NFS to work that way.

Fear not, there is a solution. It uses File System in Userspace (FUSE), a complicated name for using ssh and scp to fool my laptop into thinking that the downstairs computer's files are on the laptop's local drive.

In Ubuntu the way to do this is to install a package called sshfs. Then you just follow the directions on the linked page. Let's say I want to be able to access all the files in /home/rcjhawk on hal, my downstairs computer, from my laptop, majel:

• First make sure I have both hal and majel set up to use ssh communications. If I'm away from home, I'll also need to know hal's ip address, and make sure I can ssh to hal from outside my router.
• Next, install sshfs on majel. Just use
  $ sudo apt-get install sshfs
  from a terminal window on majel. This doesn't need to be done on hal. Hal's just going to see a series of ssh requests.
• Add myself to the FUSE group on majel:
  $ sudo gpasswd -a $USER fuse
  from the same command window.
• Log off my account on majel, log back on. Yes, it's annoying, but you're not officially added to the FUSE group until you do.
• Make a directory on majel. Doesn't matter what the name is:
  $ mkdir ~/hal
• Now we're going to mount all of hal's directory tree starting from /home/rcjhawk in majel's ~/hal directory:
  $ sshfs -o idmap=user rcjhawk@hal:/home/rcjhawk ~/hal
• If this works, then running the command
  $ ls ~/hal
  from a terminal window on majel lists hal's directory tree.
• To unmount hal, just run:
  $ fusermount -u ~/hal

To make things easier on myself, I created two files. The first, mthal, contains the mounting command, and the second, umthal, the unmount command. Make these executable, put them in your path, and it's easy to mount and unmount.

A caveat: I was getting an error remote host has disconnected. That occurred because my .cshrc file, which used a variable that's undefined when you are running ssh commands. You can find out how to diagnose that here.

Does it work? We'll, I started typing this on the laptop, and finished it up on the big machine downstairs. So, yes, it does work.

There are also versions of FUSE for BSD, OpenSolaris, and Mac OS X. In fact, I first learned about this trick using my Mac at work. With FUSE you can mount just about anything anywhere. There's even GmailFS, which transforms your Gmail account into backup storage.

Finding Your Header Files — A Better Way

I should really read my own posts. Especially since that's the original purpose of this blog.

The other day I was looking through old posts with the Ubuntu label, and found this comparison of dpkg/apt-get and rpm/yum commands. The relevant one is

dpkg -l '*'

which lists all of the packages available in your repository, installed or not.

Duh

This makes my header file lister a lot simpler to write, towit:

#! /bin/bash

# First make sure you've got all your repositories updated:
sudo apt-get update

# Now search through every repository database, looking for lines that
#  start with "Package", and end with "-dev".
# Pull out the package name, look through each package, and print out
#  the header files:  those that end in "-h"

# No matter how it wraps on your screen, this next line starts with
# "for" and ends with "awk '{print $2}'`"

for hfile in `dpkg -l '*' | grep dev | awk '{print $2}'`
do
# echo $hfile
apt-file list $hfile | grep "\.h$"
done

It's still going to take a long time to go through every package, but it's a lot neater now.

Look Familiar?

I saw this watching the Star Trek reboot on the bigscreen with our Epix OnDemand account. It's during the Old Spock/New Kirk mind-meld, at the part of the story where Spock is transported back in time. The blue box is suggestive, don't you think?

Since Epix lets you stream its movies to a Linux box, unlike Netflix, it was relatively easy to find the shot and save it. No, the resolution's not that great, but, frankly, this movie suffers even on a 48" high-def screen. To do it justice you really need to watch it in a theater.

Undercutting Yourself

Maybe I just don't understand…

The other day Borders sent me an email which gave me ten dollars in Borders' Bucks. A great promotion, to be sure. And I had a book in mind I wanted to buy, but not spend a lot of money on: Jim Butcher's collection of Harry Dresden short stories, Side Jobs.

First check it out online. We have:

List Price	$25.95
Online Discount	-$11.22
Borders' Bucks	-$10.00
Shipping	$3.98
MD Tax	$0.29
Total	$9.00

Not bad, but the shipping price sucks, it will take days to get here, and anyway I want to get out of the house for awhile, so let's go to a nearby Borders and buy it there.

I get to the Borders, and find Side Jobs. The sticker on it says 20% off. Do a little bit of math: 80% of $25.95 = $20.76 - $10 Borders' Bucks = $10.76, + 6% tax ( $0.65) = $11.41. A bit of a rip, compared to online, but I'm here, I've already spent a buck in gas, and I'll walk out with the book.

Except: I then notice that the sticker also says Borders Rewards Plus Members. I'm just an ordinary Borders Rewards member. I checked with one of the sales people, and no, I'm not entitled to the discount. That makes the cost $16.91, Borders' Bucks and tax included. I made a point of telling the guy his store's website was undercutting the store by selling the book for $5 less, including shipping, put the book down, and walked out.

I suspect some marketing guru has it all figured out, but to me this just seems plain old-fashioned stupid. If Borders gave me the online price at the store, or even got close to it, I would have gotten the book then and there. Since I was in the store, I probably would have had a coffee as well. Maybe a cookie. I would have looked at a Linux magazine while I was sipping my coffee, and possibly would have purchase that, too. I would most likely have spent the $5 in the store, maybe more, and I wouldn't have been pissed.

Instead, I went home and ordered the Dresden book online. Even with the $2 gas^† I just wasted I still saved $3 over buying it in the store. And, instead of waiting for more coupons/bucks to come my way, I went over to Amazon and ordered two books I would have otherwise eventually gotten at Borders (along with more coffee, cookies, magazines, etc.): Lois McMaster Bujold's newest Vorkosigan novel, Cryoburn, and Terry Pratchett's new Tiffany Aching/Discworld novel, I Shall Wear Midnight. Add in a book Child_II wanted, we made the supersaver price, and got free shipping.

In summary: by charging full price for a book in the store, and a deep discount online, Borders lost the sale of at least two books, two-three cups of coffee, a couple of cookies, and maybe a magazine.

It looks like Borders would rather you didn't come to their store. OK, stores cost money to keep up. Employees have to be paid, not to mention heating, lights, insurance, and all of that. But if I come into the store, I'm going to browse. I'll look around, and might well find something else I'd like to buy. Online, I'm going to get the books I want, hit the send button, and go on with my life.

Like I said, some genius with an MBA probably has figured out that this approach makes more sense for Borders, but how that could be is a mystery to me.

^† The trip wasn't wasted, I drove a mile to the local wine emporium, tasted a couple of wines, and bought a bottle.

Unintended Consequences

Article 1, Section 2, of the United States Constitution says, in part, The Number of Representatives shall not exceed one for every thirty Thousand, but each State shall have at Least one Representative. The very first amendment proposed (you can look it up) tried to change this:

After the first enumeration required by the first article of the Constitution, there shall be one Representative for every thirty thousand, until the number shall amount to one hundred, after which the proportion shall be so regulated by Congress, that there shall be not less than one hundred Representatives, nor less than one Representative for every forty thousand persons, until the number of Representatives shall amount to two hundred; after which the proportion shall be so regulated by Congress, that there shall not be less than two hundred Representatives, nor more than one Representative for every fifty thousand persons.

but this was never ratified. So we can be pretty sure that the Founders of this Republic wanted us to have one Representative for every 30,000 Americans.

Currently, however, we have 435 Congressmen, thanks to something called Public Law 62-5. I guess this was supposed to keep the number of Congressmen down to a reasonable number. Otherwise, with the current U.S. population around 300,000,000, we'd have 10,000 Congressmen running around. Think of it! Mark Russell once said something on like every night 500,000 people in the District of Columbia go to bed without a Congressman. But if we had 10,000 Congressmen, you give yours (DC residents excepted) a booty call just about every night.

The current limit of 435 Congressmen also gives rise to some startling inequities. For example, according to the 2000 Census, the total population of the United States was 281,421,906. If we exclude the District of Columbia (population 572,059) since it has no Congressional representation, then there are 280,849,847 Americans and 435 representatives, so each member of Congress should represent 645,632 people. However, Wyoming (493,782) and Montana (902,195) each have only one representative. To put it another way, Cynthia Lummis only needs to attract half as many votes as Denny Rehberg to get elected, but each gets one vote in the House. So by moving from Wyola, Montana to Parkman, Wyoming (16 miles) you can effectively double the strength of your vote for Congress.

There is a lot more information about the inequities of the current system, and the sorry history of Amendment the First, at thirty-thousand.org. I'd recommend downloading their pamphlet, Taking Back Our Republic, for more information.

One thing they didn't cover, though: the effect of all of these changes on the Electoral College. Each state gets N+2 electoral votes, where N is the number of representatives, plus the two members of the Senate allowed each state. That +2, along with Public Law 62-5, weights the Electoral College heavily toward the smaller states. Since D.C. does have 3 votes in the Electoral College, each elector theoretically represents 281,421,906/538 = 523,098 voters. However, each state has a minimum of three electoral votes. So in Wyoming each elector represents 164,594 people. In Montana, it's 300,732 people, still substantially below the national average. It's likely (I haven't checked in detail) that the state with the most people/elector is California, where there are 33,871,648 people and 55 electors, or 615,848 people/elector. Put it this way: an elector in California represents four times as many people as one in Wyoming, but each has the same say in who's elected President.

Now, as anyone can tell you, most of the smaller states, population-wise, are in the west, and most of those states vote Republican, while many of the larger states tend to vote Democratic. Another wrinkle in the system is that most states (Maine and Nebraska excepted) cast their electoral votes in a block.

Couple these two facts with the 30,000 people/representative rule and you get some interesting results. Very interesting, in some cases.

In particular, let's consider the 2000 Presidential Election. Very close, as we all recall. Al Gore got 50,999,897 votes, while George Bush got 50,456,002, but because most states cast their electoral votes as a block, Bush won in the Electoral College, where it counts, 271-266.

What if we'd apportioned Congress the way the Founders intended? And kept the current winner-take-all (mostly) electoral system? The table below shows the results. I constructed it using the following rules:

• Winner take all in each state, as now. Maine (42 votes under these rules) went to Gore, and Nebraska (54) went to Bush. Changing all of Maine's votes to Bush or all of Nebraska's to Gore isn't going to change the final result, so it doesn't matter if several congressional districts shifted from Red to Blue or vice versa.
• To get the number of representatives, I took the states' populations from the 1990 census, which governed apportionment at the time, divided by 30,000, and rounded down. Rounding to the nearest integer, or rounding up, can't change the electoral vote by more than 51, and so won't affect the result, either.
• If the District of Columbia was a state, it would have 22 electoral votes in this scheme. However, by the Twenty-Third Amendment, D.C. gets 17 electoral votes, equal to the state with the fewest, Wyoming. In the real 2000 election one D.C. elector abstained. Even assuming all 17 of the new electors abstained it wouldn't change the bottom line here, so I assume that they all voted — for Gore, naturally.

The result: Since each state has more or less the same representation per capita in the electoral college, and since most of the larger states went for Gore, he wins the election, 4,314–4,047.

Think about it. Al Gore, Forty-Third President of the United States. Somehow, I don't think that's what those currently saying we should go back to the Principles of the Founders had in mind.

State	Population	Congress	Electors	Bush	Gore	Bush	Gore
				Popular	Popular	Electoral	Electoral
Alabama	4040587	134	136	941173	692611	136	0
Alaska	550043	18	20	167398	79004	20	0
Arizona	3665228	122	124	781652	685341	124	0
Arkansas	2350725	78	80	472940	422768	80	0
California	29760021	992	994	4567429	5861203	0	994
Colorado	3294394	109	111	883748	738227	111	0
Connecticut	3287116	109	111	561094	816015	0	111
Delaware	666168	22	24	137288	180068	0	24
DC	606900	0	17	18073	171923	0	17
Florida	12937926	431	433	2912790	2912253	433	0
Georgia	6478216	215	217	1419720	1116230	217	0
Hawaii	1108229	36	38	137845	205286	0	38
Idaho	1006749	33	35	336937	138637	35	0
Illinois	11430602	381	383	2019421	2589026	0	383
Indiana	5544159	184	186	1245836	901980	186	0
Iowa	2776755	92	94	634373	638517	0	94
Kansas	2477574	82	84	622332	399276	84	0
Kentucky	3685296	122	124	872492	638898	124	0
Louisiana	4219973	140	142	927871	792344	142	0
Maine	1227928	40	42	286616	319951	0	42
Maryland	4781468	159	161	813797	1140782	0	161
Massachusetts	6016425	200	202	878502	1616487	0	202
Michigan	9295297	309	311	1953139	2170418	0	311
Minnesota	4375099	145	147	1109659	1168266	0	147
Mississippi	2573216	85	87	572844	404614	87	0
Missouri	5117073	170	172	1189924	1111138	172	0
Montana	799065	26	28	240178	137126	28	0
Nebraska	1578385	52	54	433862	231780	54	0
Nevada	1201833	40	42	301575	279978	42	0
New Hampshire	1109252	36	38	273559	266348	38	0
New Jersey	7730188	257	259	1284173	1788850	0	259
New Mexico	1515069	50	52	286417	286783	0	52
New York	17990455	599	601	2403374	4107697	0	601
North Carolina	6628637	220	222	1631163	1257692	222	0
North Dakota	638800	21	23	174852	95284	23	0
Ohio	10847115	361	363	2350363	2183628	363	0
Oklahoma	3145585	104	106	744337	474276	106	0
Oregon	2842321	94	96	713577	720342	0	96
Pennsylvania	11881643	396	398	2281127	2485967	0	398
Rhode Island	1003464	33	35	130555	249508	0	35
South Carolina	3486703	116	118	785937	565561	118	0
South Dakota	696004	23	25	190700	118804	25	0
Tennessee	4877185	162	164	1061949	981720	164	0
Texas	16986510	566	568	3799639	2433746	568	0
Utah	1722850	57	59	515096	203053	59	0
Vermont	562758	18	20	119775	149022	0	20
Virginia	6187358	206	208	1437490	1217290	208	0
Washington	4866692	162	164	1108864	1247652	0	164
West Virginia	1793477	59	61	336475	295497	61	0
Wisconsin	4891769	163	165	1237279	1242987	0	165
Wyoming	453588	15	17	147947	60481	17	0
Totals			8361	50455156	50992335	4047	4314

Sorry about the lack of commas in the table above. I used a very primitive spreadsheet to generate the data.

1990 Census data is from http://lwd.dol.state.nj.us/labor/lpa/census/1990/poptrd1.htm. Presidential vote totals are from http://www.presidency.ucsb.edu/showelection.php?year=2000.

Find Every Header File in Your Ubuntu Repositories

Last night I was following a thread on Penguin Pete's blog where Pete uses Ubuntu's dropping the Dillo browser from the distribution as a launching pad for a discussion about — heck, I can't describe what it's about, read it yourself.

Anyway, in the course of reading the discussion — if that's the word, Pete gets worked up about certain things — I was possessed by the moral imperative to install Dillo from source.

This would seem to be straightforward. You go to the Dillo home page, click Download, and follow the instructions.

Except (there's always an except) — the first install you have to make is the FLTK2 (Fast Light Toolkit) library.

Now Ubuntu offers FLTK1.1, which might well work, but hey, we need the latest and the greatest, right? So I downloaded the tarball, followed the instructions, and the make command crashed and burned with:

fatal error: X11/extensions/XInput.h

I looked on the web, found that XInput.h is in the package libxi-dev, installed that, reran make,

fatal error: GL/glu.h

and so on. It took about an hour to find every single header file needed. And that was just to install FLTK2, after which I still had to install Dillo.

So what's going on here? Well, first we have to know what a header file is. Basically, it's a file that defines a bunch of variables and statements that can be used by multiple routines. See Wikipedia for a better explanation. In particular, header files tell a program's source code what variables are used in a particular library. Thus the header XInput.h defines variables available in some X11 libraries.

OK, suppose I'm compiling a program and I find that I need a header file GrantsTomb/Buried.h. All I have to do is find the package that has that header, install it with apt-get, and I'm good to go. Right?

Well, without prior knowledge you're pretty much doomed here. I was able to find XInput.h and glu.h by web search, but who's going to tell you where is Buried.h in GrantsTomb?

Yet there has to be a solution, because:

• I know that when I use synaptic or aptitude to install packages I can see every package that is, or can be, installed on my system from my selected repositories.
• After searching around a bit, I find that apt and its children list the available files in /var/lib/apt/lists, each repository has its own file, and each file name is on a line starting with Package.
• I know about the program apt-file, in particular that the command
  apt-file list foobar
  will give me a list of all of the files in the foobar package.
• Header files are traditionally located in packages ending with the string -dev.

Given all of that, I should be able to write a script which will list every header file available in every Repository I've accessed. I can then save it in a file, and the next time a program says I need bojangles/tambourine.h I can just search the list, find that the header file is in the NittyGritty-dev package, load that up, and I'm ready to compile:

• First, install apt-file, if you don't have it already:
  sudo apt-get install apt-file
• Then run this script:
  

  #! /bin/bash
  
  # First make sure you've got all your repositories updated:
  sudo apt-get update
  
  # Next sync apt-file's database.  Note that if you do this without
  #  the sudo you'll write the data into your home directory, which is
  #  probably OK
  
  sudo apt-file update
  
  # Now search through every repository database, looking for lines that
  #  start with "Package", and end with "-dev".
  # Pull out the package name, look through each package, and print out
  #  the header files:  those that end in "-h"
  
  #! /bin/bash
  
  # No matter how it wraps on your screen, this next line starts with
  # "for" and ends with "xargs`"
  
  for hfile in `find /var/lib/apt/lists -type f -exec grep '^Package: ' {} ';' | awk '{print $2}' | grep -e '-dev' | xargs`
  do
  apt-file list $hfile | grep "\.h$"
  done
  

Be warned, this takes a long time, since every call to apt-file goes out over the network. On my system, with FIOS broadband, it took over an hour. So you'll only want to run this file once in a while, probably only when you update your repository list. You may also get an error message like
grep: /var/lib/apt/lists/lock: Permission denied
Don't worry about that.

This script should work pretty well for Debian. There are also equivalent commands for RPM-based distributions. And, of course, you can modify it to list any subset of files in any subset of repositories you want. I'll leave all of that as an exercise for the reader.