When He's Right, He's Right

Penguin Pete. Opinionated and passionate about it.

And boy, when he's right, he's right.

Baseball After the All Star Break

Some years ago, I did a predictive study on how Major League Baseball teams would rank at the end of the season, based on their records at the All Star Break and the Pythagorean projection of future wins, based on the runs scored and allowed by each team.

It didn't work all that well. In particular, I predicted that Boston would win the AL East pennant, and Washington would be the NL Wild Card. That sorta didn't happen.

Nevertheless, I'll try again. Here's the table, based on the MLB standings at the All Star Break. The method is the same as last time, so you can read all about it there.

American League
East	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB
New York Yankees	53	35	0.602	2	1	455	334	0.637	74	47.14	26.86	100.14	61.86	0.618	0.00
Boston	55	35	0.611	1	0	482	371	0.617	72	44.42	27.58	99.42	62.58	0.614	0.73
Tampa Bay	49	41	0.544	3	6	380	343	0.546	72	39.35	32.65	88.35	73.65	0.545	11.80
Toronto	45	47	0.489	4	11	426	416	0.511	70	35.76	34.24	80.76	81.24	0.498	19.39
Baltimore	36	52	0.409	5	18	355	454	0.390	74	28.85	45.15	64.85	97.15	0.400	35.29
Central	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB
Cleveland	47	42	0.528	2	0.5	386	382	0.505	73	36.85	36.15	83.85	78.15	0.518	0.00
Detroit	49	43	0.533	1	0	413	421	0.491	70	34.39	35.61	83.39	78.61	0.515	0.46
Chicago White Sox	44	48	0.478	3	5	366	383	0.479	70	33.55	36.45	77.55	84.45	0.479	6.29
Minnesota	41	48	0.461	4	6.5	347	414	0.420	73	30.69	42.31	71.69	90.31	0.443	12.16
Kansas City	37	54	0.407	5	11.5	402	449	0.450	71	31.94	39.06	68.94	93.06	0.426	14.91
West	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB
Texas	51	41	0.554	1	0	457	404	0.556	70	38.91	31.09	89.91	72.09	0.555	0.00
Los Angeles Angels	50	42	0.543	2	1	355	330	0.533	70	37.32	32.68	87.32	74.68	0.539	2.59
Seattle	43	48	0.473	3	7.5	301	319	0.474	71	33.63	37.37	76.63	85.37	0.473	13.28
Oakland	39	53	0.424	4	12	315	339	0.467	70	32.66	37.34	71.66	90.34	0.442	18.24
National League
East	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB
Philadelphia	57	34	0.626	1	0	384	295	0.618	71	43.86	27.14	100.86	61.14	0.623	0.00
Atlanta	54	38	0.587	2	3.5	365	312	0.571	70	39.96	30.04	93.96	68.04	0.580	6.89
New York Mets	46	45	0.505	3	11	399	388	0.513	71	36.40	34.60	82.40	79.60	0.509	18.46
Washington	46	46	0.500	4	11.5	352	354	0.497	70	34.82	35.18	80.82	81.18	0.499	20.04
Florida	43	48	0.473	5	14	352	396	0.447	71	31.71	39.29	74.71	87.29	0.461	26.15
Central	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB
St. Louis	49	43	0.533	2	0	433	407	0.528	70	36.97	33.03	85.97	76.03	0.531	0.00
Milwaukee	49	43	0.533	1	0	405	406	0.499	70	34.92	35.08	83.92	78.08	0.518	2.05
Pittsburgh	47	43	0.522	3	1	354	346	0.510	72	36.75	35.25	83.75	78.25	0.517	2.22
Cincinnati	45	47	0.489	4	4	437	408	0.531	70	37.18	32.82	82.18	79.82	0.507	3.79
Chicago Cubs	37	55	0.402	5	12	375	459	0.409	70	28.63	41.37	65.63	96.37	0.405	20.34
Houston	30	62	0.326	6	19	358	464	0.384	70	26.89	43.11	56.89	105.11	0.351	29.08
West	W	L	PCT	Place	GB	RS	RA	Pyth	GL	PW	PL	TW	TL	PCT	GB
San Francisco	52	40	0.565	1	0	332	322	0.514	70	35.97	34.03	87.97	74.03	0.543	0.00
Arizona	49	43	0.533	2	3	416	407	0.510	70	35.70	34.30	84.70	77.30	0.523	3.28
Colorado	43	48	0.473	3	8.5	395	407	0.486	71	34.53	36.47	77.53	84.47	0.479	10.44
Los Angeles Dodgers	41	51	0.446	4	11	340	373	0.458	70	32.06	37.94	73.06	88.94	0.451	14.92
San Diego	40	52	0.435	5	12	304	338	0.452	70	31.63	38.37	71.63	90.37	0.442	16.34

Abbreviations:

• W: Current team wins
• L: Current team loses
• PCT: Winning rate
• Place: Current place in standings
• GB: Games Behind
• RS: Total Runs scored by team
• RA: Total Runs allowed by team
• Pyth: Pythagorean expected win rate. Following MLB, I used an exponent of 1.82 rather than the original James value of 2. It doesn't make a lot of difference, and didn't change the order.
• GL: Games left in season for the team
• PW: Projected wins in remainder of season, assuming they win at the Pythagorean rate
• PL: Projected Pythagorean loses
• TW: Total wins, current + projected Pythagorean
• TL: Total loses
• PCT: Projected final winning ratio
• GB: Projected final games behind

OK, not a lot of changes going on. Despite an anemic offense, San Francisco's fantastic pitching will keep them in first in the NL West. Philadelphia will win the NL East going away, even though Atlanta wins the NL Wild Card. Texas will hang on in the AL West.

There are a few predicted swaps, highlighted in yellow: St. Louis will pull ahead of Milwaukee. And Cleveland will (yawn) edge out Detroit. Surprisingly, the only changes occur at the top, which probably says something about competitive balance in MLB.

And, finally, Red Sox will be the AL Wild Card. Which means …

Frak

The Home Team Wins Most Ties (In Baseball, Anyway)

I've been playing around with Retrosheet to see how often a baseball team wins a game if it's, say, five runs ahead at the end of the fourth inning. I plan to get that up sometime during this long weekend, but while doing the study I found another interesting result.

Retrosheet's Play-by-Play files, along with the cwevent program from Chadwick, let you extract all sorts of information from almost every MLB game played between 1950-2010. From that data I extracted every game that was tied at the end of a half-inning, and figured out who eventually won. Then I counted up the number of times the home team won for each half-inning. The results are shown below:

Click on graph to see a larger figure

The black diamond represents the situation at the start of the game, the red diamonds the situation where the game is tied in the middle of the inning, and the blue diamonds when it's tied at the end of an inning. We'll get to the error bars in a minute.

So what is all of this? Well, at the beginning of a game the score is obviously tied, so that should be part of the study. So if we look at all 115,748 games in the database, we find:

• The Home Team won 62,418 games,
• the Visiting Team won 53,192 games, and
• there were 138 games that were tied when the game was called.

If we throw out the ties, then the Home Team won 53.990% of the games that went to a decision. That's the black diamond at the far left of the graph. The 54% win rate is baseball's version of Home Field Advantage, and it has been very constant:

Click on graph to see a larger figure

I performed the same calculation for all the games which were tied at the end of a half inning. For example, if the game is tied at the end of the fifth, the home team has a 52.0% chance of eventually winning the game. Tied after the top of the sixth? It's up to 60%.

So what do the error bars represent? Basically they give you an idea of the number of games in the sample. Suppose that in a given game the home team wins with probability p. Then in an N game sample the probability that the home team wins n games follows the binomial distribution, e.g.

             N!        n      N-n
P(N,n) = -----------  p  (1-p)
          n! (N-n)!

If we look at a large number of N-game samples, then we'll find that on average the home team will win N p games, which makes sense. The standard deviation will be [N p (1-p)]^½. Since the graph normalized everything by the number of games played, the error bars are the standard deviation divided by N, or [p (1-p)/N]^½. Since most values of p are between 0.5 and 0.7, wider error bars basically tell you that fewer games have gotten to that point. And when the error bars get really wide, as they do after the fourteenth inning or so, it says there aren't enough statistics available to give you meaningful information.

What does it all mean, you ask? Well, first it says that the home team advantage is real. Why there is a home field advantage is another question, and there is not enough information here to answer that question.

Then there's the observation that the home team has a larger advantage if the game is tied in the middle of the inning than it does if the game is tied at the end of an inning. That's just common sense. In the middle of the fourth inning, the visiting team has five more innings at the plate. The home team has six — five for sure, and one more if they need it. This isn't the home team bats last advantage, it's the home team gets one more at-bat than the visitors advantage, not the same thing.

Next, we see that if the game is tied at the end of an inning, the home team's advantage decreases slightly, so that at the end of eight innings it's only 51.94%. Presumably that's because the home team does have some advantage in being at home, but as the game progresses they have less and less chance to use that advantage. After the fifth inning the home team's advantage oscillates around 52%, down from the 54% advantage they had at the start of the game.

Indeed, the fact that the blue dots go down from innings 1-4 suggests that the home team bats last advantage isn't worth a whole lot. If it was, you'd expect the advantage to be greater in tie-game situations as the game wears on, because that last at-bat becomes a larger and larger proportion of what's left of the game.

Finally, there is that dip in the red diamonds between the first and second inning. If the game is tied going into the bottom of the first, meaning that the visiting team didn't score, then the home team will win 59.15% of the time, with a standard deviation of 0.17%. If the game is tied going into the bottom of the second, however, then the home team only has a 58.01% chance of winning, with σ = 0.22%. That's a five-σ change in the probability, which I would think is statistically significant. The win rate is pretty much constant in the third inning ( 58.16% ± 0.27%) and then starts going up, as you'd expect, since the home team has proportionately more at-bats than the visitors at the middle of an inning.

Why is this so? I have no idea. All I can say is that if your a visiting baseball team, it's better to be tied with the home team in the middle of the second inning than it is to be tied before the home team comes to bat. And you better be ahead by the middle of the third.

The information used here was obtained free of charge from and is copyrighted by Retrosheet. Interested parties may contact Retrosheet at www.retrosheet.org.

Near Miss

You may have heard that Asteroid 2011 MD will pass by Monday morning. And by pass by, we mean within 8,000 miles.

That's pretty close. How close? Well, have a look at these animations:

Asteroid 2011 MD Flyby

An Update on the Average Major League Hitter

This is the promised update to my table of pretty good averages for everyday baseball players at the major league level. As before, I consider an everyday ball player to be one who was eligible for a league batting title in a given year, which from 1996-2010 means that he made at least 502 plate appearances in a given year.

The raw data for this study was taken from the mySQL database thoughtfully provided by the folks at Baseball-DataBank.org. As outlined in yesterday's post, I used mySQL to pull out the appropriate data, this time using the command:

select b.yearID as Year, m.nameLast as Last, m.nameFirst as First,
b.teamID as TEAM, b.G, b.AB+b.BB+b.HBP+b.SH+b.SF as PA, b.AB, b.R, b.H, b.2B,
b.3B, b.HR, b.RBI, b.SB, b.CS, b.BB, b.SO, b.IBB, b.HBP, b.SH, b.SF, b.GIDP
from Batting b inner join Master m
where b.playerID=m.playerID and b.yearID>1995 and
b.AB+b.BB+b.HBP+b.SH+b.SF > 502
order by b.yearID ASC, m.nameLast, m.nameFirst;

to get a list of all players from 1996 on who were eligible for a batting title. I then used mysql-query-browser to export all of the data into a spreadsheet, and there computed all of the averages and standard deviations. All the calculations are the same as in my original post, except:

• I added the batting data for the 2009 and 2010 seasons.
• My original post fraked up David Smyth's Base Runs statistic. I used the right formula (the second one on the page), but miscalculated total bases by forgetting that doubles, triples, and home runs are already counted as hits. So the numbers found in my earlier study are too high.
• I dropped 1995 from the study this time because only 144 games were scheduled for each team, so the batting eligibility criterion was 144 × 3.1 = 447 plate appearances. Just lazy.

Read more ⇒

Take Me Out to the SQualL Game

As some of you know, I have a moderate interest in Baseball and Baseball statistics. However, I'm not a database programmer, and the little bit of DB manipulation I've looked at has left me hopelessly confused.

Several years ago I did buy a copy of Baseball Hacks, but I never really got started with it. Last weekend I had some time on my hands and started playing around with it. Turns out you actually have to try to do something with a program in order to learn it (who'da thunk?). And it also turns out the Baseball Hacks, even though it's an O'Reilly book, is pretty oriented to Windows/Microsoft Access, although it does have substantial hints for Linux and Mac users, not to mention an online collection of scripts from the book (ZIP file), along with some other stuff.

So what should we do? As a first shot, how about updating my table of averages for several modern baseball statistics to include 2009 and 2010? I did the previous tables by doing some judicious editing of the Batting.csv file in Sean Lahman's baseball database, but the folks at Baseball-Databank.org have all of the data packaged neatly into a mySQL database (ZIP file), so we'll use that.

So as a start, we're going to

• Install appropriate parts of the mySQL database program in Ubuntu,
• Set it up to read the database file,
• Find the eligible batters for 2009 and 2010,
• Get their batting data into a spreadsheet,
• Find the appropriate averages, runs created, etc., and
• add the results to the appropriate tables.

That should be enough for one day. It's going to be a long journey, though, so when you've got some time join us after the break.

Read more ⇒

The Natty Narwhal Upgrade

Like Travis McGee, this week I'm taking an installment of my retirement. Unlike McGee, my retirement seems to consist of fixing up computers.

So today, Ladies and Gentlemen, Boys and Girls, Desktops and Netbooks, we're going to talk about upgrading Hal here to Ubuntu 11.04.

Yawn. This shouldn't really be that big a deal, should it? You just hit the update button, or type do-release-upgrade at the command line, right?

Well, it's not always that simple. A friend of mine did that, and he seemed to have a lot of problems. I'm not sure why, as I never got to examine his system before he reinstalled 10.10. Maybe it was the change from the Xorg window system and Gnome desktop to Wayland and the Unity desktop. Or not. Maybe it was just the way the wind was blowing that day. But I decided not to take the chance on the upgrade, but rather install 11.04 from the ground up.

I didn't have to be pushed hard, because I was leaning toward a clean install anyway. When I bought this version of Hal I was in a hurry to get Linux set up, but wanted to keep Windows 7 around. As a result, I let Ubuntu do the repartitioning of the disk, and ended up with the entire Linux system in one giant partition.

This is less than optimal. (Thanks, Dave. Don't mention it, Hal.) Ideally one wants /home and possibly /usr/local and /opt in partitions separate from the root operating system. That way when you upgrade the OS, you don't lose your previous data. (That doesn't mean you shouldn't back up that data before upgrading. There's a word for people who don't. The polite form of it is idiot)

But this requires repartitioning Hal's disk. Fortunately, all of Hal's memories reside in the extended partition, so I don't have to fiddle with the Windows partition. After about 30 secons of thought, I decided to repartition Hal's big Linux partition into 15GB for /, the root system, another 15GB for /usr/local, /opt, and /scratch (more on how to do this later), and leave the rest for /home. That's probably too much for /, and maybe not enough for /scratch, depending on what I right there, but that's how I'm going to do it this go-round. Ideally gparted would leave the data on /home as is, but that takes forever because of all the data that needs to be moved around, so we're just going to repartition everything and hope the backup holds.

Preparation

First, get the software we'll need. I downloaded and burned CDs for the 64-bit versions of:

• Ubuntu 11.04
• openSUSE 11.4
• Fedora 15
  and, last but probably most importantly,
• Gparted Live CD version

OK, Gparted is obvious: that's what's going to do the disk repartitioning. But why the alternative distributions?

Because you never know. The largely hypothetical long-time reader will remember that I used to use Fedora, but that I switched to Ubuntu when the installation of Fedora 5 failed miserably, and Ubuntu was there, ready and waiting. So having alternative Linux CDs on hand seems to be a really good idea. Besides, we bought 100 CDs maybe five years ago and still have a bunch left. They just aren't used much anymore. (That's right, rcjhawk is still a trailing indicator for device popularity.)

OK, we've got hopefully every bit of software that we need, so let's go.

Find out what's on your machine

I never remember which packages are installed from one upgrade to the next. So let's make a list:

dpkg --get-selections > installed.txt

Edit installed.txt, leaving out any packages you think will be installed by the system (e.g., kernel, window manager, etc.). If you can't remember what a package does, delete it too. If you need it, it will eventually show up as a dependency to a package you want, or you can reinstall with apt-get or synaptic. The important thing to do here is to remove cruft from the system. For example, I have the bsd-games package on here. I never play those games, so why leave them in place?

It would have been nice to make a copy of /etc/fstab, but some dingbat forgot to do that (more on this later).

Back it all up

Well, not everything. No need to back up the kernel, or Emacs, or latex, or any of that. The distribution will provide. No, I want to back up my data (basically $HOME), the Intel Fortran compiler, a royal pain to install, and my renegade version of SoX. So do the following (/backup is my ext4 formatted USB disk):

$ cd
# -rpv == recurse directories,
#  preserve permissions and timestamps,
#  and speak up about it verbosely.
$ cp -rpv dave /backup
# Assuming you're named dave, of course
$ cd /usr
$ cp -rpv local /backup
$ cd /
$ cp -rpv opt /backup

To be safe, I also had backintime, my backup manager, do an additional labeled backup, which should keep it on the disk forever and a day, or at least until the end of 2012.

Repartition

The next step is the scariest one. I've taken what was an 800+GB partition and broken it up into three parts: 15GB for /, 15GB for what will be /usr/local, /opt and /scratch, and the remainder for /home. It's scary because this wipes out the primary data, so you are now relying on the kindness of backups. It's theoretically possible to gave gparted to keep all the data in what's going to be the /home partition, but that requires moving a lot of stuff around, at least the way I tried it, and an estimated 18 hours to get it done. So I just said a prayer, bit my lip, and reformatted. Fortunately it all went well.

Installation

Not much to say here. I told Ubuntu that I wanted to do an install with a custom installation, had it mount everything where I wanted it, and let it go. The usual waiting around occurred.

Copying Files

This was rather easy: turn on the backup USB disk drive, and copy the files back. I copied back configuration files I knew I wanted to keep, e.g. things like .emacs and the .devilspie directory, but I'm letting Ubuntu set up the desktop. This means I'll have to play with preferences later, but it should get eliminate any conflicts between old and new configuration parameters.

Mounting the Backup Drive

I could just plug the backup drive in and access it at /media/really_long_hexadecimal_string, but what I really want to do is have the backup drive mounted to /backup. I've covered this before, but as I mentioned above, I forgot the UUID of the drive. You get that with the command:

blkid

which gives you all the UUID of all connected disks.

Restoring /usr/local and /opt

This is awkward. Add-on programs that you compile yourself generally go into /usr/local. Other programs, such as the Intel Fortran compiler and Google's Picasa, end up in /opt. I want both to stick around between upgrades, so they need to be out of the / partition. But I don't want two extra partitions. So here's what I did:

• Create a partition /usr/local
• Create a directory /usr/local/opt
• Copy all of the /usr/local files where they belong.
• Copy all of the /opt files to /usr/local/opt
• Then, as root, run the command
  ln -s /usr/local/opt /opt
• If you want a /scratch directory, do the same thing, but don't forget to make the original world writable:

  # mkdir /usr/local/scratch
  # chmod 777 /usr/local/scratch
  # ln -s /usr/local/scratch /scratch
  

Restoring your add-on packages.

Now we want to get back all of our old packages, the ones that aren't installed by default, at least so far as Ubuntu will let us. Remember that file installed.txt I mentioned you should create and edit? Here's how we'll use it:

• Select System => Administration => Update Manager
• Click on Settings in the lower left corner
• Under Ubuntu Software make sure all of the sources you want enabled are. If you don't want proprietary drives, etc., turn them off. Do the same under Other Software
• Open a terminal window and go the the directory where you have that installed.txt
• Run the following commands:

  sudo apt-get update
  sudo apt-get install `awk '{print $1}' installed.txt | xargs`
  

• You'll probably get some error messages, but they're pretty clear about what you need to fix. Edit the installed.txt file to match.
• This will take a while, as all the packages have to be downloaded from the net.

What Works

Pretty much everything. I selected Gnome-classic for my desktop, and after a little fiddling to with System => Preferences => Startup Applications got the system to look pretty much as it did before.

As for third party software, the Intel Fortran compiler still works fine. I was able to reinstall Picasa from Google's supplied 64-bit .deb.

Wayland seems to work just like Xorg, at least from my point of view. It pops up windows in the same way as before, and I can run, say
ssh -X majel
to get me to another machine, and then run
firefox
on majel, and the window pops up here on Hal. So no major problems for me there.

What doesn't work

Google Earth. Maybe this is a 64-bit problem, I don't know. I tried using Google's .deb file, and the official Ubuntu method. Neither worked.

By default Ubuntu installs the 3-D version of Unity, which requires a pretty good graphics accelerator. Hal lives in 2-d, so that wouldn't run. (Funny, the Unity desktop shows up when you run the live CD.) I installed the 2-D version and that works. Which brings me to

What Sucks

Unity. From my limited experience (about two minutes, after which I ran away screaming), it's an overblown and somewhat hideous version and of the Mac desktop. But don't mind me, I was the last person on Earth to use fvwm. I suppose if I played around awhile I could make Unity behave the way I wanted it to, but why bother, since Ubuntu still supplies the classic Gnome desktop. Mind you, I've seen Gnome 3, and I don't like that, either. (The word you're looking for is Luddite.)

Summary

All in all, a successful update, as long as I stay way from Unity. I have one more computer that needs updating to 11.4, I think I'll just try that as a distribution update. If it doesn't work, I can always do a full install.

More Later

Troubles will surface, they always do. When they do, I'll write about them here.

Compression

The other day an email arrived from sourceforge which mentioned that they were hosting the file compression program 7-zip. Now I had used 7-zip under windows, as an all purpose archiving tool, mostly for reading zip files. I'd never thought of it in a Linux context. But both openSuSE and Ubuntu have the command-line version available, and what more do you need?

In OpenSuSE, the RPM is called, simply enough, 7z. In Ubuntu it's a bit more complicated. There's p7zip, which provides the bare-bones standalone version of the compression program, 7zr, and a wrapper p7zip, which makes 7zr work like gzip. For the full-blown 7z program, you want the package p7zip-full, which includes 7z. While you're at it you might want to get the p7zip-rar package, which lets you decompress RAR files.

The reason you want 7z and not just 7zr is that the smaller program only compresses to the 7z format, but, as it says in the blurb,

not only does [7z] handle 7z but also ZIP, Zip64, CAB, RAR, ARJ, GZIP, BZIP2, TAR, CPIO, RPM, ISO and DEB archives.

And by handle, they mean read and write to these formats (except RAR). So I could create a zip archive with 7z, or a gzip/bzip2 compressed tarball. And guess what?

7z compression is 30-50% better than ZIP compression.

Is it? Well, let's find out.

Test 1

Presented for your consideration: an uncompressed tarball:

$ ls -l Ru.tar
-rw-r--r-- 1 dave dave 26716160 2011-05-21 15:41 Ru.tar

This is IO from the elk FP-LAPW code, so it has a lot of repetition of text, and what for our purposes are a bunch of random numbers. First we'll try compressing it with programs using their native formats. I'll try for maximum compression in all cases. Note that zip and 7z will create separate archives, while gzip and bzip2 compress the file in place.

Program	Command	File Size	Ratio
zip	zip -9 Ru Ru.tar	3899523	0.146
gzip	gzip -9 Ru.tar	3899386	0.146
bzip2	bzip2 -9 Ru.tar	2992422	0.112
7z	7zr a -mx=9 Ru.7z Ru.tar	2242708	0.084

Pretty good, huh? As advertised, 7z is about 40% better than zip/gzip, and 25% better than bzip2. But wait, there's more. Not every computer is going to have 7z available, so you may want to compress files using a more established protocol. 7z can do that, too, which is why we wanted it, not just 7zr:

Format	Command	File Size	Ratio
zip	7z a -mx=9 -tzip Ru.zip Ru.tar	3287420	0.123
gzip	7z a -mx=9 -tgzip Ru.tar.gz Ru.tar	3287335	0.123
bzip2	7z a -mx=9 -tbzip2 Ru.tar.bz2 Ru.tar	2989193	0.112

So 7z compresses to zip/gzip better than the native programs do it themselves. It doesn't really outperform bzip2 here, though. The only disadvantage compared to gzip or bzip2 is that it doesn't compress the files in place, unless you go through a script such as the one in /usr/bin/p7zip.

Test 2

Pi to 4 million Decimals has, duh, π to, actually, 4,194,034 places. The file pi.tar.gz has it in ascii, with a bit of header information. If we uncompress that file, it comes in at 4362370 bytes. Since the digits of π don't repeat, it's hard for a compression program to find blocks of bytes to compress. The following table lists the compressions achieved by our test programs, in whatever formats they can use. (See the above tables for the appropriate commands.) Let's see how everybody does:

Program	Protocol	File Size	Ratio
None	None	4362370	1.000
zip	zip	2041130	0.468
gzip	gzip	2040997	0.468
bzip2	bzip2	1863892	0.427
7z	zip	1983378	0.455
7z	gzip	1983297	0.455
7z	bzip2	1860047	0.426
7z	7z	1884999	0.432

Here, bzip2 is competitive with 7z. Oddly, though, you should use 7z to do the bzip2 compression. Weird, huh? But still, 7z is pretty good.

Wrapping Up

So what's not to like?

For one thing, 7z is slow. If you just want to quickly compress a file, go ahead and use gzip, or bzip2. There is a price to pay for better compression.

Then, too, there's a warning on the 7z man page:

DO NOT USE the 7-zip format for backup purpose on Linux/Unix because 7-zip does not store the owner/group of the file.

you can get around this by piping tar into 7z:

tar cf - directory_to_be_archived | 7z a -si directory.tar.7z

which creates the analog of a gzip/bzip2 tarball.

And finally, the native 7z format isn't standard, yet, so it's not going to be available everywhere, and might even vanish. But 7z and its compression algorithm LZMA are open source, so they are likely to stay around for awhile. A few years ago bzip2 wasn't standard, and once upon a time neither was gzip. It's probably safe to compress your files to 7z format, but if you want to be safe, use 7z to compress to gzip or bzip2 format.

Why I Hate March

I just spent the last three hours not watching the Kansas-Boston University first round NCAA game. In case you didn't join me in not watching, KU spent the first 25 or so minutes wondering why the BU players weren't genuflecting. Then they concentrated for a few minutes, and the game was over. Because of course, you know, when #1 meets #16, #16 has never won, right?

Trust me. Some day it will happen. And which team will that be? We'll, let's look at the past for some clues, starting here:

• 2010: #1 Seed, lost in second round
• 2007: #1 Seed, lost in Sweet 16
• 1998: #1 Seed, lost in second round
• 1997: #1 Seed, lost in Sweet 16
• 1995: #1 Seed, lost in Sweet 16
• 1992: #1 Seed, lost in second round

Not to mention Bradley, or Bucknell, or the suffering I did during the Ted Owens years.

It's enough to make one start a website, except a) someone beat me too it, and b) Roy was as bad, or worse, because he didn't win a Championship until he clicked his heels three times and went home.

But mark my words, the #1 Seed in NCAA history to lose in the first round will have a six-letter name on the front of the jersey, and have two Crimson and Blue mascots that look something like this.

Superbowl Commercials

There weren't many that were any good this year. I'll show you my favorite, below, but I found that a lot of people didn't get the reference. So let's go back to September, 1979, when this famous commercial aired, starting Pittsburgh's Mean Joe Green:

And now that we've set the stage, here's my favorite commercial from this year's Superbowl:

And one more reference to TV gone by:

FUSE on the Mac

Awhile ago I wrote about how one can mount remote computer filesystems using ssh. That was for Linux. You can do it with Mac OS X as well, but FUSE (File System in UserSpace) isn't installed by default, as it is in modern Linux systems.

So first you need to get FUSE installed. The package I use is called, duh, MacFUSE. It installs like a standard Mac package: Download the disk image, click on it, and go.

Next is sshfs itself. There seem to be many versions, but Google's seems to work well. Download the appropriate version for your OS (the Leopard version works on Snow Leopard), rename it sshfs, make it executable, and put it in your path. From then on it works pretty much like in Linux, with minor differences. Assuming you have a directory ~/hal on your Mac, you can look at the files on your Linux machine hal with the command

sshfs rchawk@hal:/home/rcjhawk ~/hal

So a file named /home/rcjhawk/Documents/testthis.doc on hal will appear as ~/hal/Documents/testthis.doc on your Mac.

And it's simple to unmount it:

umount ~/hal

A More Perfect World

Me, to.

Wikipedia's List of Common Misconceptions.

Happy New Year

This blog started just over seven years ago, and sometime tonight or tomorrow it will get its forty thousandth recorded hit. Thanks to all of you for listening.

Thanks to a Penguin Pete rant (see Sitemaps). I've discovered Google's webmaster tools and Analytics. My New Year's Blogging Resolution is to learn how to use these next year.^† If I come up with any neat tricks I'll let you know.

In the meantime, Happy New Year, everyone.

^† My primary resolution is to still have a job next year.

Should We Call It gnarcissism?

Linux Journal, or, more properly, the janitor at Linux Journal since everyone else is off this week, is running a pole asking if you Google® yourself.

The answer I'd choose isn't there:

No, because there are at least a half-dozen people^* with the same name who are at least as famous as I am, and that doesn't count the 1,300 or so in China.^†

^* Yes, I know, but only for the purpose of this post.^§

^†Working from the old if you're one in a million line.

^§That's my story, and I'm sticking to it.

The Christmas Toy

So here I am, Christmas Evening, waiting for BBC America to broadcast the Dr. Who Christmas special and playing with my new toy: a Logitech Harmony 650 Universal remote. I bought the thing at Radio Shack because: a) It was cheap ($69.95), and b) I'm sick and tired of having to manipulate three remotes to change from FIOS to BluRay to USB/Harddisk when watching shows on the bigscreen TV.

The neat thing about the Harmony series of remotes is that you program it on-line. The bad part is that there isn't a native Linux program to do it. There is something called Concordance which is supposed to work, but I'll try that out when I'm prepared to trash the remote if the software fraks out, OK? It's not worth the struggle (forgive me, Saint Richard).

So we boot Hal up in Windows 7 mode, and I'm blogging what's going on using a laptop.

• First: gather up the model numbers of the gadgets to be used. That's fairly easy. I've got a Samsung LCD 46 inch TV, and the model numbers on the side. A Samsung BluRay BD-P360. The hardest part is the FIOS DVR. It's by Motorola, and looking at the pictures on the FIOS site it appears to be a 6416. We'll see.
• Launch IE8 on Hal (NNNNOOOOO!!!!!!!). Go to http:www.logitech.com/myharmony and create an account. Click on My Account. No registration box. Click on the safest looking link, Go to My Stuff There's a login screen and a registration option there. For some idiot reason they want a daytime phone number. Tempted to give them the one for our security, but refrain.
• Click on Product Support. Says it will detect USB corded product. The unit comes with a USB 2 cable, but of course the connector on the remote side isn't standard USB2. Annoying, especially if I loose the cord, but I sort of forfeited the moral outrage option when I booted into Windows. So plug it in, let Windows find the driver software, and click the Detect your devices button. Of course Logitech has an IE add-on that has to be installed. It seems to work. It says I've got something called an N/N: N-10003 Harmony 650 remote. Well, I know the Harmony and 650 part are right.
• It's not particularly obvious what to do next. Click Download Software and get version 7.7.0-WIN-x86 version of LogitechHarmonyRemote, I guess. Install that. You know, I'm kind of glad I'm doing all of this on the Windows side. Installation involves the usual number of clicks to accept the license. Eventually an Icon shows up on my desktop and the thing launches. First thing it wants to know is if it should check for updates. Surprisingly, there are none.
• Log in. You'd think the account I just set up with Harmony would work, but no. So set up another account here. It's another complete registration, folks.
• Now things start. It finds my remote (again). Keep hitting Next until we get to the Add Device page. Adding devices seems straightforward, so long as you have the right device numbers. Fortunately they give you examples of how they want the information entered in.
• Once I finally got the right models in, the software found my devices. The only question I had to ask is if I wanted to turn the cable box off when it wasn't in use. I said yes. We'll see if that's a good idea or not. (I worry what's going to happen when I switch devices. Is the TV going to be out of sync with the cable box?)
• Specify that FIOS uses HDMI1 on the TV, and the BluRay uses HDMI2.
• And somehow we lost the signal, so I HAVE TO LOG IN AGAIN???? And it lost the connection to the remote. But now it has, indeed, started downloading the settings to the remote.
• Looks done. I'm told to disconnect the remote and go try it out.
• Preliminary tests done. It mostly works. There are several glitches, though:
  1. Both the TV remote and the FIOS remote have options for resizing the picture on the screen. It's especially handy when you are watching a standard definition channel, but they're broadcasting something in letterbox format (as in the above-mentioned Dr. Who special). Resize the picture, and it fits the whole screen. Not high-def, but in this case bigger is better. The 650 doesn't have such a button.
  2. The FIOS remote has a standalone DVR button, which bypasses the main menu and takes you to the stuff you use most. The 650 doesn't, you have to go through the main Menu (though the Menu, Info, and Guide buttons work exactly the way they should, as does the Favorites option on the 650's screen).
  3. The worst problem: If I want to stream Netflix using the BluRay, I press a button labeled B on the BluRay remote. One of four buttons, Red/A, Green/B, Yellow/C, Green/D. The 650 has unlabeled Red/Blue/Yellow/Green buttons, but they don't do anything to the BluRay. Since I stream Netflix as much as watch DVDs, if not more, this is a serious difficulty, and a deal-breaker.
• Back to Win-Hal. Click the I need to make some adjustments box. Log in again. At least this time it remembered my password.
• There's a Customize Buttons section for the remotes. For the BluRay, set up Blue to be Netflix, Yellow Pandora, and Green for YouTube. A/Red on the BluRay is Blockbuster, which we don't use. For the FIOS box, set A on the screen (not to be confused with A/Red below) to Aspect, which should set the screen size properly. While we're at it, set B up there to Close Caption, something we use a lot since we're old. Do that for the BluRay as well. I may have to tweak that later. Can't find a DVR button option, but I may have a way around that. Let me test out the new setup and get back to you in a minute.
• OK, those changes work. Nice touch: the buttons attached to the screen are labeled A/B/C..., but you can change those to Aspect/Closed Caption/On Demand, etc.
• Now let's tackle that DVR button problem. On the setup screen for the FIOS remote (labeled PVR in 650's language) there is a button labeled Learn Command. Click that, and go down until you find Learn a New Command. Type DVR Menu in the box and click the Learn a New Command button. Put the front of the FIOS remote to the back of the 650, press the DVR button on the FIOS remote. Wait. Go back to the Additional Buttons setup screen, and assign the button you want the new DVR Menu command.

And, for now, that's enough. If I encounter more difficulties I'll let you know. In particular, we'll see if the family can use it easily or not.

Preliminary bottom line: This isn't all that easy to program, especially if you have an exotic system. But it does seem to be straightforward to program, and you can add special keys as needed. We'll see how it all works in practice.