Office 2007 With Wine

Every once in a while I need a real honest to Bill version of Microsoft Office. Not often, but it happens.

Now I have a three license copy of Microsoft Office Home and Student Edition, purchased back in the day when Child No. 1 had a Windows computer. That computer is long gone and replace by a Mac. I could install this copy on Hal's Windows partition, but that means I have to reboot and update Windows 7 every time I want to use the software. Irritating.

Then yesterday I was updating my Harmony Remote, and found that the update software would run on Linux under Wine. True, the fonts were ugly, but hey, it ran, and given that updating a remote is a run-up-and-down-the-stairs-10-times kind of thing, having one less step was a blessing.

Now I've tried Wine before, and never been all that happy with it. But maybe it's now good enough, so what the heck.

Having installed Wine via the Synaptic Package Manager, I went looking to see if there was help for the Office install on the internet, and lo and behold, I found How to Install Microsoft Office 2007 in Ubuntu (Under Wine).

What can I say? As I said, I already had Wine installed, so it was mostly point at setup.exe and click. What do you know, it worked, right down to finding the riched20.dll library and correcting it so that PowerPoint would run.

Now this was a copy of Office 2007, so it's a little out of date. Fortunately, Microsoft is providing extended support until 2017, so it's still viable. There was a problem, though: Microsoft Update wants to work with Internet Explorer, and I don't have that installed under Wine — in fact, I don't know if I can install it properly, and I'm not inclined to try. But you can go fetch the update directly from Microsoft. Just download, make it executable, and run it from a terminal, ignoring all of those evil looking error messages.

A couple of things: LMDE puts all of the Office Suite in the Other tab of the menu. You can edit the menu, make an M$Office tab to hold everything, and move it there. Second, dialog boxes often revert to Wine's font, which is horrible. But Office itself uses native Office fonts, which is what we really want.

Now I have no idea what happens if you try this with a newer version of Office. This works for Office 2007, which was written for Windows XP. If you get a newer version running, leave a comment.

Taking POODLE to the Pound

You may have heard of POODLE (Padding Oracle On Downgraded Legacy Encryption), which exploits a bug in the ancient SSL 3.0 Encryption Protocol. This has the capability of letting malicious sites take over your browser.

The fix is to turn off SSL 3.0, forcing your browser to use better encryption. This will break some sites, but that's their problem, not yours, and they are probably working on it right now.

For most browsers the fix is not too difficult. You change some configuration setting and you're done. There's even a Firefox Extension to change the settings for you.

For Google Chrome and its open source brother Chromium, it's a little more difficult. You have to tell the browser to disable SSL 3.0 every time you launch it, e.g.

$ chromium --ssl-version-min=tls1

the --ssl-version-min=tls1 being the string that does the trick.

Let's see. I launch chromium at startup, from the Mint panel icon, and sometimes from the command line. That means three places I have to fix the call to chromium, and I have to do it for every user on the machine. There's got to be a better way.

And there is, at least when using chromium on LMDE. There is a configuration file, /etc/chromium/default, which lets you set global options for the chromium browser. To apply the fix, run the command:

$ sudo vi /etc/chromium/default

and edit the CHROMIUM_FLAGS variable. This passes a set of commands to chromium every time anyone starts the browser: look at /usr/bin/chromium to see how it works. My current variable reads

CHROMIUM_FLAGS="--password-store=detect --ssl-version-min=tls1"

Now every time chromium is started on your machine, it applies the fix.

You can test your work at https://www.poodletest.com/.

I don't know how many platforms can use this trick. On CentOS there is an analogous way to do it, but a different procedure. If you have a different way to disable SSL 3.0 for chrome/chromium on your Linux box, leave a comment below.

Updating Firefox and Thunderbird in LMDE

A couple of weeks ago, as you may recall, I switched from ever-changing Ubuntu to nearly frozen Linux Mint Debian Edition (LMDE). So how is it?

Nice. Not only does it have the Gnome 2-like MATE desktop, but because it is based on Debian Testing (currently known as jessie), LMDE is relatively slow to change.

What's irritating about LMDE is that, because it is based on Debian Testing, it is relatively slow to change.

This really isn't too much of a bother, except for Firefox and Thunderbird, which change versions at the drop of a hat, not that anyone wears a hat they can drop anymore. So, for example, as of this moment (it could change by tonight), Firefox is on version 23, and Thunderbird is on version 17.0.8. LMDE's versions, OTOH, are at 21 and 17(.0.0).

OK, this is not one of life's big tragedies. Most updates of Fox&Bird do not involve major changes. But I like to keep a little more current.

Fortunately, there is a relatively easy fix for this which won't do too much damage to your installation even if it completely craps out. The details are all in this LMDE forum post and the one above it, but I've added a little twist of my own, based on a post even further up the topic than the ones I've already mentioned and linked to.

What makes this relatively easy is that LMDE stores Firefox & Thunderbird files in the /opt directory, since pure Debian doesn't support them under their default names because of trademark issues (see iceweasel and icedove). That means any mucking around you have to do is confined to /opt, rather than such touchy directories as /usr/lib and the like.

So what to do:

1. Install Fox&Bird, if you haven't already:
   $ sudo apt-get install firefox thunderbird (N.B. If you haven't already installed Firefox or Thunderbird, and don't miss them, you might ask yourself just what you are doing reading this post.)
   
   
2. Become the superuser and go to the /opt directory.
   sudo -i
# cd /opt
   
3. Back up the firefox and thunderbird directories. This lets you get back to the original versions if you frak everything up. If you are particularly paranoid, back up your $HOME/.mozilla and $HOME/.thunderbird directories in the same way.
   # cp -rp firefox firefox_21
# cp -rp thunderbird thunderbird_17.0 The -rp options recursively copy everything and preserve all permissions and time stamps.
   
   
4. Optional step: Change the ownership of firefox and thunderbird. If you do this, then you will be able to update both programs without becoming superuser. Otherwise, you'll have to launch the apps using sudo to do the upgrades. Technically this a regression, as it allows someone to update a core component of the machine without root access, i.e., it behaves like Windows XP. On a single user machine this is probably not a big problem.
   
   Assuming your username is, say, capaldi,
   # chown -R capaldi:capaldi firefox thunderbird If you do this step you can now get out of su mode:
   # exit
logout
$
   
5. Now for the trick. You need to put Fox&Bird into the release channel:
   $ vi /opt/firefox/defaults/pref/channel-prefs.js When you get here, look for a line that says
   pref("app.update.channel", "default"); and change default to release.
   
   
6. Do the same thing for
   $ vi /opt/thunderbird/defaults/pref/channel-prefs.js In my copy, this was already set, which is why Thunderbird kept asking to update (and always failed, since it I wasn't root).
   
   
7. Restart either app. In the Help menu click on About Firefox/Thunderbird. Updates should appear normally, though I had to go through the process a couple of times to get a successful update. In Thunderbird it took forever to do the update, but when I Xed out the update window and restarted Thunderbird it was properly updated. I'll let you know if this trend continues with the next update, which might occur as early as this week.
   
   
8. Now if and when LMDE does push Fox&Bird updates, all of your lovely work will be overwritten. You can do that by locking the package. The easiest way to do this that I know is:
   
   
   1. Open synaptic. If you don't have it,
      $ sudo apt-get install synaptic
      
   2. Search for firefox and click on it.
      
      
   3. In the synaptic menu bar, click Package.
      
      
   4. Click Lock Version.
      
      
   5. Search again for Firefox and make sure it is locked. The whole Firefox line should be red.
      
      
   6. Repeat for Thunderbird.
      
      

And that should do it. Remember, if things get messed up, you can always use your backup directories to get back to the default distribution, or you can do complete remove/reinstall:
$ sudo apt-get purge firefox thunderbird
$ sudo apt-get install firefox thunderbird

LMDE

As long-threatened, I've finally followed the crowd and move from Ubuntu to Linux Mint. The main attraction was Mint's use of the MATE desktop, which is enough like Gnome 2 to keep me happy indefinitely, but I also liked the fact that the Debian edition, hereafter LMDE, is a rolling release, meaning I'm not going to have to do a complete upgrade in 6 months. The price is that some software is going to be a little behind the times, but, hey, at work we use CentOS. At least with LMDE it's unlikely that I won't be able to run Google Chrome or Chromium.

So a brief review:

Installation was simple enough. After several hours of backing up my /home partition (twice, to two separate USB disks), I downloaded the DVD and did the initial install. I left my /home and /usr/local directories, which were on separate partitions, intact, and let LMDE overwrite the root directory and install its version of Grub to run through the boot process. This took less than an hour, which was a pleasant surprise.

It then took me another couple of hours to pick out all the packages that weren't automatically installed, and get them up and running. Not too bad, really.

There were, of course, a few things that didn't quite work perfectly.

• My Gnome 3 Random Wallpaper Switcher did not, of course, work for MATE. Fortunately, the differences between gconftool and mateconftool are minor, and so I was able to rewrite the script without too much trouble. The revised program can be downloaded here.
  
  
• CUPS gave me some problems at first. I eventually found a solution in the Linux Mint Forums. Note that you have to do two things:
  • Enable 32 bit applications to work on a 64 bit platform, using the command
    sudo dpkg --add-architecture i386
    (My own humble contribution.)
  • Make sure libgs9 is properly installed:
    sudo apt-get install --reinstall libgs9
  
  More details about this are here.
  
  
• Google Earth actually worked, unlike in 64 bit Ubuntu. Well, not quite. At first, when I typed in a request, I got an invalid HTTP request message. This turns out to be a well known problem, having to do with Google's shipped version of libcurl. As noted there, the solution is to use your own version of libcurl by renaming Google's version to some nonsense name:
  
  $ sudo -i
  $ cd /opt/google/earth/free
  $ mv libcurl.so.4 libcurl_old.so.4
  
  Given the problems I had with trying to get Google-Earth running on Ubuntu, I suspect you'll need the i386 architecture available for this to work as well.
  
  
• EMACS, of all things, gave me the most problems. Whenever I'd try to start it up, I'd get the message:
  
  Font `Ubuntu Mono 13' is not defined
  
  Running emacs in a terminal window, i.e.
  
  emacs -nw
  
  worked fine. I Googled, Duck-Ducked, and even Binged for a solution, but found nothing. The only thing anyone can conclude is that something in one of my old Ubuntu configuration files was forcing Emacs to look for the Ubuntu Mono font.
  
  Which, of course, was not defined. Oh, LMDE has a package with the Ubuntu Fonts in it, but not the Mono font. I found the full set of fonts at
  
  http://mirrors.us.kernel.org/ubuntu//pool/main/u/ubuntu-font-family-sources/ttf-ubuntu-font-family_0.80-0ubuntu1~medium_all.deb
  
  
• Finally, my favorite version of Spider Solitaire is not included (even in Ubuntu, it was last seen in Lucid, 10.04). Fortunately, I have an old .deb file which I found somewhere and still installs without error. Crappy fonts, but it works. I once had the source code for this, I should really download it and install it for myself, since it seems no one is maintaining it.

After a week, those are all the problems I've found. The system is stable, MATE is as good a Desktop as you're going to find these days, and I've had no difficulties in installing other software that I want.

This could be the beginning of a beautiful friendship.

Sorting From Back to Front

The other day I was presented with a list of names and email addresses, something like this one:

Fred Flinstone <flintstone@bedrock.sag>
Barney & Wilma Rubble <bambamsfolks@bedrock.sag>
Steadholder Honor Harrington <dutchess@harrington.mdc>
Count Miles Vorkosigan <auditor@vorkosigan.byr>
Dennis & Margaret Mitchell <stilltrouble@funnies.comics.net>
Homer Simpson <donuts@springfield.st.us>
Rudolph <rednose@reindeer.np>
Kimball Kinnison <kinnison@graylens.gp>
Wile E. Coyote <genius@acme.net>
Tiberius Claudius Drusus Nero Germanicus Julius Caesar <imperator@spqr.rm>
Gen. Jack O'Neill <jack.oneill@stargate.oml>

Except that it had around 100 names. What I wanted to do was to alphabetize this by last name, to make it easier to figure out who was missing from the list, but keep the final result as
     FirstName MiddleName(s) LastName <email>
since this was input to an email list in that format.

This would not difficult if each person had exactly two names, say
     FirstName LastName <email>
in which case we'd just run the command
     sort -k 2 < elist
and we'd be done.

Unfortunately each line contains between two and eight fields, counting the email address, and we want to sort on the next to last one. As far as I can tell, sort doesn't support searches from the end of the line in.

However, the awk (or gawk) command does. For example, the command
     awk '{print $NF}' < elist
would list just the email addresses from the above file, and
     awk '{print $(NF-1)}' < elist
would list the last names — no, I don't know why you use parenthesis, but you do.

So what we need is a way to have awk pull out the last name from the file, sort those, then put everything back together. It turns out we can do that with a one-liner. I found it on the web yesterday, but I've lost the link, so I can't give proper credit. I did save the command, or my modification of it, at least:

awk '{print $(NF-1), $0}' < elist | sort | cut -f2- -d' '

Let's look at that in detail:

• awk '{print $(NF-1), $0}' < elist
  prints out the next to last column of each line, followed by the entire line ($0).
• sort
  then sorts everything on the first column, e.g. the last name. Unfortunately, that leaves you with entries like this:
   Simpson Homer Simpson <donuts@springfield.st.us>
  To get rid of these, we need
• cut -f2- -d' '
  which separates fields by whitespace (the -d' ') and prints everything out starting from the second column (-f2- . If we wanted just the second and third column it would be -f2-3).

And the correctly sorted output is:

Tiberius Claudius Drusus Nero Germanicus Julius Caesar <imperator@spqr.rm>
Wile E. Coyote <genius@acme.net>
Fred Flinstone <flintstone@bedrock.sag>
Steadholder Honor Harrington <dutchess@harrington.mdc>
Kimball Kinnison <kinnison@graylens.gp>
Dennis & Margaret Mitchell <stilltrouble@funnies.comics.net>
Gen. Jack O'Neill <jack.oneill@stargate.oml>
Barney & Wilma Rubble <bambamsfolks@bedrock.sag>
Rudolph <rednose@reindeer.np>
Homer Simpson <donuts@springfield.st.us>
Count Miles Vorkosigan <auditor@vorkosigan.byr>

Fairly simple, huh? I generalized it a bit, so that we can sort on an arbitrary column from the end:

#! /bin/bash

# Usage

# lastsort N filename
# Sorts the file filename of the field N columns from the end
# N=0 is last column of the file

awk '{print $(NF-'$1'), $0}' $2 | sort | cut -f2- -d' '

Note the single quotes around the $1 in the awk command, which passes the first argument of the calling command to awk. Without the quotes you get an error.

OK, this could have a few bells and whistles, but I'm not going to bother with that now.

DIY xkcd Password Generator

One of the things I'm thankful for today is xkcd. Specifically, I want to talk about the world-famous password strip, which points out that using a few selections from a big list of things (a dictionary) is more random, and yet easier to remember, than a lot of selections from a limited list of things (the keys on your keyboard).

There are even sites which generate xkcd-style passwords for you. Many sites, in fact.

The other day I was using one of these generators to make a password for work. The only problem was that the system that I was logging on to required my password to be between 8 and 16 letters, which is difficult to do when you're dealing with a list of random dictionary words. It also checks to see if you had a string of four or more letters that matched a dictionary word.

To fix this, I needed to have a list of, say, three letter words. Where to find them? Sergey and Larry's search engine helped. For example, here's a list of allegedly legal Scrabble words. Given that, all we need is a script to generate a list of words.

That script is below. What I didn't do was include the words. For one thing I'm not sure about the copyright status of that list. For another, you might want to use your own list. For a third, it would make this post really, really long. So add your own list of words, one per line, between the two EOF lines in the script.

While I was at it, I decided to add a few improvements, towit:

• You can specify the number of words. If you call the script xkcdpass, then
  

  xkcdpass 5
  

  will generate a password using five words from the list. The default is 4, which you can easily change.
• Given the number of words in the list, call it N, and the number of words in the password, call it M, you can generate N^M unique passwords (since strings like thethethethe are perfectly valid). That's a measure of password security, so the script tells you that.
• You have three choices of randomness. In order of security, they are: The bash variable $RANDOM, which can be seeded to the current time, and the linux scripts /dev/urandom and /dev/random. Uncomment the one you like, depending on your level of paranoia.
• It should work on any system that runs bash, including Macs.
• And, of course, I tried to document where I got everything.

So here's the script. Add a comment if you see a problem, or if you just like (or hate) it.

#! /bin/bash

# Generates an xkcd-like password from a list of three-letter words

# Usage

# xkcdpass n

# where n>0 is the number of words in the string.  The default value of
#  n is 4.

# Set the default if needed

if (( $# < 1 ))
then
    nwords=4
else
    nwords=$1
fi

# Set up an array and populate it.
declare -a array
let index=0

# There is a list of acceptable three-letter Scrabble words at
# http://www.yak.net/kablooey/scrabble/3letterwords.html
# Add additional words, if you like, or use a different list.

while read line
do
	array[$index]=$line
	let index=$index+1

# Insert your words between the two EOFs, one per line
# There is a list of acceptable three-letter Scrabble words at
# http://www.yak.net/kablooey/scrabble/3letterwords.html
# Add additional words, if you like, or use a different list.

done <<EOF

EOF

# So how secure is this string (bigger numbers are better):

echo -n $index "words in file, giving "
unique=`echo "$index^$nwords" | bc`
echo $unique unique passwords

# Uncomment this if you use $RANDOM and want a
#  unique seed.  See http://linuxgazette.net/issue55/tag/4.html
# The date +%s command gives the time from the epoch

RANDOM=$$$(date +%s)

# Select $nwords at random.  Note that you can select the
#  same word more than once.

for (( i=0 ; i<$nwords ; i++ ))
do

#   Uncomment the random technique you want to use:

#   Probably not all that random, but you can use the seed above
#   to make it better.
    let number=$RANDOM

#   More random, but slower (the sed gets rid of some annoying spaces)
#   -N3 prints out 3 bytes of data.  That's probably enough.  Note that
#   if you have 2^N words, for any integer N, it won't matter how
#   many bytes you use if the number of bytes is bigger than N
#    let number=`od -An -N3 -i /dev/urandom | sed "s/ *//"`

#   For the difference between random and urandom, see
#   http://stupefydeveloper.blogspot.com/2007/12/random-vs-urandom.html

#   Really random, though visibly slow
#    let number=`od -An -N3 -i /dev/random | sed "s/ *//"`

#   Do modulo arithmetic to get the number between 0 and $index-1
    let "number %= $index"

    echo -n ${array[$number]}
done
# Print a newline character
echo

Ubuntu Grows Up

Meaning, if you do an update, you get an update, not a complete change of your desktop or default programs. I just upgraded to Ubuntu 12.10 (Quantal Quetzal). In the fifteen minutes since the reboot, I haven't notice any difference in the machine. The Gnome desktop survived intact, even the tweaks I did to make it look like Gnome 2. Thunderbird and Firefox are at the current version. So is Flash. The Intel Fortran compiler even works.

Boring. And that's a good thing.

My Linux blog posts are usually about problems, and we just haven't had that many lately. I'll have some spare time over Christmas, maybe (don't count on it), we'll get to the statistics and baseball stuff I owe you. Maybe.

Revised GNOME3 Wallpaper Switcher

Almost a year ago, I wrote a Random Wallpaper Switcher for Gnome3. Given a directory filled with pictures, every once in a while it randomly chooses a new picture from the target directory and puts it up as Gnome3 desktop wallpaper. You can specify that "once in a while" means every XX seconds, or some random time between XX and YY seconds. And it cleans up after itself when you log off, meaning that the next time you log in you won't have multiple switchers running.

It works pretty well, IMHO, but it does have one small aesthetic flaw: If the picture doesn't fill the desktop, and most of mine don't, the color of the underlying background might be far off from the color of the picture, possibly creating a color clash. Annoying.

Then along came Penguin Pete, who showed me how to use ImageMagick to find the average color of a picture as part of his script for a wallpaper randomizer for Fluxbox. Pete then went on to merge his background with the picture, but in Gnome3 you don't have to do that, as you can set the primary background color directly. A look at man gsettings shows that you can change the background color to, say, solid purple, with the command

$ gsettings set org.gnome.desktop.background primary-color "#FF00FF"

And the rest is easy.

The script is really too long to print here, so, as before, the entire script is available on my homegrown software page. If you don't like the background changing part, I've indicated the lines that need to be eliminated to make the code work without changing the background.

Updating to Precise Pangolin

I finally got around to updating Hal to Ubuntu 12.04, Precise Pangolin. (No, I never heard of it before, either.) You could have followed the whole thing on Twitter.

Good news: the only thing that broke was the color scheme on my panel bars, which changed to black text on black background. I fixed that right away, and I'm still running a pseudo-Gnome2 desktop.

So, I'm happy with it. Besides, this is an LTS release, so if I don't want to bother updating I can keep it for a couple of years.

Just a bit later: OK, one bug. For some reason the permissions on the directory

$HOME/.config/nautilus-actions

had been reset to 555, meaning I could read or execute files in the directory, but not write to it. This meant that backintime wouldn't back up my disk. Fortunately, backintime has an excellent error log, available from inside the program, that told me what was going on. (Unlike Google-Earth, which is still failing to launch in 64-bit mode after all these years.) I just ran the command:

chmod 755 $HOME.config/nautilus-actions

and all was well.

Insert Joel and Ethan Coen Reference Here

So in today's paper there's an Office Depot ad for a Brother HL 2240 Monochrome Laser Printer for $69.95. We really need a BW printer, and Laser is obviously the way to go to keep the price per page down. The thing got decent user reviews on Amazon, as long as you remember:

• It's Cheap.
• It's USB.
• You have to replace the drum at 14,000 pages (like we'll get to that any time soon).
• The initial cartridge only lasts for 700 pages (though you can apparently fool the printer into printing more)
• It's cheap.

So I went out and bought the thing — the closest thing to an impulse electronics buy since I got that $20 Canon Printer/Scanner when I was at Duke. (The scanner still works, the paper feed mechanism for the printer is broken.)

I read all of the installation instructions, plugged the thing in, printed the test page, got the USB cable from the $20 printer and used it to connect the printer with Hal, and turned on the printer. Ubuntu's auto print installer came up, recognized the printer, and …

Wouldn't you know it, CUPS doesn't have a driver for the Brother HL2240. (Obviously I didn't do a lot of research.)

No problem, Brother provides proprietary drivers for the printer. (Forgive me, St. Richard.)

It's not a difficult process. Go to Brother site, and download the .deb packages for the HL-2240:
LPR Driver (hl2240lpr-2.1.0-1.i386.deb) and
CUPS wrapper drive (cupswrapperHL2240-2.0.4-2.i386.deb).

Install both drivers, in order:

sudo dpkg -i --force-all hl2240lpr-2.1.0-1.i386.deb
sudo dpkg -i --force-all cupswrapperHL2240-2.0.4-2.i386.deb

At some point during this you'll get a message that the Brother printer has been found, and if you click on System Settings (under your name in the upper right of your screen), followed by Printers, and then Brother HL2240, you'll be able to print out the test page.

It works fine. Except …

I also want to be able to access the printer from Harlie, the machine upstairs. I previously alluded to doing this before (see rants, uh, points, 35 and 36), but that was with native CUPS drivers. I'll provide a bit more detail this time. Remember that the printer is installed on Hal, and we want Harlie to be able to use it.

So on Hal:

• Open up a browser and go to http://localhost:631/.
• Click on the Administration tab.
• Under Server on the right, click the box that says Share printers connected to this system, and then click Change Settings.

Go upstairs to Harlie:

• Install those same Brother Printer Drivers. You'll get some message about a printer being found. Ignore it.
• Open up a browser and go to http://localhost:631/.
• Click on Administration and Add Printer.
• You should see a box which says something like
  Brother HL2240 BW Laser @ harlie (Brother HL2240 series)
  Click its radio button and then click Continue.
• Fill in the information you want to use to identify the printer. Click Share this printer if you want others to be able to use it from Harlie. Again click Continue.
• Now for the trick. The next screen will ask you for the location of the CUPS driver. It won't be in the list, since it's proprietary, but after a bit of searching, I found it. Where it says Or Provide PPD File: click Choose File, and search until you find the file
  /usr/share/ppd/HL2240.ppd
  Choose this, then Add Printer.
• If the Force is With You, your printer will now be installed.

Note that you can use CUPS to change the behavior of your printer. For example, if you go to Administration, click on Manage Printers, and click the name of your printer, you'll see a page that has two pull-down boxes, one labeled Maintenance and the other Administration. Click on the Admin box and you'll see a bunch of things you can do. In particular, under Set Default Options you can choose your paper size, print resolution, ink toner usage, etc.

Finding Files Born After a Given Date

A few weeks ago Friend TK came up with a question:

RC, how can I find all of the pictures I put on my computer since Christmas without seeing all of the pictures in the .cache directories and other junk places?

An interesting question, indeed. We want to search for files that are newer than a certain target date, and they'll probably have extensions like .jpg or .png, though there will be the occasional file with an extension .jpeg or even .JPEG. It obviously requires some form of the Unix find command, but it's not exactly obvious what that would be. TK and I hunted around on the web a bit and finally came up with this:

touch -d 20111225 tokenfile
find . -type f \( -iname "*jpg" -o -iname "*jpeg" \) -anewer tokenfile  -print | egrep -v ".cache/|.thumbnails|.kde/"

which creates a marker, tokenfile, which looks to have been born last Christmas Day, and finds all files newer than that and ending in jpg or (the -o) jpeg without regard to case (that's the -iname). We then pipe the file through egrep, striping off (-v) files who have names matching directories we don't want to see (the | is the grep equivalent of find's -o.

This is not entirely elegant. I was pretty sure I could make this into a purely find one-liner, leaving no trace behind (such as that tokenfile we created up there. To do this I knew I'd have to delve deeper into findology than I had in my 30 or so years of Unix use. If found some of the clues at Linux.ie's finder-keepers page, and other hints elsewhere. Eventually this let me put together this one-line script:

find . -type d \( -iname .\[a-z\]\* -o -iname work \) -prune -o -newermt 2011225 \( -iname "*j*g" -o -iname "*png" \) -print

Let's look at this in some detail, since it contains some things that I hadn't known:

• find, of course, is the Unix/Linux command for looking through your file system.
• . is the current directory. Find uses it as the starting directory. find will look at every file in this directory, its sub-directories, and all their children and grandchildren, to the umpteenth generation. If you only wanted to search where you thought pictures might be, you could instead write
  find $HOME/Pictures
• -type d says that the next set of files will describe directories, rather than files (which would be -type f).
• The \( and \) delineate what will be a list of file descriptors. The backslashes make sure that the parenthesis are passed to find, and not gobbled up by the shell. Sometimes you can use quotations marks within find, instead of parenthesis, but this isn't one of those times.
• Now for the heart of the matter: -iname .\[a-z\]\* tells find to look for directory names that start with a . followed by a letter ([a-z]), and then by anything else (*). Again the backslashes are there to make sure the next characters are passed to the find command. Note that this gets rid of every hidden directory in you file tree.
• As before, -o is the or command. The -iname work command identifies my work directory, which may have pictures in it but nothing I'm interested in at the present time. You can add further -o -iname commands as needed.
• -prune tells find to look at every file except those in the directories just named. This serves the purpose of the egrep -v command in our initial attempt.
• You know, I'm not sure why the -o follows -prune. You'd think it would be some kind of and command, but just dropping it doesn't work.
• -newermt 20111225 is one of the newer options in find. This one says to look at all files modified (m) after a certain time (t). The time here is a date, written in the format yyyymmdd, in this case last Christmas. If we wanted files written after noon on Christmas, we'd use -newermt "20111225 1200".
• Again we have a list of -inames, delineated by backslash-parenthesis. These are file names you want to look for. Note that j*g catches all files ending in jpg, jpeg, JPG, or JPEG. It also catches files in .jynormouslyinteresting, but you can't have everything.
• Finally, -print lists all the files. You can actually drop this, as find takes -print as its default action.

And that's it. OK, not quite. What TK wanted was to copy all of the files he found to a new directory, we'll call it recent, so he could examine them in detail. To do that we use the -exec option of find:

find . -type d \( -iname .\[a-z\]\* -o -name work -o -name recent \) -prune -o -newermt 20111225 \( -iname "*j*g" -o -iname "*png" \) -exec cp -p {} ~/recent ';'

• Note that we've added the recent directory to our list of avoided directories. Otherwise find will search recent, and give errors.
• -exec tells find to execute the following file command.
• cp -r is the usual copy command, with -r saying to preserve the original timestamps on the copied files. If you wanted to save space, and weren't going to modify the pictures, you could link with either ln or ln -s instead.
• {} is where find places its output. That is, if there is a file Pictures/cutekids.jpg, find issues the command
  cp -r Pictures/cutekids.jpg recent
• ; tells find that we're done.

A Linux Preview Script

Mac OS X comes with a program called Preview, which displays PDF files as well as images (PNG, GIF, Bitmap, etc.). Linux doesn't have anything quite like it. OK, if you use the Gnome desktop there's something called Document Viewer, usually linked to evince, which reads Postscript, PDF, Djvu, DVI, etc. In fact, it displays Documents better than Preview does.

Unfortunately, evince doesn't display images. For that we have programs like Eye of Gnome, qiv (my favorite), and for image manipulation ImageMagick or the GIMP.

And then there are plain old ASCII files, such as you'd use for source code or LaTeX. For that we could use The One True Editor, or gedit, or some other sort of pop-up display window that lists the contents of a file.

What we want, then, is a program, preferably a command line script, which is given a file name, determines what the file is, and then opens the file using the appropriate viewer/editor. How do we do that, pray tell?

Enter the file command, inherited from Unix. file is pretty useful. For example, suppose I have a JPEG file, but for some reason it has been named x1, without the extension. file figures it out pretty fast:

$ file x1
x1: JPEG image data, JFIF standard 1.02

And it will work on most standard Linux file types.

Given this, it's fairly easy to construct a script which takes a look at one or more files, determines the file type, and picks out the approriate file viewer. My script is below. I've called it gnuview. You invoke it from the command line:

$ gnuview file1.txt file2.png file3.dvi file4.bmp

and all of these files pop up on your screen, each launched using the appropriate program.

A few notes:

• The list of file types is not by any means complete, though I think I hit most of the major ones. If the script doesn't know how to handle a specific file type, it prints out a message on standard error. It's fairly obvious how to add different file types.
• I dumped my favorite programs (qiv and emacs), in favor of programs that are installed by default with most Linux distributions (eye of gnome, gedit). (qiv doesn't read bitmap files, anyway.) Feel free to change the defaults to your favorite.
• Error messages? We don't need no stinkin' error messages!
• Finally, the cascading tree structure of if statements is just annoying. This should probably be rewritten using some kind of case construction, but I didn't bother to figure it out.

No warranty for any of this, of course, and not even a license except for the Creative Commons Disclaimer down at the bottom of this web page. If you come up with a version of this code, put it in the comments or put a link to it there.

#! /bin/bash

# This is an attempt to mimic the behavior of Apple's OS X Preview
#  program.  We determine the type of the file and then use the
#  appropriate command to open it.

# Some defaults.  Change as you see fit:

# ASCII files.  Note that gedit must be invoked in standalone mode

GNTXT="/usr/bin/gedit -s"

# Picture viewer (eog reads more files than qiv, so we'll use that)

GNPIC=/usr/bin/eog

# Postscript/PDF/DjVu/DVI viewer:

GNPDF=/usr/bin/evince

# Read the command line, and scroll through each file type:

for thisfile
do

#   If the string says ASCII anywhere, do this, which should get
#     LaTeX source, source code, etc.

    file $thisfile | grep 'ASCII' 2>&1 > /dev/null
    if [ $? -eq 0 ]
    then

        $GNTXT $thisfile &

    else

#   Maybe it's a document (Postscript/PDF/DjVu/DVI):
#   (I wan't sure about the "or" construct, see the first comment in
#   http://www.cyberciti.biz/faq/searching-multiple-words-string-using-grep/)
        file $thisfile | grep 'document\|DVI' 2>&1 > /dev/null
        if [ $? -eq 0 ]
        then
            $GNPDF $thisfile &
        else

#   Or maybe it's an image
            file $thisfile | grep 'image\|bitmap' 2>&1 > /dev/null
            if [ $? -eq 0 ]
            then
                $GNPIC $thisfile &
            else

#               And if we haven't figured out the type yet,
#                leave a nice note:
                FTYPE=`file $thisfile | awk '{print $2}'`
                echo Cannot handle file $thisfile, type $FTYPE  > /dev/stderr
            fi
        fi
    fi

done

Resize a Lot of Pictures

The church web site I run has a box the displays random pictures from our photo album. The formatting of the page is easiest if these pictures are no wider than 530 pixels and no higher than 340 pixels. Of course most pictures are taken with a camera these days are a couple of thousand pixels wide, so each picture needs to be resized. You can do that by hand, of course, but I found it easiest to write a script. If you're running Linux, this uses the ImageMagick package, particularly the identify and convert programs. On a Mac, it uses sips, which has been available at least since 10.3. I suppose it could be modified for Windows or run under Cygwin, but I don't use Windows enough to make it worth my time.

Save the file as websize, make it executable, and save it to a directory in your path. Then run it with the command

websize [list of image files]

This will take a file named, say, picture1.jpg, and create a new file named picture1_web.jpg that fits into the box defined by MAXWIDTH and MAXHEIGHT. If the picture already fits into the box it will duplicate the file.

#! /bin/bash

# Resizes a picture or pictures to make sure it fits into the UPB
# frontpage box of MAXWIDTH pixels wide by MAXHEIGHT pixels tall,
# keeping the same format (PNG, JPEG, BMP, etc.)

# usage

# websize [pictures]

# If a picture is named "picturename.ext", the shrunken picture
#  will be named "picturename_web.ext".

# If a picture already fits into the website, it will simply be
#  copied with its new name.  This makes it easier to find the
#  pictures that should be posted on the web.

# Define the maximum dimensions of the picture

let MAXWIDTH=530
let MAXHEIGHT=340

# For Linux machines we need "convert" from the ImageMagick package.
#  We'll assume it's in the standard location:

# For Macs the standard program is sips, at least since 10.3

if [ -f /usr/bin/sips ]
then
    OS="Mac"
elif [ -f /usr/bin/convert ]
then
    OS="Linux"
else
    echo "Cannot find a program to do the file conversion"
    exit 1
fi

for PICTURE
do

if [ "$OS" = "Linux" ]
then

#   identify is also part of the ImageMagick package

    let WIDTH=`identify $PICTURE | sed "s/x/ /" | awk '{print $3}'`
else
    let WIDTH=`sips -g pixelWidth $PICTURE | tail -1 | awk '{print $2}'`
fi

# Need new picture name:

HEADER=${PICTURE%.*}
EXT=${PICTURE##*.}
NEWPICT=${HEADER}_web.$EXT

echo Converting $PICTURE to $NEWPICT

# Is the picture too wide?

if (( $WIDTH > $MAXWIDTH ))
then
    if [ "$OS" = "Linux" ]
    then
 convert -resize ${MAXWIDTH}x $PICTURE $NEWPICT
    else
 sips --resampleWidth $MAXWIDTH $PICTURE --out $NEWPICT
    fi
else
    cp $PICTURE $NEWPICT
fi

# Note that the new picture may still be too large:

if [ "$OS" = "Linux" ]
then
    let HEIGHT=`identify $NEWPICT | sed "s/x/ /" | awk '{print $4}'`
else
    let HEIGHT=`sips -g pixelHeight $NEWPICT | tail -1 | awk '{print $2}'`
fi

if (( $HEIGHT > $MAXHEIGHT ))
then
#   echo Resizing height of $PICTURE

    if [ "$OS" = "Linux" ]
    then
 convert -resize x$MAXHEIGHT $NEWPICT 1_${NEWPICT}
    else
 sips --resampleHeight $MAXHEIGHT $NEWPICT --out 1_$NEWPICT
    fi

    mv 1_${NEWPICT} $NEWPICT

fi

done

The obvious modification is to allow MAXWIDTH and MAXHEIGHT to be read from the command line, maybe using -w and -h flags. That's for another day.

Joining a Group

In an effort to make Ubuntu less scary to users (i.e., more Mac/Windows like), Ubuntu 11.10 removed some graphical system administration utilities that were previously available. One of those was a called Users and Groups, which let you see groups and make changes to group memberships.

The new Ubuntu would rather you just forget groups, so it includes no graphical tools for handling them. Fortunately it's still Linux, so you can fix it up yourself. Liberian Geek has details, but the first hint (using groupmod) didn't work for me, so here's my modified version. In this example, I want to add myself to the group vboxusers, which should allow me to get USB access for my Virtualbox installation. More on that some other time. For now, return with us to those thrilling days of yesteryear, before the GUI:

1. Look at the file /etc/group. Specifically, we want to see who belongs to the group vboxusers:

   $ grep vboxusers /etc/group
   vboxusers:x:125:
   

   Nobody, as we expected.
2. Now to follow the Geek and add myself:

   sudo usermod -a -G vboxusers rchawk
   

3. Finally, confirm the addition

   $ grep vboxusers /etc/group
   vboxusers:x:125:rcjhawk
   

You'll need to log out and log back in before the system will recognize the new group setting.

Installation Notes and Rants for Ubuntu 11.10

A few weeks ago I updated Hal to Ubuntu 11.10 (Oneiric Ocelot). This was relatively easy because Hal was already at 11.04, allowing me to do a distribution update from the update manager. Our other Linux box, harlie, was running Ubuntu 10.10, so that option wasn't available unless I updated first to 11.04 and then to 11.10. Instead, I downloaded 11.10 to a CD, and did an install more or less for scratch.

Now, mind you, the Ocelot has what are supposedly a lot of nifty features, including the not-quite-Mac-like-enough-to-be-sued (you might want to ask Google about that) Unity desktop interface. And, if you don't like Unity, you can get Gnome 3! I didn't really want any of that. What I wanted was a machine that had all the modern software but looked and acted exactly like the old Ubuntu. I didn't quite get that, but I came pretty close.

BUT: This release of Ubuntu sets computing back at least 10 years. In the effort to make everything super friendly, they have dumbed down the user interface so much that you MUST use the command line to make even the simplest changes: For example, up through 11.04 you could easily change theme colors through a GUI widget. This is an appropriate use of the GUI, you can see what the current colors are, select the colors you want, and if they aren't right you can fix them right then and there. But now, guess what? THE ONLY WAY TO CHANGE THEME COLORS IS TO EDIT A FRAKIN' TEXT FILE. AS THE SUPERUSER! (I only shout because I'm mad.) This is (pick one):

• Dumb
• Dumber
• So stupidly idiotic that one expects that there must be a malevolent being doing this, someone who wants everyone to throw up their hands in exasperation and switch to OS X. Or worse, Windows 8.

And this kind of idiocy exists all over the place: Things that were easy to do before are now hidden. Usually you have to edit a text file that hasn't appreciably changed since 1999. OK, I can do that, but until last month I didn't have to. Hence my frustration.

The worst of it is, this isn't just Ubuntu, a lot of the changes are driven by Gnome 3. It's enough to drive one back to the ugliness that is FVWM, where at least you know you have to edit menus going in.

But not just yet. Click below to see how I overcame the obstacles, at least so far.

Read more ⇒

Random Wallpaper Switcher for GNOME 3

20 October 2012: There's a revised version of this script. See what the changes are, then download the new version.

21 July 2013: And yet another revision, following my switch from Ubuntu to LMDE: a MATE version of this software.

Some things bore me — KU losing in March every year, the Yankees winning more than one pennant a decade, Republican Presidential Debates, etc..

One of my major points of boredom is a static background on my computer screen. Having the same picture up constantly just irks.

Apple solved this long ago, with Mac OS X you can pick a directory and have the pictures selected randomly from that directory. Linux has been a little slow to pick this up. A random wallpaper selector has never shipped with any window manager I'm aware of, leaving the field to third-parties and home-grown hacking.

Way back in the day I wrote setbg, a clunky Perl script which randomly changed the X11 background. That worked for window mangers such as FVWM, but it failed for GNOME, which has its own wallpaper protocol.

For GNOME 2 I used Scott Balneaves's Gnome background switcher, aka background.py, a Python script which let you specify a directory and a switching time. Every X seconds you got a new picture for wallpaper.

GNOME 3 does wallpaper a different way (Surprise!). Fortunately I stumbled on a one-liner which changes the background to a random picture. If you want to do it that way, you just enter

gsettings set org.gnome.desktop.background picture-uri file://$(find DIR -type f | shuf -n1`)

where DIR is the path to your picture directory.

OK, I can work with that. I decided to write a script, based on the one-liner, which combined the best features of setbg and background.py. What I want is a script that:

1. Displays a random picture as wallpaper (see the gsettings one-liner, above).
2. Changes pictures at random times. Say someplace between every 10 minutes and every 20 minutes, but not always every 10 minutes and not always every 20 minutes. setbg did this, but background.py did not. If I knew more Python I would have fixed that, but I never had time.
3. Can be loaded as a Startup Application (now conveniently located in Applications/Other on your Ubuntu menu).
4. Works on any Linux computer running GNOME 3. Basically requires a common scripting language. setbg used Perl, background.py Python. For kicks, I decided to do this one in bash.
5. Only selects pictures to display. This was a little tricky. I don't know of a native GNOME command that says this is a picture or that is not a picture. I cheated on the previous requirement a bit and used the identify command from the ImageMagick package. No defense, except that it's available in every Linux distribution I know of.
6. Shuts off when you log off, so that there aren't multiple copies running if you log back in. background.py did this. setbg relied on an FVWM command to explicitly kill it. I fudge this one a bit. Basically, my script checks every minute or so to see if its parent (usually gnome-session) is still running. If it's not, the script dies.

As I said, I wrote the script in bash. It's not really a long script, but it comes in at 173 lines because I commented everything. You can download my Gnome 3 Random Wallpaper Switcher script from my software page. It's not licensed, since it's all based on someone else's one-liner. But if you make changes to the script, or find a bug, please let me know about it.

To run the script, put it in your path and type

gnome3_random_wallpaper T1 T2 DIR

Where T1 and T2 are times in seconds, and you want the wallpaper to change at some time T where T1 < T < T2. DIR is the full path to a directory that contains pictures. If it contains some other files, that's OK, the script will skip over them. You and put the command in your startup applications.

I've only tried this with Ubuntu 11.10, as that's the only GNOME 3 desktop I have at the moment, but I don't see why it wouldn't work elsewhere.

Download the official RCJHawk Gnome 3 Random Wallpaper Switcher

I For One, Welcome Our New Canonical Overlords

I've just updated Hal to Ubuntu 11.10, Oneiric Ocelot.

I don't regret it — quite. It is however, going to take a long time to get it to where I want it to be, e.g., something that looks like my old Gnome 2 desktop.

Why not just change to a distribution that still uses Gnome 2? Because Gnome 2 is going away. openSUSE 12.1 is out now, and it uses Gnome 3. Fedora uses Gnome 3. You see where this is going.

So this post, and probably many more, will be rough notes on how to make Hal's desktop look like it did before. I'm not going to go cursing Mark Shuttleworth. (I've learned from the last ten years of cursing Dan Snyder — it just doesn't work.) Ubuntu is, for now, a useable distribution, and I can, eventually, tweak everything to get to where I want to go. That's the beauty of Linux.

Besides, it makes for a lot more blog posts.

Hal, I should point out, is just fine. He can still do fancy LAPW electronic structure calculations faster than the Cray XMP of my youth, and he still reads my email and browses the web with aplomb. He can still develop code with Emacs. He just doesn't look very good right now, but we're dealing with that.

Eventually I'm going to update the spouse's computer, Harlie. When I do that I hope to do one of those step-by-step guides, which will be more coherent. But for now, think of this as a set of Google Notebook entries. Oh, wait, Google pulled the plug on that and is now moving all my files to Google Docs.

Oh, well, let's just get on with it.

Oneiric comes with the brand-spanking new Unity desktop, which, as I said once before, is so like Mac OS X that it comes with a lawyer to deflect lawsuits from Cupertino.

I was wrong — it's worse than Mac OS.

Like OS X, Oneiric comes with a dock, or maybe it's called a Launcher. On the Mac the dock can be moved, it can change size, it can automatically hide if you don't want it on the screen, and it's quite easy to drag applications over to it. In Oneiric the dock is fixed on the left, you can't change its size, it only hides itself when a window pushes it away. You can drag applications onto the dock, but it seems harder, somehow, than on the Mac — that's my subjective point of view, of course.

So I quickly changed over to Gnome 3.2. Well, not quickly, because Gnome isn't installed in Oneiric. And since my 11.04 distribution used Gnome 2, the upgrade process ripped out all of that software as well. So one must install all things Gnome. I used to do this with Synaptic, but, guess what, Synaptic isn't installed either.

So first open up a terminal window. Can't find a terminal? Click on the Dash Home icon at the top of the dock, click More Apps, click to show all of the installed apps, scroll down and find the terminal, and click it. (First you might want to drag the icon over to the dock, for later use.)

With the terminal finally open, you can install Synaptic by typing

sudo apt-get install synaptic

You could then find synaptic in the Dash Home menu, but it's easier to just say:

sudo /usr/sbin/synaptic

and go from there.

Once in synaptic, search for Gnome and install it. You'll probably want to install some Gnome themes, etc. While you're there, also install the Gnome Tweak Tool as well.

Then log out. When the login screen appears, click on the little gear by your name and select Gnome Classic (no effects) or whatever other version of Gnome you desire.

It's OK to throw up your hands and go off to figure out Unity at this point. Really.

Because when you log in, you'll see something that looks only vaguely like your old Gnome desktop, sort of like the difference between Darrel Hammond and Bill Clinton. There's a menu on the top, but it only says Applications and Places. And the clock seems to be stuck to the middle of the upper panel, neither movable nor removable. Also the Theme will be hideous. You can fix that, sort of. Click Applications/Other/Advanced Settings, (that's the Gnome Tweak Tool), click Theme in the box that pops up, and play with the Window Theme and the GTK+ Theme until you get something you like. One of my missions is to figure out how to edit the theme colors, to create what used to be called a Custom Theme. No luck, so far, but if I find it I'll let you know.

You should have two toolbars, one at the top and one at the bottom of your window. You add things to these bars, change their properties, etc., as before, but instead of just right-clicking on the bar, you have to hold down the Alt key and right-click. That's easy enough.

You'll also have to search for some of your options. The old Gnome menu used to have a Preference section and an Administrative section. Those functions are now hidden in the Applications menu, under Other and System Settings, apparently with neither rhyme nor reason. Play around with that until you're comfortable in finding things.

Finally, for now, go to Things to Tweak After Installing Oneiric Ocelot 11.10, and see what's useful there. And don't forget 10 things to do after installing Ubuntu 11.10.

More as I find more stuff.

Take Me Out to the SQualL Game

As some of you know, I have a moderate interest in Baseball and Baseball statistics. However, I'm not a database programmer, and the little bit of DB manipulation I've looked at has left me hopelessly confused.

Several years ago I did buy a copy of Baseball Hacks, but I never really got started with it. Last weekend I had some time on my hands and started playing around with it. Turns out you actually have to try to do something with a program in order to learn it (who'da thunk?). And it also turns out the Baseball Hacks, even though it's an O'Reilly book, is pretty oriented to Windows/Microsoft Access, although it does have substantial hints for Linux and Mac users, not to mention an online collection of scripts from the book (ZIP file), along with some other stuff.

So what should we do? As a first shot, how about updating my table of averages for several modern baseball statistics to include 2009 and 2010? I did the previous tables by doing some judicious editing of the Batting.csv file in Sean Lahman's baseball database, but the folks at Baseball-Databank.org have all of the data packaged neatly into a mySQL database (ZIP file), so we'll use that.

So as a start, we're going to

• Install appropriate parts of the mySQL database program in Ubuntu,
• Set it up to read the database file,
• Find the eligible batters for 2009 and 2010,
• Get their batting data into a spreadsheet,
• Find the appropriate averages, runs created, etc., and
• add the results to the appropriate tables.

That should be enough for one day. It's going to be a long journey, though, so when you've got some time join us after the break.

Read more ⇒

The Natty Narwhal Upgrade

Like Travis McGee, this week I'm taking an installment of my retirement. Unlike McGee, my retirement seems to consist of fixing up computers.

So today, Ladies and Gentlemen, Boys and Girls, Desktops and Netbooks, we're going to talk about upgrading Hal here to Ubuntu 11.04.

Yawn. This shouldn't really be that big a deal, should it? You just hit the update button, or type do-release-upgrade at the command line, right?

Well, it's not always that simple. A friend of mine did that, and he seemed to have a lot of problems. I'm not sure why, as I never got to examine his system before he reinstalled 10.10. Maybe it was the change from the Xorg window system and Gnome desktop to Wayland and the Unity desktop. Or not. Maybe it was just the way the wind was blowing that day. But I decided not to take the chance on the upgrade, but rather install 11.04 from the ground up.

I didn't have to be pushed hard, because I was leaning toward a clean install anyway. When I bought this version of Hal I was in a hurry to get Linux set up, but wanted to keep Windows 7 around. As a result, I let Ubuntu do the repartitioning of the disk, and ended up with the entire Linux system in one giant partition.

This is less than optimal. (Thanks, Dave. Don't mention it, Hal.) Ideally one wants /home and possibly /usr/local and /opt in partitions separate from the root operating system. That way when you upgrade the OS, you don't lose your previous data. (That doesn't mean you shouldn't back up that data before upgrading. There's a word for people who don't. The polite form of it is idiot)

But this requires repartitioning Hal's disk. Fortunately, all of Hal's memories reside in the extended partition, so I don't have to fiddle with the Windows partition. After about 30 secons of thought, I decided to repartition Hal's big Linux partition into 15GB for /, the root system, another 15GB for /usr/local, /opt, and /scratch (more on how to do this later), and leave the rest for /home. That's probably too much for /, and maybe not enough for /scratch, depending on what I right there, but that's how I'm going to do it this go-round. Ideally gparted would leave the data on /home as is, but that takes forever because of all the data that needs to be moved around, so we're just going to repartition everything and hope the backup holds.

Preparation

First, get the software we'll need. I downloaded and burned CDs for the 64-bit versions of:

• Ubuntu 11.04
• openSUSE 11.4
• Fedora 15
  and, last but probably most importantly,
• Gparted Live CD version

OK, Gparted is obvious: that's what's going to do the disk repartitioning. But why the alternative distributions?

Because you never know. The largely hypothetical long-time reader will remember that I used to use Fedora, but that I switched to Ubuntu when the installation of Fedora 5 failed miserably, and Ubuntu was there, ready and waiting. So having alternative Linux CDs on hand seems to be a really good idea. Besides, we bought 100 CDs maybe five years ago and still have a bunch left. They just aren't used much anymore. (That's right, rcjhawk is still a trailing indicator for device popularity.)

OK, we've got hopefully every bit of software that we need, so let's go.

Find out what's on your machine

I never remember which packages are installed from one upgrade to the next. So let's make a list:

dpkg --get-selections > installed.txt

Edit installed.txt, leaving out any packages you think will be installed by the system (e.g., kernel, window manager, etc.). If you can't remember what a package does, delete it too. If you need it, it will eventually show up as a dependency to a package you want, or you can reinstall with apt-get or synaptic. The important thing to do here is to remove cruft from the system. For example, I have the bsd-games package on here. I never play those games, so why leave them in place?

It would have been nice to make a copy of /etc/fstab, but some dingbat forgot to do that (more on this later).

Back it all up

Well, not everything. No need to back up the kernel, or Emacs, or latex, or any of that. The distribution will provide. No, I want to back up my data (basically $HOME), the Intel Fortran compiler, a royal pain to install, and my renegade version of SoX. So do the following (/backup is my ext4 formatted USB disk):

$ cd
# -rpv == recurse directories,
#  preserve permissions and timestamps,
#  and speak up about it verbosely.
$ cp -rpv dave /backup
# Assuming you're named dave, of course
$ cd /usr
$ cp -rpv local /backup
$ cd /
$ cp -rpv opt /backup

To be safe, I also had backintime, my backup manager, do an additional labeled backup, which should keep it on the disk forever and a day, or at least until the end of 2012.

Repartition

The next step is the scariest one. I've taken what was an 800+GB partition and broken it up into three parts: 15GB for /, 15GB for what will be /usr/local, /opt and /scratch, and the remainder for /home. It's scary because this wipes out the primary data, so you are now relying on the kindness of backups. It's theoretically possible to gave gparted to keep all the data in what's going to be the /home partition, but that requires moving a lot of stuff around, at least the way I tried it, and an estimated 18 hours to get it done. So I just said a prayer, bit my lip, and reformatted. Fortunately it all went well.

Installation

Not much to say here. I told Ubuntu that I wanted to do an install with a custom installation, had it mount everything where I wanted it, and let it go. The usual waiting around occurred.

Copying Files

This was rather easy: turn on the backup USB disk drive, and copy the files back. I copied back configuration files I knew I wanted to keep, e.g. things like .emacs and the .devilspie directory, but I'm letting Ubuntu set up the desktop. This means I'll have to play with preferences later, but it should get eliminate any conflicts between old and new configuration parameters.

Mounting the Backup Drive

I could just plug the backup drive in and access it at /media/really_long_hexadecimal_string, but what I really want to do is have the backup drive mounted to /backup. I've covered this before, but as I mentioned above, I forgot the UUID of the drive. You get that with the command:

blkid

which gives you all the UUID of all connected disks.

Restoring /usr/local and /opt

This is awkward. Add-on programs that you compile yourself generally go into /usr/local. Other programs, such as the Intel Fortran compiler and Google's Picasa, end up in /opt. I want both to stick around between upgrades, so they need to be out of the / partition. But I don't want two extra partitions. So here's what I did:

• Create a partition /usr/local
• Create a directory /usr/local/opt
• Copy all of the /usr/local files where they belong.
• Copy all of the /opt files to /usr/local/opt
• Then, as root, run the command
  ln -s /usr/local/opt /opt
• If you want a /scratch directory, do the same thing, but don't forget to make the original world writable:

  # mkdir /usr/local/scratch
  # chmod 777 /usr/local/scratch
  # ln -s /usr/local/scratch /scratch
  

Restoring your add-on packages.

Now we want to get back all of our old packages, the ones that aren't installed by default, at least so far as Ubuntu will let us. Remember that file installed.txt I mentioned you should create and edit? Here's how we'll use it:

• Select System => Administration => Update Manager
• Click on Settings in the lower left corner
• Under Ubuntu Software make sure all of the sources you want enabled are. If you don't want proprietary drives, etc., turn them off. Do the same under Other Software
• Open a terminal window and go the the directory where you have that installed.txt
• Run the following commands:

  sudo apt-get update
  sudo apt-get install `awk '{print $1}' installed.txt | xargs`
  

• You'll probably get some error messages, but they're pretty clear about what you need to fix. Edit the installed.txt file to match.
• This will take a while, as all the packages have to be downloaded from the net.

What Works

Pretty much everything. I selected Gnome-classic for my desktop, and after a little fiddling to with System => Preferences => Startup Applications got the system to look pretty much as it did before.

As for third party software, the Intel Fortran compiler still works fine. I was able to reinstall Picasa from Google's supplied 64-bit .deb.

Wayland seems to work just like Xorg, at least from my point of view. It pops up windows in the same way as before, and I can run, say
ssh -X majel
to get me to another machine, and then run
firefox
on majel, and the window pops up here on Hal. So no major problems for me there.

What doesn't work

Google Earth. Maybe this is a 64-bit problem, I don't know. I tried using Google's .deb file, and the official Ubuntu method. Neither worked.

By default Ubuntu installs the 3-D version of Unity, which requires a pretty good graphics accelerator. Hal lives in 2-d, so that wouldn't run. (Funny, the Unity desktop shows up when you run the live CD.) I installed the 2-D version and that works. Which brings me to

What Sucks

Unity. From my limited experience (about two minutes, after which I ran away screaming), it's an overblown and somewhat hideous version and of the Mac desktop. But don't mind me, I was the last person on Earth to use fvwm. I suppose if I played around awhile I could make Unity behave the way I wanted it to, but why bother, since Ubuntu still supplies the classic Gnome desktop. Mind you, I've seen Gnome 3, and I don't like that, either. (The word you're looking for is Luddite.)

Summary

All in all, a successful update, as long as I stay way from Unity. I have one more computer that needs updating to 11.4, I think I'll just try that as a distribution update. If it doesn't work, I can always do a full install.

More Later

Troubles will surface, they always do. When they do, I'll write about them here.

Compression

The other day an email arrived from sourceforge which mentioned that they were hosting the file compression program 7-zip. Now I had used 7-zip under windows, as an all purpose archiving tool, mostly for reading zip files. I'd never thought of it in a Linux context. But both openSuSE and Ubuntu have the command-line version available, and what more do you need?

In OpenSuSE, the RPM is called, simply enough, 7z. In Ubuntu it's a bit more complicated. There's p7zip, which provides the bare-bones standalone version of the compression program, 7zr, and a wrapper p7zip, which makes 7zr work like gzip. For the full-blown 7z program, you want the package p7zip-full, which includes 7z. While you're at it you might want to get the p7zip-rar package, which lets you decompress RAR files.

The reason you want 7z and not just 7zr is that the smaller program only compresses to the 7z format, but, as it says in the blurb,

not only does [7z] handle 7z but also ZIP, Zip64, CAB, RAR, ARJ, GZIP, BZIP2, TAR, CPIO, RPM, ISO and DEB archives.

And by handle, they mean read and write to these formats (except RAR). So I could create a zip archive with 7z, or a gzip/bzip2 compressed tarball. And guess what?

7z compression is 30-50% better than ZIP compression.

Is it? Well, let's find out.

Test 1

Presented for your consideration: an uncompressed tarball:

$ ls -l Ru.tar
-rw-r--r-- 1 dave dave 26716160 2011-05-21 15:41 Ru.tar

This is IO from the elk FP-LAPW code, so it has a lot of repetition of text, and what for our purposes are a bunch of random numbers. First we'll try compressing it with programs using their native formats. I'll try for maximum compression in all cases. Note that zip and 7z will create separate archives, while gzip and bzip2 compress the file in place.

Program	Command	File Size	Ratio
zip	zip -9 Ru Ru.tar	3899523	0.146
gzip	gzip -9 Ru.tar	3899386	0.146
bzip2	bzip2 -9 Ru.tar	2992422	0.112
7z	7zr a -mx=9 Ru.7z Ru.tar	2242708	0.084

Pretty good, huh? As advertised, 7z is about 40% better than zip/gzip, and 25% better than bzip2. But wait, there's more. Not every computer is going to have 7z available, so you may want to compress files using a more established protocol. 7z can do that, too, which is why we wanted it, not just 7zr:

Format	Command	File Size	Ratio
zip	7z a -mx=9 -tzip Ru.zip Ru.tar	3287420	0.123
gzip	7z a -mx=9 -tgzip Ru.tar.gz Ru.tar	3287335	0.123
bzip2	7z a -mx=9 -tbzip2 Ru.tar.bz2 Ru.tar	2989193	0.112

So 7z compresses to zip/gzip better than the native programs do it themselves. It doesn't really outperform bzip2 here, though. The only disadvantage compared to gzip or bzip2 is that it doesn't compress the files in place, unless you go through a script such as the one in /usr/bin/p7zip.

Test 2

Pi to 4 million Decimals has, duh, π to, actually, 4,194,034 places. The file pi.tar.gz has it in ascii, with a bit of header information. If we uncompress that file, it comes in at 4362370 bytes. Since the digits of π don't repeat, it's hard for a compression program to find blocks of bytes to compress. The following table lists the compressions achieved by our test programs, in whatever formats they can use. (See the above tables for the appropriate commands.) Let's see how everybody does:

Program	Protocol	File Size	Ratio
None	None	4362370	1.000
zip	zip	2041130	0.468
gzip	gzip	2040997	0.468
bzip2	bzip2	1863892	0.427
7z	zip	1983378	0.455
7z	gzip	1983297	0.455
7z	bzip2	1860047	0.426
7z	7z	1884999	0.432

Here, bzip2 is competitive with 7z. Oddly, though, you should use 7z to do the bzip2 compression. Weird, huh? But still, 7z is pretty good.

Wrapping Up

So what's not to like?

For one thing, 7z is slow. If you just want to quickly compress a file, go ahead and use gzip, or bzip2. There is a price to pay for better compression.

Then, too, there's a warning on the 7z man page:

DO NOT USE the 7-zip format for backup purpose on Linux/Unix because 7-zip does not store the owner/group of the file.

you can get around this by piping tar into 7z:

tar cf - directory_to_be_archived | 7z a -si directory.tar.7z

which creates the analog of a gzip/bzip2 tarball.

And finally, the native 7z format isn't standard, yet, so it's not going to be available everywhere, and might even vanish. But 7z and its compression algorithm LZMA are open source, so they are likely to stay around for awhile. A few years ago bzip2 wasn't standard, and once upon a time neither was gzip. It's probably safe to compress your files to 7z format, but if you want to be safe, use 7z to compress to gzip or bzip2 format.