Tuesday, January 01, 2013

Sorting From Back to Front

The other day I was presented with a list of names and email addresses, something like this one:

Fred Flinstone <flintstone@bedrock.sag>
Barney & Wilma Rubble <bambamsfolks@bedrock.sag>
Steadholder Honor Harrington <dutchess@harrington.mdc>
Count Miles Vorkosigan <auditor@vorkosigan.byr>
Dennis & Margaret Mitchell <stilltrouble@funnies.comics.net>
Homer Simpson <donuts@springfield.st.us>
Rudolph <rednose@reindeer.np>
Kimball Kinnison <kinnison@graylens.gp>
Wile E. Coyote <genius@acme.net>
Tiberius Claudius Drusus Nero Germanicus Julius Caesar <imperator@spqr.rm>
Gen. Jack O'Neill <jack.oneill@stargate.oml>

Except that it had around 100 names. What I wanted to do was to alphabetize this by last name, to make it easier to figure out who was missing from the list, but keep the final result as
     FirstName MiddleName(s) LastName <email>
since this was input to an email list in that format.

This would not difficult if each person had exactly two names, say
     FirstName LastName <email>
in which case we'd just run the command
     sort -k 2 < elist
and we'd be done.

Unfortunately each line contains between two and eight fields, counting the email address, and we want to sort on the next to last one. As far as I can tell, sort doesn't support searches from the end of the line in.

However, the awk (or gawk) command does. For example, the command
     awk '{print $NF}' < elist
would list just the email addresses from the above file, and
     awk '{print $(NF-1)}' < elist
would list the last names — no, I don't know why you use parenthesis, but you do.

So what we need is a way to have awk pull out the last name from the file, sort those, then put everything back together. It turns out we can do that with a one-liner. I found it on the web yesterday, but I've lost the link, so I can't give proper credit. I did save the command, or my modification of it, at least:

awk '{print $(NF-1), $0}' < elist | sort | cut -f2- -d' '

Let's look at that in detail:

  • awk '{print $(NF-1), $0}' < elist
    prints out the next to last column of each line, followed by the entire line ($0).
  • sort
    then sorts everything on the first column, e.g. the last name. Unfortunately, that leaves you with entries like this:
     Simpson Homer Simpson <donuts@springfield.st.us>
    To get rid of these, we need
  • cut -f2- -d' '
    which separates fields by whitespace (the -d' ') and prints everything out starting from the second column (-f2- . If we wanted just the second and third column it would be -f2-3).

And the correctly sorted output is:

Tiberius Claudius Drusus Nero Germanicus Julius Caesar <imperator@spqr.rm>
Wile E. Coyote <genius@acme.net>
Fred Flinstone <flintstone@bedrock.sag>
Steadholder Honor Harrington <dutchess@harrington.mdc>
Kimball Kinnison <kinnison@graylens.gp>
Dennis & Margaret Mitchell <stilltrouble@funnies.comics.net>
Gen. Jack O'Neill <jack.oneill@stargate.oml>
Barney & Wilma Rubble <bambamsfolks@bedrock.sag>
Rudolph <rednose@reindeer.np>
Homer Simpson <donuts@springfield.st.us>
Count Miles Vorkosigan <auditor@vorkosigan.byr>

Fairly simple, huh? I generalized it a bit, so that we can sort on an arbitrary column from the end:

#! /bin/bash

# Usage

# lastsort N filename
# Sorts the file filename of the field N columns from the end
# N=0 is last column of the file

awk '{print $(NF-'$1'), $0}' $2 | sort | cut -f2- -d' '

Note the single quotes around the $1 in the awk command, which passes the first argument of the calling command to awk. Without the quotes you get an error.

OK, this could have a few bells and whistles, but I'm not going to bother with that now.

Monday, February 19, 2007

Almost One Line Archiving

This is one of those “well, if I don't write it down, I'll forget it,” posts, but someone else may find it helpful as well.

I recently did a bunch of updates on my favorite website. I keep a duplicate copy of the site where I can work on it without making a big mess of the active site. All of the files are in a directory we'll call $HOME/website. So I'm going to modify the files in my local copy of $HOME/website, check to make sure I haven't frakked anything, and then upload everything to the hosting computer, pause, take a deep breath and pray I really haven't frakked anything, and update the real website.

So what I want to create is a tarball consisting of all the files I've changed in $HOME/website since the last time I did an update. This is an almost one-liner. I say that because you need to do one step before you make any changes at all:

$ touch $HOME/newupdates

This creates a zero-length file. The important thing is that it will be older than any of the files you are about to change, and younger than any of the files that remain unchanged. If you have another file like this around, then you don't need this step.

Now make all the changes you want, making sure you have valid spelling, HTML, physics, tax advice, etc.

Now do the archive:

cd $HOME ; tar cvzf updates.tar.gz \
`find $HOME/website -type f -newer newupdates \
| xargs`; touch newupdates

(That's really all one line, of course, though you should be able to enter it as is with the backslashes.)

This uses the find command to list all regular (-type f) files in the directory $HOME/website which are newer than newupdates, then uses xargs to put them into a single string, and creates a gzipped tarball file named updates.tar.gz. Then it again touches the newupdates file, so that the next time you do this you won't pick up the files you've already archived.

Then copy updates.tar.gz wherever you want it.

Monday, January 16, 2006

Command Line "mail" with SMTP

Back in the olden days*, when you wanted to send email from a Unix terminal, it went something like this:

$ mail  joeschmo@foobar.net
Subject:  Here's my email
Joe,

Mom said we should talk more often.  So I'm sending you this email.
Bye.
^D

And Joe would eventually get the email. (The ^D is Control-D, used to denote an end of file.) Now the nice thing about the old "mail" command is that it would accept anything from standard input. So if I wanted to send Joe a really long email I could put it in a big file, and pipe it into the mail command:

$ cat bigfile.txt | mail -s "Dear Joe" joeschmo@foobar.net

or

$ mail -s "Dear Joe" joeschmo@foobar.net < bigfile.txt

(The -s indicates that the next entry is the subject line of the email, the subject itself is in quotes to keep the command shell from thinking it's a bunch of options.)

This is fine so long as your local computer handles its own mail. However, these days home users mostly deal with ISPs, who have their own mail systems, which we usually access with an email client that understands the Simple Mail Transfer Protocol. These days, your email clients are usually graphical ones like Evolution, Thunderbird, Eudora, or Mozilla, but the all-you-can-eat text editor, Emacs, also can send mail via SMTP.

The program that can't send mail via SMTP is the old Unix "mail" command. It isn't the fault of "mail" itself, it's the fault of your Linux box. "mail" looks for a program called "sendmail", or its replacement, to send mail through the system. Now most sendmail-like programs assume that your computer, or one on your network, is your mailserver. They don't speak to your ISP via SMTP.

Why do I care? Well, a friend recently asked me if he could generate a file on his computer and send it to another friend without him having to do anything else. On a system with "mail" working, it's easy. Suppose the command "genbignovel" creates a large text file "pulitzer.txt". Then to write the novel and send it to Joe, who's on the Pulizter committee, I could create a small text file:

genbignovel
mail joeschmo@foobar.net < pulitzer.txt

which sends the file to Joe as soon as it's generated. If I'm using Evolution, Thunderbird, etc., then I have to send the file as an attachment, which means a bunch of mouse-clicks, at the very least.

So what we need is a replacement for sendmail which understands SMTP. A Google search reveals many, but the most popular one seems to be msmtp. I decided to try it out, so I downloaded the file, installed it using checkinstall, and tried to figure out how to set it up.

It's not too hard, though most people seem to use msmtp with mutt rather than mail. First, you need to create a file to tell msmtp which SMTP server you want to use, and the information needed to access that account. This is done through a file called ~/.msmtprc, i.e., a file named .msmtprc that lives in your home directory. Here's mine, with a few obvious changes:

account default
auto_from off
host smtp.gmail.com
from dubya@gmail.com
auth on
password xyzzymouse
user dubya@gmail.com
tls on

This obviously works on gmail, but you should be able to modify it for any SMTP server. You can look at the msmtp manual to see what everything means.

Now how do I get "mail" to use msmtp? Simplicity itself. mail uses a command file called ~/.mailrc. Create it, if you don't have one yourself, and add a line:

set sendmail=/usr/local/bin/msmtp

if you put msmtp in /usr/local/bin.

And now mail should work. It does from here, anyway.

Good God, how many posts of mine start with "back in the olden days?" Must remember that not everyone sent their first email in 1983.

Wednesday, May 11, 2005

Making a Linux Computer Talk

I'm back from the trip. I've got a lot of topics I want to post, but first, today I read an article from this month's issue of Linux Gazette (.net), titled, Shelling your Linux box with Festival, which tells you about a Linux speech synthesis program, Festival.

The article has some neat tricks, such as having your computer read the headlines from the BBC News Feed.

However, what I want to talk about isn't in the article. What I want to do is to create a greeting which can be heard online, such as this one.

It's not too hard. In the Festival distribution is a program called text2wav, which seems to create .wav files. Anyway, this works:

$ echo "Please reed Linux and Things" | text2wav > plsrd.wav

where we have to say "reed" rather than "read", since Festival wants to put the latter in the past tense.

OK, that's a .wav file, which is 66498 bytes. I can compress it into an MP3 file using the MP3-enabled version of sox:

$ sox plsrd.wav plsrd.mp3

which comes in at 6588 bytes. Of course, I should use a politically correct format such as Ogg, but this way more of you can hear the voice.

Festival has other tricks, including a British accent, but I haven't figured out how to use them yet.

Saturday, November 20, 2004

Hotplugs

It used to take some work to set up a hotplug, that is, to mount an external device such as a memory key or digital camera. No more. Just plug the device in, and do

$ ls -l /media/
total 24
drwxr-xr-x  2 root root 4096 Nov 20 09:58 cdrecorder
drwxr-xr-x  2 root root 4096 Nov 20 09:58 floppy
drwxr-xr-x  2 root root 4096 Nov 20 19:00 usbdisk

usbdisk is what's in the USB port, so do

$ mount /media/usbdisk/

Move files around as usual, and then

$ umount /media/usbdisk

before unpluging.