Sunday, November 28, 2010

Mounting Remote Computer Files with FUSE

I'm sitting here, upstairs, typing on my laptop. There's a file I want on my office machine downstairs. So I should go downstairs and look for the file, right?

Wrong. The TV up here is on NFL Red Zone ( (sigh) Red Zone). I'm addicted, OK? It's hard enough to break away for a bathroom trip.

Several options exist, of course. I could ssh to the downstairs machine, search for the file, and use scp to copy it upstairs. But the file is a picture, and I while I remember what it looks like, I don't remember what name I used for it. So ideally I'd like to use an image viewer, say gthumb or F-Spot to look through the directory and find the file. But I can't, because this computer doesn't mount the downstairs computer's disk.

Ah, but it can. The classic way to do this is to use the Network File System protocol, aka NFS. But that requires work to set up. Besides, some days I might want to search my computer from work, or from a hotel. It's tricky to get NFS to work that way.

Fear not, there is a solution. It uses File System in Userspace (FUSE), a complicated name for using ssh and scp to fool my laptop into thinking that the downstairs computer's files are on the laptop's local drive.

In Ubuntu the way to do this is to install a package called sshfs. Then you just follow the directions on the linked page. Let's say I want to be able to access all the files in /home/rcjhawk on hal, my downstairs computer, from my laptop, majel:

  • First make sure I have both hal and majel set up to use ssh communications. If I'm away from home, I'll also need to know hal's ip address, and make sure I can ssh to hal from outside my router.
  • Next, install sshfs on majel. Just use
    $ sudo apt-get install sshfs
    from a terminal window on majel. This doesn't need to be done on hal. Hal's just going to see a series of ssh requests.
  • Add myself to the FUSE group on majel:
    $ sudo gpasswd -a $USER fuse
    from the same command window.
  • Log off my account on majel, log back on. Yes, it's annoying, but you're not officially added to the FUSE group until you do.
  • Make a directory on majel. Doesn't matter what the name is:
    $ mkdir ~/hal
  • Now we're going to mount all of hal's directory tree starting from /home/rcjhawk in majel's ~/hal directory:
    $ sshfs -o idmap=user rcjhawk@hal:/home/rcjhawk ~/hal
  • If this works, then running the command
    $ ls ~/hal
    from a terminal window on majel lists hal's directory tree.
  • To unmount hal, just run:
    $ fusermount -u ~/hal

To make things easier on myself, I created two files. The first, mthal, contains the mounting command, and the second, umthal, the unmount command. Make these executable, put them in your path, and it's easy to mount and unmount.

A caveat: I was getting an error remote host has disconnected. That occurred because my .cshrc file, which used a variable that's undefined when you are running ssh commands. You can find out how to diagnose that here.

Does it work? We'll, I started typing this on the laptop, and finished it up on the big machine downstairs. So, yes, it does work.

There are also versions of FUSE for BSD, OpenSolaris, and Mac OS X. In fact, I first learned about this trick using my Mac at work. With FUSE you can mount just about anything anywhere. There's even GmailFS, which transforms your Gmail account into backup storage.

Saturday, November 27, 2010

Finding Your Header Files — A Better Way

I should really read my own posts. Especially since that's the original purpose of this blog.

The other day I was looking through old posts with the Ubuntu label, and found this comparison of dpkg/apt-get and rpm/yum commands. The relevant one is

dpkg -l '*'

which lists all of the packages available in your repository, installed or not.

Duh

This makes my header file lister a lot simpler to write, towit:

#! /bin/bash

# First make sure you've got all your repositories updated:
sudo apt-get update

# Now search through every repository database, looking for lines that
#  start with "Package", and end with "-dev".
# Pull out the package name, look through each package, and print out
#  the header files:  those that end in "-h"

# No matter how it wraps on your screen, this next line starts with
# "for" and ends with "awk '{print $2}'`"

for hfile in `dpkg -l '*' | grep dev | awk '{print $2}'`
do
# echo $hfile
apt-file list $hfile | grep "\.h$"
done

It's still going to take a long time to go through every package, but it's a lot neater now.

Thursday, November 25, 2010

Look Familiar?

I saw this watching the Star Trek reboot on the bigscreen with our Epix OnDemand account. It's during the Old Spock/New Kirk mind-meld, at the part of the story where Spock is transported back in time. The blue box is suggestive, don't you think?

Since Epix lets you stream its movies to a Linux box, unlike Netflix, it was relatively easy to find the shot and save it. No, the resolution's not that great, but, frankly, this movie suffers even on a 48" high-def screen. To do it justice you really need to watch it in a theater.

Saturday, November 20, 2010

Undercutting Yourself

Maybe I just don't understand…

The other day Borders sent me an email which gave me ten dollars in Borders' Bucks. A great promotion, to be sure. And I had a book in mind I wanted to buy, but not spend a lot of money on: Jim Butcher's collection of Harry Dresden short stories, Side Jobs.

First check it out online. We have:

List Price $25.95
Online Discount -$11.22
Borders' Bucks -$10.00
Shipping $3.98
MD Tax $0.29

Total $9.00

Not bad, but the shipping price sucks, it will take days to get here, and anyway I want to get out of the house for awhile, so let's go to a nearby Borders and buy it there.

I get to the Borders, and find Side Jobs. The sticker on it says 20% off. Do a little bit of math: 80% of $25.95 = $20.76 - $10 Borders' Bucks = $10.76, + 6% tax ( $0.65) = $11.41. A bit of a rip, compared to online, but I'm here, I've already spent a buck in gas, and I'll walk out with the book.

Except: I then notice that the sticker also says Borders Rewards Plus Members. I'm just an ordinary Borders Rewards member. I checked with one of the sales people, and no, I'm not entitled to the discount. That makes the cost $16.91, Borders' Bucks and tax included. I made a point of telling the guy his store's website was undercutting the store by selling the book for $5 less, including shipping, put the book down, and walked out.

I suspect some marketing guru has it all figured out, but to me this just seems plain old-fashioned stupid. If Borders gave me the online price at the store, or even got close to it, I would have gotten the book then and there. Since I was in the store, I probably would have had a coffee as well. Maybe a cookie. I would have looked at a Linux magazine while I was sipping my coffee, and possibly would have purchase that, too. I would most likely have spent the $5 in the store, maybe more, and I wouldn't have been pissed.

Instead, I went home and ordered the Dresden book online. Even with the $2 gas I just wasted I still saved $3 over buying it in the store. And, instead of waiting for more coupons/bucks to come my way, I went over to Amazon and ordered two books I would have otherwise eventually gotten at Borders (along with more coffee, cookies, magazines, etc.): Lois McMaster Bujold's newest Vorkosigan novel, Cryoburn, and Terry Pratchett's new Tiffany Aching/Discworld novel, I Shall Wear Midnight. Add in a book ChildII wanted, we made the supersaver price, and got free shipping.

In summary: by charging full price for a book in the store, and a deep discount online, Borders lost the sale of at least two books, two-three cups of coffee, a couple of cookies, and maybe a magazine.

It looks like Borders would rather you didn't come to their store. OK, stores cost money to keep up. Employees have to be paid, not to mention heating, lights, insurance, and all of that. But if I come into the store, I'm going to browse. I'll look around, and might well find something else I'd like to buy. Online, I'm going to get the books I want, hit the send button, and go on with my life.

Like I said, some genius with an MBA probably has figured out that this approach makes more sense for Borders, but how that could be is a mystery to me.

The trip wasn't wasted, I drove a mile to the local wine emporium, tasted a couple of wines, and bought a bottle.

Thursday, November 11, 2010

Unintended Consequences

Article 1, Section 2, of the United States Constitution says, in part, The Number of Representatives shall not exceed one for every thirty Thousand, but each State shall have at Least one Representative. The very first amendment proposed (you can look it up) tried to change this:

After the first enumeration required by the first article of the Constitution, there shall be one Representative for every thirty thousand, until the number shall amount to one hundred, after which the proportion shall be so regulated by Congress, that there shall be not less than one hundred Representatives, nor less than one Representative for every forty thousand persons, until the number of Representatives shall amount to two hundred; after which the proportion shall be so regulated by Congress, that there shall not be less than two hundred Representatives, nor more than one Representative for every fifty thousand persons.

but this was never ratified. So we can be pretty sure that the Founders of this Republic wanted us to have one Representative for every 30,000 Americans.

Currently, however, we have 435 Congressmen, thanks to something called Public Law 62-5. I guess this was supposed to keep the number of Congressmen down to a reasonable number. Otherwise, with the current U.S. population around 300,000,000, we'd have 10,000 Congressmen running around. Think of it! Mark Russell once said something on like every night 500,000 people in the District of Columbia go to bed without a Congressman. But if we had 10,000 Congressmen, you give yours (DC residents excepted) a booty call just about every night.

The current limit of 435 Congressmen also gives rise to some startling inequities. For example, according to the 2000 Census, the total population of the United States was 281,421,906. If we exclude the District of Columbia (population 572,059) since it has no Congressional representation, then there are 280,849,847 Americans and 435 representatives, so each member of Congress should represent 645,632 people. However, Wyoming (493,782) and Montana (902,195) each have only one representative. To put it another way, Cynthia Lummis only needs to attract half as many votes as Denny Rehberg to get elected, but each gets one vote in the House. So by moving from Wyola, Montana to Parkman, Wyoming (16 miles) you can effectively double the strength of your vote for Congress.

There is a lot more information about the inequities of the current system, and the sorry history of Amendment the First, at thirty-thousand.org. I'd recommend downloading their pamphlet, Taking Back Our Republic, for more information.

One thing they didn't cover, though: the effect of all of these changes on the Electoral College. Each state gets N+2 electoral votes, where N is the number of representatives, plus the two members of the Senate allowed each state. That +2, along with Public Law 62-5, weights the Electoral College heavily toward the smaller states. Since D.C. does have 3 votes in the Electoral College, each elector theoretically represents 281,421,906/538 = 523,098 voters. However, each state has a minimum of three electoral votes. So in Wyoming each elector represents 164,594 people. In Montana, it's 300,732 people, still substantially below the national average. It's likely (I haven't checked in detail) that the state with the most people/elector is California, where there are 33,871,648 people and 55 electors, or 615,848 people/elector. Put it this way: an elector in California represents four times as many people as one in Wyoming, but each has the same say in who's elected President.

Now, as anyone can tell you, most of the smaller states, population-wise, are in the west, and most of those states vote Republican, while many of the larger states tend to vote Democratic. Another wrinkle in the system is that most states (Maine and Nebraska excepted) cast their electoral votes in a block.

Couple these two facts with the 30,000 people/representative rule and you get some interesting results. Very interesting, in some cases.

In particular, let's consider the 2000 Presidential Election. Very close, as we all recall. Al Gore got 50,999,897 votes, while George Bush got 50,456,002, but because most states cast their electoral votes as a block, Bush won in the Electoral College, where it counts, 271-266.

What if we'd apportioned Congress the way the Founders intended? And kept the current winner-take-all (mostly) electoral system? The table below shows the results. I constructed it using the following rules:

  • Winner take all in each state, as now. Maine (42 votes under these rules) went to Gore, and Nebraska (54) went to Bush. Changing all of Maine's votes to Bush or all of Nebraska's to Gore isn't going to change the final result, so it doesn't matter if several congressional districts shifted from Red to Blue or vice versa.
  • To get the number of representatives, I took the states' populations from the 1990 census, which governed apportionment at the time, divided by 30,000, and rounded down. Rounding to the nearest integer, or rounding up, can't change the electoral vote by more than 51, and so won't affect the result, either.
  • If the District of Columbia was a state, it would have 22 electoral votes in this scheme. However, by the Twenty-Third Amendment, D.C. gets 17 electoral votes, equal to the state with the fewest, Wyoming. In the real 2000 election one D.C. elector abstained. Even assuming all 17 of the new electors abstained it wouldn't change the bottom line here, so I assume that they all voted — for Gore, naturally.

The result: Since each state has more or less the same representation per capita in the electoral college, and since most of the larger states went for Gore, he wins the election, 4,314–4,047.

Think about it. Al Gore, Forty-Third President of the United States. Somehow, I don't think that's what those currently saying we should go back to the Principles of the Founders had in mind.

State Population Congress Electors Bush Gore Bush Gore
Popular Popular Electoral Electoral
Alabama 4040587 134 136 941173 692611 136 0
Alaska 550043 18 20 167398 79004 20 0
Arizona 3665228 122 124 781652 685341 124 0
Arkansas 2350725 78 80 472940 422768 80 0
California 29760021 992 994 4567429 5861203 0 994
Colorado 3294394 109 111 883748 738227 111 0
Connecticut 3287116 109 111 561094 816015 0 111
Delaware 666168 22 24 137288 180068 0 24
DC 606900 0 17 18073 171923 0 17
Florida 12937926 431 433 2912790 2912253 433 0
Georgia 6478216 215 217 1419720 1116230 217 0
Hawaii 1108229 36 38 137845 205286 0 38
Idaho 1006749 33 35 336937 138637 35 0
Illinois 11430602 381 383 2019421 2589026 0 383
Indiana 5544159 184 186 1245836 901980 186 0
Iowa 2776755 92 94 634373 638517 0 94
Kansas 2477574 82 84 622332 399276 84 0
Kentucky 3685296 122 124 872492 638898 124 0
Louisiana 4219973 140 142 927871 792344 142 0
Maine 1227928 40 42 286616 319951 0 42
Maryland 4781468 159 161 813797 1140782 0 161
Massachusetts 6016425 200 202 878502 1616487 0 202
Michigan 9295297 309 311 1953139 2170418 0 311
Minnesota 4375099 145 147 1109659 1168266 0 147
Mississippi 2573216 85 87 572844 404614 87 0
Missouri 5117073 170 172 1189924 1111138 172 0
Montana 799065 26 28 240178 137126 28 0
Nebraska 1578385 52 54 433862 231780 54 0
Nevada 1201833 40 42 301575 279978 42 0
New Hampshire 1109252 36 38 273559 266348 38 0
New Jersey 7730188 257 259 1284173 1788850 0 259
New Mexico 1515069 50 52 286417 286783 0 52
New York 17990455 599 601 2403374 4107697 0 601
North Carolina 6628637 220 222 1631163 1257692 222 0
North Dakota 638800 21 23 174852 95284 23 0
Ohio 10847115 361 363 2350363 2183628 363 0
Oklahoma 3145585 104 106 744337 474276 106 0
Oregon 2842321 94 96 713577 720342 0 96
Pennsylvania 11881643 396 398 2281127 2485967 0 398
Rhode Island 1003464 33 35 130555 249508 0 35
South Carolina 3486703 116 118 785937 565561 118 0
South Dakota 696004 23 25 190700 118804 25 0
Tennessee 4877185 162 164 1061949 981720 164 0
Texas 16986510 566 568 3799639 2433746 568 0
Utah 1722850 57 59 515096 203053 59 0
Vermont 562758 18 20 119775 149022 0 20
Virginia 6187358 206 208 1437490 1217290 208 0
Washington 4866692 162 164 1108864 1247652 0 164
West Virginia 1793477 59 61 336475 295497 61 0
Wisconsin 4891769 163 165 1237279 1242987 0 165
Wyoming 453588 15 17 147947 60481 17 0
Totals     8361 50455156 50992335 4047 4314

Sorry about the lack of commas in the table above. I used a very primitive spreadsheet to generate the data.

1990 Census data is from http://lwd.dol.state.nj.us/labor/lpa/census/1990/poptrd1.htm. Presidential vote totals are from http://www.presidency.ucsb.edu/showelection.php?year=2000.

Monday, November 08, 2010

Find Every Header File in Your Ubuntu Repositories

Last night I was following a thread on Penguin Pete's blog where Pete uses Ubuntu's dropping the Dillo browser from the distribution as a launching pad for a discussion about — heck, I can't describe what it's about, read it yourself.

Anyway, in the course of reading the discussion — if that's the word, Pete gets worked up about certain things — I was possessed by the moral imperative to install Dillo from source.

This would seem to be straightforward. You go to the Dillo home page, click Download, and follow the instructions.

Except (there's always an except) — the first install you have to make is the FLTK2 (Fast Light Toolkit) library.

Now Ubuntu offers FLTK1.1, which might well work, but hey, we need the latest and the greatest, right? So I downloaded the tarball, followed the instructions, and the make command crashed and burned with:

fatal error: X11/extensions/XInput.h

I looked on the web, found that XInput.h is in the package libxi-dev, installed that, reran make,

fatal error: GL/glu.h

and so on. It took about an hour to find every single header file needed. And that was just to install FLTK2, after which I still had to install Dillo.

So what's going on here? Well, first we have to know what a header file is. Basically, it's a file that defines a bunch of variables and statements that can be used by multiple routines. See Wikipedia for a better explanation. In particular, header files tell a program's source code what variables are used in a particular library. Thus the header XInput.h defines variables available in some X11 libraries.

OK, suppose I'm compiling a program and I find that I need a header file GrantsTomb/Buried.h. All I have to do is find the package that has that header, install it with apt-get, and I'm good to go. Right?

Well, without prior knowledge you're pretty much doomed here. I was able to find XInput.h and glu.h by web search, but who's going to tell you where is Buried.h in GrantsTomb?

Yet there has to be a solution, because:

  • I know that when I use synaptic or aptitude to install packages I can see every package that is, or can be, installed on my system from my selected repositories.
  • After searching around a bit, I find that apt and its children list the available files in /var/lib/apt/lists, each repository has its own file, and each file name is on a line starting with Package.
  • I know about the program apt-file, in particular that the command
    apt-file list foobar
    will give me a list of all of the files in the foobar package.
  • Header files are traditionally located in packages ending with the string -dev.

Given all of that, I should be able to write a script which will list every header file available in every Repository I've accessed. I can then save it in a file, and the next time a program says I need bojangles/tambourine.h I can just search the list, find that the header file is in the NittyGritty-dev package, load that up, and I'm ready to compile:

  • First, install apt-file, if you don't have it already:
    sudo apt-get install apt-file
  • Then run this script:
    #! /bin/bash
    
    # First make sure you've got all your repositories updated:
    sudo apt-get update
    
    # Next sync apt-file's database.  Note that if you do this without
    #  the sudo you'll write the data into your home directory, which is
    #  probably OK
    
    sudo apt-file update
    
    # Now search through every repository database, looking for lines that
    #  start with "Package", and end with "-dev".
    # Pull out the package name, look through each package, and print out
    #  the header files:  those that end in "-h"
    
    #! /bin/bash
    
    # No matter how it wraps on your screen, this next line starts with
    # "for" and ends with "xargs`"
    
    for hfile in `find /var/lib/apt/lists -type f -exec grep '^Package: ' {} ';' | awk '{print $2}' | grep -e '-dev' | xargs`
    do
    apt-file list $hfile | grep "\.h$"
    done
    

Be warned, this takes a long time, since every call to apt-file goes out over the network. On my system, with FIOS broadband, it took over an hour. So you'll only want to run this file once in a while, probably only when you update your repository list. You may also get an error message like
grep: /var/lib/apt/lists/lock: Permission denied
Don't worry about that.

This script should work pretty well for Debian. There are also equivalent commands for RPM-based distributions. And, of course, you can modify it to list any subset of files in any subset of repositories you want. I'll leave all of that as an exercise for the reader.