I recently set up a tunnel from my AirPort Extreme to Hurricane Electric so I can access hosts using IPv6. I had done this previously using m0n0wall and documented it so others could follow in my footsteps. This post does the same but for AirPort Extreme.

The first step is to create an account with Hurricane Electric's Tunnel Broker service. From there select Create A Regular Tunnel. Enter your IP address and then submit the form. Once the tunnel is created click Tunnel Details. The page should look something like this.

Now open AirPort Utility and go to Advanced ➔ IPv6. For IPv6 Mode select Tunnel. From there you want to change Configure IPv6 to Manually. This will give you a couple of text fields which you can fill out using values from the tunnel detail page. Here is my configuration.

In other words, here is the mapping from Tunnel Broker to AirPort Utility.

To make this even more useful, Hurricane Electric runs a nameserver that supports Google over IPv6. If you set this as your nameserver you will receive AAAA records for certain Google properties. To enable this, in AirPort Utility go to Internet ➔ TCP/IP and under DNS Server enter the address found in Tunnel Broker's Anycasted IPv4 Caching Nameserver field.

To test that everything worked you can try pinging Google.

Google Wave used to have a realtime language translation robot named Rosy Etta. Unfortunately it isn't currently available, so I had to write my own. I named my translation bot Aunt Rosie after my aunt and as a tribute to the original Rosy.

If you include a special keyword, Aunt Rosie will reply to your message with a translation in the language you specified. Watch the video for a demonstration.

To get started, you should add aunt-rosie@appspot.com to your contacts. Now invite her to a Wave.

To begin translation, type "/translate:xx" in your blip. The XX is an ISO 639-1 code which specifies the target language. Chinese is the exception. Use "zh-CN" for simplified Chinese and "zh-TW" for traditional. Note: Aunt Rosie is in beta and still has some bugs. As such, you need to add a space and a newline after the language target. Rosie should appear as soon as you you type in this keyword. If she doesn't, then you might be doing something wrong.

Rosie automatically detects the language you are writing. However, this means you'll need to type enough text for her to recognize the language. Usually it takes 5-6 words before she starts translation. If you are using short words or proper nouns, it might take even longer. Try not to put too many proper nouns in one sentence or she'll get confused.

Aunt Rosie uses Google's machine translation software as her backend. She speaks all the languages Google supports. She is written in Java and runs on Google's App Engine.

Update 2009-10-13: I just finished making Aunt Rosie a lot easier to use. Instead of having to type special keywords and remember two letter codes, Aunt Rosie will automatically insert a language drop down into your blip when you've typed enough for her to recognize your language. Select the language you'd like to translate to and she'll reply with the translation. Easy as pie.

Update 2009-10-23: Occasionally the bot stops working due to a bug in Google Wave. I've mirrored the bot at webmaster@appspot.com, so switch to webmaster if Aunt Rosie is broken. You can track bug 278 on Google Wave's developer site

Update 2010-07-19: It took a long time to get work approval and then I was lazy uploading it somewhere (and in the meantime I switched to Git), but the source code is finally available on GitHub.

Hadoop 0.20 was released on April 22nd. I was curious about some of the changes in 0.20 so I did a little research and decided to blog about what I found. These are the changes, features, and improvements that were interesting to me. For a full list of changes, see the Hadoop 0.20 changelog.

My first impression was that 0.20 is less ambitious feature-wise than 0.19. That is not a denigration of Hadoop's committers, but an artifact of Hadoop's quarterly release cycle. You can read about Hadoop 0.19's features in Cloudera's excellent write up.

Context Object For Mapper and Reducer
The biggest change in 0.20 is a large refactoring of the core MapReduce classes (HADOOP-1230). The commit message lists the changes best:

1. All of the methods take Context objects that allow us to add new methods without breaking compatability.
2. Mapper and Reducer now have a "run" method that is called once and contains the control loop for the task, which lets applications replace it.
3. Mapper and Reducer by default are Identity Mapper and Reducer.
4. The FileOutputFormats use part-r-00000 for the output of reduce 0 and part-m-00000 for the output of map 0.
5. The reduce grouping comparator now uses the raw compare instead of object compare.
6. The number of maps in FileInputFormat is controlled by min and max split size rather than min size and the desired number of maps.

An example of the changes can be seen in map's method signature (before and after):

void map(K1 key, V1 value, OutputCollector<K2, V2> output, Reporter reporter)
protected void map(KEYIN key, VALUEIN value, Context context)

OutputCollector and Reporter can both be accessed through the new Context object.

This is a large change, so Hadoop added the classes in a new package at org.apache.hadoop.mapreduce. The old classes have been deprecated but can still be found under org.apache.hadoop.mapred.

This feature will allow Hadoop Core to quickly iterate and add new features without breaking end user code every release. This is an important milestone on the path to stability as Hadoop approaches 1.0.

Vaidya
Hadoop 0.20 also includes a new performance tool called Vaidya (HADOOP-4179). Vaidya scans your job logs using a collection of rules to identify potential performance issues. For example, if it notices that your Mapper is writing a lot of data to disk, it'll suggest you create a Combiner. It reminds me a bit of Fortify or FindBugs (or if you're feeling less generous: Clippy) in that it suggests improvements when using bad practices (although, not using powerful static analysis like the aforementioned software).

BloomMapFile
HADOOP-3063 gives us BloomMapFile, a fast-failing implementation of MapFile. MapFiles don't seem to get much attention, although having used them for a project in my CSE 490h class with Hadoop 0.7.2, I appreciate the addition.

Removal of LZO
Finally, the last change that caught my attention was the removal of the LZO compression codec in HADOOP-4874. The LZO code was infected with GPL (I hear that's been going around recently) and thus had to be removed. However, the LZO code lives on as a side project on Google Code.

Last week it was #savejon and this week it is #amazonfail. What do these hashtags have in common? They were both markers used in modern day witch hunts.

The basic pattern is similar. Someone posts an unsubstantiated claim on the internet. Thousands of "twactivists" repeat and forward the claims creating an internet swarm. They write outraged blog posts and send angry e-mails to the supposedly offending company. The few people that actually possess critical thinking skills e-mail the company for more information or start doing research on their own. Eventually the big, slow moving company responds to the issue, but by then all the low-attention span sheeple have already moved on to the next big drama.

In #savejon's case, a guy named Jon claimed claimed a stock art company had stolen his art and was now suing him. Turns out that was all a lie.

With #amazonfail, Amazon stopped giving sales rank to some GLBT books. People thought they were pressured by the religious right to remove the books from the sales charts. This was a perfect example of confirmation bias because people would search for other gay books and see that they didn't have a sales rank either. It is easy to think it only affects GLBT books when you are only searching for those books. Additionally, people searched for [homosexuality] and only got books by anti-gay authors. Of course they did! Gay people usually describe themselves as gay not homosexual!

Google has encountered similar issues with their automated ranking algorithms. A few years ago searches for [Jew] returned an anti-semetic website first. Since people don't understand how search engines work, Google had to post an explanation. There is no ulterior motive, big complicated systems can produce wonky results sometimes.

Hack journalists who think they know computers know that this could never, ever be a computer glitch. No, computers and software always work perfectly. Most people don't realize how hard it is to maintain large ontologies and sets of metadata, especially when that data comes from publishers, third party sellers, and users.

What's even worse is that some people still refuse to believe it was a mistake. These people are as bad as moon landing conspiracy theorists and 9/11 truthers. They maintain their beliefs no matter how much evidence or logic you throw at them. Some people just need to feel persecuted.

I find it interesting that this has happened twice in as many weeks. I'm a little worried about the reactionary witch hunt mentality that people are showing. Instead of taking some random internet claim with a bit of skepticism, people automatically assume the worst. Whatever happened to giving someone the benefit of the doubt or innocent until proven guilty? Drop the persecution complex and just chill, okay?

At the beginning of 2008 I made a resolution to write one blog post a week. I fell short of my goal and only clocked in 21 posts. Nevertheless, I'm going to highlight some of the posts and provide a recap of what I've been up to here in the last year.

I wrote a couple of posts about cloud computing. For earth day, I talked about the potential environmental and economic benefits of using cloud computing. This is relevant given the recent attention paid to datacenter infrastructure by the popular media and the often incorrect studies they propagate. Microsoft launched Photosynth as Software + Services, which I wasn't too fond of. Finally, I explained Microsoft Azure to some ignorant slashdotters.

Speaking of cloud computing, I wrote two apps for Google App Engine. The first was a BigTable-like RESTful web service. It was intended as a joke, but it gets a fair amount of traffic (the website, not the service). I also ported my album cloud mashup over to App Engine.

I completed one of my life goals by finally getting IPv6 set up. Afterwards I wrote a HOWTO on configuring an IPv6 tunnel using m0n0wall. Hopefully it'll help a few people connect to the future of the internet.

And of course, anything I do wouldn't be complete with rants… lots and lots of rants about Leopard, Safeway, Photosynth, bailouts, slashdotters, and the auto industry.

Here's to a great 2009.

Recently there has been a lot of talk about bailing out the automobile industry. I'm against bailouts in general and for the auto industry in particular. They've resisted market changes and are being punished for it, deservedly so. They also seem to be inefficiently run.

People have been rationalizing the bailout on a number of grounds: "The banks got a bailout!", "A million jobs would be lost!". First, the American demand for automobiles isn't going to magically drop to zero (as much as I'd like it to). If one company completely disappears, the demand for cars (and the jobs that accompany it) will be picked up by the other manufacturers. Politicians talk about preserving jobs, but what goes unsaid is that they want existing jobs protected at all costs. Heaven forbid a job be created at a competitor, or that the new efficiencies that made that job obsolete go on to create a new job in another field or industry. A job in and of itself is not worth anything. Keeping someone employed who creates a net negative on the economy is not sustainable. If we improve the operating efficiency of these automakers (at a cost of jobs), then new jobs would be created in other areas with the newly found savings.

Also, I'm tired of people saying manufacturing jobs are important because they "make something". Service oriented jobs make something too, its called wealth, have you heard of it? Just because there is a physical product at the end of the day doesn't mean the job has more value than a service based job. Someone at a financial company that deals with money all day also adds value, if they are improving the efficiency of their business and the economy, otherwise no one would pay them. I work in the service industry, I write software, and judging from the average salaries in my field, I'd say companies believe we add value. If you are creating wealth as part of your job, you are a positive contribution to the economy.

The notion that we, as a nation, don't produce goods is inaccurate. Just because we no longer make cheap steel and other raw materials, doesn't mean we don't make things. There is little money in basic raw materials, which is why we let others do it where the labor is cheaper. America makes advanced and expensive products, such as jet-engines, airplanes, and train locomotives. Intel and IBM have fabs in the United States and a whole green manufacturing industry is popping up in Silicon Valley. We make stuff, alright?

Finally, a bankruptcy could be good for these auto companies. It'd let them restructure themselves and lower costs. If we just keep propping them up with money, they'll keep burning through it and won't improve. By taking money from efficient, profitable businesses and giving it to the automotive companies, which is effectively what would happen, we are removing efficiency from the market and making us all that much poorer.

I just installed the updated version of Google's Mobile App for iPhone because I wanted to test out the voice recognition search. My experience with speech recognition (excluding systems that recognize only a small subset of English, such as digits) has been spotty at best. I know Google has been collecting data using their 411 service for a while, so perhaps they can improve.

I had no faith that the first query I chose would work. Google surprised me when it properly interpreted the query: [can has cheezburger]. Not only did it get the first two words correct, it also used the correct spelling for "cheezburger", even though that word exists in no dictionary. If I just say "cheeseburger", it figures out the correct spelling for that too.

This is one of the reasons I'm so excited about Computer Science. A product like this couldn't have been created even a few years ago because of the vast amount of data that is required to power it (data such as a web corpus, search logs, and 411 voice samples). In fact, in this day and age, data is perhaps more important than algorithms. By throwing enormous amounts of data at the problem, Google's voice recognition algorithm can perform better than any of the best algorithms that don't utilize data. This has only been possible recently because of the affordability of processing terabytes and petabytes of data. I recommend Programming Collective Intelligence for a quick introduction to some of the algorithms enabled by massive amounts of information.

So what's next? I think Google should hook up their excellent voice recognition system with their best of breed machine translation software and create a universal translator. Imagine an iPhone app where you select an output language, speak into the phone, and get back a text or voice translation in the target language. It is still a pipe dream for now. For one, they'd need to port their voice recognition system to other languages, or it'd be a one way conversation. Actually, port is probably the wrong word, "train" would be more accurate. Algorithms created using data—and not code—are language independent from the get go. Instead of writing a whole new algorithm for each language, the system would just need to be trained on an additional corpus, with perhaps a few tweaks to improve precision and recall. To make a truly Star Trek-worthy translator, they'd also have to work on reproducing human speech. Considering how far we've come in the past few years, I don't think a universal translator would take more than 2-4 years to come to fruition.

I'm sick and tired of people making stupid comments when articles about "The Cloud" are posted. If you don't understand the topics on which you speak, then please don't speak! I finally got fed up and replied to a discussion about Microsoft's Azure. Here's the meat of my reply:

"This offering from Microsoft isn't about a web based office suite or webmail, it is foundational web services that allow businesses and developers to build websites and services while offloading the heavy lifting (such as writing distributed systems or load balancing). The primitives Microsoft is offering are similar to those Amazon already has: storage, database, compute, queueing. In general, you don't access these through your browser.

"This isn't some new AJAXy Web 2.0 website. "The Cloud" is about outsourcing the building blocks of software—database, storage, compute—to someone else and paying for exactly what you use. Instead of buying your own machines, managing the fleet, and building or buying scalable software, you pay for a service and someone else takes care of all of that for you.

"It is like the transition to the electric grid. Instead of paying for a generator and diesel upfront, you just pay for what you use from the electric company, and benefit from their economies of scale. This is utility computing."

Disclaimer: I work for Amazon Web Services. However these opinions are mine and mine alone. They do not represent the opinions of Amazon.

My heart goes out to those affected.

I'm getting more disgusted by the week, as the government nationalizes one company after another. I'm being punished for the reckless actions and mistakes of others. I looked into buying a house last summer but decided against it because I thought the market was too shaky and because it would significantly limit my freedom for years to come (for example, the freedom to create a startup).

Now we are faced with an enormous government bailout that will cost each taxpayer thousands of dollars. And this couldn't have happened at a worse time. Since the election is almost upon us, both candidates are veering even more towards populism in order to win votes. They are both blaming the free market for getting us into this mess. However, the government has been engineering this crisis for years by keeping the interest rates low and propping up Freddie and Fannie so people with low income can buy houses. Not only did the government cause the crisis through their policies, but now they are going to exacerbate it with even more broken legislation.

I didn't make irresponsible investments or take out a loan I couldn't afford, but I'm being punished for other people's mistakes. This is an outrage. I want my bailout.

Today Steve Jobs introduced a series of new iPods, including a redesigned nano that comes in seven bright colors (and two drab ones).

It seems Cupertino may have looked south to Los Angeles for color and branding inspiration. The iPod's "nano-chromatic" branding is eerily similar to the thermo-chromatic sign that has recently appeared at my local American Apparel. The iPod nano colors also map well to those of American Apparel's t-shirts.

If I had the money, I'd get a shirt and matching nano for every day of the week. It is the ultimate accessory.

I should note, this wouldn't be the first time Apple has taken "inspiration" from others for use in advertisements and branding.

Wonderful Photoshopping provided by Jeff Hitchcock.

I've been excited about Microsoft's Photosynth since I first saw their demo. Photosynth recreates 3D scenes based only on photos of the location. It was created with technology from the University of Washington and Seadragon Software (who Microsoft acquired). On Friday, Microsoft released the first version to support user-created Synths.

I was excited to try the app, but it was having trouble handling the launch day load. I figured the load would die down that night, but it persisted all the way through Saturday night as–apparently–the employees were desperately searching for more hardware. My first successful Synth didn't happen until Sunday afternoon. On Friday, I tried creating a large Synth, but it kept having trouble connecting to the servers and uploading the images. I tried a simpler one on Saturday, and it made it all the way to the end, but it failed on the last step (publishing). Sunday afternoon I made an attempt with a different dataset and it worked. I went back and successfully redid the first two, which was their third attempt each.

I wonder, though, what was so tough about keeping the website up? From what I can tell, most of the heavy lifting is done by the users or Limelight. The client software appears to do all the feature detection and scene creation. Limelight handles the distribution and perhaps (I'm not sure about this last part) ingestion as well. From what I can tell, their servers just handle the management of synths.

I'm a little irked with this whole "Software + Services" thing. The way it was done, I got none of the advantages of "the cloud", but all of the disadvantages. Even though I was uploading the data to Microsoft, my machine had to do all of the transformations and calculations itself. The whole point of shipping my photos off to a datacenter is to let them handle the heavy lifting. And this brings me to the disadvantage: I had to max out my upload for 24 hours to create my big synth. The time would have been shorter had I not been required to upload the meta data and image pyramids as well. Of course, it would have been fastest if I could just run the software and viewer locally. That would only take about 2 hours.

The annoyance of having the servers down was magnified by the bad handling of failure cases. When something went wrong, you would see typical Microsoft error messages such as "A000000B" or "A0000009". Additionally, when one of my Synths failed during publishing, I tried to recreate it with the same title. However, it gave me an error message because of the duplicate title and closed the creation window. It is very frustrating to have to reenter the tags and license info and then wait for 500 image thumbnails to be regenerated before you can try again.

Now that it works, it is a lot of fun to play with, but this launch has left a bad taste in my mouth. Microsoft coupled what could have been great client software with unnecessary "+ Services". Photosynth doesn't take advantage of offloading computation to the cloud, so the "+ Services" is wasted and just ends up adding extra complication and—as the launch shows—unreliability.

Update 2008-08-24: I forgot to mention another opportunity lost because of Microsoft's implementation of Photosynth: group synths. One advantage of forcing users to upload their photos to a shared service is that different users' photos could be combined into one synth. I have some photos of the Forbidden City and Chichen Itza that—by themselves— aren't enough to create a synth, but that I'd love to donate to a community synth. However, this isn't possible with the current implementation, and the architecture of the service prevents it, since all computations are done client side.

After much trouble and delay, I finally made my first successful Photosynth. Here is a my Synth of Cloudy Pass:

About every year or so for the last few years I unsuccessfully attempt to set up an IPv6 connection for my home network. Today I tried again—triggered by Slashdot's article about IPv6 adoption—and this time I succeeded.

For a few years I've been using m0n0wall as a router on one of my old computers. It provides advanced features such as a stateful firewall, VPN endpoint, and QoS traffic shaper. I'll show how to configure a Hurricane Electric IPv6 tunnel with m0n0wall.

In my latest attempt to configure IPv6, I noticed that m0n0wall recently released a beta that supports tunneling IPv6 over IPv4. If I could set up the tunnel such that it works for all computers on the network and not just my desktop, that'd be great. If you are coming from the 1.2x series, make sure you upgrade to 1.3b7 before 1.3b13+.

For the IPv6 tunnel, I went with Hurricane Electric's tunnel broker. They offer a free /48 to anyone with an e-mail address who is willing and able to set up the tunnel. After registration, confirm your e-mail address and login. Create a tunnel by specifying your IP address and the closest endpoint. Once created, you'll be presented with the tunnel detail page. You only need to remember the "Server IPv4 address", "Client IPv6 address", and "Routed /64" fields.

Now return to your m0n0wall installation. Click Advanced under System and then check "Enable IPv6 support".

To setup the tunnel, click WAN under Interfaces. Halfway down the page you'll see the IPv6 configuration. Select "Tunnel" for IPv6 mode. The IPv6 address is the client address you saved from the tunnel detail page. Leave the drop down as /64. IPv6 gateway is greyed out, so continue down to IPv6 tunnel endpoint and fill in the server IPv4 address you saved from the tunnel detail page.

Now move up to the LAN configuration. Set the IPv6 mode to static. The server IPv6 address I was given ends in ::1 and the client address ends in ::2, so I used the same prefix and ended this address with ::3. Enter the "Routed /64" from the tunnel configuration as your IPv6 address. Check the box that says "Send IPv6 router advertisements". Click Save. It'll want you to restart the router, but that can wait for a minute.

Saving the LAN settings disables the DHCP server. If you need DHCP like I do, click on DHCP server under Services. Check "Enable DHCP server on LAN interface" and then Save. Finally, you need to add a firewall rule to allow traffic through. Go to IPv6 Rules under Firewall and click LAN. Add a new rule with Action set to "Pass", Interface as "LAN", and over any protocol. Save it and then delete any default rules that may have already been defined. Now you can restart the router using the Diagnostics menu. If everything worked, you should now have a functioning IPv6 tunnel.

Update 2008-08-26: A reader contacted me after having trouble with the instructions. It turns out I had forgotten to include the step about adding a firewall rule to let traffic out through IPv6. Also, he corrected me about the IP address to use on the LAN configuration page, it should be the routed /64. I've update the post to include the new information

I just got back from the Safeway in Seattle's University District where I bought one of my favorite foods: Amy's Refried Beans With Green Chiles. I went to the shortest checkout line. I didn't realize until I was already being checked out that I had had this man before.

He asked me if I knew how refried beans were made. "Uhh, no…". He explained, "They use lard. Do you think they used organic lard while making these beans?"

These are vegan refried beans, so no lard was used. I didn't really care, so I let it slip. However, he continued, "Do you think it is right that they charge more for organic food? Don't you feel like you are being taken advantage of? I mean, I don't think that is right."

Oh shit. I remember this guy. He's the asshole that tried to convince me last time that I was being swindled by the organic food industry.

I was tired of his shit and pissed off at his poor customer service. I replied flat out with the truth, "If anyone is taking advantage of me it is Safeway." I pointed to the beans, "This is cheaper at Whole Foods." I was infuriated and didn't talk to him for the rest of the transaction.

Seriously, I don't know how they let people like this into retail. You shouldn't comment on someone's purchases, but if you must, you don't insist they are getting scammed. The only reason I shop at this Safeway is because of the convenient location, otherwise their prices are high and their produce is disgusting.

Maybe it is because Amazon has drilled being customer centric into my brain, but I find this unacceptable.

In honor of Earth Day, I'm going to talk a little bit about why these new utility computing services (or "cloud computing") are good for the environment (and business too).

One of the tenets of cloud computing is that you use what you need and you pay for what you use. Amazon S3 and Simple DB and Google App Engine all charge based on storage, bandwidth, and CPU time. Additionally, these services run on shared infrastructure, so you don't have separate physical boxes serving your traffic. This allows Amazon and Google to run at high utilization knowing that—statistically—not all of their users will be hammering the service at once. For a detailed discussion of these statistics, read Power Provisioning for a Warehouse-sized Computer [PDF]. I guess you could say the same thing about Dreamhost (or any other shared server provider) since they cram many people on a box and run at high utilization, but they don't provide the same level of scalability as Amazon or Google (you are limited to a single box).

Why does utilization matter? If you can do the same amount of work on fewer computers by having higher utilization, you save the environmental cost of building those computers. Additionally, computers use lots of energy idling. Even though you increase single-computer power usage when consolidating, net power decreases as you take other systems offline. For a concrete example, imagine you have two computers that use 100W idling and 200W at full load. If each machine is one-third utilized, it takes 266W to perform your work (133W for each computer). However, consolidating this onto a single machine results in a total draw of 166W, or a 37% savings. Now… imagine running a datacenter at 80% or more load.

In the second paragraph I didn't mention Amazon's EC2. EC2 differs from S3 or GAE because you do pay for what you don't use. EC2 charges based on how long you have a computer under your control, not by the utilization of that computer. However, this is still a much more granular level of control than colocation, since you can scale your fleet up or down as needed to meet load. When you return a machine, someone else is free to grab it. The end result is high utilization, because people won't hold on to a machine they are idling, and will thus reduce their fleet size to compensate.

In my opinion this makes Hadoop the killer app for EC2. Users can spin up a cluster, run their job at full bore, and then return the computers back to "the cloud". Companies such as Powerset and the NY Times have used EC2 for this very reason. This is a triumphant example of the market's ability to reduce energy consumption and resource usage (in the form of unneeded computers), because it directly translates into monetary savings—no heavy handed government mandates required.

Let me use an example of my own usage of S3 for backups. All the numbers will be based on those in the power provisioning paper mentioned above. I have 20 GB of data backed up in S3 and I want it to be available for instant access. This necessitates a constantly spinning hard disk. If I were to do it myself, I could just add another hard disk in my computer and stick a copy on that. Power usage of hard disk: 12W. This is cheap and easy, but doesn't give me offsite backup in case of a disaster. For the storage server—using the numbers from the paper—we have 200W base power plus 12W for each disk. Giving the server eight disks results in a total draw of 296W and 8192 GB of disk space. Let's double that to take into account battery backup and air conditioning in the datacenter, so 592W. Since the server is shared with others (we are aiming for high utilization, remember?), I'll have to figure out my usage as a proportion of the total. Before I do that, I should take durability into account. Distributed file systems store multiple copies of each file because hard disks and computers constantly fail in large systems. The industry standard seems to be three copies—judging by GFS and HDFS. Three copies results in 60 GB dedicated to my backups. Adding this all up results in the follow equation: 592W * (60 GB / 8192 GB) = 4.33W. As you can see, even with all the additional overhead and infrastructure, using S3 saves two-thirds the power versus having an extra spinning hard disk in my own machine. Plus, my apartment is no longer a single point of failure.

Hopefully this article has provided you a new look at utility computing. Next time the media tries to make a fuss over the growing power usage of datacenters, think back to this and realize they might actually be saving energy. Happy Earth Day, Andrew.

Last week, rumors started to surface that Google would be releasing BigTable as a web service for developers. While open source clones are being created, so far only Googlers have had access to the real BigTable. This announcement would have also been the biggest competition to Amazon's Web Services to date. The actual launch was App Engine—hosted middleware on Google's platform. App Engine runs on BigTable, but doesn't expose all the nitty gritty details of BigTable. This makes it simpler but less flexible, and despite all the comparisons to AWS online, the two offerings are quite different.

I registered the night App Engine was released and soon got my invite. I then came up with the crazy idea to offer BigTable as a web service using App Engine. It would be an infinitely scalable database running in Google's datacenters. I spent my weekend learning Python and hacking together an implementation. Now I'm happy to present the BigTable Web Service. It models the API of Hbase—a BigTable clone. Now you can have simulated BigTable running atop App Engine, which itself provides an abstraction on top of the real BigTable.

The site describes the API of BigTable and gives examples of how to call it. I've also included a Python client for writing software against it. You must register for an account and create tables using the site, but everything after that is done through pseudo-RESTful service calls. I'm allowing free, unlimited access of the service… up to the limits imposed by Google.

Last Sunday was .83's third annual Fucking Hills Race on Bainbridge Island. I met my hobo friends under the Alaskan Way Viaduct at the ungodly hour of 8:45. I registered with Derrickito and got my paramilitary-styled pirate flag made with scraps from OR's dumpster. After registering, I ate the breakfast of champions: half a donut and a swig of vodka. We swarmed the ferry, paid our dues, and lined up to board. We were the first on, but unfortunately the cars disembarked before us.

On the ferry I had a cup of coffee and sausage, egg, and cheese muffin. My heart is still thanking me. I almost got stuck in a swarm of squids while finding .83's seats. Fortunately the spandex allowed me to slip right past. Apparently the Chilly Hilly was scheduled for the same day? Odd. I also ate some stale Clif Shot Blocks for dessert.

We gathered in a pullout after getting off the ferry. Derrick said we couldn't start until we finished a bottle of Evan Williams. Ben the Angry Hippy–owner of said bottle–complained and lifted the requirement. And then we were off! I wore the same outfit as last year: a pair of blue jeans and my Bob Marley jacket. Fortunately the weather was nicer, so my pants didn't get soaking wet during the journey.

Knowing the approximate route made the ride go faster because I didn't always think the end was right around the corner. Even though I was out of shape, I was still able to ride at a fairly quick pace. I don't know where all the energy came from.

The advantage of being a second year rider is that I knew where the finish line was located. I placed about 25th. Fortunately, they had plenty of chili this year. I had two bowls of that and then went for beer. While waiting for people to roll in, I ended up drinking four Mongoose IPAs. My friend Kevin eventually rolled in sporting his Amazon.com jersey. I selected a cool Cadence t-shirt for my prize, which I had been eyeing for a while.

While waiting for the ferry, a few of us went to a coffee shop and I had a piece of coffee cake. I had a corn dog on the ferry back, in true .83 fashion. I eventually made it home–sore–and took a shower.

This year's FHR was great. They had plenty of chili, I got a cool prize because I didn't stop to drink a beer on the way, and the weather was amazing. Thanks to Derrickito for organizing, our sponsors for the great prizes, btah and Kalen for manning the finish line, and anyone else I'm forgetting. See you next year!

More than a decade after first watching the movie, I finally made it to a live midnight performance of The Rocky Horror Picture Show. The show was put on by The Vicarious Theater Company at the Admiral theatre in West Seattle. The price was right at only $5. My friend Martin organized the outing and brought the requisite rice, newspapers, and (Amazon.com branded) playing cards.

The devirginizing was mild and painless. They were overwhelmed by first timers and so–unfortunately–didn't have enough supplies for all of us to partake in the food-based rituals.

The average age of the audience skewed younger than I expected–I was probably older than more than half of the audience. Still, many of the youngsters and first timers had impressive costumes. The audience was very enthusiastic with feedback and everyone partook in the Time Warp. Some of the quips were quite clever, but the part about Brad being an asshole got old. I'd definitely go again, I just wish the location wasn't so inconvenient.

Earlier today Google posted its opinions about Microsoft's bid for Yahoo on its blog. I must say, I'm disappointed. In the post, Google not-so-subtly asks for government intervention in the takeover. Google claims they want this in the name of openness and innovation, but what freedom? Specifically, the freedom from government intervention in private business. The software industry is healthy and prosperous with minimum government contact. Allowing government interference would result in corruption, cronyism, and inefficiency, just like every industry it touches.

Regarding the purchase, neither are majority players in search or advertising–likely the main reasons for the acquisition–even combined. Contrast this to Google and DoubleClick at the time of that purchase. Demanding the right to buy DoubleClick and then asking the government to restrict Microsoft's purchase of Yahoo smacks of hypocrisy. Last year Microsoft tried to prevent the DoubleClick acquisition, but I expect this sort of hypocrisy from Microsoft. Now it seems Google is trying to pull the same thing. You have let me down Google, I thought better of you.

Anyway, I don't see what they are worried about. Google should be happy about the acquisition, the merger will more than likely result in a clusterfuck that drives more users away from Yahoo properties. I know that if Microsoft starts fucking with the Yahoo services I use, I'll jump ship.

"Holy crap" was the first thing that came to mind this morning when I read the news about Microsoft's bid for Yahoo. Then I immediately thought about Flickr, del.icio.us, and Hadoop and what this would mean for them. Flickr already went through a painful migration to Yahoo's login service, would that have to be repeated when they move to Passport? Fortunately, del.icio.us has come out mostly unscathed, so I'm not too worried about it. Hadoop is open source with a large number of contributors, but losing the Yahoo employees would definitely slow progress. Pig would also take a big hit.

To be honest, I don't see how this could possibly work. Integrating two companies of this size would be a nightmare. Have we learned nothing from eBay/PayPal or AOL/TW? From my outsider perspective, it seems the two companies have fundamental cultural differences. Yahoo is a fairly large contributor to open source and seems to support web standards. Microsoft is the anti-thesis of web standards. Does Microsoft expect Yahoo to move away from their current operating systems (which I assume are some sort of unix) to Windows? That worked oh so well for Hotmail. I believe any value Microsoft sees in Yahoo would probably be destroyed as part of the transition (even if it is just the customer base).

Finally, Microsoft is still having branding issues with their internet properties (MSN Hotmail redirects me to mail.live.com… WTF?) and now they want to throw Yahoo into the mix? Yikes.

On Tuesday, Google celebrated the opening of their new Fremont office with an open house. When I received my invitation, I reflected on the open house they had for their Kirkland office, over three years ago. It is impressive to think how far they have come. The last open house was right before the release of Google Maps, which has made impressive strides itself.

The day of the open house, I saw a Metro bus modeled after their master plan whiteboard that was advertising the new development center.

The open house was quite the party. It was much larger and hipper than Kirkland–matching the neighborhood, I guess. The office was completely illuminated with colored lights and there was loud dance music playing. The drink glasses contained a glowing liquid, like a giant glow stick that flowed when used. I subsisted on free beer and tapenade. They had a demo area with Googlers demoing their projects. I already knew about all the products, but I wished the Reader team was there so I could complain about Reader's 30 day memory. I thought it was funny the Webmaster Central group was using Matt Cutts' blog for their demo.

The party was a veritable who's who of the last three years of my life. About a half dozen of my former professors were there, along with nearly all of my CS friends. I met with bike hobos and coworkers (intersecting groups, mind you). I ran into Amanda Camp–who ran last summer's scalability conference–while getting my massage. Jeff Dean and Alan Eustace were there. Fortunately, Robert Scoble was absent this time.

In my opinion, the party was great. It was the closest I've come to a true VC-funded San Francisco dot-com party. Two days later I met some Yahoo! recruiters and not-so-inconspicuously hinted that they should have an open house for their new Bellevue offices. I made sure to slip them my card so I'd get an invite. We'll see if anything comes up.

Tiger introduced a really neat feature that gave every Cocoa app an instant, inline dictionary. Holding cmd-ctrl-D while mousing over a word causes a pop up to appear with the definition. If you keep holding down the key combination and move the mouse, any word you hover over is displayed. If the word wasn't in the dictionary, it would say so. However, often times this feature would just downright not work. I'd hold down the key combination while moving my mouse around and nothing would appear. At least it was easy to tell when it was broken. Leopard's dictionary lookup is faster and more reliable, but unfortunately takes a step backwards and no longer tells you when it doesn't recognize a world. Instead, it just doesn't display the pop up. In these situations–having been trained for the last few years on the shoddier Tiger implementation–I'm siting there with my mouse hovering over a word like a dumbshit waiting for it to load. Grr.

Note: Titling this entry as "Pet Peeve #1" doesn't necessarily mean this is my biggest peeve with Leopard, nor does it imply future entries in the Pet Peeve series.

I've long been a fan of Apple's advertising and copy (even though their best advertisements are often copies). A few weeks ago, I was delighted by the tagline for the new iMac.

Amen. If I can't play my marimba ringtone on your exposed ribcage, you aren't thin enough. Of course, some bulimatards had to come along and force them to remove it. Oh well.

There has been much debate about the lack of copy and paste on the iPhone. Some people think it is a desperately lacking feature, while others believe the phone creates a new standard in UI design in which copy/paste is unnecessary. Either way, here is my idea of how copy and paste could be implemented:

Copy: Position the text cursor as you normally would, by pressing in a text field until the loupe appears and then moving your finger to the correct location. Once you have navigated to the correct location, release your finger and then immediately tap and hold. After a second, the loupe will reappear, but this time moving your finger highlights the text between the initial location and your new one. To complete the selection, remove you finger. A dialog will pop up prompting you to Copy, Search, or E-mail the selection.

Paste: To paste, navigate to the correct location using the loupe as normal. Now, release and briefly tap the screen (without holding). The software could paste the text immediately, or prompt you to either paste or cancel. The correct behavior would have to be determined by user tests, to see if spurious taps result in unwanted pastes or if the dialog box is unnecessary and cumbersome.

There is still the problem of selecting text in an area where holding your finger drags the view. Perhaps if you immediately move your finger after pressing the screen, then the view moves, but if you don't move after a short delay, the loupe appears. However, I often find myself touching the screen for a few seconds before I pan, so this would likely need some refinement.

Fess up, who was running third party apps on their iPhone? A large part of ATT's EDGE network was down yesterday, so someone must have been.

Blogged from my iPhone.

With the recent news from WWDC and other developments, I've gone from wanting to be the first on the block with iPhone to sitting on the fence, taking a wait-and-see approach. Here are a few things that irk me:

• No MMS. I don't use this much now, but if I got iPhone, I'd get a plan that allows lots of data and texts, so I'd start using this much more frequently.
• No iChat. If iPhone had "just… the internet" then I should be able to use iChat to chat with my AIM and Jabber friends. The crappy free phone I got five years ago supported AIM. WFT?
• No 3rd party apps. This is one that really bothers me. Apple gave developers and users a big "fuck you" at WWDC by telling them to build web apps that could be used on iPhone. I'm sorry Apple, but that's bullshit. Even with the shitty stream I was watching, I could see the UI differences between the native address book app and the web one. It's insulting to claim otherwise. Also, web apps won't support Multi-Touch. Finally, there are already some apps I want that Apple doesn't offer and would suck in web form, such as Terminal and Remote Desktop.
• No 3G. I might as well just wait until they release a 3G version (Wifi isn't as ubiquitous as Steve claims).

I'm a big fan of foods that can be eaten just using chips (such as beans, guacamole, or salsa). Here I'll present my guacamole recipe. I intend to publish more recipes in the future, so keep an eye out.

• 1 avocado
• 2 slices red onion, finely chopped
• ~1 teaspoon lemon juice
• 1-2 serrano chiles, chopped
• 1 vine ripened tomato, seeds removed and chopped
• ~1 cup cilantro, chopped
• Salt and pepper to taste

1. Cut the avocado in half, put your knife into the seed, and twist to remove. Scoop the innards into a bowl. Mix in the lemon juice and combine.
2. Add the rest of the ingredients and stir until it is roughly homogeneous.
3. Cover the bowl and refrigerate for an hour.

All I can say is, "About fracking time!" The existence of DRM on iTunes tracks has limited the number of songs I purchase, so this will result in more sales (and those sales will be for more money). Here's how it works: the current tracks all stay the same, but there will be a new offering where you can purchase a 256 kbps version with no DRM for $1.29. Albums are the same price, but use the higher bitrate and lack DRM.

Since DRM is usually cracked, I wouldn't pay more for a DRM-free copy. However, the price increase also comes with a doubling of the bit rate, which is worth a little bit more money. I think it will be difficult for Apple to know why people are buying the higher priced songs, because they are changing two variables and there is no control. But, I guess the reason won't matter if they are making more money.

This is not the only new feature. A few days ago, Apple introduced "Complete My Album", which allows you to buy a full album and get discounted the price of the singles from the album you already purchased. This is a smart feature to have, but I am disappointed that it only works on purchases that are less than 180 days old. Sometimes it takes me a while to discover I like more than one or two songs on an album, and I am less likely to upgrade to the album if I can't get the discount, so they are still hurting themselves here.

Something they don't have, but I would like to see, is a feature that allows you to buy the music video for a song you already purchased for just $1. Right now, you can buy the music video and song for $2, but if you've already purchased the song, you are out of luck.

What's wrong with this picture?

• It's St. Patrick's Day
• It was my 21st birthday today
• I'm sober

Wow, what a week. I"ve been coding (or reading about coding) for about six days straight. It is hard to believe it was that long, since it is all a blur of for loops and parenthesis.

I spent most of my waking hours last weekend in the underground labs at the Allen Center working on CAMPS, my 403 group project. We developed a campus mapping system that is all AJAXy and feels like Google Maps. I was responsible for the controls on the left side of the page, and I was also in charge of the client/server protocol. As part of that, I wrote this really neat JavaScript that does autocomplete when you type text. I am proud of it not because of its technical merits necessarily, but because it behaves very much like a desktop drop down and does what you'd expect. Anyway, I'd link to the hosted demo we have, but the server is going to wiped in a few days, at the end of the quarter. However, at least a few of us are going to continue to work on this in our spare time and hopefully we can get it hosted somewhere permanently.

So, not only was I busy all weekend, but my sleep, that precious me time, was cut short by those jokers in Congress. The DST change happened the weekend before finals week, which is just great. I'd like to do some data analysis to see how the grades suffered because of that (I would not be surprised to see a dip in normalized grades (versus past years) on Monday exams that is slowly erased as the week goes on). I'm mainly upset because I had an 08:30 final on Monday.

As soon as that final was done and we presented our 403 projects, I immediately started reading about Core Data. I had a Computer Vision assignment due this week where we had to write a content based image retrieval system (or CBIR). The program had to be able to save and load data and we get extra points for having a GUI. I had been wanting to try out Cocoa and Objective-C again, so this sounded like a perfect opportunity. I knew about Core Data, but none of the details, so I quickly read as much as I could about it to see if it was feasible. The last time I tried delving into Cocoa (it has been about two years now, so longer than I've officially been a CS major), I was overwhelmed by all the unfamiliar terms and large number of classes. Now that I've learned more and have a better understanding of the concepts (and no longer call methods functions), it wasn't all that challenging. My GUI was pretty basic, just listing essential information and allowing you to perform queries, but it was better than a command line interface. I had lots of ideas about how to make the UI better, but I didn't have time to play around with that.

Okay, enough about CS and CBIR. I wouldn't mind seeing some beer myself, but alas I missed the .83 ride today. Oh well, I just have to wait about 22 hours…

San Francisco, CA — The innertubes were nearly full today as millions of geeks self flagellated while watching Steve Jobs announce the iPhone.

Servers were hammered with the release. News of the iPhone quickly became the most dugg story of the year and currently has over 17,000 diggs (with almost 1400 comments). The Flickr photo linked to in the article has over 101k views. Nine of the top ten stories on Digg today were related to Apple's announcement.

Additionally, Apple's market cap rose approximately 6 billion dollars as the stock skyrocketed over 8%. Owners of Research in Motion, maker of the Blackberry, weren't so lucky, as RIMM shed over 2 billion dollars of market cap and fell nearly 8%.

It was very difficult for those of us trying to avoid spoilers (I wanted to watch the Keynote video without knowing any of the new products). Nearly every website I follow carried the news (even non-tech sites). A Burger Place, where I had lunch, was tuned to a station showing Apple's new gadgets, but my TV-B-Gone took care of that. It was especially difficult to avoid spoilers, as Apple's own website added a new iPhone tab and the website for the stream mentioned the iPhone. Bastards.

It is funny, I had an iPhone premonition on Monday. I said to my brother over Jabber, "It'd be cool if Apple released a PADD: a big screen iPod/iPhone device/PDA." I thought the PADD would be neat, but I didn't actually expect it to get released.

I was surprised that MWSF07 didn't include any info about the new version of OS X or any news about updated iApps (iLife and iWork). However, fortunately, Steve Jobs wasn't retiring as some folks were predicting (or perhaps worrying).

Being a Pastafarian, I'm not huge into Christmas, but I do enjoy seeing the extended family and wearing funny sweaters. In honor of pagan rituals co-opted by Christians and eventually turned into an excuse for excess consumerism, I decided to whip out the ole PVC and attempt to play a few festive jingles. I couldn't convince my brother to play with me, so there is no accompaniment. Additionally, I was rushed while recording both of these pieces, so I didn't have as much time to practice as I would have liked, which inevitably leads to me dragging in parts as I try to remember which notes to play (muscle memory hadn't completely kicked in at the time these were both recorded).

I chose the easier song first, in order to have it posted on YouTube before the family Christmas dinner. A few days after Christmas, I decided to tackle one of my favorite Christmas songs—Carol of the Bells. I was never able to get all the way through the song without messing up (I was short on time and this song required wild gesticulations), so this audio file was created by snipping the different parts of the song together (which is why there might be funny delays in parts). I think this song sounds particularly cool on the PVC.

Another pet peeve of mine is when people refer to their cell phone reception in bars. What the fuck is that? Is bar an IEEE standard? There are three possibilities that I see.

First, bars have a standard meaning in decibels. In this instance, saying your bars is a perfectly valid method of stating reception. I highly doubt this is the case. Different phones have different numbers of bars, so this is unlikely.

Second, the max receptions on all phones are equal and represented by full bars. I don't believe this is the case. However, if this is how it works, then comparing bars is still completely useless unless you state your reception as a fraction, including the total number of bars. This would let others know that 3/5 bars means 60% reception.

The third and most likely explanation is that bars mean jack shit and are only valid for people using the same model of phone (and same software/firmware revision).

However, there is one valid definition of bar as a unit–which Chris pointed out during our discussions today–and that is a unit of pressure. I don't think cell phones measure reception in terms of pressure, but I can play along. From now on I'll say, "I'm getting approximately two atmospheres of reception." Or, even better "My reception rocks, I'm getting over than 2250 mm Hg."

Well, that is the end of my rant. I should leave now because the coffee shop I'm sitting in has horrible reception, I'm not even getting 100 kilopascals of signal!

I touched a Zune for the first time yesterday. The first thing I tried to do was use the "scrollwheel", before realizing that they were just four buttons. That's a design flaw: don't mislead users about using the interface. If they are just buttons, make sure I notice that at a glance.

Unfortunately, there were no other Zunes around, so I couldn't try the infamous squirting feature. I imagine this will often be the case in real life usage as well.

I have mixed feelings about the Zune coverage and reviews. One on hand, as an Apple shareholder, I'm glad to see that the much hyped Zune is no threat to iPod. However, I also feel a little sorry for the Zune team at Microsoft. There are a lot of people who I'm sure put a lot of hard work into making this a reality, only to have it ridiculed in nearly every review. Although, what did they expect by making it incompatible with PlaysForSure and requiring users to buy Zune points?

This is just another example where Apple Gets It™ and Microsoft is too tied to their special interests.

It's been a while since I've posted, so I thought I'd give you an update. I wanted to have one last adventure before school started (which happened last week), so on September 22nd, I rode my bike to Vancouver. I got to visit my brother and finally get a tour of UBC. I enjoyed visiting Vancouver, but was disappointed by their liquor store's selection of beer. Fortunately the on-campus pub had a good selection.

In other news, I've gotten a part time job helping with content over at Programmable Web. The website is based in Seattle, but I get to work from home. I think it is a good match because I am very familiar with the space.

In computing news, I've switch from using Safari to Firefox. This is big news because I've used Safari as my main browser on OS X since day one. Well, more precisely, I've used Safari since hour one. I switched because I don't have much free hard drive space on my computer anymore (I keep acquiring music), and Safari was wont to consume hundreds of megs of space. I switched about the time FF 2.0b2 came out, which was part of the reason for the switch. Some of the new features are really nice (like rearranging tabs). Each tab now has its own close button, which is how Safari has always done it. I do miss the integration with the rest of the system and being able to use some of my favorite commands (ctrl-cmd-d), though.

By switching to Firefox, I no longer had frequent access to my RSS feeds. Fortunately, not too long ago, Google released a new version of Reader. I dropped all my original feeds (which were created just to play around with Reader) and put in my current reading list. Hopefully it'll be good enough that I won't have to switch to something else for a little while (I've already noticed that it is slow at picking up new stories).

Lastly, I've started working on a presentation about carfree cities. I was inspired by the success of An Inconvenient Truth, so I started creating a Keynote slideshow in the style of Steve Jobs. I'm imitating Steve Jobs because he is one of the best speakers I've ever seen. I've discovered it is way more difficult designing simple, concise slides than I anticipated. As I near completion of the slideshow, I'll post some of the slides and perhaps other information.

Facebook recently launched a "feed" feature which lists every public action you or your friends make, including changing your profile or relationship status and RSVPing to a party. I don't like the feature because instead of just having the information public (in which case someone would have to be dedicated to noticing changes in my profile, which is okay by me), it is now being advertised to every single friend. It is also really unpopular among my friends (which I can tell because of the new feed feature). In fact, there is now an anti-feed group in Facebook with over half a million users. Time has a really succinct article reiterating my complaints about Facebook's new feature. If I wasn't getting free music because of Apple's partnership with Facebook, I'd probably have deleted my account.

I've had many great bike adventures this week. Every Tuesday and Thursday, I go on a bike ride with Point 83—a bike group with a drinking problem. This Tuesday's ride was a lot of fun and a prime example of what happens on these rides. After meeting at Red Square at 19:00, the group decided to head down to Georgetown. To make the ride longer and more interesting, we rode through Fremont, up Nickerson St, and across a bridge to the other side of the Interbay train yard. We followed the path through the train yard and Myrtle Edwards Park, where a number of hippies were still stoned and lost from Hempfest that weekend. We biked along the waterfront for a few miles before turning left towards Safeco Field, where we could hear the Mariners playing the Yankees inside. After climbing the elevated roadway, we turned right onto 4th Ave and continued deeper into SODO. Folks were hungry, so we decided to go to Stellar Pizza and Ale. Along the way, I almost got in an accident with Daniel Featherhead—who recently got back from a bike ride to Nova Scotia. I was riding behind him and close to the sidewalk when he turned in front of me to get on the sidewalk. I'm not sure how either of us managed to stay up, because my front wheel went in between the spokes of his back wheel, but we did.

At Stellar, some people ordered beer (they sell a pitcher of PBR for $4 on Tuesdays) and I decided to split a pizza with a few friends. It was my choice, so I ordered the Corson Classic: "Sliced Yukon spuds, gorgonzola cheese & sweet white onions". The pizza was slow to arrive, but I passed the time drinking Arrogant Bastard Ale and PBR. The pizza was great, but a little expensive. Once the drinks were gone, we decided to hit up all the important South Seattle dumpsters. First on the list was the infamous chocolate dumpster. The employees just happened to bring out some espresso chocolate as we arrived, so it hadn't even hit the dumpster yet. Everyone had their fix and Abby kept the extras (as it turns out, I ran into that chocolate again at a party on Saturday). We continued zig-zagged through Georgetown, hitting up the juice, bread, and bagel dumpsters. I got a large bottle of orange juice at the Naked dumpster, which I drank throughout the week and volunteered for screwdrivers at Mike's house. It retails for $6. We finished by riding through downtown and then headed towards our respective neighborhoods.

All in all, it was a great ride. We put in a fair number of miles and I ate good food, got free stuff, and hung out with cool folks. This was one of many bike events this week, but I thought it was notable and gives a good introduction to .83. Also, this post is foreshadowing something I'm working on, but I'll talk more about that later.

Today I had the good fortune to experience flying out of London's Heathrow airport on a US-bound flight. Sarcasm aside, I was lucky that my flight wasn't canceled.

My dad flew out last Thursday when the plot was revealed and the airport chaos began. The rest of my family assumed he was spending the night in London. A day or two later we checked our e-mail and discovered he managed to get out with only a four hour delay.

On the BBC the morning after the plot was revealed, I was aghast to learn passengers were allowed only the vital essentials (passport, napkins feminine hygiene products (curiously, I don't think pocket knives and space blankets were part of the airlines' ideas on 10 essentials)) in clear plastic bags. If this is the future of air travel, I want no part of it. Alas, I still had to get myself and my valuables home. I loathed the thought of sending my laptop in my checked luggage, and I don't have any faith in FedEx.

This morning I went down to breakfast in our B&B and turned on the telly. The BBC was talking about the relaxed restrictions. I did a little dance, but was disappointed when I learned that the airports weren't going to adopt the new practices until tomorrow morning. Blast. Nevertheless, I still had to fly home and I knew there was nothing I could do.

After saying goodbye to my brother Jeff at Paddington station (he is continuing on to see more of Europe by himself), my mom and I caught a train to Heathrow. I was curious about the whole ordeal, but still not looking forward to the chance of delays and not having any entertainment on the flight. The conductor of the train told us to head toward arrivals instead of departures. The directions stopped once we backtracked to baggage claim. We asked one of the many uniformed employees and were directed out to the parking lot and up the elevator. There were large numbers of people sitting around at the top floor of the parking garage. There were a few large tents set up employees had food, water, and blankets on hand. I felt like a refugee. After asking, we were told we should stay in the area until two hours before the flight when they would call our flight. We waited until the time came near, and then moved down to Area B, where our checking was located. Once you flight is called, you were allowed to cross the pedestrian bridges and enter the terminal. Not seeing any signs, we asked another employee where we should go. They directed us to a line in Area B, but someone soon came buy asking for our destination. It turns out we were in the wrong line and would have to head to Area A.

Walking the short distance to Area A, we passed a number of, presumably, BBC correspondents. One man was discussing luggage size, and demonstrating how standard carry-on luggage wouldn't fit the new restrictions (he was using a hastily constructed wood box painted white and emblazoned with "BBC"). A woman was interviewing a passenger and later doing a monologue in while walking past the line.

We waiting in that line for over an hour. About halfway through, employees starting asking people destined for Washington, DC (and later Vancouver and Montreal) to come to the front of the line for expedited processing. Eventually they asked for everyone destined for Seattle, and the first two and a half switchbacks raised their hands. At this point the plane was supposed to depart in 55 minutes. It was ten more minutes before we reached the front of the line, and the majority of Seattle bound passengers were still behind us. Note, the bottle neck here was completely the fault of British Airways. The actual check-in procedure was quite quick, despite the fact that I reached into my bag for a few items while at the counter. Only one third or one fourth of the stations had staff, so the process could have easily been sped up.

Upon entering the terminal, I saw passengers carrying non-essential items in their plastic bags (including Pringles). After inquiring, I learned that magazines and even laptops were allowed. I was still skeptical, so I delayed removing my laptop until the last minute before checking the bags. I triple checked with the check-in agent before removing my laptop and headphones.

After checking our luggage, we made our way to security and, presumably, more queues. My mom commented that she felt naked without her carry-ons and I agreed. At the entrance to the security area, where they checked our tickets and passports, they had one final garbage collection (a godsend in a country with a conspicuous lack of rubbish bins). The security line was quick, and would have been even shorter had we not been ambushed by a government surveyor asking about our traveling experience in England (in an attempt to gauge the economic benefit of tourism in the UK). At the x-ray machine, they had me remove my laptop from a clear plastic bag (filled with just two magazines and my headphones) and put it in a clear plastic bag by itself. I also took off my relatively new belt because I remembered it set off the alarms in the States. Upon navigating the arch, I was frisked by a male attendant. I had caught a glimpse of my mom with the female attendant, but had assumed she was being wanded. The boy was thorough, I'll give him that. He went quick, and even felt around the waist of my jeans. I was afraid they might drop, since I was missing my belt. The security check was amazingly quick—contrary to everything I had been hearing—and it took more time to just get dressed again.

My mother looked for lip balm for sale in the inner sanctum, but all shops were sold out. We eventually made it to our gate and got in line to board the plane (which was already in progress). Walking down the walkway, we passed a final inspection point which seemed to be only interested in Muslims. People boarded relatively quickly, an advantage of not having any carry-on luggage. We taxied for a little while before stopping… for an hour and a half. Finally US immigrations gave us the A-OK (which garnered much applause in the plane) and we took off. I'm currently above Canada's Ungava Bay.

Surprisingly, the delays were not caused by security, which I presumed and the media reported, but by the lack of employees checking-in luggage and the extremely long delay getting approval from US immigration. The delays were not inherent in the level of security checks, but by the apparent incompetence in planning and execution by the airlines and mysterious delay from the US (perhaps based on increased background checks on passengers?). Assuming those two problems are resolved, I see no reason why service can't return to normal. I don't know what the situation is like at the domestic terminal, but if it is anything like that at the international terminal, then BA need only hire more people to work the baggage checkins to increase throughput. The domestic flights aren't slowed by US immigration, which was the main cause of delay for our flight.

About six months ago I felt my website design was starting to get a little old and that I should probably get rid of the puke colors I lifted from ThinkGeek over four and a half years ago. Also, I wanted to make my page more accessible and truly separate the content from the design. The design should be much more flexible now, and the content comes before the navigation, which helps those people using screen readers.

I was originally playing around with blues, but the page always looked too cold. I found this online color wheel which helped me pick a color scheme for my website. I found some colors that complimented each other, and then I got their pastel versions to make it more pleasant. I spent a lot of time rearranging colors to see which worked well in which area. I settled on the tan for the title to make the page warm. I had trouble working in the purple, so I saved that for visited links.

The text size and color variety really adds to the page and makes it more interesting. Also, the website has much better SEO features (like using the title of posts as the link, instead of "Permanent Link"). The page feels a lot more modern and I'm glad I'm finally able to roll it out. The old version is still available if you want to compare.

Update 14:59
I forgot to mention, the script now properly throws a 404 if you request a page that doesn't exist, instead of just showing the design with no content.

Today Jeff Dean gave a talk at the University of Washington about BigTable - their system for storing large amounts of data in a semi-structured manner. I was unable to find much info about BigTable on the internet, so I decided to take notes and write about it myself.

First an overview. BigTable has been in development since early 2004 and has been in active use for about eight months (about February 2005). There are currently around 100 cells for services such as Print, Search History, Maps, and Orkut. Following Google's philosophy, BigTable was an in-house development designed to run on commodity hardware. BigTable allows Google to have a very small incremental cost for new services and expanded computing power (they don't have to buy a license for every machine, for example). BigTable is built atop their other services, specifically GFS, Scheduler, Lock Service, and MapReduce.

Each table is a multi-dimensional sparse map. The table consists of rows and columns, and each cell has a time version. There can be multiple copies of each cell with different times, so they can keep track of changes over time. In his examples, the rows were URLs and the columns had names such as "contents:" (which would store the file data) or "language:" (which would contain a string such as "EN").

In order to make each manage the huge tables, the tables are split at row boundaries and saved as tablets. Tablets are each around 100-200 MB and each machine stores about 100 of them (they are stored in GFS). This setup allows fine grain load balancing (if one tablet is receiving lots of queries, it can shed other tablets or move the busy tablet to another machine) and fast rebuilding (when a machine goes down, other machines take one tablet from the downed machine, so 100 machines get new tablet, but the load on each machine to pick up the new tablet is fairly small).

Tablets are stored on systems as immutable SSTables and a tail of logs (one log per machine). When system memory is filled, it compacts some tablets. He went kind of fast through this, so I didn't have time to write everything down, but here is the overview: There are minor and major compactions. Minor compactions involve only a few tablets, while major ones involve the whole system. Major compactions can reclaim hard disk space. The location of the tablets are actually stored in special BigTable cells. The lookup is a three-level system. The clients get a pointer to the META0 tablet (there is only one). This tablet is heavily used, and so one machine usually ends up shedding all its other tablets to support the load. The META0 tablet keeps track of all the META1 tablets. These tables contain the location of the actual tablet being looked up. There is no big bottleneck in the system, because they make heavy use of pre-fetching and caching.

Back to columns. Columns are in the form of "family:optional_qualifier". In his example, the row "www.cnn.com" might have the columns "contents:" with the HTML of the page, "anchor:cnn.com/news" with the anchor text of that link ("CNN Homepage"), and "anchor:stanford.edu/" with that anchor text ("CNN"). Columns have type information. Columns families can have attributes/rules that apply to their cells, such as "keep n time entries" or "keep entries less than n days old". When tablets are rebuilt, these rules are applied to get rid of any expired entries. Because of the design of the system, columns are easy to create (and are created implicitly), while column families are heavy to create (since you specify things like type and attributes). In order to optimize access, column families can be split into locality groups. Locality groups cause the columns to be split into different SSTables (or tablets?). This increases performance because small, frequently accessed columns can be stored in a different spot than the large, infrequent columns.

All the tablets on one machine share a log; otherwise, one million tablets in a cluster would result in way too many files opened for writing (there seems to be a discrepancy here, he said 100 tablets per machine and 1000 machines, but that doesn't equal one million tablets). New log chunks are created every so often (like 64 MB, which would correspond with the size of GFS chunks). When a machine goes down, the master redistributes its log chunks to other machines to process (and these machines store the processed results locally). The machines that pick up the tablets then query the master for the location of the processed results (to update their recently acquired tablet) and then go directly to the machine for their data.

There is a lot of redundant data in their system (especially through time), so they make heavy use of compression. He went kind of fast and I only followed part of it, so I'm just going to give an overview. Their compression looks for similar values along the rows, columns, and times. They use variations of BMDiff and Zippy. BMDiff gives them high write speeds (~100MB/s) and even faster read speeds (~1000MB/s). Zippy is similar to LZW. It doesn't compresses as highly as LZW or gzip, but it is much faster. He gave an example of a web crawl they compressed with the system. The crawl contained 2.1B pages and the rows were named in the following form: "com.cnn.www/index.html:http". The size of the uncompressed web pages was 45.1 TB and the compressed size was 4.2 TB, yielding a compressed size of only 9.2%. The links data compressed to 13.9% and the anchors data compressed to 12.7% the original size.

They have their eye on the future with some features under consideration. 1. Expressive data manipulation, including having scripts sent to clients to modify data. 2. Multi-row transaction support. 3. General performance for larger cells. 4. BigTable as a service. It sounds like each service (such as Maps or Search History) have their own cluster running BigTable. They are considering running a Google-wide BigTable system, but that would require fairly splitting resources and compute time, etc.

The talk was very interesting and it contained lots new information I had never heard before (I haven't seen a BigTable paper yet). This information is accurate to the best of my knowledge, but if you see a mistake, please e-mail me. I wonder what the next Google talk will be about... I can't wait.

Update 2006-08-31
Google has finally released the official BigTable paper.

Today I saw my second Cinerama movie, "This is Cinerama." It was the first movie made on the new process and is just a showcase of what Cinerama brings to the theatre (7 channel sound and a huge, super-wide screen with three projectors). We arrived about an hour before the show started and we were able to walk right in. I was afraid we were too late to get good seats, but luckily we were able to sit in the second row. About a 3 dm behind me and one 1 dm to the right was the X that marks the "sweet spot."

My mom, brother, and I went up to check out the balcony. On our way down, we noticed the door to one of the projection rooms was open, so I snapped a few pictures. The nice projectionist (from the middle booth) noticed our interest and offered to take us to Projection Booth Beta (the center one). I happily agreed and followed him.

In the main room, we saw an old backup 70 mm projector, the film for our movie, and the Boeing digital projector with anamorphic lenses. We also saw the servers that hold the approximately 60 GB movies and a DLP chip. Along with the four projectors and three racks located in this room was a bathroom. He may have noticed our interest and my reasonable amount of knowledge about film and digital projectors, so he offered to take us to the Charlie booth, where another Cinerama projector and the Cinerama sound reels are located.

When the movie started, I was very, very happy to see that it was kept authentic to the original showing because they only opened part of the screen during the introduction. After his history of motion and movies, the man said (I had wanted to see this part of this movie for a few years), "Welcome to CINERAMA!" The curtain opened to reveal a huge screen (it was big during How the West Was Won, but I was sitting a lot farther away) and we were taken on a roller coaster ride.

As the presentor mentioned, Cinerama is in anagram for American. If you have seen any of the movies (This is Cinerama and How the West Was Won (the ones I've seen), especially), then you can really see the patriotic influence. For example, in HTWW they say stuff like, "let's get us some Injuns." You could also see the influence of patriotism in this movie. They dedicated half the movie to America (and American water-skiing babes) and they played songs like God Bless America, America the Beautiful, and a whole bunch of religious songs I don't know the name of. They also had a section in the beginning showing a singing church choir, located in a "place that we are all familiar with," yeah right.

During the tour of the United States, they flew around Mt. Rainier *applause*... and they flew past the "weird coast of the Pacific Northwest." In the first half (I am splitting up half based on the intermission), they took us on a trip around the world, and by world I of course mean Europe (this was the '50's, Asia hadn't yet been discovered yet, right?). After the movie was over, I went up to check out the screen (because other people were showing interest in it and I wanted to see what was so special). The screen is made up of about 2000 venetian blind-like strips, which prevent one side of the screen from reflecting light onto another (I guess that would be a problem when it is so damned curved). I could see a few vertical black stripes during the movie, but I didn't realize it was because the screen was actually in pieces.

Check out the rest of my photos.

Update 2009-01-24: Edited post to improve grammar, sentence structure, and flow. Also added paragraphs and fixed broken links. See the original post at Archive.org.