Hadoop Installation Gotchas

Over the last few years, with most large scale data processors such as Google, Yahoo, Amazon, Twitter and Facebook open-sourcing their internal data crunching algorithms/software, "BigData" projects have taken off exponentially. We are now in the BigData era of computing, where a large number of companies and individuals are contributing to or adopting software stacks such as  Hadoop, Cassandra, MongoDB, Riak, Redis etc. for large scale distributed data processing. Several sub industries have been spawned to provide technical support, hardware and SaaS for BigData as well. What is most appealing and revolutionary about this industry is the fact that open source software is at the heart of it. Any company or individual can dip into the BigData pot without spending large amounts of money and without being held hostage by vendors.


For a recent experiment, I had to install and configure a simple two node Hadoop cluster. In this post, I will highlight a few gotchas that I encountered, the solutions to which were not properly documented anywhere. Hopefully this will save someone else a little bit of time.

I will not go through the process of installing Hadoop, as it is well documented in numerous places. (Michael G. Noll's Hadoop tutorial is an excellent starting point for newbies.)

Gotcha: Server at x not available yet, Zzzzz...
I encountered this error message after I setup my second node and started the whole cluster up. The datanode logs were full of the above message and the dfs health check showed the available space as 0 bytes.

Solution: 
This seems to be a Java bug in resolving names from /etc/hosts. Java networking libraries cache host name resolutions forever. (see: http://myhowto.org/java/42-understanding-host-name-resolution-and-dns-behavior-in-java/) Setting the networkaddress.cache.ttl JVM property to a reasonable number such as 10 should solve this problem, but I had no luck with it. In the end, I managed to get around the issue by using IP addresses instead of hostnames in Hadoop *-site.xml configuration files. Since I was setting up a development cluster, this was acceptable. In a production environment, it is very unlikely that you will be relying on /etc/hosts to resolve hosts anyway. Therefore, it's a minor annoyance that you will only encounter during a trial installation like mine.



Gotcha:  Error reading task output , Too many fetch-failures
I started seeing a bunch of these messages while a job was running in the cluster. Although the job completed successfully, these failures delayed the job quite a lot.

Solution:
This is a problem with /etc/hosts configuration. If you have multiple aliases for the localhost, make sure the hostname alias that is in the Hadoop configuration file, is the first in the list.
For example, if the current /etc/hosts file looks like:
127.0.0.1 localhost.localdomain localhost hadoop-slave1

Change it to:
127.0.0.1 hadoop-slave1 localhost.localdomain localhost



Gotcha:  java.lang.NoClassDefFoundError for classes defined in external libraries
Any non trivial MapReduce job will need to reference external libraries. However, even if the HADOOP_CLASSPATH variable is correctly set, when the job is run by Hadoop, you will get NoClassDef errors for any classes defined in these external library jars.

Solution: 
This took a bit of searching to find out. Hadoop expects all external dependencies to be in a directory named lib inside the job jar. Use the following Maven assembly descriptor to create a job jar that conforms to this convention. (Make sure the core Hadoop dependencies in the POM are set to the provided scope.)


<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
<id>job-dist</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<unpack>false</unpack>
<scope>runtime</scope>
<outputDirectory>lib</outputDirectory>
<excludes>
<exclude>${groupId}:${artifactId}</exclude>
</excludes>
</dependencySet>
<dependencySet>
<unpack>true</unpack>
<includes>
<include>${groupId}:${artifactId}</include>
</includes>
</dependencySet>
</dependencySets>
</assembly>



I will conttinue to update this post as I continue to experiment with Hadoop. If you spot any errors or know of a more elgant solution to some of these problems, please leave a comment.

Fedora 14 64 bit : Distorted Sounds From Flash Player "SQUARE"

If you have been using the new 64 bit beta of Flash Player ("Square") on Fedora 14, you might have noticed that some audio streams are distorted by a weird metallic noise that seems to emanate from the background. Apparently this is caused by a patch to glibc - which removes support for overlapping regions in memcpy. Although this is the right thing to do and clearly Adobe is using memcpy in a non-standard way, it's an annoying bug for non-techie users who don't really care much about whether an application is doing the "right thing" under the hood. 

There are several solutions for the problem at the moment:

• Use Linus Torvalds' custom memcpy function
• Use Ray Strode's flashplayer patch which replaces all memcpy calls with memmove
• Use the 32bit Flash plugin

I have tried the first two and they both work perfectly.

For those interested, the full Bugzilla thread can be found at https://bugzilla.redhat.com/show_bug.cgi?id=638477

Evince failing to open PDF files

I noticed that Evince was suddenly failing to open PDF files with the error message:

File type PDF document (application/pdf) is not supported

Trying to debug the problem, I tried opening a PDF document through the command line and got the following error message.

(evince:1514): EvinceDocument-WARNING **: /usr/lib64/libpoppler-glib.so.5: undefined symbol: cairo_surface_set_mime_data

(evince:1514): EvinceDocument-WARNING **: Cannot load backend 'pdfdocument' since file '/usr/lib64/evince/3/backends/libpdfdocument.so' cannot be read.

(evince:1514): EvinceDocument-WARNING **: /usr/lib64/evince/3/backends/libpdfdocument.so: undefined symbol: cairo_region_union_rectangle

(evince:1514): EvinceDocument-WARNING **: Cannot load backend 'pdfdocument' since file '/usr/lib64/evince/3/backends/libpdfdocument.so' cannot be read.

Now it suddenly makes sense! Fedora 14 has a broken implementation of Cairo graphics library which causes desktop themes to look absolutely ghastly. To get over the problem, I downgraded the Cairo libraries to F13 versions. However, this now seems to have broken Evince. Sure enough; upgrading cairo back to F14 versions made evince work again. Now I am stuck between having a working PDF reader and a nice desktop. It sure is a tough choice!

Haiku - Remember when...

Windows 95 was "awesome" and Windows 98 was slated to be "awesomer"? Linux was taking its baby steps and Mac OS was "pretty" and ruled the DTP circuit??

I was a young 'un, getting my feet wet in the magical world of computers. A Pentium 200Mhz with 32MB of RAM was my gateway in to the realm. (Yes.. it had a turbo button). Now, I do realise that there were even more magical times when people hacked away on mainframes and owning a Zinclair or Apple ][e was considered to be the coolest thing ever. I am too young to have lived in that era. So all of you grey beards, shut up and I promise to get off your lawn in a jiffy.

Anyway, the reason I am going off on a nostalgic tangent is because I remember being really excited when BeOS was announced somewhere in the mid 90's. It was a multimedia OS from the ground up; something alien to us Windows drones who had to contend with belts-and-braces multimedia extensions that never worked as advertised. BeOS had support for multi-threading and pre-emptive scheduling at its core - making processor heavy multimedia apps work faster and better. It had a new file system that supported larger files than FAT32 and most importantly, it had a cool new interface!  I was watching with an open mouth as an engineer demoed playing 4 videos at the same time in a single box running BeOS. My computer couldn't even play a single video without losing frames or freezing up. In that time period where hardware was getting faster and cheaper, an OS that could really utilise those features was a wonderful thing indeed.

Unforutnately, after the initial hype, BeOS faded into the background and was never heard of again. (The conspiracy theorist in me thinks that a certain turtleneck clad individual had something to do with it. BeOS almost managed to become the new Apple OS before they did an about turn and bought NEXT instead). Things moved on and now multimedia is not such a big deal any more. Even my phone has more multimedia capabilities than my first computer now.

Relevance aside, I was pleasantly surprised to learn that BeOS still survives in the form of the Haiku project - a project run by enthusiasts to develop and maintain BeOS. (Hands up if you stumbled on it after reading xkcd). The current version is alpha 2 and the first thing that stuck me after first booting it up is its 90's style interface. (That's not a bad thing).  The 16bit icons and pointers really made me feel nostalgic. However, regardless of the retro look, Haiku seems to be a fairly modern operating system. I was particularly impressed with the task manager - which showed a cascading menu of processes and child-processes with their memory usage embedded in the menu. That is something I would like to have in my regular desktop as well.

I don't think I am qualified enough to formally review Haiku. It is still in the alpha stage, so there aren't that many applications or features to talk about either. But, it looks to be a good enough operating system for basic day-to-day tasks. Given enough contributors, who knows? It might even become a mainstream OS someday.

If you grew up in the 90's, I definitely recommend booting up Haiku and letting the nostalgia sweep over you.

Adobe Flash 'Square' for Linux 64bit

After putting 64bit native Linux Flash player on hold indefinitely, Adobe have done an about turn and released a version of Flash Player 10.2 'Square' preview for 64 bit Linux as well. Good things about this release are that:

• After some serious security flaws were discovered, Adobe advised on an upgrade. But for us poor 64bit Linux users, there was no 64 bit version to upgrade to until now.
• It's a native Linux 64bit binary which should integrate well with Pulseaudio etc and use less reources. This funny comment on Slashdot illustrates the previous state of things really well.
• I suddenly seem to be able to watch 1080p HD videos on Youtube. Can't recall ever seeing that option while I was using the old player.

So go ahead and grab the new version from here. To install the plugin in Fedora, just untar and copy libflashplayer.so to /usr/lib64/mozilla/plugins

Ugly font rendering in Chrome Linux Beta and Unstable

Just upgraded to Google Chrome Beta (6.xx) and then to Unstable (7.xx) in my Fedora box and noticed the  butt-ugly font rendering in both versions. Apparently it's a known issue with gtk-webkit, but I would have expected Google to fix something as elementary and important as font rendering before concentrating on anything else.

A bit of Googling (ofcourse!) dug up this excellent post that fixed the font rendering issue in seconds!

Compare the before and after shots to see how severe the issue is:

Before

After

Firefox 4 Beta On Fedora

Just stumbled on this blog post while skimming through Google Reader and thought I should share it.

To get the latest Beta installed, follow these steps:


sudo wget -O /etc/yum.repos.d/fedora-firefox4.repo http://repos.fedorapeople.org/repos/spot/firefox4/fedora-firefox4.repo

sudo yum install firefox4