Bufferbloat

Interesting Google tech talk by Jim Gettys about Bufferbloat:  http://www.youtube.com/user/GoogleTechTalks#p/u/0/qbIozKVz73g

Also checkout netalyzr to test and gather information about your network connection : http://netalyzr.icsi.berkeley.edu/

 

Strong,Soft,Weak and Phantom References in Java

If you read any documentation on Java garbage collection, the term "strong/weak/soft/phantom reference"  crops up quite frequently. It's easy to guess what strong references are, but what the heck is a phantom/weak/soft reference?

I stumbled on a very informative post about each of the reference types at http://weblogs.java.net/blog/enicholas/archive/2006/05/understanding_w.html. It is an article that perhaps every Java programmer not familiar with this topic should read.  To quote the author of the article: "If you don't know what they are, how will you know when to use them?"

 

I am paraphrasing the main contents of the article below for posterity and for my own reference:

Strong references:

Just like it says on the tin, the vanilla object references that we are all used to. For example, in the following code snippet, someClassObj and someOtherObj are both strong references.

 

public class SomeClass
{
     private SomeObject someOtherObject = new SomeObject();
     ...
     ...

     public static void main(String[] args)
     {
          SomeClass someClassObj = new SomeClass();
          ...
          ...
     }
}

As long as there is a strong reference to an object, it cannot be garbage collected.

 

Weak References

In certain situations, strong references can be a pain to manage and lead to nasty memory leaks. Imagine a global object cache where object references are stored for faster retrieval. The burden is on the programmer to manage the memory used by the cache. If objects that are no longer needed elsewhere are not removed from the cache manually, they will keep keep using memory unnecessarily. As far as the garbage collector is concerned, these objects are still alive because the cache is holding strong references to them.  Therefore, the programmer ends up doing extra work to manage the memory used by the program.

The above unfortunate situation can be made much better by offloading bulk of the work to the garbage collector itself. The way to achieve this is to store weak references in the cache. Weak references are a hint to the garbage collector that the memory occupied by those objects can be reclaimed if no other links to them are found during the link traversal phase of the collector.

 

WeakReference myWeakReference = new WeakReference(someClassObj);
objectRef = myWeakReference.get();

The get method of the weak reference object will return null if the object that it points to has been garbage collected. This takes care of the memory used by the unused object, but what about the WeakReference object itself? ReferenceQueue to the rescue!

ReferenceQueue refQueue = new ReferenceQueue();
WeakReference myWeakReference1 = new WeakReference(someClassObj1,refQueue);
WeakReference myWeakReference2 = new WeakReference(someClassObj2,refQueue);

If a reference to a ReferenceQueue object is passed to the WeakReference object when it is constructed, the garbage collector will automatically put the WeakReference object into the queue once the object is garbage collected. Now all we need to do is to periodically look through the reference queue and dispose of any useless weak refs.
Java provides a handy WeakHashMap class that automatically handles the cleanup when it's entries are garbage collected.

 

Soft References

Soft references are a less eager form of weak references. Generally, objects pointed to by soft references will stay in memory as long as there's enough memory to go around.

 

Phantom References

The main difference between a weak and a phantom reference is that the get method of a phantom reference will always return null.  The get method of a weak reference returning null does not necessarily mean that the pointed object is removed from the memory. The garbage collector is yet to call the finalizer on that object, and hence there is a slim chance that the object could resurrect itself by virtue of having a weird finalize method that creates a strong reference to it.

Objects pointed to be phantom references  have had their finalizers executed and are already physically removed from memory. Therefore, phantom refs only serve to indicate that a certain piece of memory has been reclaimed.

The Python Challenge

The Python challenge is an online riddle inspired by notpron. The catch is that each level requires you to write some Python code (or Perl, Ruby etc. etc. if you are so inclined) to arrive at the solution. The hardest part is figuring out the cryptic clues. The Python bit is easy; so far, the longest piece of Python I have written to solve a riddle is about 6 lines. (Admittedly, I am still on level 6. But according to the forum posts, none of the challenges require a lot of code)

It's a very challenging but fun way to learn the intricacies of Python, regardless of whether you are a novice or a pro. Give it a try at http://www.pythonchallenge.com.

Javac type inference bug

Scenario: A generics based configuration reader that worked perfectly fine under Eclipse, suddenly started failing inside a Jenkins build. The cause was a compilation error: "type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds int,java.lang.Object".

According to this bug report, this is an almost 6 years old javac bug. Apprently Java 1.7 fixes it, but I haven't had a chance to verify it. Here's some sample code to reproduce it:

package com.lucidelectricdreams;

public class GenericsTest
{
	@SuppressWarnings("unchecked")
	public <T> T genericReturnTest(int typeToReturn)
	{
		
		switch(typeToReturn)
		{
			case 1: // integer
				return (T)(new Integer(12));
				
			case 2: // double
				return (T)(new Double(56.2D));
				
			default: // String
				return (T)"test";			
		}
	}
	
	
	public static void main(String[] args)
	{
		GenericsTest gt = new GenericsTest();
		
		int intRetVal = gt.genericReturnTest(1);
		double doubleRetVal = gt.genericReturnTest(2);
		String stringRetVal = gt.genericReturnTest(3);
		
		System.out.println(intRetVal);
		System.out.println(doubleRetVal);
		System.out.println(stringRetVal);		
	}
}

 

Changing the main method as follows, gets rid of the compiler error:

        public static void main(String[] args)
	{
		GenericsTest gt = new GenericsTest();
		
		int intRetVal = gt.<Integer>genericReturnTest(1);
		double doubleRetVal = gt.<Double>genericReturnTest(2);
		String stringRetVal = gt.genericReturnTest(3);
		
		System.out.println(intRetVal);
		System.out.println(doubleRetVal);
		System.out.println(stringRetVal);		
	}

Kill tasks in Windows through the command line

I received a frantic phone call from my Dad last night, who lives a few thousand miles away, across a few oceans. "My computer is not working. Something is wrong with my hard disk" - he said. "Fix it!". Although it was  flattering that my dad thinks that I could magically fix a bad hard drive from a few thousand miles away, it didn't sound quite right because he had just recently bought that machine. I will spare you my dear readers, of the painful 10 minutes that I spent talking in a very slow and calm voice to my dad to figure out what was really wrong. Eventually I managed to figure out that he was infected with the "Windows Recovery" virus. (http://www.bleepingcomputer.com/virus-removal/remove-windows-recovery).

Removing this infection is simple enough if you follow the excellent instructions from the bleepingcomputer guide above. However,  what can you do when you don't have physical access to the machine? Luckily the LogmeIn app I had installed sometime back was still running and accessible, so I could access the computer remotely. However, trying to download the rkill application to stop the virus was impossible because it was blocking all DNS requests out of the machine. The task manager was disabled by the virus as well, so pressing Ctrl+Alt+Del didn't work either. Restarting the computer in safe mode would cut off my remote access through LogMeIn. Asking my dad to press even a single key takes more than 5 minutes of explanations and several wrong attempts - so telling him what to do was not an option either.

The Solution:

tasklist

There is a little known command in Windows named tasklist, which does the same thing as the Linux ps command. Running the command on a command prompt will display a list of all running processes along with their PIDs. To kill any process, type tskill followed by the PID. For example, to kill PID 2476, type:

tskill 2476

Pretty simple, but very handy command for those sticky situations!

 

 

 

Hadoop Installation Gotchas

Over the last few years, with most large scale data processors such as Google, Yahoo, Amazon, Twitter and Facebook open-sourcing their internal data crunching algorithms/software, "BigData" projects have taken off exponentially. We are now in the BigData era of computing, where a large number of companies and individuals are contributing to or adopting software stacks such as  Hadoop, Cassandra, MongoDB, Riak, Redis etc. for large scale distributed data processing. Several sub industries have been spawned to provide technical support, hardware and SaaS for BigData as well. What is most appealing and revolutionary about this industry is the fact that open source software is at the heart of it. Any company or individual can dip into the BigData pot without spending large amounts of money and without being held hostage by vendors.


For a recent experiment, I had to install and configure a simple two node Hadoop cluster. In this post, I will highlight a few gotchas that I encountered, the solutions to which were not properly documented anywhere. Hopefully this will save someone else a little bit of time.

I will not go through the process of installing Hadoop, as it is well documented in numerous places. (Michael G. Noll's Hadoop tutorial is an excellent starting point for newbies.)

Gotcha: Server at x not available yet, Zzzzz...
I encountered this error message after I setup my second node and started the whole cluster up. The datanode logs were full of the above message and the dfs health check showed the available space as 0 bytes.

Solution: 
This seems to be a Java bug in resolving names from /etc/hosts. Java networking libraries cache host name resolutions forever. (see: http://myhowto.org/java/42-understanding-host-name-resolution-and-dns-behavior-in-java/) Setting the networkaddress.cache.ttl JVM property to a reasonable number such as 10 should solve this problem, but I had no luck with it. In the end, I managed to get around the issue by using IP addresses instead of hostnames in Hadoop *-site.xml configuration files. Since I was setting up a development cluster, this was acceptable. In a production environment, it is very unlikely that you will be relying on /etc/hosts to resolve hosts anyway. Therefore, it's a minor annoyance that you will only encounter during a trial installation like mine.



Gotcha:  Error reading task output , Too many fetch-failures
I started seeing a bunch of these messages while a job was running in the cluster. Although the job completed successfully, these failures delayed the job quite a lot.

Solution:
This is a problem with /etc/hosts configuration. If you have multiple aliases for the localhost, make sure the hostname alias that is in the Hadoop configuration file, is the first in the list.
For example, if the current /etc/hosts file looks like:
127.0.0.1 localhost.localdomain localhost hadoop-slave1

Change it to:
127.0.0.1 hadoop-slave1 localhost.localdomain localhost



Gotcha:  java.lang.NoClassDefFoundError for classes defined in external libraries
Any non trivial MapReduce job will need to reference external libraries. However, even if the HADOOP_CLASSPATH variable is correctly set, when the job is run by Hadoop, you will get NoClassDef errors for any classes defined in these external library jars.

Solution: 
This took a bit of searching to find out. Hadoop expects all external dependencies to be in a directory named lib inside the job jar. Use the following Maven assembly descriptor to create a job jar that conforms to this convention. (Make sure the core Hadoop dependencies in the POM are set to the provided scope.)


<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
<id>job-dist</id>
<formats>
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<unpack>false</unpack>
<scope>runtime</scope>
<outputDirectory>lib</outputDirectory>
<excludes>
<exclude>${groupId}:${artifactId}</exclude>
</excludes>
</dependencySet>
<dependencySet>
<unpack>true</unpack>
<includes>
<include>${groupId}:${artifactId}</include>
</includes>
</dependencySet>
</dependencySets>
</assembly>



I will conttinue to update this post as I continue to experiment with Hadoop. If you spot any errors or know of a more elgant solution to some of these problems, please leave a comment.

Fedora 14 64 bit : Distorted Sounds From Flash Player "SQUARE"

If you have been using the new 64 bit beta of Flash Player ("Square") on Fedora 14, you might have noticed that some audio streams are distorted by a weird metallic noise that seems to emanate from the background. Apparently this is caused by a patch to glibc - which removes support for overlapping regions in memcpy. Although this is the right thing to do and clearly Adobe is using memcpy in a non-standard way, it's an annoying bug for non-techie users who don't really care much about whether an application is doing the "right thing" under the hood. 

There are several solutions for the problem at the moment:

• Use Linus Torvalds' custom memcpy function
• Use Ray Strode's flashplayer patch which replaces all memcpy calls with memmove
• Use the 32bit Flash plugin

I have tried the first two and they both work perfectly.

For those interested, the full Bugzilla thread can be found at https://bugzilla.redhat.com/show_bug.cgi?id=638477