No FileSystem for scheme: hdfs

I recently came across the "No FileSystem for scheme: hdfs" error from a Scala application that I wrote to work with some HDFS files. I was bundling the Cloudera Hadoop 2.0.0-cdh4.4.0 libraries with my application and it turns out that the FileSystem service definition for HDFS was missing from the META-INF/services directory. The fix is as follows:

1. Create a META-INF/services directory in src/main/resources and add a file named org.apache.hadoop.fs.FileSystem

   mkdir -p src/main/resources/META-INF/services
   touch src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
   

2. Edit the org.apache.hadoop.fs.FileSystem file and add the following content. You can omit almost everything except the last line

   org.apache.hadoop.fs.LocalFileSystem
   org.apache.hadoop.fs.viewfs.ViewFileSystem
   org.apache.hadoop.fs.s3.S3FileSystem
   org.apache.hadoop.fs.s3native.NativeS3FileSystem
   org.apache.hadoop.fs.kfs.KosmosFileSystem
   org.apache.hadoop.fs.ftp.FTPFileSystem
   org.apache.hadoop.fs.HarFileSystem
   org.apache.hadoop.hdfs.DistributedFileSystem
   

3. If you are using the Maven assembly plugin to create a fat jar, the following stanza should be added to assembly.xml inside the <assembly> tag

       <containerDescriptorHandlers>
           <containerDescriptorHandler>
               <handlerName>metaInf-services</handlerName>
           </containerDescriptorHandler>
       </containerDescriptorHandlers>
   

GTIN Validation with Python

GTINs (Global Trade Item Number) are ubiquitous. We commonly see them as barcodes on products. They come in several different types and names such as EAN, UPC or ISBN etc.

Recently, I needed to validate a set of GTINs stored in a file. To my surprise (unless my Google-fu is getting weak), I could not find any libraries written in Python for doing this. The algorithm (http://www.gs1.org/barcodes/support/check_digit_calculator) is simple enough to knock together in a few minutes, but I think the requirement is common enough to warrant a ready-made library.

My first attempt at this can be found at https://github.com/charithe/gtin-validator. This module can validate GTIN-8, GTIN-12, GTIN-13 and GTIN-14 codes in either numeric or string forms. Dashes in the code (as is common with ISBN numbers) are supported.

I have never worked on a publicly available Python module before. Neither am I a professional Python developer. The chances are that there are certain parts in the code that do not conform to conventions. However, it's fully open source - so anybody can contribute to make it better.

Using System.getProperty() with Jenkins builds

One of my recent projects needed to be able to manipulate file paths according to a set of rules and read and process files directly from the file system. When it came to writing unit tests for this module, I was faced with a dilemma. My usual approach for this kind of system is to use the "Test Double" pattern to hide environment specific details from the tests to make them portable across different platforms and different developer environments. However, the module was about 80% file processing code - which made that approach impractical. So I ended up with a suite of tests that required absolute file paths to work properly.

At this point I should mention how it's near impossible to obtain an absolute file path in Java - specially if you are trying to be portable as possible. All my test data was in the src/test/resources/testjobs directory of the project. The absolute path to this was: /home/charithe/workspace/mercury/src/test/resources/testjobs. Obviously, when somebody else checks out this code, the first part of this path would be something else entirely. I played around quite a bit with ClassLoader.getResource et al, but had no luck whatsoever in making the path portable. In desperation, I even tried accessing Maven properties like project.source.directory - which didn't work out as is to be expected.

With time running out, I did the quickest hack imaginable. I added a custom JVM property named testdata.path to the run configuration and the test suite was changed to call System.getProperty("testdata.path") to retrieve the first half of the path, which was then prepended to the unchanging portion of the path to get the full absolute path. This worked out quite well. I could run the tests successfully from within Eclipse by adding the property definition to the run configuration and ditto for maven with the command:

mvn test -Dtestdata.path=/home/charithe/workspace/mercury

It was a short reprieve though. Once my code was checked-in, the Jenkins build failed miserably. My custom property was resolving to null even though I had added it explicitly to the maven command line of the Jenkins job configuration. After doing a quick Google search, it turns out that this is a known bug/feature of Jenkins (http://jenkins.361315.n4.nabble.com/System-properties-hidden-by-hudson-while-test-execution-td380285.html)

The fix was quite simple. In fact, it was so simple, when I found it out, I had to bang my head against the desk for not figuring that out earlier. The Maven sure-fire plugin has the ability to pass properties to the unit tests. This can be done through the plugin configuration section of the pom. (http://maven.apache.org/plugins/maven-surefire-plugin/examples/system-properties.html). The relevant section of my pom.xml looks as follows:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<version>2.5</version>
	<configuration>
		<systemPropertyVariables>
		  <testdata.path>${basedir}/src/test/resources</testdata.path>
		</systemPropertyVariables>
	</configuration>
</plugin>

Strong,Soft,Weak and Phantom References in Java

If you read any documentation on Java garbage collection, the term "strong/weak/soft/phantom reference"  crops up quite frequently. It's easy to guess what strong references are, but what the heck is a phantom/weak/soft reference?

I stumbled on a very informative post about each of the reference types at http://weblogs.java.net/blog/enicholas/archive/2006/05/understanding_w.html. It is an article that perhaps every Java programmer not familiar with this topic should read.  To quote the author of the article: "If you don't know what they are, how will you know when to use them?"

 

I am paraphrasing the main contents of the article below for posterity and for my own reference:

Strong references:

Just like it says on the tin, the vanilla object references that we are all used to. For example, in the following code snippet, someClassObj and someOtherObj are both strong references.

 

public class SomeClass
{
     private SomeObject someOtherObject = new SomeObject();
     ...
     ...

     public static void main(String[] args)
     {
          SomeClass someClassObj = new SomeClass();
          ...
          ...
     }
}

As long as there is a strong reference to an object, it cannot be garbage collected.

 

Weak References

In certain situations, strong references can be a pain to manage and lead to nasty memory leaks. Imagine a global object cache where object references are stored for faster retrieval. The burden is on the programmer to manage the memory used by the cache. If objects that are no longer needed elsewhere are not removed from the cache manually, they will keep keep using memory unnecessarily. As far as the garbage collector is concerned, these objects are still alive because the cache is holding strong references to them.  Therefore, the programmer ends up doing extra work to manage the memory used by the program.

The above unfortunate situation can be made much better by offloading bulk of the work to the garbage collector itself. The way to achieve this is to store weak references in the cache. Weak references are a hint to the garbage collector that the memory occupied by those objects can be reclaimed if no other links to them are found during the link traversal phase of the collector.

 

WeakReference myWeakReference = new WeakReference(someClassObj);
objectRef = myWeakReference.get();

The get method of the weak reference object will return null if the object that it points to has been garbage collected. This takes care of the memory used by the unused object, but what about the WeakReference object itself? ReferenceQueue to the rescue!

ReferenceQueue refQueue = new ReferenceQueue();
WeakReference myWeakReference1 = new WeakReference(someClassObj1,refQueue);
WeakReference myWeakReference2 = new WeakReference(someClassObj2,refQueue);

If a reference to a ReferenceQueue object is passed to the WeakReference object when it is constructed, the garbage collector will automatically put the WeakReference object into the queue once the object is garbage collected. Now all we need to do is to periodically look through the reference queue and dispose of any useless weak refs.
Java provides a handy WeakHashMap class that automatically handles the cleanup when it's entries are garbage collected.

 

Soft References

Soft references are a less eager form of weak references. Generally, objects pointed to by soft references will stay in memory as long as there's enough memory to go around.

 

Phantom References

The main difference between a weak and a phantom reference is that the get method of a phantom reference will always return null.  The get method of a weak reference returning null does not necessarily mean that the pointed object is removed from the memory. The garbage collector is yet to call the finalizer on that object, and hence there is a slim chance that the object could resurrect itself by virtue of having a weird finalize method that creates a strong reference to it.

Objects pointed to be phantom references  have had their finalizers executed and are already physically removed from memory. Therefore, phantom refs only serve to indicate that a certain piece of memory has been reclaimed.

The Python Challenge

The Python challenge is an online riddle inspired by notpron. The catch is that each level requires you to write some Python code (or Perl, Ruby etc. etc. if you are so inclined) to arrive at the solution. The hardest part is figuring out the cryptic clues. The Python bit is easy; so far, the longest piece of Python I have written to solve a riddle is about 6 lines. (Admittedly, I am still on level 6. But according to the forum posts, none of the challenges require a lot of code)

It's a very challenging but fun way to learn the intricacies of Python, regardless of whether you are a novice or a pro. Give it a try at http://www.pythonchallenge.com.

Javac type inference bug

Scenario: A generics based configuration reader that worked perfectly fine under Eclipse, suddenly started failing inside a Jenkins build. The cause was a compilation error: "type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds int,java.lang.Object".

According to this bug report, this is an almost 6 years old javac bug. Apprently Java 1.7 fixes it, but I haven't had a chance to verify it. Here's some sample code to reproduce it:

package com.lucidelectricdreams;

public class GenericsTest
{
	@SuppressWarnings("unchecked")
	public <T> T genericReturnTest(int typeToReturn)
	{
		
		switch(typeToReturn)
		{
			case 1: // integer
				return (T)(new Integer(12));
				
			case 2: // double
				return (T)(new Double(56.2D));
				
			default: // String
				return (T)"test";			
		}
	}
	
	
	public static void main(String[] args)
	{
		GenericsTest gt = new GenericsTest();
		
		int intRetVal = gt.genericReturnTest(1);
		double doubleRetVal = gt.genericReturnTest(2);
		String stringRetVal = gt.genericReturnTest(3);
		
		System.out.println(intRetVal);
		System.out.println(doubleRetVal);
		System.out.println(stringRetVal);		
	}
}

 

Changing the main method as follows, gets rid of the compiler error:

        public static void main(String[] args)
	{
		GenericsTest gt = new GenericsTest();
		
		int intRetVal = gt.<Integer>genericReturnTest(1);
		double doubleRetVal = gt.<Double>genericReturnTest(2);
		String stringRetVal = gt.genericReturnTest(3);
		
		System.out.println(intRetVal);
		System.out.println(doubleRetVal);
		System.out.println(stringRetVal);		
	}