top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Hadoop native and snappy library?

+8 votes
1,398 views

I'm trying to enable the Hadoop native library and the snappy library for compression in Hadoop 2.2.0, but I always end up with:

./hadoop/bin/hadoop checknative -a
Native library checking:
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false

I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting .so in hadoop/lib/native/. I also compiled snappy from scratch and put it there. In a different approach I installed snappy via sudo apt-get
and then linked the resulting .so to hadoop/lib/native/libsnappy.so, still no luck.

What is going on here? Why won't Hadoop find my native libraries? Is there any log where I can check what went wrong during loading?

posted Feb 11, 2014 by Abhay Kulkarni

Looking for an answer?  Promote on:
Facebook Share Button Twitter Share Button LinkedIn Share Button
Thanks... after looking at the sourcecode of the native library loading process and activating the DEBUG messages I found this in the "hadoop.log":

DEBUG org.apache.hadoop.util.NativeCodeLoader:
Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)

Is it possible to compile the native library with glibc 2.12, because at the moment I am not able to upgrade glibc on the servers which run Hadoop.
Where do you compile your libhadoop.so.1.0.0? It is more like that you compiled libhadoop.so.1.0.0 in a environment with glibc 2.14, but tried to use it in an environment only have glibc 2.12.

If you are using a hadoop compiled by yourself, then it is best to compile in an environment matching with your production.

Similar Questions
+1 vote

I am running on Hadoop 1.0.4 and I would like to use Snappy for map output compression. I am adding the configurations:

configuration.setBoolean("mapred.compress.map.output", true);
 configuration.set("mapred.map.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec");

And Ive added libsnappy.so.1 to $HADOOP_HOME/lib/native/Linux-amd64-64/
Still, all map tasks fail with "native snappy library not available". Could anyone elaborate on how to install Snappy for Hadoop ?

...