[PIG-5075] Pig ORCStorage with Snappy Compression will fail with NoClassDefFoundError org/iq80/snappy/Snappy

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.16.0
• Fix Version/s: 0.18.0
• Component/s: impl
• Assignee: Unassigned
• Reporter: Prabhu Joseph
• Created: 09/Dec/16 05:59
• Updated: 24/May/17 23:03

Description

Pig Script to store a text file into ORC with Snappy compression enabled fails with java.lang.NoClassDefFoundError: org/iq80/snappy/Snappy

The hive-exec jar which comes with Pig does not have snappy jars whereas the hive-exec jar comes with hive has that.

[root@prabhuSpark3 lib]# jar tvf /usr/hdp/2.4.2.0-258/pig/lib/hive-exec-1.2.1000.2.4.2.0-258-core.jar | grep iq80
[root@prabhuSpark3 lib]#
[root@prabhuSpark3 lib]# jar tvf /usr/hdp/2.4.2.0-258/hive/lib/hive-exec-1.2.1000.2.4.2.0-258.jar | grep iq80
0 Mon Apr 25 06:49:28 UTC 2016 org/iq80/
0 Mon Apr 25 06:49:28 UTC 2016 org/iq80/snappy/
1577 Mon Apr 25 06:49:28 UTC 2016 org/iq80/snappy/Snappy.class 

Repro:

[root@prabhuSpark3 lib]# hadoop fs -cat /tmp/data
hadoop,5
hive,4
pig,3
tez,2
hawq,1

MYFILE = LOAD '/tmp/data' using PigStorage(',') As (name:chararray,age:int);
Store MYFILE into '/tmp/orcsnappydata' using OrcStorage('-c SNAPPY');

2016-09-22 03:29:06,830 [main] ERROR org.apache.pig.backend.hadoop.executionengine.Launcher - Backend error message
Error: org/iq80/snappy/Snappy
2016-09-22 03:29:06,831 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 0: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: Error: org/iq80/snappy/Snappy
2016-09-22 03:29:06,831 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!

Workaround:
Register /usr/hdp/2.4.2.0-258/hive/lib/hive-exec-1.2.1000.2.4.2.0-258.jar;

As part of this Bug, we want to include the Snappy jars which comes into Pig hive-exec jar.