Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
2.6.1
-
None
-
None
Description
Many queries, similar to the below InsertOverwrite, are failing after performing an HDP->HDP upgrade. The cause is that Hive does not localize the Tez TAR file, but instead uses the MapReduce2 property mapreduce.admin.user.env
>>> analyze table studenttab10k compute statistics for columns age ; INFO : Session is already open INFO : Dag name: analyze table studenttab10k compute st...age(Stage-0) INFO : Status: Running (Executing on YARN cluster with App id application_1512568339263_0003) [2K-------------------------------------------------------------------------------- [2K[36;1m VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED [22;0m[2K-------------------------------------------------------------------------------- [2KMap 1 RUNNING 1 0 1 0 2 0 Reducer 2 INITED 1 0 0 1 0 0 [2K-------------------------------------------------------------------------------- [2K[31;1mVERTICES: 00/02 [>>--------------------------] 0% ELAPSED TIME: 9.38 s [22;0m[2K-------------------------------------------------------------------------------- [8A[2K-------------------------------------------------------------------------------- [2K[36;1m VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED [22;0m[2K-------------------------------------------------------------------------------- [2KMap 1 RUNNING 1 0 0 1 4 0 Reducer 2 INITED 1 0 0 1 0 0 [2K-------------------------------------------------------------------------------- [2K[31;1mVERTICES: 00/02 [>>--------------------------] 0% ELAPSED TIME: 14.37 s [22;0m[2K-------------------------------------------------------------------------------- ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1512568339263_0003_2_00, diagnostics=[Task failed, taskId=task_1512568339263_0003_2_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec. at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec. at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:117) at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:143) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:138) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149) ... 14 more Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:208) ... 18 more ], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec.
Attachments
Attachments
Issue Links
- links to