Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7926 long-lived daemons for query fragment execution, I/O and caching
  3. HIVE-11524

LLAP: tez.runtime.compress doesn't appear to be honored for LLAP

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: llap
    • Component/s: llap
    • Labels:
      None

      Description

      When running llap on an openstack cluster without snappy installed, with tez.runtime.compress set to false and codec set to snappy, one still gets the exceptions due to snappy codec being absent:

      2015-08-10 11:14:30,440 [TezTaskRunner_attempt_1438943112941_0015_2_00_000000_0(attempt_1438943112941_0015_2_00_000000_0)] ERROR org.apache.hadoop.io.compress.snappy.SnappyCompressor: failed to load SnappyCompressor
      java.lang.NoSuchFieldError: clazz
      	at org.apache.hadoop.io.compress.snappy.SnappyCompressor.initIDs(Native Method)
      	at org.apache.hadoop.io.compress.snappy.SnappyCompressor.<clinit>(SnappyCompressor.java:57)
      	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:69)
      	at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
      	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
      	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
      	at org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.<init>(IFile.java:153)
      	at org.apache.tez.runtime.library.common.sort.impl.IFile$Writer.<init>(IFile.java:138)
      	at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter$SpillCallable.callInternal(UnorderedPartitionedKVWriter.java:406)
      	at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter$SpillCallable.callInternal(UnorderedPartitionedKVWriter.java:367)
      	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      	at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.finalSpill(UnorderedPartitionedKVWriter.java:612)
      	at org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.close(UnorderedPartitionedKVWriter.java:521)
      	at org.apache.tez.runtime.library.output.UnorderedKVOutput.close(UnorderedKVOutput.java:128)
      	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:376)
      	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:79)
      	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655)
      	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
      	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:35)
      	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      When it's set to true, the client complains about snappy. When it's set to fails, the client doesn't complain but it still tries to use it.

        Attachments

          Activity

            People

            • Assignee:
              sseth Siddharth Seth
              Reporter:
              sershe Sergey Shelukhin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: