Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21496

Automatic sizing of unordered buffer can overflow

    XMLWordPrintableJSON

Details

    Description

      HIVE-21329 added automatic sizing of tez unordered partitioned KV buffer based on group by statistics. However, some corner cases for group by statistics sets Long.MAX for data size. This ends up setting Integer.MAX for unordered KV buffer size. This buffer size is expected to be in MB. Converting Integer.MAX value from MB to bytes will overflow and following exception is thrown.

      2019-03-23T01:35:17,760 INFO [Dispatcher thread {Central}] HistoryEventHandler.criticalEvents: [HISTORY][DAG:dag_1553330105749_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 1, taskAttemptId=attempt_1553330105749_0001_1_00_000000_0, creationTime=1553330117468, allocationTime=1553330117524, startTime=1553330117562, finishTime=1553330117755, timeTaken=193, status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Error while running task ( failure ) : attempt_1553330105749_0001_1_00_000000_0:java.lang.IllegalArgumentException
      at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
      at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
      at org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:110)
      at org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:214)
      at org.apache.tez.runtime.library.output.UnorderedPartitionedKVOutput.initialize(UnorderedPartitionedKVOutput.java:76)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:537)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:520)
      at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:505)
      at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

       

      Stats for GBY operator is getting Long.MAX_VALUE as seen below

      2019-03-23T01:35:16,466 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: [0] STATS-TS[0] (logs): numRows: 1795 dataSize: 4443078 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity= colName: severity colType: string countDistincts: 359 numNulls: 89 avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true}
      2019-03-23T01:35:16,466 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: Estimating row count for GenericUDFOPEqual(Column[severity], Const string ERROR) Original num rows: 1795 New num rows: 5
      2019-03-23T01:35:16,467 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: [1] STATS-FIL[8]: numRows: 5 dataSize: 12376 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity= colName: severity colType: string countDistincts: 359 numNulls: 89 avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true}
      2019-03-23T01:35:16,467 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] exec.FilterOperator: Setting stats (Num rows: 5 Data size: 12376 Basic stats: PARTIAL Column stats: NONE) on: FIL[8]
      2019-03-23T01:35:16,468 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] exec.SelectOperator: Setting stats (Num rows: 5 Data size: 12376 Basic stats: PARTIAL Column stats: NONE) on: SEL[2]
      2019-03-23T01:35:16,468 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: [1] STATS-SEL[2]: numRows: 5 dataSize: 12376 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity= colName: severity colType: string countDistincts: 359 numNulls: 89 avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true}
      2019-03-23T01:35:16,471 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: STATS-GBY[3]: inputSize: 4443078 maxSplitSize: 256000000 parallelism: 1 containsGroupingSet: false sizeOfGroupingSet: 1
      2019-03-23T01:35:16,471 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: [Case 1] STATS-GBY[3]: cardinality: 5
      2019-03-23T01:35:16,472 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] exec.GroupByOperator: Setting stats (Num rows: 1 Data size: 9223372036854775807 Basic stats: PARTIAL Column stats: NONE) on: GBY[3]
      2019-03-23T01:35:16,472 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: [0] STATS-GBY[3]: numRows: 1 dataSize: 9223372036854775807 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity= colName: severity colType: string countDistincts: 1 numNulls: 18 avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true, _col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: false}
      2019-03-23T01:35:16,473 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] exec.ReduceSinkOperator: Setting stats (Num rows: 1 Data size: 9223372036854775807 Basic stats: PARTIAL Column stats: NONE) on: RS[4]
      2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: [0] STATS-RS[4]: numRows: 1 dataSize: 9223372036854775807 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity= colName: severity colType: string countDistincts: 1 numNulls: 18 avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true, _col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: false}
      2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: STATS-GBY[5]: inputSize: 1 maxSplitSize: 256000000 parallelism: 1 containsGroupingSet: false sizeOfGroupingSet: 1
      2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: [Case 7] STATS-GBY[5]: cardinality: 0
      2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] stats.StatsUtils: STATS-GBY[5]: Equals 0 in number of rows. 0 rows will be set to 1
      2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] exec.GroupByOperator: Setting stats (Num rows: 1 Data size: 9223372036854775807 Basic stats: PARTIAL Column stats: NONE) on: GBY[5]
      2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: [0] STATS-GBY[5]: numRows: 1 dataSize: 9223372036854775807 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity= colName: severity colType: string countDistincts: 1 numNulls: 18 avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true, _col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: false}
      2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main] annotation.StatsRulesProcFactory: [0] STATS-FS[7]: numRows: 1 dataSize: 9223372036854775807 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity= colName: severity colType: string countDistincts: 1 numNulls: 36 avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true, _col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0 avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: false}

      Attachments

        1. hive.log
          1.53 MB
          Prasanth Jayachandran
        2. HIVE-21496.01.patch
          1 kB
          Jesus Camacho Rodriguez
        3. HIVE-21496.02.patch
          12 kB
          Jesus Camacho Rodriguez

        Activity

          People

            jcamachorodriguez Jesus Camacho Rodriguez
            prasanth_j Prasanth Jayachandran
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: