Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4529

Pig on tez hit counter limit imposed by MR

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.15.0
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      See the following stack in test:

      org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 131 max=130
      	at org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:101)
      	at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:108)
      	at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:78)
      	at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:95)
      	at org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:87)
      	at org.apache.hadoop.mapred.Counters$Group.addCounter(Counters.java:354)
      	at org.apache.pig.tools.pigstats.tez.TezDAGStats.covertToHadoopCounters(TezDAGStats.java:276)
      	at org.apache.pig.tools.pigstats.tez.TezDAGStats.accumulateStats(TezDAGStats.java:192)
      	at org.apache.pig.tools.pigstats.tez.TezPigScriptStats.accumulateStats(TezPigScriptStats.java:181)
      	at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:198)
      	at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:174)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

      Actually I've set tez.counters.max to a much higher value, TezLauncher should not be limited by MR limits.

        Attachments

        1. PIG-4529-2.patch
          2 kB
          Jianyong Dai
        2. PIG-4529-1.patch
          2 kB
          Jianyong Dai

          Activity

            People

            • Assignee:
              daijy Jianyong Dai
              Reporter:
              daijy Jianyong Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: