Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4516

RANK in Tez generates OOM: Java heap space

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.14.0
    • None
    • tez
    • None

    Description

      Running a script with a RANK operator using Tez generates a OOM error. The script continues and ends successfully, but during the execution there are several failed tasks.

      2015-04-23 12:48:54,881 INFO [AsyncDispatcher event handler] history.HistoryEventHandler: [HISTORY][DAG:dag_1429175510653_0318_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=scope-75, taskAttemptId=attempt_1429175510653_0318_1_01_000111_0, startTime=1429786121169, finishTime=1429786134880, timeTaken=13711, status=FAILED, diagnostics=Error: Fatal Error cause TezChild exit.:java.lang.OutOfMemoryError: Java heap space
      	at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.<init>(DefaultSorter.java:140)
      	at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:114)
      	at org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.initializeOutputs(PigProcessor.java:299)
      	at org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.run(PigProcessor.java:181)
      	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
      	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            Carlos Balduz Carlos Balduz

            Dates

              Created:
              Updated:

              Slack

                Issue deployment