Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1800

Integer overflow in ExternalSorter.getInitialMemoryRequirement()

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When trying to implement support for tez.runtime.io.sort.mb > 2 GB in pipelinedsorter, I encountered integer overflow exception in ExternalSorter.

      Exception thrown is given below.

      ], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.IllegalArgumentException
              at com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
              at org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:158)
              at org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:97)
              at org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:192)
              at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.initialize(OrderedPartitionedKVOutput.java:95)
              at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:430)
              at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.call(LogicalIOProcessorRuntimeTask.java:409)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      

      Actual place where the integer overflow happens is in ExternalSorter.getInitialMemoryRequirement()

      int initialMemRequestMb = 
              conf.getInt(
                  TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB, 
                  TezRuntimeConfiguration.TEZ_RUNTIME_IO_SORT_MB_DEFAULT);
      long reqBytes = initialMemRequestMb << 20;
      
      1. TEZ-1800.1.patch
        0.9 kB
        Rajesh Balamohan

        Activity

        Hide
        sseth Siddharth Seth added a comment -

        +1.

        Show
        sseth Siddharth Seth added a comment - +1.
        Hide
        rajesh.balamohan Rajesh Balamohan added a comment -

        Thanks Siddharth Seth. Committed to master.
        >>>
        commit 79edb49b245bb274ddb377d53683187f8a640a0d
        >>>

        Show
        rajesh.balamohan Rajesh Balamohan added a comment - Thanks Siddharth Seth . Committed to master. >>> commit 79edb49b245bb274ddb377d53683187f8a640a0d >>>
        Hide
        hitesh Hitesh Shah added a comment -

        This should be backported to 0.5.x.

        Show
        hitesh Hitesh Shah added a comment - This should be backported to 0.5.x.
        Hide
        rajesh.balamohan Rajesh Balamohan added a comment -

        committed to branch-0.5

        >>>
        commit 1fb43bd29e20b1718e6652c5175ffe9b2d05fac1
        >>>

        Show
        rajesh.balamohan Rajesh Balamohan added a comment - committed to branch-0.5 >>> commit 1fb43bd29e20b1718e6652c5175ffe9b2d05fac1 >>>
        Hide
        sseth Siddharth Seth added a comment -

        Setting fix version to 0.5.4 since this is already in branch-0.5

        Show
        sseth Siddharth Seth added a comment - Setting fix version to 0.5.4 since this is already in branch-0.5
        Hide
        hitesh Hitesh Shah added a comment -

        Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

        Show
        hitesh Hitesh Shah added a comment - Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

          People

          • Assignee:
            rajesh.balamohan Rajesh Balamohan
            Reporter:
            rajesh.balamohan Rajesh Balamohan
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development