Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5288

ResourceEstimator#getEstimatedTotalMapOutputSize suffers from divide by zero issues

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0
    • Component/s: mrv1
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The computation in the above mentioned class-method is below:

            long estimate = Math.round(((double)inputSize * 
                completedMapsOutputSize * 2.0)/completedMapsInputSize);
      

      Given http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(double), its possible that the returned estimate could be Long.MAX_VALUE if completedMapsInputSize is determined to be zero.

      This can be proven with a simple code snippet:

      class Foo {
          public static void main(String... args) {
              long inputSize = 600L + 2;
              long estimate = Math.round(((double)inputSize *
                                    1L * 2.0)/0L);
              System.out.println(estimate);
          }
      }
      

      The above conveniently prints out: 9223372036854775807, which is Long.MAX_VALUE (or 8 Exbibytes per MapReduce).

      1. mr-5288-1.patch
        5 kB
        Karthik Kambatla

        Activity

        Harsh J created issue -
        Karthik Kambatla made changes -
        Field Original Value New Value
        Assignee Karthik Kambatla [ kkambatl ]
        Hide
        Fengdong Yu added a comment -

        This was fixed in hadoop-1.1.2 stable release.
        if we determine completedMapsInputSize is zero, then job's map tasks MUST be zero, so the estimated output size is zero.
        below is the code:

          long getEstimatedMapOutputSize() {
            long estimate = 0L;
            if (job.desiredMaps() > 0) {
              estimate = getEstimatedTotalMapOutputSize()  / job.desiredMaps();
            }
            return estimate;
          }
        
        Show
        Fengdong Yu added a comment - This was fixed in hadoop-1.1.2 stable release. if we determine completedMapsInputSize is zero, then job's map tasks MUST be zero, so the estimated output size is zero. below is the code: long getEstimatedMapOutputSize() { long estimate = 0L; if (job.desiredMaps() > 0) { estimate = getEstimatedTotalMapOutputSize() / job.desiredMaps(); } return estimate; }
        Hide
        Karthik Kambatla added a comment -

        Harsh J, is it a problem if the estimate is Long.MAX_VALUE? To frame it differently, if no maps completed (may be due to the infinite amount of work each map does), what other value would we set this to.

        Fengdong Yu, I am not sure if the snippet posted addresses the issue Harsh reported.

        Show
        Karthik Kambatla added a comment - Harsh J , is it a problem if the estimate is Long.MAX_VALUE? To frame it differently, if no maps completed (may be due to the infinite amount of work each map does), what other value would we set this to. Fengdong Yu , I am not sure if the snippet posted addresses the issue Harsh reported.
        Hide
        Harsh J added a comment -

        Hi Karthik,

        The estimate blocks out scheduling so its kinda critical. We hold back on scheduling if the estimate (which is incorrect) is higher than the node's disk free space report. This is ending up blocking up all map tasks from getting scheduled.

        I think what Fengdong Yu is pointing out is that do not try to estimate if completed maps isn't > 0. The code he points out, says this shouldn't ever happen but over the mailing list we did see this happen.

        Show
        Harsh J added a comment - Hi Karthik, The estimate blocks out scheduling so its kinda critical. We hold back on scheduling if the estimate (which is incorrect) is higher than the node's disk free space report. This is ending up blocking up all map tasks from getting scheduled. I think what Fengdong Yu is pointing out is that do not try to estimate if completed maps isn't > 0. The code he points out, says this shouldn't ever happen but over the mailing list we did see this happen.
        Hide
        Karthik Kambatla added a comment -

        Uploading a patch that sets the thresholdToUse value to at least 1. This value is used in getEstimatedTotalMapOutputSize() to decide whether to estimate or not.

        Show
        Karthik Kambatla added a comment - Uploading a patch that sets the thresholdToUse value to at least 1. This value is used in getEstimatedTotalMapOutputSize() to decide whether to estimate or not.
        Karthik Kambatla made changes -
        Attachment mr-5288-1.patch [ 12586371 ]
        Karthik Kambatla made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12586371/mr-5288-1.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3737//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586371/mr-5288-1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3737//console This message is automatically generated.
        Hide
        Harsh J added a comment -

        +1

        Show
        Harsh J added a comment - +1
        Hide
        Alejandro Abdelnur added a comment -

        Thanks Karthik. Thanks Harsh for reviewing it. Committed to branch-1.

        Show
        Alejandro Abdelnur added a comment - Thanks Karthik. Thanks Harsh for reviewing it. Committed to branch-1.
        Alejandro Abdelnur made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 1.3.0 [ 12324153 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Karthik Kambatla
            Reporter:
            Harsh J
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development