Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Resolves the test failure by modifying the test to base it on spill counters rather than on bytes read/written. It also introduces a new configuration parameter "mapred.job.shuffle.input.buffer.percent" to provide finer grained control on the memory limit to be used during shuffle.
      Show
      Resolves the test failure by modifying the test to base it on spill counters rather than on bytes read/written. It also introduces a new configuration parameter "mapred.job.shuffle.input.buffer.percent" to provide finer grained control on the memory limit to be used during shuffle.

      Description

      Testcase: testReduceFromMem took 23.625 sec
      	FAILED
      Non-zero read from local: 83
      junit.framework.AssertionFailedError: Non-zero read from local: 83
      	at org.apache.hadoop.mapred.TestReduceFetch.testReduceFromMem(TestReduceFetch.java:289)
      	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
      	at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
      	at junit.extensions.TestSetup.run(TestSetup.java:27)
      

      Ran TestReduceFetch a few times on a clean trunk. It failed consistently.

      1. 6029-0.patch
        5 kB
        Chris Douglas
      2. 6029-1.patch
        5 kB
        Chris Douglas
      3. 6029-2.patch
        6 kB
        Chris Douglas
      4. FAILING-PARTIALMEM-TEST-org.apache.hadoop.mapred.TestReduceFetch.txt
        288 kB
        Devaraj Das
      5. M433-0v20.patch
        5 kB
        Chris Douglas
      6. M433-1v20.patch
        5 kB
        Chris Douglas
      7. M433-1y20.patch
        5 kB
        Chris Douglas
      8. M433-v20.another.patch
        5 kB
        Devaraj Das
      9. TEST-org.apache.hadoop.mapred.TestReduceFetch.txt
        288 kB
        Tsz Wo Nicholas Sze
      10. TEST-org.apache.hadoop.mapred.TestReduceFetch.txt
        302 kB
        Devaraj Das
      11. testReduceFromMem.txt
        287 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Chris Douglas made changes -
          Attachment M433-1y20.patch [ 12436678 ]
          Chris Douglas made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Chris Douglas made changes -
          Attachment M433-1v20.patch [ 12431409 ]
          Chris Douglas made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Chris Douglas made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Fix Version/s 0.20.2 [ 12314205 ]
          Fix Version/s 0.21.0 [ 12314045 ]
          Resolution Fixed [ 1 ]
          Devaraj Das made changes -
          Attachment M433-v20.another.patch [ 12431249 ]
          Chris Douglas made changes -
          Link This issue is duplicated by MAPREDUCE-1172 [ MAPREDUCE-1172 ]
          Chris Douglas made changes -
          Attachment M433-0v20.patch [ 12431178 ]
          Chris Douglas made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Jothi Padmanabhan made changes -
          Release Note Resolves the test failure by modifying the test to base it on spill counters rather than on bytes read/written. It also introduces a new configuration parameter "mapred.job.shuffle.input.buffer.percent" to provide finer grained control on the memory limit to be used during shuffle.
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12314045 ]
          Resolution Fixed [ 1 ]
          Chris Douglas made changes -
          Attachment 6029-2.patch [ 12412129 ]
          Chris Douglas made changes -
          Affects Version/s 0.20.1 [ 12314047 ]
          Chris Douglas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.20.1 [ 12314047 ]
          Assignee Chris Douglas [ chris.douglas ]
          Chris Douglas made changes -
          Attachment 6029-1.patch [ 12411504 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-6029 MAPREDUCE-433
          Component/s mapred [ 12310690 ]
          Chris Douglas made changes -
          Attachment 6029-0.patch [ 12411019 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment testReduceFromMem.txt [ 12410996 ]
          Tsz Wo Nicholas Sze made changes -
          Devaraj Das made changes -
          Field Original Value New Value
          Attachment FAILING-PARTIALMEM-TEST-org.apache.hadoop.mapred.TestReduceFetch.txt [ 12410802 ]
          Attachment TEST-org.apache.hadoop.mapred.TestReduceFetch.txt [ 12410803 ]
          Tsz Wo Nicholas Sze created issue -

            People

            • Assignee:
              Chris Douglas
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development