Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2664

MR 279: Implement JobCounters for MRv2 + Fix for Map Data Locality

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MRv2 is currently not setting any Job Counters.

      1. MR2664_2.patch
        48 kB
        Siddharth Seth
      2. MR2664.patch
        47 kB
        Siddharth Seth

        Activity

        Hide
        Sharad Agarwal added a comment -

        I committed this MR-279 branch. Thanks Siddharth!

        Show
        Sharad Agarwal added a comment - I committed this MR-279 branch. Thanks Siddharth!
        Hide
        Siddharth Seth added a comment -

        Removed some TODOs
        Added one to resolve host names in a better way.
        Changed testSleepJob to not fail if the app-jar is missing (i.e. skip the test).

        Show
        Siddharth Seth added a comment - Removed some TODOs Added one to resolve host names in a better way. Changed testSleepJob to not fail if the app-jar is missing (i.e. skip the test).
        Hide
        Siddharth Seth added a comment -

        Sharad, thanks for taking a look. The test cases do pass for me (patch applied to revision e3d9a2bcbcab817043b1c4c41efb7036ce00904f).
        Could you please post details of which tests are failing along with the trace and env.

        The Data locality fix in this patch is a temporary patch to allow testing. A more efficient patch incorporating MR-2489 will be required later.

        Show
        Siddharth Seth added a comment - Sharad, thanks for taking a look. The test cases do pass for me (patch applied to revision e3d9a2bcbcab817043b1c4c41efb7036ce00904f). Could you please post details of which tests are failing along with the trace and env. The Data locality fix in this patch is a temporary patch to allow testing. A more efficient patch incorporating MR-2489 will be required later.
        Hide
        Sharad Agarwal added a comment -

        The patch looks fine. minor - some TODOs and FIXME will go away from JobImpl after this patch.

        @Sid - Do the testcases pass for you? Some testcases are failing for me with the patch.

        Show
        Sharad Agarwal added a comment - The patch looks fine. minor - some TODOs and FIXME will go away from JobImpl after this patch. @Sid - Do the testcases pass for you? Some testcases are failing for me with the patch.
        Hide
        Siddharth Seth added a comment -

        Sending job counter update events via the eventHandler as and when they happen so that the counters can be queried while the job is running and to avoid any kind of sync issues in JobImpl.
        Converting the split location to an IP address (Containers currently use IPs). This should enable node data locality.

        Show
        Siddharth Seth added a comment - Sending job counter update events via the eventHandler as and when they happen so that the counters can be queried while the job is running and to avoid any kind of sync issues in JobImpl. Converting the split location to an IP address (Containers currently use IPs). This should enable node data locality.

          People

          • Assignee:
            Siddharth Seth
            Reporter:
            Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development