Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1752

Implement getFileBlockLocations in HarFilesystem

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: harchive
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      To efficiently run map reduce on the data that has been HAR'ed it will be great to actually implement getFileBlockLocations for a given filename.
      This way the JobTracker will have information about data locality and will schedule tasks appropriately.
      I believe the overhead introduced by doing lookups in the index files can be smaller than that of copying data over the wire.
      Will upload the patch shortly, but would love to get some feedback on this. And any ideas on how to test it are very welcome.

      1. MAPREDUCE-1752.2.patch
        5 kB
        Dmytro Molkov
      2. MAPREDUCE-1752.3.patch
        11 kB
        Patrick Kling
      3. MR-1752.patch
        2 kB
        Dmytro Molkov

        Issue Links

          Activity

          Dmytro Molkov created issue -
          Mahadev konar made changes -
          Field Original Value New Value
          Fix Version/s 0.22.0 [ 12314184 ]
          dhruba borthakur made changes -
          Assignee Dmytro Molkov [ dms ]
          Dmytro Molkov made changes -
          Attachment MR-1752.patch [ 12445380 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to MAPREDUCE-1712 [ MAPREDUCE-1712 ]
          Tsz Wo Nicholas Sze made changes -
          Component/s harchive [ 12312903 ]
          Dmytro Molkov made changes -
          Attachment MAPREDUCE-1752.2.patch [ 12458296 ]
          Dmytro Molkov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Patrick Kling made changes -
          Link This issue blocks MAPREDUCE-2156 [ MAPREDUCE-2156 ]
          Patrick Kling made changes -
          Attachment MAPREDUCE-1752.3.patch [ 12460153 ]
          dhruba borthakur made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.23.0 [ 12315570 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue blocks MAPREDUCE-2156 [ MAPREDUCE-2156 ]
          Gavin made changes -
          Link This issue is depended upon by MAPREDUCE-2156 [ MAPREDUCE-2156 ]

            People

            • Assignee:
              Dmytro Molkov
              Reporter:
              Dmytro Molkov
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development