Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17362

Doing hadoop ls on Har file triggers too many RPC calls

    XMLWordPrintableJSON

    Details

      Description

      Daryn Sharp has noticed that Invoking hadoop ls on HAR is taking too much of time.

      The har system has multiple deficiencies that significantly impacted performance:

      1. Parsing the master index references ranges within the archive index. Each range required re-opening the hdfs input stream and seeking to the same location where it previously stopped.
      2. Listing a har stats the archive index for every "directory". The per-call cache used a unique key for each stat, rendering the cache useless and significantly increasing memory pressure.
      3. Determining the children of a directory scans the entire archive contents and filters out children. The cached metadata already stores the exact child list.
      4. Globbing a har's contents resulted in unnecessary stats for every leaf path.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ahussein Ahmed Hussein
                Reporter:
                ahussein Ahmed Hussein
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m