Hadoop Common
  1. Hadoop Common
  2. HADOOP-6732

Improve FsShell's heap consumption by switching to listStatus that returns an iterator

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When listing a large directory from the command line using the default heap configuration, FsShell often runs out of memory. This is because all stats of the entries under the directory need to be in memory before printing them. The new API listStatus that returns an iterator of FileStatus, which implemented in HDFS-1091, no longer requires that all entries are fetched first. Thus switching to this new API will greatly improve the use of heap space.

        Issue Links

          Activity

          Hairong Kuang created issue -
          Hairong Kuang made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-6424 [ HADOOP-6424 ]
          Nigel Daley made changes -
          Fix Version/s 0.22.0 [ 12314296 ]
          Daryn Sharp made changes -
          Assignee Daryn Sharp [ daryn ]
          Daryn Sharp made changes -
          Fix Version/s 0.23.0 [ 12315569 ]
          Arun C Murthy made changes -
          Fix Version/s 0.24.0 [ 12317652 ]
          Fix Version/s 0.23.0 [ 12315569 ]
          Harsh J made changes -
          Fix Version/s 0.24.0 [ 12317652 ]

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Hairong Kuang
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development