Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10987

Provide an iterator-based listing API for FileSystem

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.7.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Iterator based listing methods already exist in FileContext for both simple listing and listing with locations. However, FileSystem lacks the former. From what I understand, it wasn't added to FileSystem because it was believed to be phased out soon. Since FileSystem is very well alive today and new features are getting added frequently, I propose adding an iterator based listStatus method. As for the name of the new method, we can use the same name used in FileContext : listStatusIterator().

      It will be particularly useful when listing giant directories. Without this, the client has to build up a huge data structure and hold it in memory. We've seen client JVMs running out of memory because of this.

      Once this change is made, we can modify FsShell, etc. in followup jiras.

        Attachments

        1. HADOOP-10987.patch
          13 kB
          Kihwal Lee
        2. HADOOP-10987.v2.patch
          22 kB
          Kihwal Lee
        3. HADOOP-10987.v3.patch
          22 kB
          Kihwal Lee
        4. HADOOP-10987.v4_with_comment_fix.branch-2.patch
          15 kB
          Kihwal Lee
        5. HADOOP-10987.v4_with_comment_fix.patch
          15 kB
          Kihwal Lee
        6. HADOOP-10987.v4.branch-2.patch
          15 kB
          Kihwal Lee
        7. HADOOP-10987.v4.patch
          15 kB
          Kihwal Lee

          Issue Links

            Activity

              People

              • Assignee:
                kihwal Kihwal Lee
                Reporter:
                kihwal Kihwal Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: