Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10987

Provide an iterator-based listing API for FileSystem

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.7.0
    • 2.7.0
    • None
    • None
    • Reviewed

    Description

      Iterator based listing methods already exist in FileContext for both simple listing and listing with locations. However, FileSystem lacks the former. From what I understand, it wasn't added to FileSystem because it was believed to be phased out soon. Since FileSystem is very well alive today and new features are getting added frequently, I propose adding an iterator based listStatus method. As for the name of the new method, we can use the same name used in FileContext : listStatusIterator().

      It will be particularly useful when listing giant directories. Without this, the client has to build up a huge data structure and hold it in memory. We've seen client JVMs running out of memory because of this.

      Once this change is made, we can modify FsShell, etc. in followup jiras.

      Attachments

        1. HADOOP-10987.v4_with_comment_fix.patch
          15 kB
          Kihwal Lee
        2. HADOOP-10987.v4_with_comment_fix.branch-2.patch
          15 kB
          Kihwal Lee
        3. HADOOP-10987.v4.patch
          15 kB
          Kihwal Lee
        4. HADOOP-10987.v4.branch-2.patch
          15 kB
          Kihwal Lee
        5. HADOOP-10987.v3.patch
          22 kB
          Kihwal Lee
        6. HADOOP-10987.v2.patch
          22 kB
          Kihwal Lee
        7. HADOOP-10987.patch
          13 kB
          Kihwal Lee

        Issue Links

          Activity

            People

              kihwal Kihwal Lee
              kihwal Kihwal Lee
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: