Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-786

Implement getContentSummary(..) in HftpFileSystem

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HftpFileSystem does not override getContentSummary(..). As a result, it uses FileSystem's default implementation, which computes content summary on the client side by calling listStatus(..) recursively. In contrast, DistributedFileSystem has overridden getContentSummary(..) and does the computation on the NameNode.

      As a result, running "fs -dus" on hftp is much slower than running it on hdfs.

      1. h786_20091223.patch
        11 kB
        Tsz Wo Nicholas Sze
      2. h786_20091224.patch
        12 kB
        Tsz Wo Nicholas Sze
      3. h786_20100104.patch
        12 kB
        Tsz Wo Nicholas Sze
      4. h786_20100106_0.20.patch
        12 kB
        Tsz Wo Nicholas Sze
      5. h786_20100106.patch
        12 kB
        Tsz Wo Nicholas Sze
      6. h786_20100223_0.20.patch
        12 kB
        Suresh Srinivas

        Activity

        Tsz Wo Nicholas Sze created issue -
        Tsz Wo Nicholas Sze made changes -
        Field Original Value New Value
        Attachment h786_20091223.patch [ 12428879 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h786_20091224.patch [ 12428929 ]
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.22.0 [ 12314241 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h786_20100104.patch [ 12429402 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h786_20100106.patch [ 12429613 ]
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Chris Douglas made changes -
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h786_20100106_0.20.patch [ 12430314 ]
        Suresh Srinivas made changes -
        Attachment h786_20100223_0.20.patch [ 12436792 ]
        Tom White made changes -
        Fix Version/s 0.21.0 [ 12314046 ]
        Fix Version/s 0.22.0 [ 12314241 ]
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Tsz Wo Nicholas Sze
            Reporter:
            Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development