Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15196

RBF: RouterRpcServer getListing cannot list large dirs correctly

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.3.1, 3.4.0
    • 3.3.1, 3.4.0
    • rbf
    • None
    • Reviewed

    Description

      In RouterRpcServer, getListing function is handled as two parts:

      1. Union all partial listings from destination ns + paths
      2. Append mount points for the dir to be listed

      In the case of large dir which is bigger than DFSConfigKeys.DFS_LIST_LIMIT (with default value 1k), the batch listing will be used and the startAfter will be used to define the boundary of each batch listing. However, step 2 here will add existing mount points, which will mess up with the boundary of the batch, thus making the next batch startAfter wrong.

      The initial fix is just to append the mount points when there is no more batch query necessary, but this will break the order of returned entries. Therefore more complex logic is added to make sure the order is kept. At the same time the remainingEntries variable inside DirectoryListing is also updated to include the remaining mount points.

      Attachments

        1. HDFS-15196.001.patch
          10 kB
          Fengnan Li
        2. HDFS-15196.002.patch
          7 kB
          Fengnan Li
        3. HDFS-15196.003.patch
          7 kB
          Fengnan Li
        4. HDFS-15196.003.patch
          7 kB
          Fengnan Li
        5. HDFS-15196.004.patch
          7 kB
          Fengnan Li
        6. HDFS-15196.005.patch
          7 kB
          Fengnan Li
        7. HDFS-15196.006.patch
          7 kB
          Fengnan Li
        8. HDFS-15196.007.patch
          8 kB
          Fengnan Li
        9. HDFS-15196.008.patch
          8 kB
          Fengnan Li
        10. HDFS-15196.009.patch
          8 kB
          Fengnan Li
        11. HDFS-15196.010.patch
          8 kB
          Fengnan Li
        12. HDFS-15196.011.patch
          9 kB
          Fengnan Li
        13. HDFS-15196.012.patch
          9 kB
          Fengnan Li
        14. HDFS-15196.013.patch
          10 kB
          Fengnan Li
        15. HDFS-15196.014.patch
          10 kB
          Fengnan Li

        Activity

          People

            fengnanli Fengnan Li
            fengnanli Fengnan Li
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: