Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12594

snapshotDiff fails if the report exceeds the RPC response limit

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: hdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The snapshotDiff command fails if the snapshotDiff report size is larger than the configuration value of ipc.maximum.response.length which is by default 128 MB.

      Worst case, with all Renames ops in sanpshots each with source and target name equal to MAX_PATH_LEN which is 8k characters, this would result in at 8192 renames.

      SnapshotDiff is currently used by distcp to optimize copy operations and in case of the the diff report exceeding the limit , it fails with the below exception:

      Test set: org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
      -------------------------------------------------------------------------------
      Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
      testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) Time elapsed: 111.906 sec <<< ERROR!
      java.io.IOException: Failed on local exception: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; Host Details : local host is: "hw15685.local/10.200.5.230"; destination host is: "localhost":59808;

      Attached is the proposal for the changes required.

        Attachments

        1. HDFS-12594.001.patch
          86 kB
          Shashikant Banerjee
        2. HDFS-12594.002.patch
          87 kB
          Shashikant Banerjee
        3. HDFS-12594.003.patch
          79 kB
          Shashikant Banerjee
        4. HDFS-12594.004.patch
          81 kB
          Shashikant Banerjee
        5. HDFS-12594.005.patch
          83 kB
          Shashikant Banerjee
        6. HDFS-12594.006.patch
          82 kB
          Shashikant Banerjee
        7. HDFS-12594.007.patch
          82 kB
          Shashikant Banerjee
        8. HDFS-12594.008.patch
          85 kB
          Shashikant Banerjee
        9. HDFS-12594.009.patch
          85 kB
          Shashikant Banerjee
        10. HDFS-12594.010.patch
          84 kB
          Shashikant Banerjee
        11. SnapshotDiff_Improvemnets .pdf
          122 kB
          Shashikant Banerjee

          Issue Links

            Activity

              People

              • Assignee:
                shashikant Shashikant Banerjee
                Reporter:
                shashikant Shashikant Banerjee
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: