Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20579

Improve snapshot manifest copy in ExportSnapshot

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.5.0, 2.0.0
    • Fix Version/s: 3.0.0, 2.1.0
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This patch adds an FSUtil.copyFilesParallel() to help copy files in parallel, and it will return all the paths of directories and files traversed. Thus when we copy manifest in ExportSnapshot, we can copy reference files concurrently and use the paths it returns to help setOwner and setPermission.
      The size of thread pool is determined by the configuration snapshot.export.copy.references.threads, and its default value is the number of runtime available processors.
      Show
      This patch adds an FSUtil.copyFilesParallel() to help copy files in parallel, and it will return all the paths of directories and files traversed. Thus when we copy manifest in ExportSnapshot, we can copy reference files concurrently and use the paths it returns to help setOwner and setPermission. The size of thread pool is determined by the configuration snapshot.export.copy.references.threads, and its default value is the number of runtime available processors.

      Description

      ExportSnapshot need to copy snapshot manifest to destination cluster first, then setOwner and setPermission for those paths. But it's done with one thread, which lead to a long time to submit the job if your snapshot is big. I tried to make them processing in parallel, which can reduce the total time of submitting dramatically.

        Attachments

        1. HBASE-20579.branch-2.addendum.patch
          2 kB
          Josh Elser
        2. HBASE-20579.master.001.patch
          9 kB
          Jingyun Tian
        3. HBASE-20579.master.002.patch
          11 kB
          Jingyun Tian

          Issue Links

            Activity

              People

              • Assignee:
                tianjingyun Jingyun Tian
                Reporter:
                tianjingyun Jingyun Tian
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: