Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15087 RBF: Balance/Rename across federation namespaces
  3. HDFS-15294

RBF: Balance data across federation namespaces with DistCp and snapshot diff

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This jira introduces a new balance command 'fedbalance' that is ran by the administrator. The process is:
      1. Use distcp and snapshot diff to sync data between src and dst until they are the same.
      2. Update mount table in Router.
      3. Delete the src to trash.

       

      The patch is too big to review, so I split it into 2 patches:

      Phase 1 / The State Machine(BalanceProcedureScheduler): Including the abstraction of job and scheduler model.   <See HDFS-15340>

      org.apache.hadoop.hdfs.procedure.BalanceProcedureScheduler;
      org.apache.hadoop.hdfs.procedure.BalanceProcedureConfigKeys;
      org.apache.hadoop.hdfs.procedure.BalanceProcedure;
      org.apache.hadoop.hdfs.procedure.BalanceJob;
      org.apache.hadoop.hdfs.procedure.BalanceJournal;
      org.apache.hadoop.hdfs.procedure.HDFSJournal;
      

      Phase 2 / The DistCpFedBalance: It's an implementation of BalanceJob.    <See HDFS-15346>

      org.apache.hadoop.hdfs.server.federation.procedure.MountTableProcedure;
      org.apache.hadoop.tools.DistCpFedBalance;
      org.apache.hadoop.tools.DistCpProcedure;
      org.apache.hadoop.tools.FedBalance;
      org.apache.hadoop.tools.FedBalanceConfigs;
      org.apache.hadoop.tools.FedBalanceContext;
      org.apache.hadoop.tools.TrashProcedure;
      

        Attachments

        1. BalanceProcedureScheduler.png
          47 kB
          Jinglun
        2. distcp-balance.pdf
          86 kB
          Jinglun
        3. distcp-balance.v2.pdf
          99 kB
          Jinglun
        4. HDFS-15294.001.patch
          134 kB
          Jinglun
        5. HDFS-15294.002.patch
          134 kB
          Jinglun
        6. HDFS-15294.003.patch
          151 kB
          Jinglun
        7. HDFS-15294.003.reupload.patch
          151 kB
          Jinglun
        8. HDFS-15294.004.patch
          152 kB
          Jinglun
        9. HDFS-15294.005.patch
          153 kB
          Jinglun
        10. HDFS-15294.006.patch
          154 kB
          Jinglun
        11. HDFS-15294.007.patch
          154 kB
          Jinglun

          Issue Links

            Activity

              People

              • Assignee:
                LiJinglun Jinglun
                Reporter:
                LiJinglun Jinglun
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: