Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15294

Federation balance tool

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This jira introduces a new HDFS federation balance tool to balance data across different federation namespaces. It uses Distcp to copy data from the source path to the target path.

      The process is:
      1. Use distcp and snapshot diff to sync data between src and dst until they are the same.
      2. Update mount table in Router if we specified RBF mode.
      3. Deal with src data, move to trash, delete or skip them.

      The design of fedbalance tool comes from the discussion in HDFS-15087.

        Attachments

        1. HDFS-15294.007.patch
          154 kB
          Jinglun
        2. HDFS-15294.006.patch
          154 kB
          Jinglun
        3. HDFS-15294.005.patch
          153 kB
          Jinglun
        4. HDFS-15294.004.patch
          152 kB
          Jinglun
        5. HDFS-15294.003.reupload.patch
          151 kB
          Jinglun
        6. HDFS-15294.003.patch
          151 kB
          Jinglun
        7. HDFS-15294.002.patch
          134 kB
          Jinglun
        8. HDFS-15294.001.patch
          134 kB
          Jinglun
        9. distcp-balance.v2.pdf
          99 kB
          Jinglun
        10. distcp-balance.pdf
          86 kB
          Jinglun
        11. BalanceProcedureScheduler.png
          47 kB
          Jinglun

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                LiJinglun Jinglun
                Reporter:
                LiJinglun Jinglun
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: