Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15294

Federation balance tool

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • rbf, tools
    • None
    • Reviewed

    Description

      This jira introduces a new HDFS federation balance tool to balance data across different federation namespaces. It uses Distcp to copy data from the source path to the target path.

      The process is:
      1. Use distcp and snapshot diff to sync data between src and dst until they are the same.
      2. Update mount table in Router if we specified RBF mode.
      3. Deal with src data, move to trash, delete or skip them.

      The design of fedbalance tool comes from the discussion in HDFS-15087.

      Attachments

        1. distcp-balance.pdf
          86 kB
          Jinglun
        2. HDFS-15294.001.patch
          134 kB
          Jinglun
        3. HDFS-15294.002.patch
          134 kB
          Jinglun
        4. HDFS-15294.003.patch
          151 kB
          Jinglun
        5. BalanceProcedureScheduler.png
          47 kB
          Jinglun
        6. HDFS-15294.003.reupload.patch
          151 kB
          Jinglun
        7. HDFS-15294.004.patch
          152 kB
          Jinglun
        8. HDFS-15294.005.patch
          153 kB
          Jinglun
        9. HDFS-15294.006.patch
          154 kB
          Jinglun
        10. HDFS-15294.007.patch
          154 kB
          Jinglun
        11. distcp-balance.v2.pdf
          99 kB
          Jinglun

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16087 RBF balance process is stuck at DisableWrite stage

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-16116 Fix Hadoop federationBanance markdown bug.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. HDFS-15087 RBF: Balance/Rename across federation namespaces

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HDFS-15747 RBF: Rename across sub-namespaces.

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-13123 RBF: Add a balancer tool to move data across subcluster

          • Major - Major loss of function.
          • Patch Available

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-15662 Complete the javadoc of hadoop-federation-balance.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Patch Available
          links to

          Web Link Design doc

          Activity

            People

              LiJinglun Jinglun
              LiJinglun Jinglun
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m