Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15655

Add option to make balancer prefer to get cold blocks

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 3.4.0
    • balancer & mover
    • None
    • Reviewed

    Description

      We met two issues when using balancer.

      1. Moving hot files may cause failing of dfsclient reading.
      2. Some blocks of temporary files are moved and they are deleted soon.

      Add a config key  " dfs.namenode.hot.block.time.interval",  the balancer prefer to get the blocks which are belong to the cold files created before this time period.

      Also add a option "-hotBlockTimeInterval" to the command line of balancer for inputting.

      Attachments

        1. HDFS-15655.001.patch
          10 kB
          Yang Yun
        2. HDFS-15655.002.patch
          9 kB
          Yang Yun
        3. HDFS-15655.003.patch
          31 kB
          Yang Yun
        4. HDFS-15655.004.patch
          31 kB
          Yang Yun
        5. HDFS-15655.005.patch
          33 kB
          Yang Yun
        6. HDFS-15655.006.patch
          33 kB
          Yang Yun
        7. HDFS-15655.007.patch
          33 kB
          Yang Yun
        8. HDFS-15655.008.patch
          33 kB
          Yang Yun
        9. HDFS-15655.009.patch
          33 kB
          Yang Yun

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hadoop_yangyun Yang Yun
            hadoop_yangyun Yang Yun
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment