Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3570

Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Minor
    • Resolution: Unresolved
    • 2.0.0-alpha
    • None
    • balancer & mover
    • None

    Description

      Report from a user here: https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ, post archived at http://pastebin.com/eVFkk0A0

      This user had a specific DN that had a large non-DFS usage among dfs.data.dirs, and very little DFS usage (which is computed against total possible capacity).

      Balancer apparently only looks at the usage, and ignores to consider that non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a DFS Usage report from DN is 8% only, its got a lot of free space to write more blocks, when that isn't true as shown by the case of this user. It went on scheduling writes to the DN to balance it out, but the DN simply can't accept any more blocks as a result of its disks' state.

      I think it would be better if we computed the actual utilization based on (100-(actual remaining space))/(capacity), as opposed to the current (dfs used)/(capacity). Thoughts?

      This isn't very critical, however, cause it is very rare to see DN space being used for non DN data, but it does expose a valid bug.

      Attachments

        1. HDFS-3570.aash.1.patch
          3 kB
          Andrew Ash
        2. HDFS-3570.2.patch
          15 kB
          Akira Ajisaka
        3. HDFS-3570.003.patch
          5 kB
          Akira Ajisaka

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            aajisaka Akira Ajisaka
            qwertymaniac Harsh J

            Dates

              Created:
              Updated:

              Issue deployment