Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5517

Lower the default maximum number of blocks per file

    Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      The default value of "dfs.namenode.fs-limits.max-blocks-per-file" has been reduced from 1M to 10K.

      Description

      We introduced the maximum number of blocks per file in HDFS-4305, but we set the default to 1MM. In practice this limit is so high as to never be hit, whereas we know that an individual file with 10s of thousands of blocks can cause problems. We should lower the default value, in my opinion to 10k.

        Attachments

        1. HDFS-5517.003.patch
          4 kB
          Andrew Wang
        2. HDFS-5517.002.patch
          2 kB
          Andrew Wang
        3. HDFS-5517.patch
          2 kB
          Aaron T. Myers

          Activity

            People

            • Assignee:
              atm Aaron T. Myers
              Reporter:
              atm Aaron T. Myers
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: