Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4305

Add a configurable limit on number of blocks per file, and min block size

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.4, 2.0.4-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      This change introduces a maximum number of blocks per file, by default one million, and a minimum block size, by default 1MB. These can optionally be changed via the configuration settings "dfs.namenode.fs-limits.max-blocks-per-file" and "dfs.namenode.fs-limits.min-block-size", respectively.
      Show
      This change introduces a maximum number of blocks per file, by default one million, and a minimum block size, by default 1MB. These can optionally be changed via the configuration settings "dfs.namenode.fs-limits.max-blocks-per-file" and "dfs.namenode.fs-limits.min-block-size", respectively.

      Description

      We recently had an issue where a user set the block size very very low and managed to create a single file with hundreds of thousands of blocks. This caused problems with the edit log since the OP_ADD op was so large (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To prevent users from making such mistakes, we should:

      • introduce a configurable minimum block size, below which requests are rejected
      • introduce a configurable maximum number of blocks per file, above which requests to add another block are rejected (with a suitably high default as to not prevent legitimate large files)
      1. hdfs-4305-1.patch
        7 kB
        Andrew Wang
      2. hdfs-4305-2.patch
        9 kB
        Andrew Wang
      3. hdfs-4305-3.patch
        9 kB
        Andrew Wang

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          +1. These seem like useful limits.

          Show
          Suresh Srinivas added a comment - +1. These seem like useful limits.
          Hide
          Konstantin Shvachko added a comment -

          Was it OP_ADD or OP_UPDATE_BLOCKS?

          Should we instead change OP_UPDATE_BLOCKS transaction to save only updated blocks rather than the entire set of file blocks.
          OP_CLOSE used to save all blocks since we persisted blocks only at the end of file. Now we don't.

          I also see that in trunk we persist all file blocks twice, first in dir.removeBlock() using OP_ADD, then immediately after that in dir.persistBlocks() using OP_UPDATE_BLOCKS.

          Show
          Konstantin Shvachko added a comment - Was it OP_ADD or OP_UPDATE_BLOCKS? Should we instead change OP_UPDATE_BLOCKS transaction to save only updated blocks rather than the entire set of file blocks. OP_CLOSE used to save all blocks since we persisted blocks only at the end of file. Now we don't. I also see that in trunk we persist all file blocks twice, first in dir.removeBlock() using OP_ADD, then immediately after that in dir.persistBlocks() using OP_UPDATE_BLOCKS.
          Hide
          Todd Lipcon added a comment -

          I believe in this case it was OP_ADD, but I'm not certain of that (I don't have a copy of the logs unfortunately).

          I agree we could probably optimize these further, but regardless, I think it makes sense to limit users to a "reasonable" number of blocks to prevent them from shooting themselves in the foot.

          Show
          Todd Lipcon added a comment - I believe in this case it was OP_ADD, but I'm not certain of that (I don't have a copy of the logs unfortunately). I agree we could probably optimize these further, but regardless, I think it makes sense to limit users to a "reasonable" number of blocks to prevent them from shooting themselves in the foot.
          Hide
          Andrew Wang added a comment -

          Took a hack at this, patch attached.

          I set the default to 0 for the min block size. Going non-zero I think would break a lot of tests since they set small block sizes, and would be an incompatible change.

          For the max number of blocks per file, I chose 1 million. I figure with the default 64MB block size, that's a 64TB file.

          Feedback appreciated, especially if these defaults should be altered.

          Show
          Andrew Wang added a comment - Took a hack at this, patch attached. I set the default to 0 for the min block size. Going non-zero I think would break a lot of tests since they set small block sizes, and would be an incompatible change. For the max number of blocks per file, I chose 1 million. I figure with the default 64MB block size, that's a 64TB file. Feedback appreciated, especially if these defaults should be altered.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12574943/hdfs-4305-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4135//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4135//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12574943/hdfs-4305-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4135//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4135//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          This looks good to me.

          One possible change: we can set the default min block size to 1MB or so, and then add a minimum of 0 to the hdfs-site.xml in src/test/resources to avoid having to rewrite a bunch of tests.

          It would be an incompatible change, but perhaps a good one. What do folks think?

          Show
          Todd Lipcon added a comment - This looks good to me. One possible change: we can set the default min block size to 1MB or so, and then add a minimum of 0 to the hdfs-site.xml in src/test/resources to avoid having to rewrite a bunch of tests. It would be an incompatible change, but perhaps a good one. What do folks think?
          Hide
          Suresh Srinivas added a comment -

          It would be an incompatible change, but perhaps a good one. What do folks think?

          +1 for adding limiting max number of blocks per file, even though it is incompatible.

          I am not sure if minimum block size is really required. I would rather make it a namenode WebUI status to say, your block size is way too small.

          Show
          Suresh Srinivas added a comment - It would be an incompatible change, but perhaps a good one. What do folks think? +1 for adding limiting max number of blocks per file, even though it is incompatible. I am not sure if minimum block size is really required. I would rather make it a namenode WebUI status to say, your block size is way too small.
          Hide
          Todd Lipcon added a comment -

          I am not sure if minimum block size is really required. I would rather make it a namenode WebUI status to say, your block size is way too small.

          What about for the case that just one or a few files have the small block size? You wouldn't want to put this on the NN web UI.

          The issue I've seen in the past is that some well-meaning but naive user wants to get their MR job to generate more splits. They don't know how to do this properly within MR, so instead they create the file with a tiny block size like 1KB, then are surprised when they have really bad performance, etc. Having some reasonable limit should help keep them from shooting themselves in the foot.

          Show
          Todd Lipcon added a comment - I am not sure if minimum block size is really required. I would rather make it a namenode WebUI status to say, your block size is way too small. What about for the case that just one or a few files have the small block size? You wouldn't want to put this on the NN web UI. The issue I've seen in the past is that some well-meaning but naive user wants to get their MR job to generate more splits. They don't know how to do this properly within MR, so instead they create the file with a tiny block size like 1KB, then are surprised when they have really bad performance, etc. Having some reasonable limit should help keep them from shooting themselves in the foot.
          Hide
          Suresh Srinivas added a comment -

          The change of adding max number of blocks per file, even though incompatible, is more likely not going to be an issue.

          What about for the case that just one or a few files have the small block size? You wouldn't want to put this on the NN web UI.

          Setting default block size smaller and having all the files affected by is a big issue. Is setting small block files for a few files such a big issue. One could just copy the files right?

          Show
          Suresh Srinivas added a comment - The change of adding max number of blocks per file, even though incompatible, is more likely not going to be an issue. What about for the case that just one or a few files have the small block size? You wouldn't want to put this on the NN web UI. Setting default block size smaller and having all the files affected by is a big issue. Is setting small block files for a few files such a big issue. One could just copy the files right?
          Hide
          Kihwal Lee added a comment -

          We had a user who accidentally set the block size to something small in his big job. That job alone added 10s of millions of blocks when it was still in early stage. The blocks/file was large but did not grow large enough to cause trouble (0.23 was without HDFS-4304) before taking an admin action, since the progress was so slow. The blocks were allocated and completed so frequently that the namenode couldn't keep up, let alone the pressure on the namespace and heap.

          In cases like this, max blocks per file may not be useful for detecting the issue, but enforcing min block size will help.

          Show
          Kihwal Lee added a comment - We had a user who accidentally set the block size to something small in his big job. That job alone added 10s of millions of blocks when it was still in early stage. The blocks/file was large but did not grow large enough to cause trouble (0.23 was without HDFS-4304 ) before taking an admin action, since the progress was so slow. The blocks were allocated and completed so frequently that the namenode couldn't keep up, let alone the pressure on the namespace and heap. In cases like this, max blocks per file may not be useful for detecting the issue, but enforcing min block size will help.
          Hide
          Suresh Srinivas added a comment -

          We had a user who accidentally set the block size to something small in his big job.

          Out of curiosity what was the size of the block?

          The blocks were allocated and completed so frequently that the namenode couldn't keep up

          This is very interesting. I would have thought given namenode allocates only waits for first block to be reported from a datanode before allocating third block, I thought block allocation would be slower and cannot overwhelm the namenode. Perhaps the users job was also creating a lot of files.

          I am reluctant about this change since this incompatible change will affect QA tests and integration tests. Thinking about it I am okay if by default the min block size is 0 and an admin sets up minimum block size in production deployment. Either admin sets it up as best practice or after an incident where they get impacted by a job choosing very small block size.

          Show
          Suresh Srinivas added a comment - We had a user who accidentally set the block size to something small in his big job. Out of curiosity what was the size of the block? The blocks were allocated and completed so frequently that the namenode couldn't keep up This is very interesting. I would have thought given namenode allocates only waits for first block to be reported from a datanode before allocating third block, I thought block allocation would be slower and cannot overwhelm the namenode. Perhaps the users job was also creating a lot of files. I am reluctant about this change since this incompatible change will affect QA tests and integration tests. Thinking about it I am okay if by default the min block size is 0 and an admin sets up minimum block size in production deployment. Either admin sets it up as best practice or after an incident where they get impacted by a job choosing very small block size.
          Hide
          Todd Lipcon added a comment -

          I am reluctant about this change since this incompatible change will affect QA tests and integration tests. Thinking about it I am okay if by default the min block size is 0 and an admin sets up minimum block size in production deployment. Either admin sets it up as best practice or after an incident where they get impacted by a job choosing very small block size.

          Why push the effort on users to configure it to a reasonable 'best practice' level, instead of pushing it on QA (hadoop experts) to set it to 0 in test clusters where they want to abuse block size to stress test the NN?

          I think we have too many defaults where we say "any reasonable production cluster should not leave this at the default". Sure, it helps vendors pimp our management tools (which would ship with the best practice defaults overriding whatever's in hdfs-default.xml), but I don't think it's in the best interest of the project if the setup instructions involve changing a bunch of things away from unsafe defaults.

          Similarly, waiting until after a production incident to make the user then go and learn about a better default is not very user-centric.

          Show
          Todd Lipcon added a comment - I am reluctant about this change since this incompatible change will affect QA tests and integration tests. Thinking about it I am okay if by default the min block size is 0 and an admin sets up minimum block size in production deployment. Either admin sets it up as best practice or after an incident where they get impacted by a job choosing very small block size. Why push the effort on users to configure it to a reasonable 'best practice' level, instead of pushing it on QA (hadoop experts) to set it to 0 in test clusters where they want to abuse block size to stress test the NN? I think we have too many defaults where we say "any reasonable production cluster should not leave this at the default". Sure, it helps vendors pimp our management tools (which would ship with the best practice defaults overriding whatever's in hdfs-default.xml), but I don't think it's in the best interest of the project if the setup instructions involve changing a bunch of things away from unsafe defaults. Similarly, waiting until after a production incident to make the user then go and learn about a better default is not very user-centric.
          Hide
          Suresh Srinivas added a comment -

          I don't think it's in the best interest of the project if the setup instructions involve changing a bunch of things away from unsafe defaults.

          Fair point.

          Similarly, waiting until after a production incident to make the user then go and learn about a better default is not very user-centric.

          How many incidents of this nature you have observed so far?

          Show
          Suresh Srinivas added a comment - I don't think it's in the best interest of the project if the setup instructions involve changing a bunch of things away from unsafe defaults. Fair point. Similarly, waiting until after a production incident to make the user then go and learn about a better default is not very user-centric. How many incidents of this nature you have observed so far?
          Hide
          Todd Lipcon added a comment -

          I've seen it twice among our customers, sounds like the Yahoo guys have seen it at least once, so that makes at least 3 times in the wild. As more new folks start using Hadoop, I'd imagine the incidence rate will only increase.

          Show
          Todd Lipcon added a comment - I've seen it twice among our customers, sounds like the Yahoo guys have seen it at least once, so that makes at least 3 times in the wild. As more new folks start using Hadoop, I'd imagine the incidence rate will only increase.
          Hide
          Andrew Wang added a comment -

          Thanks for the feedback everyone. Here's a new patch implementing Todd's suggestion of a 1MB min block size. I also added the new params to hdfs-default.xml, which I forgot to do before.

          Ran mvn test -Dtest=TestDistributedFileSystem,TestFileLimit,TestFileCreation,TestLargeBlock,TestSmallBlock successfully, so I think changing the test hdfs-site.xml works as expected.

          Show
          Andrew Wang added a comment - Thanks for the feedback everyone. Here's a new patch implementing Todd's suggestion of a 1MB min block size. I also added the new params to hdfs-default.xml, which I forgot to do before. Ran mvn test -Dtest=TestDistributedFileSystem,TestFileLimit,TestFileCreation,TestLargeBlock,TestSmallBlock successfully, so I think changing the test hdfs-site.xml works as expected.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12578293/hdfs-4305-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4226//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4226//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12578293/hdfs-4305-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4226//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4226//console This message is automatically generated.
          Hide
          Andrew Wang added a comment -

          I'll add that Todd's on vacation for the next two weeks, so if someone else wants to review this, that'd be great.

          Show
          Andrew Wang added a comment - I'll add that Todd's on vacation for the next two weeks, so if someone else wants to review this, that'd be great.
          Hide
          Aaron T. Myers added a comment -

          +1, the patch looks good to me. I'm going to commit this momentarily.

          Show
          Aaron T. Myers added a comment - +1, the patch looks good to me. I'm going to commit this momentarily.
          Hide
          Aaron T. Myers added a comment -

          Whoops, I spoke too soon. Looks like the patch no longer compiles on the latest trunk.

          Andrew, mind updating the patch? Sorry about this.

          Show
          Aaron T. Myers added a comment - Whoops, I spoke too soon. Looks like the patch no longer compiles on the latest trunk. Andrew, mind updating the patch? Sorry about this.
          Hide
          Suresh Srinivas added a comment -

          Given that the problems due to very small block is seen very few times, my preference is not to add a minimum size. That said, I will not block this change.

          Show
          Suresh Srinivas added a comment - Given that the problems due to very small block is seen very few times, my preference is not to add a minimum size. That said, I will not block this change.
          Hide
          Andrew Wang added a comment -

          Here's a rebased version of the patch.

          Show
          Andrew Wang added a comment - Here's a rebased version of the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580426/hdfs-4305-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestBlockReaderLocalLegacy

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4315//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4315//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580426/hdfs-4305-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestBlockReaderLocalLegacy +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4315//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4315//console This message is automatically generated.
          Hide
          Andrew Wang added a comment -

          I believe the test failure is unrelated. It's a null pointer in MiniDFSCluster#shutdown (so harmless), and it didn't happen when I ran the test a few times locally.

          Show
          Andrew Wang added a comment - I believe the test failure is unrelated. It's a null pointer in MiniDFSCluster#shutdown (so harmless), and it didn't happen when I ran the test a few times locally.
          Hide
          Aaron T. Myers added a comment -

          Thanks for confirming you're OK with this, Suresh.

          +1, the latest patch looks good to me. I agree that the test failure seems unrelated. I'm going to commit this momentarily.

          Show
          Aaron T. Myers added a comment - Thanks for confirming you're OK with this, Suresh. +1, the latest patch looks good to me. I agree that the test failure seems unrelated. I'm going to commit this momentarily.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3691/)
          HDFS-4305. Add a configurable limit on number of blocks per file, and min block size. Contributed by Andrew Wang. (Revision 1477354)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477354
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileLimit.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.xml
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3691/ ) HDFS-4305 . Add a configurable limit on number of blocks per file, and min block size. Contributed by Andrew Wang. (Revision 1477354) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477354 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileLimit.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.xml
          Hide
          Aaron T. Myers added a comment -

          I've just committed this to trunk and branch-2.

          Thanks a lot for the contribution, Andrew.

          Show
          Aaron T. Myers added a comment - I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Andrew.
          Hide
          Zhijie Shen added a comment -

          The patch seems to have broken M/R tests (see MAPREDUCE-5156 and MAPREDUCE-5157). The problem is related to the value of "dfs.namenode.fs-limits.min-block-size".

          https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3557//testReport/
          https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3558//testReport/

          Show
          Zhijie Shen added a comment - The patch seems to have broken M/R tests (see MAPREDUCE-5156 and MAPREDUCE-5157 ). The problem is related to the value of "dfs.namenode.fs-limits.min-block-size". https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3557//testReport/ https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3558//testReport/
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for the report, Zhijie. I've filed an MR JIRA to address this here: MAPREDUCE-5193. I'll upload a patch shortly.

          Show
          Aaron T. Myers added a comment - Thanks a lot for the report, Zhijie. I've filed an MR JIRA to address this here: MAPREDUCE-5193 . I'll upload a patch shortly.
          Hide
          Zhijie Shen added a comment -

          No problem. Thanks for your quick response

          Show
          Zhijie Shen added a comment - No problem. Thanks for your quick response
          Hide
          Suresh Srinivas added a comment -

          Aaron T. Myers Please mark this as incompatible change. Please also update Release note to describe the nature of incompatibility and how to get around it.

          Show
          Suresh Srinivas added a comment - Aaron T. Myers Please mark this as incompatible change. Please also update Release note to describe the nature of incompatibility and how to get around it.
          Hide
          Suresh Srinivas added a comment -

          One more comment, this should be in the incompatible section in CHANGES.txt.

          Show
          Suresh Srinivas added a comment - One more comment, this should be in the incompatible section in CHANGES.txt.
          Hide
          Aaron T. Myers added a comment -
          Show
          Aaron T. Myers added a comment - Suresh Srinivas Done.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3699 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3699/)
          Move the CHANGES.txt entry for HDFS-4305 to the incompatible changes section. (Revision 1477488)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477488
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3699 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3699/ ) Move the CHANGES.txt entry for HDFS-4305 to the incompatible changes section. (Revision 1477488) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477488 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #199 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/199/)
          Move the CHANGES.txt entry for HDFS-4305 to the incompatible changes section. (Revision 1477488)
          HDFS-4305. Add a configurable limit on number of blocks per file, and min block size. Contributed by Andrew Wang. (Revision 1477354)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477488
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477354
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileLimit.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #199 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/199/ ) Move the CHANGES.txt entry for HDFS-4305 to the incompatible changes section. (Revision 1477488) HDFS-4305 . Add a configurable limit on number of blocks per file, and min block size. Contributed by Andrew Wang. (Revision 1477354) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477488 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477354 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileLimit.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1388 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1388/)
          Move the CHANGES.txt entry for HDFS-4305 to the incompatible changes section. (Revision 1477488)
          HDFS-4305. Add a configurable limit on number of blocks per file, and min block size. Contributed by Andrew Wang. (Revision 1477354)

          Result = FAILURE
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477488
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477354
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileLimit.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1388 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1388/ ) Move the CHANGES.txt entry for HDFS-4305 to the incompatible changes section. (Revision 1477488) HDFS-4305 . Add a configurable limit on number of blocks per file, and min block size. Contributed by Andrew Wang. (Revision 1477354) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477488 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477354 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileLimit.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1415 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1415/)
          Move the CHANGES.txt entry for HDFS-4305 to the incompatible changes section. (Revision 1477488)
          HDFS-4305. Add a configurable limit on number of blocks per file, and min block size. Contributed by Andrew Wang. (Revision 1477354)

          Result = FAILURE
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477488
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477354
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileLimit.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1415 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1415/ ) Move the CHANGES.txt entry for HDFS-4305 to the incompatible changes section. (Revision 1477488) HDFS-4305 . Add a configurable limit on number of blocks per file, and min block size. Contributed by Andrew Wang. (Revision 1477354) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477488 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1477354 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileLimit.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hdfs-site.xml
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #173 (See https://builds.apache.org/job/hbase-0.95/173/)
          HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478204)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          Show
          Hudson added a comment - Integrated in hbase-0.95 #173 (See https://builds.apache.org/job/hbase-0.95/173/ ) HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478204) Result = FAILURE jmhsieh : Files : /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #517 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/517/)
          HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478203)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #517 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/517/ ) HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478203) Result = FAILURE jmhsieh : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #87 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/87/)
          HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478204)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #87 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/87/ ) HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478204) Result = FAILURE jmhsieh : Files : /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #4095 (See https://builds.apache.org/job/HBase-TRUNK/4095/)
          HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478368)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/trunk/hbase-server/src/test/resources/hdfs-site.xml
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #4095 (See https://builds.apache.org/job/HBase-TRUNK/4095/ ) HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478368) Result = FAILURE jmhsieh : Files : /hbase/trunk/hbase-server/src/test/resources/hdfs-site.xml
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95 #177 (See https://builds.apache.org/job/hbase-0.95/177/)
          HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478369)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/resources/hdfs-site.xml
          Show
          Hudson added a comment - Integrated in hbase-0.95 #177 (See https://builds.apache.org/job/hbase-0.95/177/ ) HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478369) Result = FAILURE jmhsieh : Files : /hbase/branches/0.95/hbase-server/src/test/resources/hdfs-site.xml
          Hide
          Hudson added a comment -

          Integrated in hbase-0.95-on-hadoop2 #89 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/89/)
          HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478369)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/branches/0.95/hbase-server/src/test/resources/hdfs-site.xml
          Show
          Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #89 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/89/ ) HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478369) Result = FAILURE jmhsieh : Files : /hbase/branches/0.95/hbase-server/src/test/resources/hdfs-site.xml
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #518 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/518/)
          HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478368)

          Result = FAILURE
          jmhsieh :
          Files :

          • /hbase/trunk/hbase-server/src/test/resources/hdfs-site.xml
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #518 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/518/ ) HBASE-8469 [hadoop2] Several tests break because of HDFS-4305 (Revision 1478368) Result = FAILURE jmhsieh : Files : /hbase/trunk/hbase-server/src/test/resources/hdfs-site.xml
          Hide
          Suresh Srinivas added a comment -

          See - https://builds.apache.org/job/Hadoop-Hdfs-trunk/1399/

          The following failures seem related to this jira.

          Running org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem
          Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.436 sec
          
          Results :
          
          Tests in error:
            testOperation[4](org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem): Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 1024 < 1048576(..)
            testOperationDoAs[4](org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem): Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 1024 < 1048576(..)
            testOperation[4](org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem): Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 1024 < 1048576(..)
            testOperationDoAs[4](org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem): Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 1024 < 1048576(..)
          
          Show
          Suresh Srinivas added a comment - See - https://builds.apache.org/job/Hadoop-Hdfs-trunk/1399/ The following failures seem related to this jira. Running org.apache.hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.436 sec Results : Tests in error: testOperation[4](org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem): Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 1024 < 1048576(..) testOperationDoAs[4](org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem): Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 1024 < 1048576(..) testOperation[4](org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem): Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 1024 < 1048576(..) testOperationDoAs[4](org.apache.hadoop.fs.http.client.TestHttpFSWithHttpFSFileSystem): Specified block size is less than configured minimum value (dfs.namenode.fs-limits.min-block-size): 1024 < 1048576(..)
          Hide
          Andrew Wang added a comment -

          Thanks for the finding this, Suresh. I put a tiny patch up on HDFS-4825 to fix this.

          Show
          Andrew Wang added a comment - Thanks for the finding this, Suresh. I put a tiny patch up on HDFS-4825 to fix this.

            People

            • Assignee:
              Andrew Wang
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development