Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5517

Lower the default maximum number of blocks per file

    Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      The default value of "dfs.namenode.fs-limits.max-blocks-per-file" has been reduced from 1M to 10K.

      Description

      We introduced the maximum number of blocks per file in HDFS-4305, but we set the default to 1MM. In practice this limit is so high as to never be hit, whereas we know that an individual file with 10s of thousands of blocks can cause problems. We should lower the default value, in my opinion to 10k.

      1. HDFS-5517.002.patch
        2 kB
        Andrew Wang
      2. HDFS-5517.003.patch
        4 kB
        Andrew Wang
      3. HDFS-5517.patch
        2 kB
        Aaron T. Myers

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10921 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10921/)
        HDFS-5517. Lower the default maximum number of blocks per file. (wang: rev 7226a71b1f684f562bd88ee121f1dd7aa8b73816)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10921 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10921/ ) HDFS-5517 . Lower the default maximum number of blocks per file. (wang: rev 7226a71b1f684f562bd88ee121f1dd7aa8b73816) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed, thanks for the review Akira and sorry again for missing the broken tests earlier.

        Show
        andrew.wang Andrew Wang added a comment - Committed, thanks for the review Akira and sorry again for missing the broken tests earlier.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        LGTM, +1. Thanks Andrew.

        Show
        ajisakaa Akira Ajisaka added a comment - LGTM, +1. Thanks Andrew.
        Hide
        andrew.wang Andrew Wang added a comment -

        Checkstyles are ones I'm apt to ignore (line length in DFSConfigKeys, method length in existing test). Test failure is due to a port conflict, and not one of the ones that failed last time.

        Show
        andrew.wang Andrew Wang added a comment - Checkstyles are ones I'm apt to ignore (line length in DFSConfigKeys, method length in existing test). Test failure is due to a port conflict, and not one of the ones that failed last time.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 9m 39s trunk passed
        +1 compile 0m 59s trunk passed
        +1 checkstyle 0m 37s trunk passed
        +1 mvnsite 1m 10s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 2m 1s trunk passed
        +1 javadoc 0m 48s trunk passed
        +1 mvninstall 1m 0s the patch passed
        +1 compile 0m 56s the patch passed
        +1 javac 0m 56s the patch passed
        -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 455 unchanged - 3 fixed = 457 total (was 458)
        +1 mvnsite 1m 5s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 2m 16s the patch passed
        +1 javadoc 0m 43s the patch passed
        -1 unit 77m 14s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 28s The patch does not generate ASF License warnings.
        101m 52s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestMaintenanceState



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-5517
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840947/HDFS-5517.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 9f5f06870799 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / dd9a96c
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17704/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17704/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17704/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 9m 39s trunk passed +1 compile 0m 59s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 1m 10s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 1m 0s the patch passed +1 compile 0m 56s the patch passed +1 javac 0m 56s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 455 unchanged - 3 fixed = 457 total (was 458) +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 16s the patch passed +1 javadoc 0m 43s the patch passed -1 unit 77m 14s hadoop-hdfs in the patch failed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 101m 52s Reason Tests Failed junit tests hadoop.hdfs.TestMaintenanceState Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-5517 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840947/HDFS-5517.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 9f5f06870799 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dd9a96c Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17704/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17704/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17704/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17704/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        New patch to address the two unit test issues.

        Show
        andrew.wang Andrew Wang added a comment - New patch to address the two unit test issues.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10907 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10907/)
        Revert "HDFS-5517. Lower the default maximum number of blocks per file. (wang: rev b407d5319597f01ebd51736e299239022bd08028)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10907 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10907/ ) Revert " HDFS-5517 . Lower the default maximum number of blocks per file. (wang: rev b407d5319597f01ebd51736e299239022bd08028) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Hide
        andrew.wang Andrew Wang added a comment -

        Sorry about the fuss, reverted for now.

        Show
        andrew.wang Andrew Wang added a comment - Sorry about the fuss, reverted for now.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        -1 unit 63m 3s hadoop-hdfs in the patch failed.

        The failure looks related to this patch. Hi Andrew Wang and Aaron T. Myers, would you fix it?

        Show
        ajisakaa Akira Ajisaka added a comment - -1 unit 63m 3s hadoop-hdfs in the patch failed. The failure looks related to this patch. Hi Andrew Wang and Aaron T. Myers , would you fix it?
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10904 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10904/)
        HDFS-5517. Lower the default maximum number of blocks per file. (wang: rev 09451252fae90a3ec192b8d7f0c49508df65e8c5)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10904 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10904/ ) HDFS-5517 . Lower the default maximum number of blocks per file. (wang: rev 09451252fae90a3ec192b8d7f0c49508df65e8c5) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        andrew.wang Andrew Wang added a comment -

        Since this was a trivial rebase, I'm committing this based on Uma and Vinay's previous +1s. Thanks all!

        Show
        andrew.wang Andrew Wang added a comment - Since this was a trivial rebase, I'm committing this based on Uma and Vinay's previous +1s. Thanks all!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 1s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 52s trunk passed
        +1 compile 0m 45s trunk passed
        +1 checkstyle 0m 31s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 41s trunk passed
        +1 javadoc 0m 39s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 398 unchanged - 1 fixed = 399 total (was 399)
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 43s the patch passed
        +1 javadoc 0m 36s the patch passed
        -1 unit 63m 3s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        81m 28s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
          hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-5517
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840722/HDFS-5517.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 8dcf2ac8bc4a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a2b1ff0
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17680/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17680/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17680/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17680/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 1s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 52s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 398 unchanged - 1 fixed = 399 total (was 399) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 43s the patch passed +1 javadoc 0m 36s the patch passed -1 unit 63m 3s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 81m 28s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-5517 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840722/HDFS-5517.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8dcf2ac8bc4a 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a2b1ff0 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17680/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17680/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17680/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17680/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        I did the trival rebase for this change, patch attached.

        Show
        andrew.wang Andrew Wang added a comment - I did the trival rebase for this change, patch attached.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 7s HDFS-5517 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue HDFS-5517
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12614116/HDFS-5517.patch
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17189/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HDFS-5517 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-5517 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12614116/HDFS-5517.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17189/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        I'm also updating the target version, since strictly speaking we shouldn't commit incompat changes to branch-2.

        Show
        andrew.wang Andrew Wang added a comment - I'm also updating the target version, since strictly speaking we shouldn't commit incompat changes to branch-2.
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Aaron T. Myers, looks like this patch has gone stale. Do you want to update it for inclusion in 3.0.0?

        Show
        andrew.wang Andrew Wang added a comment - Hi Aaron T. Myers , looks like this patch has gone stale. Do you want to update it for inclusion in 3.0.0?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12614116/HDFS-5517.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / d3797f9
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11542/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12614116/HDFS-5517.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d3797f9 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11542/console This message was automatically generated.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Moving bugs out of previously closed releases into the next minor release 2.8.0.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Moving bugs out of previously closed releases into the next minor release 2.8.0.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12614116/HDFS-5517.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / f1a152c
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10579/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12614116/HDFS-5517.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10579/console This message was automatically generated.
        Hide
        vinayrpet Vinayakumar B added a comment -

        +1 ATM. Thanks for Jira and patch.

        Show
        vinayrpet Vinayakumar B added a comment - +1 ATM. Thanks for Jira and patch.
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 thanks Aaron.

        Show
        andrew.wang Andrew Wang added a comment - +1 thanks Aaron.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12614116/HDFS-5517.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5449//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5449//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614116/HDFS-5517.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5449//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5449//console This message is automatically generated.
        Hide
        atm Aaron T. Myers added a comment -

        Thanks a lot for the support, Uma. Here's a little patch which just changes the default from 1MM to 10,000.

        Show
        atm Aaron T. Myers added a comment - Thanks a lot for the support, Uma. Here's a little patch which just changes the default from 1MM to 10,000.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        +1 for lowering the value. Thanks Aaron for proposing this change. Recently we also faced this issue. When file having 2L blocks itself NN is behaving something abnormal.
        For every block addition we do update blocks, so this big array will be persisted every time incrementally I think. This will be overhead really, if allow to write more and more blocks for same file with the current code But in practice I am not sure what will be the scenario to configure less block sizes and more blocks for same file. But we saw this in testing cluster.

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - +1 for lowering the value. Thanks Aaron for proposing this change. Recently we also faced this issue. When file having 2L blocks itself NN is behaving something abnormal. For every block addition we do update blocks, so this big array will be persisted every time incrementally I think. This will be overhead really, if allow to write more and more blocks for same file with the current code But in practice I am not sure what will be the scenario to configure less block sizes and more blocks for same file. But we saw this in testing cluster.

          People

          • Assignee:
            atm Aaron T. Myers
            Reporter:
            atm Aaron T. Myers
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development