Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11466

Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
    • Component/s: namenode
    • Labels:
      None

      Description

      Per discussion on HDFS-10798, it might make sense to change the default value for long write lock holds to 5000ms like the read threshold, to avoid spamming the log.

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        Trivial patch attached.

        Show
        andrew.wang Andrew Wang added a comment - Trivial patch attached.
        Hide
        zhz Zhe Zhang added a comment -

        Thanks for the patch Andrew. Quick question, did you check that the old default has not been included in any releases?

        Show
        zhz Zhe Zhang added a comment - Thanks for the patch Andrew. Quick question, did you check that the old default has not been included in any releases?
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Zhe, so far it hasn't been released, though 2.8.0 is close to release. I think there's still time to sneak in this simple change if you think it's important to get this done before it gets released, though I don't consider this change incompatible.

        Ping Junping Du for awareness.

        Show
        andrew.wang Andrew Wang added a comment - Hi Zhe, so far it hasn't been released, though 2.8.0 is close to release. I think there's still time to sneak in this simple change if you think it's important to get this done before it gets released, though I don't consider this change incompatible. Ping Junping Du for awareness.
        Hide
        djp Junping Du added a comment -

        Thanks Andrew Wang for pinging me on this. Yes. 2.8.0 is pending on HADOOP-13866 decision which should get resolved soon. About patch here, if it is really important (seems unlikely at first glance), we can get it to 2.8.0. Otherwise, my suggestion is better to leave it to 2.8.1.

        Show
        djp Junping Du added a comment - Thanks Andrew Wang for pinging me on this. Yes. 2.8.0 is pending on HADOOP-13866 decision which should get resolved soon. About patch here, if it is really important (seems unlikely at first glance), we can get it to 2.8.0. Otherwise, my suggestion is better to leave it to 2.8.1.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 1s trunk passed
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 36s trunk passed
        +1 mvnsite 0m 54s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 46s trunk passed
        +1 javadoc 0m 40s trunk passed
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 41s the patch passed
        +1 javac 0m 41s the patch passed
        +1 checkstyle 0m 32s the patch passed
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 50s the patch passed
        +1 javadoc 0m 37s the patch passed
        -1 unit 64m 51s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        90m 9s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11466
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855022/HDFS-11466.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 3713ab1f93f0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5f5b031
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18463/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18463/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 1s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 64m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 90m 9s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11466 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855022/HDFS-11466.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 3713ab1f93f0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5f5b031 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18463/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18463/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18463/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhz Zhe Zhang added a comment -

        +1; I just committed the patch to trunk, branch-2, branch-2.8, and branch-2.7. Thanks Andrew for the fix.

        Show
        zhz Zhe Zhang added a comment - +1; I just committed the patch to trunk, branch-2, branch-2.8, and branch-2.7. Thanks Andrew for the fix.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11319 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11319/)
        HDFS-11466. Change dfs.namenode.write-lock-reporting-threshold-ms (zhz: rev d269b488a71a158d3ddcbdea96992abe29725c69)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11319 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11319/ ) HDFS-11466 . Change dfs.namenode.write-lock-reporting-threshold-ms (zhz: rev d269b488a71a158d3ddcbdea96992abe29725c69) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks Zhe for the quick review and commit!

        Show
        andrew.wang Andrew Wang added a comment - Thanks Zhe for the quick review and commit!
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          People

          • Assignee:
            andrew.wang Andrew Wang
            Reporter:
            andrew.wang Andrew Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development