Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There are a number of tickets (HDFS-10742, HDFS-10817, HDFS-10713, this subtask's story HDFS-10475) which are adding/improving logging/metrics around the FSNamesystemLock. All of this is done in FSNamesystem right now, which is polluting the namesystem with ThreadLocal variables, timing counters, etc. which are only relevant to the lock itself and the number of these increases as the logging/metrics become more sophisticated. It would be best to move these all into FSNamesystemLock to keep the metrics/logging tied directly to the item of interest.

      1. HDFS-10896.000.patch
        33 kB
        Erik Krogen
      2. HDFS-10896.001.patch
        32 kB
        Erik Krogen
      3. HDFS-10896.002.patch
        45 kB
        Erik Krogen
      4. HDFS-10896.003.patch
        45 kB
        Erik Krogen
      5. HDFS-10896.004.patch
        43 kB
        Erik Krogen
      6. HDFS-10896-branch-2.7.004.patch
        44 kB
        Erik Krogen

        Issue Links

          Activity

          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik for opening the JIRA. +1 on the idea.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik for opening the JIRA. +1 on the idea.
          Hide
          xkrogen Erik Krogen added a comment -

          Attaching v000 patch. Note that this will conflict heavily with HDFS-10713.

          Show
          xkrogen Erik Krogen added a comment - Attaching v000 patch. Note that this will conflict heavily with HDFS-10713 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HDFS-10896 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HDFS-10896
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830121/HDFS-10896.000.patch
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16868/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HDFS-10896 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HDFS-10896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830121/HDFS-10896.000.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16868/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment -

          Rebased for patch v001.

          Show
          xkrogen Erik Krogen added a comment - Rebased for patch v001.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 35s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 33s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 51s trunk passed
          +1 javadoc 1m 4s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 48s the patch passed
          +1 javac 0m 48s the patch passed
          -0 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 6 fixed = 587 total (was 591)
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 1m 0s the patch passed
          -1 unit 59m 24s hadoop-hdfs in the patch failed.
          -1 asflicense 0m 17s The patch generated 1 ASF License warnings.
          80m 45s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Format string should use %n rather than n in org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.readUnlock() At FSNamesystemLock.java:rather than n in org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.readUnlock() At FSNamesystemLock.java:[line 93]
            Format string should use %n rather than n in org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock() At FSNamesystemLock.java:rather than n in org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock() At FSNamesystemLock.java:[line 123]
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
            hadoop.tools.TestHdfsConfigFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10896
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830403/HDFS-10896.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 87e203bc340e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8e06d86
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16871/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16871/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 35s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 51s trunk passed +1 javadoc 1m 4s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 48s the patch passed +1 javac 0m 48s the patch passed -0 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 6 fixed = 587 total (was 591) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 findbugs 2m 3s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 1m 0s the patch passed -1 unit 59m 24s hadoop-hdfs in the patch failed. -1 asflicense 0m 17s The patch generated 1 ASF License warnings. 80m 45s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Format string should use %n rather than n in org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.readUnlock() At FSNamesystemLock.java:rather than n in org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.readUnlock() At FSNamesystemLock.java: [line 93]   Format string should use %n rather than n in org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock() At FSNamesystemLock.java:rather than n in org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock() At FSNamesystemLock.java: [line 123] Failed junit tests hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA   hadoop.tools.TestHdfsConfigFields Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830403/HDFS-10896.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 87e203bc340e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8e06d86 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16871/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16871/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16871/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xkrogen Erik Krogen added a comment - - edited

          Attaching v002 patch that rebases on top of HDFS-10713 and also fixes the Findbugs / checkstyle / whitespace / unit test / asflicense issues

          Show
          xkrogen Erik Krogen added a comment - - edited Attaching v002 patch that rebases on top of HDFS-10713 and also fixes the Findbugs / checkstyle / whitespace / unit test / asflicense issues
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 21s trunk passed
          +1 compile 0m 51s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 1m 0s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 4 fixed = 587 total (was 589)
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 0m 58s the patch passed
          +1 unit 65m 49s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          88m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10896
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830542/HDFS-10896.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 4314dabe7a6c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 875062b
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16883/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16883/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16883/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 21s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 1m 0s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 585 unchanged - 4 fixed = 587 total (was 589) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 0s the patch passed +1 javadoc 0m 58s the patch passed +1 unit 65m 49s hadoop-hdfs in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 88m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830542/HDFS-10896.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4314dabe7a6c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 875062b Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16883/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16883/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16883/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          The patch is mostly moving code to the new classes.

          +1 pending the checkstyle fix.

          Thanks Erik Krogen

          Show
          eddyxu Lei (Eddy) Xu added a comment - The patch is mostly moving code to the new classes. +1 pending the checkstyle fix. Thanks Erik Krogen
          Hide
          xkrogen Erik Krogen added a comment -

          Attached v003 patch with checkstyle fix. Thanks for the review, Lei (Eddy) Xu.

          Show
          xkrogen Erik Krogen added a comment - Attached v003 patch with checkstyle fix. Thanks for the review, Lei (Eddy) Xu .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 12s trunk passed
          +1 compile 1m 2s trunk passed
          +1 checkstyle 0m 38s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 1m 1s trunk passed
          +1 mvninstall 0m 53s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 585 unchanged - 4 fixed = 586 total (was 589)
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 9s the patch passed
          +1 javadoc 0m 58s the patch passed
          -1 unit 60m 24s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          82m 45s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSShell
            hadoop.hdfs.TestDistributedFileSystem
            hadoop.hdfs.server.mover.TestStorageMover



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10896
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830754/HDFS-10896.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 4e136ba819a8 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e19b37e
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16914/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16914/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16914/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16914/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 12s trunk passed +1 compile 1m 2s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 1s trunk passed +1 mvninstall 0m 53s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 31s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 585 unchanged - 4 fixed = 586 total (was 589) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 9s the patch passed +1 javadoc 0m 58s the patch passed -1 unit 60m 24s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 82m 45s Reason Tests Failed junit tests hadoop.hdfs.TestDFSShell   hadoop.hdfs.TestDistributedFileSystem   hadoop.hdfs.server.mover.TestStorageMover Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830754/HDFS-10896.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4e136ba819a8 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e19b37e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16914/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16914/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16914/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16914/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hanishakoneru Hanisha Koneru added a comment -

          +1 for v3 patch.
          Thanks Erik Krogen for including changes from HDFS-10713.

          Show
          hanishakoneru Hanisha Koneru added a comment - +1 for v3 patch. Thanks Erik Krogen for including changes from HDFS-10713 .
          Hide
          xkrogen Erik Krogen added a comment -

          After looking at HDFS-5239 it turns out the lack of documentation for dfs.namenode.fslock.fair was intentional. Attaching v004 patch which leaves it undocumented.

          Show
          xkrogen Erik Krogen added a comment - After looking at HDFS-5239 it turns out the lack of documentation for dfs.namenode.fslock.fair was intentional. Attaching v004 patch which leaves it undocumented.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik for the work and Lei (Eddy) Xu, Hanisha Koneru for the review.

          I think the dfs.namenode.fslock.fair should be separated out because it was intentionally hard-coded in HDFS-5239. +1 pending this change.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik for the work and Lei (Eddy) Xu , Hanisha Koneru for the review. I think the dfs.namenode.fslock.fair should be separated out because it was intentionally hard-coded in HDFS-5239 . +1 pending this change.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik for the update. One last nit:

            FSNamesystemLock(Configuration conf) {
              this(conf, new Timer());
            }
          
            @VisibleForTesting
            FSNamesystemLock(Configuration conf, Timer timer) {
          

          I guess we can simplify the logic by removing the constructor with Timer and just always initializing Timer in the constructor?

          Show
          zhz Zhe Zhang added a comment - Thanks Erik for the update. One last nit: FSNamesystemLock(Configuration conf) { this (conf, new Timer()); } @VisibleForTesting FSNamesystemLock(Configuration conf, Timer timer) { I guess we can simplify the logic by removing the constructor with Timer and just always initializing Timer in the constructor?
          Hide
          xkrogen Erik Krogen added a comment -

          The constructor which accepts a Timer parameter is used for testing so that you can substitute a Timer that does not actually follow wall-clock time (note @VisibleForTesting)

          Show
          xkrogen Erik Krogen added a comment - The constructor which accepts a Timer parameter is used for testing so that you can substitute a Timer that does not actually follow wall-clock time (note @VisibleForTesting )
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the clarification. Agreed we should keep it that way. +1 and will commit v4 patch after Jenkins.

          Show
          zhz Zhe Zhang added a comment - Thanks for the clarification. Agreed we should keep it that way. +1 and will commit v4 patch after Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 50s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 39s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 191 unchanged - 4 fixed = 192 total (was 195)
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 53s the patch passed
          -1 unit 56m 46s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          75m 44s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
            hadoop.hdfs.server.namenode.TestDecommissioningStatus



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10896
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831128/HDFS-10896.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux da3be1245d1d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0670149
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16946/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16946/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16946/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 50s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 39s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 191 unchanged - 4 fixed = 192 total (was 195) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 53s the patch passed -1 unit 56m 46s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 75m 44s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.server.namenode.TestDecommissioningStatus Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831128/HDFS-10896.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux da3be1245d1d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0670149 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16946/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16946/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16946/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16946/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10522 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10522/)
          HDFS-10896. Move lock logging logic from FSNamesystem into (zhz: rev 434c5ea75dc3d87513e49290ac9999148ff5163c)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemLock.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10522 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10522/ ) HDFS-10896 . Move lock logging logic from FSNamesystem into (zhz: rev 434c5ea75dc3d87513e49290ac9999148ff5163c) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemLock.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
          Hide
          zhz Zhe Zhang added a comment -

          I committed to trunk~branch-2.8. But the backport the branch-2.7 is pretty messy. Erik Krogen Do you mind posting a branch-2.7 patch? Or suggest additional backports to branch-2.7 to help resolve conflict?

          Show
          zhz Zhe Zhang added a comment - I committed to trunk~branch-2.8. But the backport the branch-2.7 is pretty messy. Erik Krogen Do you mind posting a branch-2.7 patch? Or suggest additional backports to branch-2.7 to help resolve conflict?
          Hide
          xkrogen Erik Krogen added a comment -

          Attaching branch-2.7 patch. Backport wasn't as messy as it seemed.

          Show
          xkrogen Erik Krogen added a comment - Attaching branch-2.7 patch. Backport wasn't as messy as it seemed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 49s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 1s branch-2.7 passed
          +1 compile 0m 57s branch-2.7 passed with JDK v1.8.0_101
          +1 compile 1m 0s branch-2.7 passed with JDK v1.7.0_111
          +1 checkstyle 0m 29s branch-2.7 passed
          +1 mvnsite 0m 59s branch-2.7 passed
          +1 mvneclipse 0m 17s branch-2.7 passed
          +1 findbugs 2m 57s branch-2.7 passed
          +1 javadoc 1m 0s branch-2.7 passed with JDK v1.8.0_101
          +1 javadoc 1m 41s branch-2.7 passed with JDK v1.7.0_111
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 56s the patch passed with JDK v1.8.0_101
          +1 javac 0m 56s the patch passed
          +1 compile 1m 1s the patch passed with JDK v1.7.0_111
          +1 javac 1m 1s the patch passed
          -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 361 unchanged - 10 fixed = 368 total (was 371)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 2532 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 1m 2s The patch 78 line(s) with tabs.
          +1 findbugs 3m 4s the patch passed
          +1 javadoc 0m 57s the patch passed with JDK v1.8.0_101
          +1 javadoc 1m 50s the patch passed with JDK v1.7.0_111
          -1 unit 44m 17s hadoop-hdfs in the patch failed with JDK v1.7.0_111.
          -1 asflicense 0m 25s The patch generated 3 ASF License warnings.
          132m 48s



          Reason Tests
          JDK v1.8.0_101 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.web.TestHttpsFileSystem
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
            hadoop.hdfs.TestRollingUpgrade
          JDK v1.7.0_111 Failed junit tests hadoop.hdfs.qjournal.TestSecureNNWithQJM
            hadoop.hdfs.web.TestHttpsFileSystem
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Issue HDFS-10896
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831440/HDFS-10896-branch-2.7.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5f2eec5373aa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / c08346e
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16989/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16989/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 49s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 1s branch-2.7 passed +1 compile 0m 57s branch-2.7 passed with JDK v1.8.0_101 +1 compile 1m 0s branch-2.7 passed with JDK v1.7.0_111 +1 checkstyle 0m 29s branch-2.7 passed +1 mvnsite 0m 59s branch-2.7 passed +1 mvneclipse 0m 17s branch-2.7 passed +1 findbugs 2m 57s branch-2.7 passed +1 javadoc 1m 0s branch-2.7 passed with JDK v1.8.0_101 +1 javadoc 1m 41s branch-2.7 passed with JDK v1.7.0_111 +1 mvninstall 0m 52s the patch passed +1 compile 0m 56s the patch passed with JDK v1.8.0_101 +1 javac 0m 56s the patch passed +1 compile 1m 1s the patch passed with JDK v1.7.0_111 +1 javac 1m 1s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 7 new + 361 unchanged - 10 fixed = 368 total (was 371) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 2532 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 1m 2s The patch 78 line(s) with tabs. +1 findbugs 3m 4s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_101 +1 javadoc 1m 50s the patch passed with JDK v1.7.0_111 -1 unit 44m 17s hadoop-hdfs in the patch failed with JDK v1.7.0_111. -1 asflicense 0m 25s The patch generated 3 ASF License warnings. 132m 48s Reason Tests JDK v1.8.0_101 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.TestRollingUpgrade JDK v1.7.0_111 Failed junit tests hadoop.hdfs.qjournal.TestSecureNNWithQJM   hadoop.hdfs.web.TestHttpsFileSystem   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-10896 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831440/HDFS-10896-branch-2.7.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5f2eec5373aa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / c08346e Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16989/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/16989/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16989/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik! +1 on the branch-2.7 patch. I just verified reported test failures and committed to branch-2.7.

          Show
          zhz Zhe Zhang added a comment - Thanks Erik! +1 on the branch-2.7 patch. I just verified reported test failures and committed to branch-2.7.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Hi Zhe Zhang

          Seems you are not updating in changes.txt for branch-2.7.I think,As we maintain changes.txt we need to update. Please see HADOOP-13670

          Show
          brahmareddy Brahma Reddy Battula added a comment - Hi Zhe Zhang Seems you are not updating in changes.txt for branch-2.7.I think,As we maintain changes.txt we need to update. Please see HADOOP-13670

            People

            • Assignee:
              xkrogen Erik Krogen
              Reporter:
              xkrogen Erik Krogen
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development