Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12319

DirectoryScanner will throw IllegalStateException when Multiple BP's are present

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: None
    • Labels:
      None

      Description

      Scenario:
      Configure "dfs.datanode.directoryscan.interval" as 60 and start federated cluster atleast with two nameservices.

      2017-08-18 19:06:37,150 [java.util.concurrent.ThreadPoolExecutor$Worker@37d68b4e[State = -1, empty queue]] ERROR datanode.DirectoryScanner (DirectoryScanner.java:getDiskReport(551)) - Error compiling report for the volume, StorageId: DS-258b5e16-caa3-48c8-a0c8-b16934eb8a0c
      java.util.concurrent.ExecutionException: java.lang.IllegalStateException: StopWatch is already running
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:542)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:392)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:373)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:318)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalStateException: StopWatch is already running
      	at org.apache.hadoop.util.StopWatch.start(StopWatch.java:49)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:612)
      	at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner$ReportCompiler.call(DirectoryScanner.java:579)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	... 3 more
      
      1. HDFS-12319-001.patch
        4 kB
        Brahma Reddy Battula
      2. HDFS-12319-002.patch
        5 kB
        Brahma Reddy Battula
      3. TestCase_to_Reproduce.patch
        2 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Arpit Agarwal thanks for review and commit. Vinayakumar B thanks for additional review.

          Committed this through to branch-2.8

          Even you committed to branch-2.8.2,looks you forgot to mention.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Arpit Agarwal thanks for review and commit. Vinayakumar B thanks for additional review. Committed this through to branch-2.8 Even you committed to branch-2.8.2 ,looks you forgot to mention.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Committed this through to branch-2.8. For branch-2.8 I also pulled in HDFS-7764 to resolve merge conflicts in the test case.

          Thanks all.

          Show
          arpitagarwal Arpit Agarwal added a comment - Committed this through to branch-2.8. For branch-2.8 I also pulled in HDFS-7764 to resolve merge conflicts in the test case. Thanks all.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12242 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12242/)
          HDFS-12319. DirectoryScanner will throw IllegalStateException when (arp: rev 4b2c442d4e34f4708fa2ca442208427ca10798c1)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12242 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12242/ ) HDFS-12319 . DirectoryScanner will throw IllegalStateException when (arp: rev 4b2c442d4e34f4708fa2ca442208427ca10798c1) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          Hide
          djp Junping Du added a comment -

          Got it. Thanks Arpit Agarwal for confirm. I will hold on 2.8.2 RC0 for this.

          Show
          djp Junping Du added a comment - Got it. Thanks Arpit Agarwal for confirm. I will hold on 2.8.2 RC0 for this.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Straightforward fix. +1 for the v2 patch. Thank you for finding and fixing this Brahma. Not committing today in case Vinayakumar B has additional comments.

          Hi Junping Du, yes this should block 2.8.2. We should be able to commit it by tomorrow.

          Show
          arpitagarwal Arpit Agarwal added a comment - Straightforward fix. +1 for the v2 patch. Thank you for finding and fixing this Brahma . Not committing today in case Vinayakumar B has additional comments. Hi Junping Du , yes this should block 2.8.2. We should be able to commit it by tomorrow.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 14m 13s trunk passed
          +1 compile 0m 50s trunk passed
          +1 checkstyle 0m 38s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 41s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 46s the patch passed
          +1 javac 0m 46s the patch passed
          +1 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 74 unchanged - 4 fixed = 74 total (was 78)
          +1 mvnsite 0m 51s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 38s the patch passed
                Other Tests
          -1 unit 72m 11s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          98m 23s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingReconstruction
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-12319
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883507/HDFS-12319-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 93e83ad31451 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3123895
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20841/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20841/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 13s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 0m 54s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 41s trunk passed       Patch Compile Tests +1 mvninstall 0m 49s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed +1 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 74 unchanged - 4 fixed = 74 total (was 78) +1 mvnsite 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 38s the patch passed       Other Tests -1 unit 72m 11s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 98m 23s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingReconstruction   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12319 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883507/HDFS-12319-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 93e83ad31451 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3123895 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20841/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20841/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20841/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Vinayakumar B thanks for taking look.

          In Test, scanner.reconcile(); is called multiple times

          Refactored now..Uploaded the patch.Kindly Review.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Vinayakumar B thanks for taking look. In Test, scanner.reconcile(); is called multiple times Refactored now..Uploaded the patch.Kindly Review.
          Hide
          vinayrpet Vinayakumar B added a comment -

          perfTimer and throttleTimer should start per volume not per BP since both the BP's are under same dir.

          I think you are right.
          Source changes looks good to me.
          In Test, scanner.reconcile(); is called multiple times. Once in test and inside scan(..) for each blockpool. It would be better to refactor the test code, so that scan done once by calling scanner.reconsile() and verification for each blockbool'f diff and stats done later on scan results.

          +1 once done

          Show
          vinayrpet Vinayakumar B added a comment - perfTimer and throttleTimer should start per volume not per BP since both the BP's are under same dir. I think you are right. Source changes looks good to me. In Test, scanner.reconcile(); is called multiple times. Once in test and inside scan(..) for each blockpool. It would be better to refactor the test code, so that scan done once by calling scanner.reconsile() and verification for each blockbool'f diff and stats done later on scan results. +1 once done
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Arpit Agarwal can you please review..? perfTimer and throttleTimer should start per volume not per BP since both the BP's are under same dir.
          Please correct me If I am wrong.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Arpit Agarwal can you please review..? perfTimer and throttleTimer should start per volume not per BP since both the BP's are under same dir. Please correct me If I am wrong.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          IMO, this should be blocker for 2.8.2 release since Directory scanner might not work in federated cluster.

          Show
          brahmareddy Brahma Reddy Battula added a comment - IMO, this should be blocker for 2.8.2 release since Directory scanner might not work in federated cluster.
          Hide
          djp Junping Du added a comment -

          Hi Arpit Agarwal, can you confirm that if it should be blocker for 2.8.2? I am planning to kick off a RC within this week.

          Show
          djp Junping Du added a comment - Hi Arpit Agarwal , can you confirm that if it should be blocker for 2.8.2? I am planning to kick off a RC within this week.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 17m 8s trunk passed
          +1 compile 1m 7s trunk passed
          +1 checkstyle 0m 46s trunk passed
          +1 mvnsite 1m 14s trunk passed
          +1 findbugs 2m 20s trunk passed
          +1 javadoc 0m 52s trunk passed
                Patch Compile Tests
          +1 mvninstall 1m 14s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          -0 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 78 unchanged - 0 fixed = 79 total (was 78)
          +1 mvnsite 0m 51s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 39s the patch passed
                Other Tests
          -1 unit 74m 24s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          105m 42s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.TestReconstructStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-12319
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882857/HDFS-12319-001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 04c828989e67 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 267e19a
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20781/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20781/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20781/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20781/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 17m 8s trunk passed +1 compile 1m 7s trunk passed +1 checkstyle 0m 46s trunk passed +1 mvnsite 1m 14s trunk passed +1 findbugs 2m 20s trunk passed +1 javadoc 0m 52s trunk passed       Patch Compile Tests +1 mvninstall 1m 14s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed -0 checkstyle 0m 33s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 78 unchanged - 0 fixed = 79 total (was 78) +1 mvnsite 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 39s the patch passed       Other Tests -1 unit 74m 24s hadoop-hdfs in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 105m 42s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestReconstructStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12319 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882857/HDFS-12319-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 04c828989e67 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 267e19a Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/20781/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/20781/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20781/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20781/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Uploaded the patch.Kindly Review.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Uploaded the patch.Kindly Review.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Arpit Agarwal thanks for taking look

          I haven't tried out multiple nameservices in a while but if this is true it should be a 3.0 beta blocker.

          Attached the test-case,it's straight scenario. Even it's blocker for 2.8.2..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Arpit Agarwal thanks for taking look I haven't tried out multiple nameservices in a while but if this is true it should be a 3.0 beta blocker. Attached the test-case,it's straight scenario. Even it's blocker for 2.8.2 ..?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for reporting this Brahma Reddy Battula.

          I haven't tried out multiple nameservices in a while but if this is true it should be a 3.0 beta blocker.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for reporting this Brahma Reddy Battula . I haven't tried out multiple nameservices in a while but if this is true it should be a 3.0 beta blocker.

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              brahmareddy Brahma Reddy Battula
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development