Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9619

SimulatedFSDataset sometimes can not find blockpool for the correct namenode

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode, test
    • Labels:
    • Environment:

      Jenkins

    • Hadoop Flags:
      Reviewed

      Description

      We sometimes see TestBalancerWithMultipleNameNodes.testBalancer failed to replicate a file, because a data node is excluded.

      File /tmp.txt could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
       at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1745)
       at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:299)
       at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2390)
       at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:797)
       at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:500)
       at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
       at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
       at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2305)
       at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2301)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:415)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1705)
       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2299)
      

      Relevent logs suggest root cause is due to block pool not found.

      2016-01-03 22:11:43,174 [DataXceiver for client DFSClient_NONMAPREDUCE_849671738_1 at /127.0.0.1:47318 [Receiving block BP-1927700312-172.26.2.1-1451887902222:blk_1073741825_1001]] ERROR datanode.DataNode (DataXceiver.java:run(280)) - host0.foo.com:49997:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:47318 dst: /127.0.0.1:49997
      java.io.IOException: Non existent blockpool BP-1927700312-172.26.2.1-1451887902222
      at org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset.getMap(SimulatedFSDataset.java:583)
      at org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset.createTemporary(SimulatedFSDataset.java:955)
      at org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset.createRbw(SimulatedFSDataset.java:941)
      at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:203)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver(DataXceiver.java:1235)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:678)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166)
      at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
      at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253)
      at java.lang.Thread.run(Thread.java:745)
      

      For a bit more context, this test starts a cluster with two name nodes and one data node. The block pools are added, but one of them is not found after added. The root cause is due to an undetected concurrent access in a hash map in SimulatedFSDataset (two block pools are added simultaneously). I added some logs to print blockMap, and saw a few ConcurrentModificationExceptions. The solution would be to use a thread safe class instead, like ConcurrentHashMap.

      1. HDFS-9619.001.patch
        1 kB
        Wei-Chiu Chuang
      2. HDFS-9619.002.patch
        4 kB
        Wei-Chiu Chuang

        Activity

        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thank you very much Vinayakumar B for the quick commit and reviews!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thank you very much Vinayakumar B for the quick commit and reviews!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9065 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9065/)
        HDFS-9619. SimulatedFSDataset sometimes can not find blockpool for the (vinayakumarb: rev 6702e7d66a4949a40bcdbd9f51ca1e746c8991e1)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9065 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9065/ ) HDFS-9619 . SimulatedFSDataset sometimes can not find blockpool for the (vinayakumarb: rev 6702e7d66a4949a40bcdbd9f51ca1e746c8991e1) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
        Hide
        vinayrpet Vinayakumar B added a comment -

        Committed to trunk, branch-2 and branch-2.8.
        Thanks Wei-Chiu Chuang.

        Show
        vinayrpet Vinayakumar B added a comment - Committed to trunk, branch-2 and branch-2.8. Thanks Wei-Chiu Chuang .
        Hide
        vinayrpet Vinayakumar B added a comment -

        +1, committing shortly

        Show
        vinayrpet Vinayakumar B added a comment - +1, committing shortly
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        The failed tests appear to be flaky ones, unrelated to this patch.

        Meanwhile, I ran TestBalancerWithMultipleNameNodes.testBalancer locally for more than 600 times so far without any failures.

        Show
        jojochuang Wei-Chiu Chuang added a comment - The failed tests appear to be flaky ones, unrelated to this patch. Meanwhile, I ran TestBalancerWithMultipleNameNodes.testBalancer locally for more than 600 times so far without any failures.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 36s trunk passed
        +1 compile 0m 40s trunk passed with JDK v1.8.0_66
        +1 compile 0m 41s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 52s trunk passed
        +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 46s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 35s the patch passed with JDK v1.8.0_66
        +1 javac 0m 35s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.7.0_91
        +1 javac 0m 38s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 59s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91
        -1 unit 54m 7s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        -1 unit 49m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 24s Patch does not generate ASF License warnings.
        130m 0s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.datanode.TestDataNodeMetrics
        JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780814/HDFS-9619.002.patch
        JIRA Issue HDFS-9619
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 54993fa1453d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 67c9780
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14048/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14048/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 36s trunk passed +1 compile 0m 40s trunk passed with JDK v1.8.0_66 +1 compile 0m 41s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 52s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 45s the patch passed +1 compile 0m 35s the patch passed with JDK v1.8.0_66 +1 javac 0m 35s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_91 +1 javac 0m 38s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 59s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_91 -1 unit 54m 7s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 49m 47s hadoop-hdfs in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 130m 0s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.datanode.TestDataNodeMetrics JDK v1.7.0_91 Failed junit tests hadoop.hdfs.server.namenode.TestNNThroughputBenchmark   hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780814/HDFS-9619.002.patch JIRA Issue HDFS-9619 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 54993fa1453d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67c9780 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14048/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14048/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14048/console This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        TestBlockReplacement.testBlockReplacement is a flaky test that often fails.
        TestBlockStoragePolicy.testChangeHotFileRep appears to be a flaky test too.

        Show
        jojochuang Wei-Chiu Chuang added a comment - TestBlockReplacement.testBlockReplacement is a flaky test that often fails. TestBlockStoragePolicy.testChangeHotFileRep appears to be a flaky test too.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 33s trunk passed
        +1 compile 0m 39s trunk passed with JDK v1.8.0_66
        +1 compile 0m 43s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 53s trunk passed
        +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 46s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 46s the patch passed
        +1 compile 0m 36s the patch passed with JDK v1.8.0_66
        +1 javac 0m 36s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_91
        +1 javac 0m 39s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 49s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 2s the patch passed
        +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 43s the patch passed with JDK v1.7.0_91
        -1 unit 53m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
        +1 unit 50m 30s hadoop-hdfs in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        130m 1s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.hdfs.TestBlockStoragePolicy



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780801/HDFS-9619.001.patch
        JIRA Issue HDFS-9619
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 30744bddc127 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b993668
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14045/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14045/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14045/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 75MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14045/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 33s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_66 +1 compile 0m 43s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 6s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 46s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 46s the patch passed +1 compile 0m 36s the patch passed with JDK v1.8.0_66 +1 javac 0m 36s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_91 +1 javac 0m 39s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 1m 5s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 43s the patch passed with JDK v1.7.0_91 -1 unit 53m 32s hadoop-hdfs in the patch failed with JDK v1.8.0_66. +1 unit 50m 30s hadoop-hdfs in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 130m 1s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.TestBlockStoragePolicy Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780801/HDFS-9619.001.patch JIRA Issue HDFS-9619 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 30744bddc127 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b993668 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14045/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14045/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14045/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14045/console This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Well, maybe the test case is not needed. It's pretty obvious there is a concurrency bug using HashMap without synchronized block.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Well, maybe the test case is not needed. It's pretty obvious there is a concurrency bug using HashMap without synchronized block.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Rev02: Added a test case.
        In this test case TestSimulatedFSDataset.testConcurrentAddBlockPool(), it starts two threads, which add different block pools concurrently, and then attempt to add a block into the pool. If the block pool is not found, it throws an IOException.

        Without the rev01 patch that uses ConcurrentHashMap, this test case always fail because it can not find an added block pool; after the patch, I am not seeing any failures.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Rev02: Added a test case. In this test case TestSimulatedFSDataset.testConcurrentAddBlockPool() , it starts two threads, which add different block pools concurrently, and then attempt to add a block into the pool. If the block pool is not found, it throws an IOException. Without the rev01 patch that uses ConcurrentHashMap, this test case always fail because it can not find an added block pool; after the patch, I am not seeing any failures.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Rev01. Use ConcurrentHashMap instead of HashMap in SimulatedFSDataset to store block pools.

        Tested locally. Before the patch, the test failed 1 in 10 runs. After the patch, I've been running for ~100 runs without seeing any failures.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Rev01. Use ConcurrentHashMap instead of HashMap in SimulatedFSDataset to store block pools. Tested locally. Before the patch, the test failed 1 in 10 runs. After the patch, I've been running for ~100 runs without seeing any failures.

          People

          • Assignee:
            jojochuang Wei-Chiu Chuang
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development