HBase
  1. HBase
  2. HBASE-9639

SecureBulkLoad dispatches file load requests to all Regions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.98.0, 0.96.0
    • Component/s: Client, Coprocessors
    • Labels:
      None
    • Environment:

      Hadoop2, Kerberos

    • Hadoop Flags:
      Reviewed

      Description

      When running a bulk load on a secure environment and loading data into the first region of a table, the request to load the HFile set is dispatched to all Regions for the table. This is reproduced consistently by running IntegrationTestBulkLoad on a secure cluster. The load fails with an exception that looks like:

      2013-08-30 07:37:22,993 INFO  [main] mapreduce.LoadIncrementalHFiles: Split occured while grouping HFiles, retry attempt 1 with 3 files remaining to group or split
      2013-08-30 07:37:22,999 ERROR [main] mapreduce.LoadIncrementalHFiles: IOException during splitting
      java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File does not exist: /user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/L/bf41ea13997b4e228d05e67ba7b1b686
      at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
      at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:51)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1489)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1438)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1418)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1392)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:438)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:269)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59566)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
      at java.util.concurrent.FutureTask.get(FutureTask.java:83)
      at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:403)
      at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:284)
      at org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad.runLinkedListMRJob(IntegrationTestBulkLoad.java:200)
      at org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad.testBulkLoad(IntegrationTestBulkLoad.java:133)
      
      1. HBASE-9639.00.patch
        8 kB
        Nick Dimiduk
      2. HBASE-9639.96.01.patch
        8 kB
        Nick Dimiduk
      3. HBASE-9639.trunk.01.patch
        8 kB
        Nick Dimiduk
      4. HBASE-9639.96.02.patch
        8 kB
        Nick Dimiduk
      5. HBASE-9639.trunk.02.patch
        8 kB
        Nick Dimiduk

        Activity

        Hide
        Nick Dimiduk added a comment -

        I looked again at the patch. In an earlier revision I had toyed with some of the HConstants; those changes didn't make it into the final patch. No public API was impacted after all. Please disregard.

        Show
        Nick Dimiduk added a comment - I looked again at the patch. In an earlier revision I had toyed with some of the HConstants; those changes didn't make it into the final patch. No public API was impacted after all. Please disregard.
        Hide
        Enis Soztutar added a comment -

        It is just a nit. If you can get an addendum in today, it would be fine. Otherwise, let's leave that as it is.

        Show
        Enis Soztutar added a comment - It is just a nit. If you can get an addendum in today, it would be fine. Otherwise, let's leave that as it is.
        Hide
        Nick Dimiduk added a comment -

        Ok, I now understood the bug. +1 for the patch.
        Nit: change LAST_ROW to EMPTRY_STOP_KEY.

        Enis Soztutar do you want this resolved before the next RC makes it's API? (cc stack)

        Show
        Nick Dimiduk added a comment - Ok, I now understood the bug. +1 for the patch. Nit: change LAST_ROW to EMPTRY_STOP_KEY. Enis Soztutar do you want this resolved before the next RC makes it's API? (cc stack )
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in hbase-0.96 #90 (See https://builds.apache.org/job/hbase-0.96/90/)
        HBASE-9639 SecureBulkLoad dispatches file load requests to all Regions (ndimiduk: rev 1526032)

        • /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java
        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Show
        Hudson added a comment - SUCCESS: Integrated in hbase-0.96 #90 (See https://builds.apache.org/job/hbase-0.96/90/ ) HBASE-9639 SecureBulkLoad dispatches file load requests to all Regions (ndimiduk: rev 1526032) /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4556 (See https://builds.apache.org/job/HBase-TRUNK/4556/)
        HBASE-9639 SecureBulkLoad dispatches file load requests to all Regions (ndimiduk: rev 1526033)

        • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4556 (See https://builds.apache.org/job/HBase-TRUNK/4556/ ) HBASE-9639 SecureBulkLoad dispatches file load requests to all Regions (ndimiduk: rev 1526033) /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #760 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/760/)
        HBASE-9639 SecureBulkLoad dispatches file load requests to all Regions (ndimiduk: rev 1526033)

        • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #760 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/760/ ) HBASE-9639 SecureBulkLoad dispatches file load requests to all Regions (ndimiduk: rev 1526033) /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in hbase-0.96-hadoop2 #56 (See https://builds.apache.org/job/hbase-0.96-hadoop2/56/)
        HBASE-9639 SecureBulkLoad dispatches file load requests to all Regions (ndimiduk: rev 1526032)

        • /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java
        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Show
        Hudson added a comment - SUCCESS: Integrated in hbase-0.96-hadoop2 #56 (See https://builds.apache.org/job/hbase-0.96-hadoop2/56/ ) HBASE-9639 SecureBulkLoad dispatches file load requests to all Regions (ndimiduk: rev 1526032) /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
        Hide
        Enis Soztutar added a comment -

        oops, too late. nvm.

        Show
        Enis Soztutar added a comment - oops, too late. nvm.
        Hide
        Enis Soztutar added a comment -

        Ok, I now understood the bug. +1 for the patch.
        Nit: change LAST_ROW to EMPTRY_STOP_KEY.

        Show
        Enis Soztutar added a comment - Ok, I now understood the bug. +1 for the patch. Nit: change LAST_ROW to EMPTRY_STOP_KEY.
        Hide
        Nick Dimiduk added a comment -

        Committed to 0.96 and trunk. Thanks for the review Ted, Enis.

        Show
        Nick Dimiduk added a comment - Committed to 0.96 and trunk. Thanks for the review Ted, Enis.
        Hide
        Nick Dimiduk added a comment -

        Updated to correct the whitespace nit.

        Show
        Nick Dimiduk added a comment - Updated to correct the whitespace nit.
        Hide
        Nick Dimiduk added a comment -

        Ted Yu ACK. will fix on commit.

        Show
        Nick Dimiduk added a comment - Ted Yu ACK. will fix on commit.
        Hide
        Ted Yu added a comment -

        +1
        nit:

        +              + Bytes.toStringBinary(getRow()) + "with hfile group " + famPaths);
        

        Insert space before 'with'

        Show
        Ted Yu added a comment - +1 nit: + + Bytes.toStringBinary(getRow()) + "with hfile group " + famPaths); Insert space before 'with'
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12604826/HBASE-9639.trunk.01.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604826/HBASE-9639.trunk.01.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7355//console This message is automatically generated.
        Hide
        Nick Dimiduk added a comment -

        Rebased patches for 0.96 and trunk.

        Show
        Nick Dimiduk added a comment - Rebased patches for 0.96 and trunk.
        Hide
        Nick Dimiduk added a comment -

        Oops. More drift than I expected. Let me handle that.

        Show
        Nick Dimiduk added a comment - Oops. More drift than I expected. Let me handle that.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12604661/HBASE-9639.00.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7345//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604661/HBASE-9639.00.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7345//console This message is automatically generated.
        Hide
        Nick Dimiduk added a comment -

        To answer a question asked by Ted Yu offline, the log I pasted above "client.HTable: For key range ..." was from an additional log statement I added while debugging. In hind-sight, I think it would be clear to see the extraneous calls by enabling IPC trace logging.

        Show
        Nick Dimiduk added a comment - To answer a question asked by Ted Yu offline, the log I pasted above "client.HTable: For key range ..." was from an additional log statement I added while debugging. In hind-sight, I think it would be clear to see the extraneous calls by enabling IPC trace logging.
        Hide
        Nick Dimiduk added a comment -

        Yes, I think the comparator is in fact correct. The problem appears to be from SecureBulkLoadClient#bulkLoadHFiles sending bulk load requests to all regions in the table rather than the single region it intended. This is because is passes in EMPTY_BYTE_ARRAY for both start and end rows, which is also the invocation structure necessary for calling the whole table.

        Show
        Nick Dimiduk added a comment - Yes, I think the comparator is in fact correct. The problem appears to be from SecureBulkLoadClient#bulkLoadHFiles sending bulk load requests to all regions in the table rather than the single region it intended. This is because is passes in EMPTY_BYTE_ARRAY for both start and end rows, which is also the invocation structure necessary for calling the whole table.
        Hide
        Enis Soztutar added a comment -

        It looks like a comparator is broken for byte[0]. With a little extra logging, you see this come out of HTable#getStartKeysInRange

        This looks correct to me. It should give out every start key for range empty to empty, right?

        Show
        Enis Soztutar added a comment - It looks like a comparator is broken for byte [0] . With a little extra logging, you see this come out of HTable#getStartKeysInRange This looks correct to me. It should give out every start key for range empty to empty, right?
        Hide
        Nick Dimiduk added a comment -

        This patch explicitly limits the number of calls made by SecureBulkLoadClient#bulkLoadHFiles to one region per HFile batch.

        Show
        Nick Dimiduk added a comment - This patch explicitly limits the number of calls made by SecureBulkLoadClient#bulkLoadHFiles to one region per HFile batch.
        Hide
        Nick Dimiduk added a comment -

        It looks like a comparator is broken for byte[0]. With a little extra logging, you see this come out of HTable#getStartKeysInRange:

        2013-09-21 00:40:24,850 DEBUG [LoadIncrementalHFiles-9] client.HTable: For key range ['',''] got regions [, \x08\x00\x00\x00\x00\x00\x00\x00, \x10\x00\x00\x00\x00\x00\x00\x00, \x18\x00\x00\x00\x00\x00\x00\x00,  \x00\x00\x00\x00\x00\x00\x00, (\x00\x00\x00\x00\x00\x00\x00, 0\x00\x00\x00\x00\x00\x00\x00, 8\x00\x00\x00\x00\x00\x00\x00, @\x00\x00\x00\x00\x00\x00\x00, H\x00\x00\x00\x00\x00\x00\x00, P\x00\x00\x00\x00\x00\x00\x00, X\x00\x00\x00\x00\x00\x00\x00, `\x00\x00\x00\x00\x00\x00\x00, h\x00\x00\x00\x00\x00\x00\x00, p\x00\x00\x00\x00\x00\x00\x00, x\x00\x00\x00\x00\x00\x00\x00]
        

        This results in the SecureBulkLoadClient sending secureBulkLoadHFiles to all region servers in the table.

        Show
        Nick Dimiduk added a comment - It looks like a comparator is broken for byte[0]. With a little extra logging, you see this come out of HTable#getStartKeysInRange : 2013-09-21 00:40:24,850 DEBUG [LoadIncrementalHFiles-9] client.HTable: For key range ['',''] got regions [, \x08\x00\x00\x00\x00\x00\x00\x00, \x10\x00\x00\x00\x00\x00\x00\x00, \x18\x00\x00\x00\x00\x00\x00\x00, \x00\x00\x00\x00\x00\x00\x00, (\x00\x00\x00\x00\x00\x00\x00, 0\x00\x00\x00\x00\x00\x00\x00, 8\x00\x00\x00\x00\x00\x00\x00, @\x00\x00\x00\x00\x00\x00\x00, H\x00\x00\x00\x00\x00\x00\x00, P\x00\x00\x00\x00\x00\x00\x00, X\x00\x00\x00\x00\x00\x00\x00, `\x00\x00\x00\x00\x00\x00\x00, h\x00\x00\x00\x00\x00\x00\x00, p\x00\x00\x00\x00\x00\x00\x00, x\x00\x00\x00\x00\x00\x00\x00] This results in the SecureBulkLoadClient sending secureBulkLoadHFiles to all region servers in the table.
        Hide
        Nick Dimiduk added a comment -

        On the region server, you see the HFile group bulkloaded successfully into one region and then an attempt to load the same file group into others. The source files have been moved into place, to the file not found is thrown for subsequent loads. Which Region loads the files appears to be non-deterministic – there's a race for whichever region can complete the load first.

        2013-08-30 07:37:22,536 INFO  [RpcServer.handler=5,port=60020] regionserver.HStore: Validating hfile at hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 for inclusion in store D region IntegrationTestBulkLoad,,1377848151704.ba6689d409333b857c4c06f0b0485ca0.
        2013-08-30 07:37:22,655 DEBUG [RpcServer.handler=5,port=60020] access.SecureBulkLoadEndpoint: Moving hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 to hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20
        2013-08-30 07:37:22,680 DEBUG [RpcServer.handler=5,port=60020] regionserver.HRegionFileSystem: Committing store file hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 as hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_
        2013-08-30 07:37:22,731 INFO  [RpcServer.handler=5,port=60020] regionserver.HStore: Loaded HFile hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 into store 'D' as hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_ - updating store file list.
        2013-08-30 07:37:22,732 INFO  [RpcServer.handler=5,port=60020] regionserver.HStore: Successfully loaded store file hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 into store D (new location: hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_)
        2013-08-30 07:37:22,732 DEBUG [RpcServer.handler=5,port=60020] access.SecureBulkLoadEndpoint: Bulk Load done for: hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
        2013-08-30 07:37:22,911 INFO  [RpcServer.handler=8,port=60020] regionserver.HStore: Validating hfile at hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 for inclusion in store D region IntegrationTestBulkLoad,x\x00\x00\x00\x00\x00\x00\x00,1377848151709.925bd31a52e5b70a61c2216b5c1d3683.
        2013-08-30 07:37:22,914 INFO  [RpcServer.handler=4,port=60020] regionserver.HStore: Validating hfile at hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 for inclusion in store D region IntegrationTestBulkLoad,@\x00\x00\x00\x00\x00\x00\x00,1377848151707.57e30d9da6cf9af4763fe9968dfcf564.
        org.apache.hadoop.io.MultipleIOException: 3 exceptions [java.io.FileNotFoundException: File does not exist: /user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20
        	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
        	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:51)
        	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1489)
        	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1438)
        	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1418)
        	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1392)
        	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:438)
        	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:269)
        	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59566)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
        	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:396)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
        	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
        ...
        
        Show
        Nick Dimiduk added a comment - On the region server, you see the HFile group bulkloaded successfully into one region and then an attempt to load the same file group into others. The source files have been moved into place, to the file not found is thrown for subsequent loads. Which Region loads the files appears to be non-deterministic – there's a race for whichever region can complete the load first. 2013-08-30 07:37:22,536 INFO [RpcServer.handler=5,port=60020] regionserver.HStore: Validating hfile at hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 for inclusion in store D region IntegrationTestBulkLoad,,1377848151704.ba6689d409333b857c4c06f0b0485ca0. 2013-08-30 07:37:22,655 DEBUG [RpcServer.handler=5,port=60020] access.SecureBulkLoadEndpoint: Moving hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 to hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 2013-08-30 07:37:22,680 DEBUG [RpcServer.handler=5,port=60020] regionserver.HRegionFileSystem: Committing store file hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 as hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_ 2013-08-30 07:37:22,731 INFO [RpcServer.handler=5,port=60020] regionserver.HStore: Loaded HFile hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 into store 'D' as hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_ - updating store file list. 2013-08-30 07:37:22,732 INFO [RpcServer.handler=5,port=60020] regionserver.HStore: Successfully loaded store file hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/staging/hbase__IntegrationTestBulkLoad__b2pb1f9udm9it9vhfg57qbm4l4g1n201ubdchlpp1q2v1chulmkno08t4b7occnu/D/8b0aaa6fd40d41e090aab25e0c32bf20 into store D (new location: hdfs://hor15n14.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestBulkLoad/ba6689d409333b857c4c06f0b0485ca0/D/f1eb3cff651745aa950c6e7f5c0866e6_SeqId_2_) 2013-08-30 07:37:22,732 DEBUG [RpcServer.handler=5,port=60020] access.SecureBulkLoadEndpoint: Bulk Load done for: hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 2013-08-30 07:37:22,911 INFO [RpcServer.handler=8,port=60020] regionserver.HStore: Validating hfile at hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 for inclusion in store D region IntegrationTestBulkLoad,x\x00\x00\x00\x00\x00\x00\x00,1377848151709.925bd31a52e5b70a61c2216b5c1d3683. 2013-08-30 07:37:22,914 INFO [RpcServer.handler=4,port=60020] regionserver.HStore: Validating hfile at hdfs://hor15n14.gq1.ygridcore.net:8020/user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 for inclusion in store D region IntegrationTestBulkLoad,@\x00\x00\x00\x00\x00\x00\x00,1377848151707.57e30d9da6cf9af4763fe9968dfcf564. org.apache.hadoop.io.MultipleIOException: 3 exceptions [java.io.FileNotFoundException: File does not exist: /user/hbase/test-data/c45ddfe9-ee30-4d32-8042-928db12b1cee/IntegrationTestBulkLoad-0/D/8b0aaa6fd40d41e090aab25e0c32bf20 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:51) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1489) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1438) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1418) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1392) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:438) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:269) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59566) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) ...

          People

          • Assignee:
            Nick Dimiduk
            Reporter:
            Nick Dimiduk
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development