HBase
  1. HBase
  2. HBASE-8521

Cells cannot be overwritten with bulk loaded HFiles

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.1
    • Fix Version/s: 0.94.13
    • Component/s: None
    • Labels:
      None

      Description

      Let's say you have a pre-built HFile that contains a cell:

      ('rowkey1', 'family1', 'qual1', 1234L, 'value1')

      We bulk load this first HFile. Now, let's create a second HFile that contains a cell that overwrites the first:

      ('rowkey1', 'family1', 'qual1', 1234L, 'value2')

      That gets bulk loaded into the table, but the value that HBase bubbles up is still 'value1'.

      It seems that there's no way to overwrite a cell for a particular timestamp without an explicit put operation. This seems to be the case even after minor and major compactions happen.

      My guess is that this is pretty closely related to the sequence number work being done on the compaction algorithm via HBASE-7842, but I'm not sure if one of would fix the other.

      1. HBASE-8521-v4-addendum-0.94.patch
        1.0 kB
        Jean-Marc Spaggiari
      2. HBASE-8521-v4-0.94.patch
        27 kB
        Jean-Marc Spaggiari
      3. HBASE-8521-v3-0.94.patch
        27 kB
        Jean-Marc Spaggiari
      4. HBASE-8521-v2-0.94.patch
        28 kB
        Jean-Marc Spaggiari
      5. HBASE-8521-v1-0.94.patch
        24 kB
        Jean-Marc Spaggiari
      6. HBASE-8521-v0-0.94.patch
        23 kB
        Jean-Marc Spaggiari
      7. HBASE-8521.diff
        6 kB
        Jonathan Natkins
      8. hfileDirs.tar.gz
        0.6 kB
        Jonathan Natkins

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          @Jonathan:
          Are you able to write a unit test exhibiting this behavior ?

          Thanks

          Show
          Ted Yu added a comment - @Jonathan: Are you able to write a unit test exhibiting this behavior ? Thanks
          Hide
          Jonathan Natkins added a comment -

          @Ted:
          Seemingly, I can't. It seems the mini cluster that HBaseTestingUtility creates has a different behavior than an actual cluster. What I will do, however, is upload a diff of the test that I wrote to show the behavior that I tried to create, and then I'll also attach the files necessary to recreate on a real cluster.

          Show
          Jonathan Natkins added a comment - @Ted: Seemingly, I can't. It seems the mini cluster that HBaseTestingUtility creates has a different behavior than an actual cluster. What I will do, however, is upload a diff of the test that I wrote to show the behavior that I tried to create, and then I'll also attach the files necessary to recreate on a real cluster.
          Hide
          Elliott Clark added a comment -

          @Ted he opened this because of me. It's a pretty tough thing to repro.

          Bulk load files (on trunk) are sorted by see StoreFile#Comparator:

          SeqID
          Size (reversed)
          TimeStamp
          filename

          So bulkoad files that are larger than the bulkloaded files that come before them can overwrite key values with the same timestamp. We should adopt what 0.89-fb does. Assign seqid to bulk loaded files at bulk load time, by renaming the files.

          This should mean that bulk load files are treated in much the same way as flushed/compacted files.

          Show
          Elliott Clark added a comment - @Ted he opened this because of me. It's a pretty tough thing to repro. Bulk load files (on trunk) are sorted by see StoreFile#Comparator : SeqID Size (reversed) TimeStamp filename So bulkoad files that are larger than the bulkloaded files that come before them can overwrite key values with the same timestamp. We should adopt what 0.89-fb does. Assign seqid to bulk loaded files at bulk load time, by renaming the files. This should mean that bulk load files are treated in much the same way as flushed/compacted files.
          Hide
          Jonathan Natkins added a comment -

          There are two directories in this tarball: familyDir1, and familyDir2. Each contains a single HFile, and each of them has one cell of data in them.

          The table was created as:
          create 'test',

          {NAME => 'myfam', VERSIONS => 100000, TTL => 1000000000}

          In familyDir1, the HFile's cell contains the value "oldVal" for myfam:myqual.

          In familyDir2, the HFile's cell contains the value "newVal" for myfam:myqual.

          Show
          Jonathan Natkins added a comment - There are two directories in this tarball: familyDir1, and familyDir2. Each contains a single HFile, and each of them has one cell of data in them. The table was created as: create 'test', {NAME => 'myfam', VERSIONS => 100000, TTL => 1000000000} In familyDir1, the HFile's cell contains the value "oldVal" for myfam:myqual. In familyDir2, the HFile's cell contains the value "newVal" for myfam:myqual.
          Hide
          Jonathan Natkins added a comment -

          Basically, my process for reproducing is this:

          hadoop fs -put familyDir1 familyDir1
          hadoop fs -put familyDir2 familyDir2
          
          create 'test', {NAME => 'myfam', VERSIONS => 100000, TTL => 1000000000}
          
          hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs://localhost:8020/user/natty/familyDir1 test
          hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs://localhost:8020/user/natty/familyDir2 test
          

          The result after this set of operations is:

          1.9.2p320 :001 > scan 'test'
          ROW                                           COLUMN+CELL                                                                                                                          
           aaaa                                         column=myfam:myqual, timestamp=1368157470713, value=oldVal                                                                           
          1 row(s) in 0.6260 seconds
          

          If I only load familyDir2, the output is this:

          1.9.2p320 :001 > scan 'test'
          ROW                                           COLUMN+CELL                                                                                                                          
           aaaa                                         column=myfam:myqual, timestamp=1368157470713, value=newVal                                                                           
          1 row(s) in 0.5930 seconds
          
          Show
          Jonathan Natkins added a comment - Basically, my process for reproducing is this: hadoop fs -put familyDir1 familyDir1 hadoop fs -put familyDir2 familyDir2 create 'test', {NAME => 'myfam', VERSIONS => 100000, TTL => 1000000000} hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs://localhost:8020/user/natty/familyDir1 test hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs://localhost:8020/user/natty/familyDir2 test The result after this set of operations is: 1.9.2p320 :001 > scan 'test' ROW COLUMN+CELL aaaa column=myfam:myqual, timestamp=1368157470713, value=oldVal 1 row(s) in 0.6260 seconds If I only load familyDir2, the output is this: 1.9.2p320 :001 > scan 'test' ROW COLUMN+CELL aaaa column=myfam:myqual, timestamp=1368157470713, value=newVal 1 row(s) in 0.5930 seconds
          Hide
          Jonathan Natkins added a comment -

          Here's a diff of the test I wrote. You'll have to forgive the quick-and-dirtiness of it, but it gets the point across. This test passes, despite the real cluster failing the same functional test.

          Also, with full disclosure, this diff was made against 0.92.1 included in CDH4.1.4, so if it doesn't nicely line up with Apache source, I'll go and diff it against the real source code.

          Show
          Jonathan Natkins added a comment - Here's a diff of the test I wrote. You'll have to forgive the quick-and-dirtiness of it, but it gets the point across. This test passes, despite the real cluster failing the same functional test. Also, with full disclosure, this diff was made against 0.92.1 included in CDH4.1.4, so if it doesn't nicely line up with Apache source, I'll go and diff it against the real source code.
          Hide
          Ted Yu added a comment -

          We should adopt what 0.89-fb does. Assign seqid to bulk loaded files at bulk load time, by renaming the files.

          +1

          @Jonathan:
          Thanks for outlining the procedure for reproducing the issue.

          Show
          Ted Yu added a comment - We should adopt what 0.89-fb does. Assign seqid to bulk loaded files at bulk load time, by renaming the files. +1 @Jonathan: Thanks for outlining the procedure for reproducing the issue.
          Hide
          Jeff Kolesky added a comment -

          According to HBASE-6630, 0.89-fb's fix was ported to 0.95. Any chance it is reasonable to backport to 0.92 or 0.94?

          Show
          Jeff Kolesky added a comment - According to HBASE-6630 , 0.89-fb's fix was ported to 0.95. Any chance it is reasonable to backport to 0.92 or 0.94?
          Hide
          Ted Yu added a comment -

          Lars Hofhansl:
          What's your opinion on the backport ?

          Show
          Ted Yu added a comment - Lars Hofhansl : What's your opinion on the backport ?
          Hide
          Lars Hofhansl added a comment -

          The patch in HBASE-6630 is easy enough to understand (and small too when disregarding the protobuf boilerplate).

          Show
          Lars Hofhansl added a comment - The patch in HBASE-6630 is easy enough to understand (and small too when disregarding the protobuf boilerplate).
          Hide
          Lars Hofhansl added a comment -

          +0 from me (seems like a rare use case to me).

          It seems like Jeff and Jonathan need that, so let's get it in.

          Show
          Lars Hofhansl added a comment - +0 from me (seems like a rare use case to me). It seems like Jeff and Jonathan need that, so let's get it in.
          Hide
          Ted Yu added a comment -

          Looks like the following new method needs to be added to HRegionInterface (note the 3rd parameter):

            public boolean bulkLoadHFiles(List<Pair<byte[], String>> familyPaths, byte[] regionName, assignSeqIds)
            throws IOException;
          

          This means that if some region servers use 0.94.7 jars, the call in LoadIncrementalHFiles utilizing the new method would fail.

          Fallback to existing, two argument, method can be used in above scenario.

          I want to other people's opinion on the compatibility issue.

          Show
          Ted Yu added a comment - Looks like the following new method needs to be added to HRegionInterface (note the 3rd parameter): public boolean bulkLoadHFiles(List<Pair< byte [], String >> familyPaths, byte [] regionName, assignSeqIds) throws IOException; This means that if some region servers use 0.94.7 jars, the call in LoadIncrementalHFiles utilizing the new method would fail. Fallback to existing, two argument, method can be used in above scenario. I want to other people's opinion on the compatibility issue.
          Hide
          Jean-Marc Spaggiari added a comment -

          FYI, I have backported HBASE-6630 to 0.94 and used also some of 0.89fb diff too.

          I will run some tests locally with the data attached to this defect and try to reproduce what Jonathan Natkins get.

          Show
          Jean-Marc Spaggiari added a comment - FYI, I have backported HBASE-6630 to 0.94 and used also some of 0.89fb diff too. I will run some tests locally with the data attached to this defect and try to reproduce what Jonathan Natkins get.
          Hide
          Jean-Marc Spaggiari added a comment -

          So. First try attached.

          Passed all the tests successfuly. (3 tests failed because of compilation issues not related to this patch. Servlets, etc.)

          hbase(main):001:0> create 'test', {NAME => 'myfam', VERSIONS => 100000, TTL => 1000000000}
          
          bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir1 test
          
          hbase(main):001:0> scan 'test'
          ROW                                                   COLUMN+CELL                                                                                                                                                  
           aaaa                                                 column=myfam:myqual, timestamp=1368157470713, value=oldVal                                                                                                   
          1 row(s) in 0.4190 seconds
          
          bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir2 test
          
          hbase(main):001:0> scan 'test'
          ROW                                                   COLUMN+CELL                                                                                                                                                  
           aaaa                                                 column=myfam:myqual, timestamp=1368157470713, value=newVal                                                                                                   
          1 row(s) in 0.4330 seconds
          

          So as said above, there is come incompatibility between he version.

          One thing that I can change is to not pass the parameter if it's false so new clients can still call old servers.

          Open to comments.

          Show
          Jean-Marc Spaggiari added a comment - So. First try attached. Passed all the tests successfuly. (3 tests failed because of compilation issues not related to this patch. Servlets, etc.) hbase(main):001:0> create 'test', {NAME => 'myfam', VERSIONS => 100000, TTL => 1000000000} bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir1 test hbase(main):001:0> scan 'test' ROW COLUMN+CELL aaaa column=myfam:myqual, timestamp=1368157470713, value=oldVal 1 row(s) in 0.4190 seconds bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir2 test hbase(main):001:0> scan 'test' ROW COLUMN+CELL aaaa column=myfam:myqual, timestamp=1368157470713, value=newVal 1 row(s) in 0.4330 seconds So as said above, there is come incompatibility between he version. One thing that I can change is to not pass the parameter if it's false so new clients can still call old servers. Open to comments.
          Hide
          Jean-Marc Spaggiari added a comment -

          Small update.

          I have defaulted the assignSeqIds to false in LoadIncrementalHFiles so if no additionnal parameter is used, it will behave the same way as before.

          Also, if assignSeqIds is false, I'm using the previous existing method signatures in the serverCallable to make sure new client -> old server is still compatible.

          If someone specificaly turn assignSeqIds to true, then they are most probably aware of this new feature and will most probably also know that they need to upgrade the server side too.

          What I can add is a message in the console when this is turned to true to informe that this is working only with a server >= 0.94.8.

          Again, here are the tests results of this change:

          Tests in error: 
            testBasic(org.apache.hadoop.hbase.regionserver.TestRSStatusServlet): Unresolved compilation problems: (..)
            testWithRegions(org.apache.hadoop.hbase.regionserver.TestRSStatusServlet): Unresolved compilation problems: (..)
            testGetEmpty(org.apache.hadoop.hbase.avro.TestAvroUtil): Unresolved compilation problems: (..)
          
          Tests run: 682, Failures: 0, Errors: 3, Skipped: 0
          

          I have compilation errors in my repository for thos 2 classes, not related to the changes I made.

          bin/hbase -Dhbase.mapreduce.bulkload.assign.sequenceNumbers=true org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir1 test
          
          hbase(main):001:0> scan 'test'
          ROW                                                   COLUMN+CELL                                                                                                                                                  
           aaaa                                                 column=myfam:myqual, timestamp=1368157470713, value=oldVal                                                                                                   
          1 row(s) in 0.3860 seconds
          
          
          bin/hbase -Dhbase.mapreduce.bulkload.assign.sequenceNumbers=true org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir2 test
          
          
          hbase(main):001:0> scan 'test'
          ROW                                                   COLUMN+CELL                                                                                                                                                  
           aaaa                                                 column=myfam:myqual, timestamp=1368157470713, value=newVal                                                                                                   
          1 row(s) in 0.3800 seconds
          

          And without the parameter...

          echo "create 'test', {NAME => 'myfam', VERSIONS => 100000, TTL => 1000000000}" | bin/hbase shell
          
          bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir1 test
          bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir2 test
          
          echo "scan 'test'" | bin/hbase shell
          HBase Shell; enter 'help<RETURN>' for list of supported commands.
          Type "exit<RETURN>" to leave the HBase Shell
          Version 0.94.8-SNAPSHOT, r, Thu May 16 08:43:22 EDT 2013
          
          scan 'test'
          ROW                                                   COLUMN+CELL                                                                                                                                                  
           aaaa                                                 column=myfam:myqual, timestamp=1368157470713, value=oldVal  
          
          Show
          Jean-Marc Spaggiari added a comment - Small update. I have defaulted the assignSeqIds to false in LoadIncrementalHFiles so if no additionnal parameter is used, it will behave the same way as before. Also, if assignSeqIds is false, I'm using the previous existing method signatures in the serverCallable to make sure new client -> old server is still compatible. If someone specificaly turn assignSeqIds to true, then they are most probably aware of this new feature and will most probably also know that they need to upgrade the server side too. What I can add is a message in the console when this is turned to true to informe that this is working only with a server >= 0.94.8. Again, here are the tests results of this change: Tests in error: testBasic(org.apache.hadoop.hbase.regionserver.TestRSStatusServlet): Unresolved compilation problems: (..) testWithRegions(org.apache.hadoop.hbase.regionserver.TestRSStatusServlet): Unresolved compilation problems: (..) testGetEmpty(org.apache.hadoop.hbase.avro.TestAvroUtil): Unresolved compilation problems: (..) Tests run: 682, Failures: 0, Errors: 3, Skipped: 0 I have compilation errors in my repository for thos 2 classes, not related to the changes I made. bin/hbase -Dhbase.mapreduce.bulkload.assign.sequenceNumbers= true org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir1 test hbase(main):001:0> scan 'test' ROW COLUMN+CELL aaaa column=myfam:myqual, timestamp=1368157470713, value=oldVal 1 row(s) in 0.3860 seconds bin/hbase -Dhbase.mapreduce.bulkload.assign.sequenceNumbers= true org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir2 test hbase(main):001:0> scan 'test' ROW COLUMN+CELL aaaa column=myfam:myqual, timestamp=1368157470713, value=newVal 1 row(s) in 0.3800 seconds And without the parameter... echo "create 'test', {NAME => 'myfam', VERSIONS => 100000, TTL => 1000000000}" | bin/hbase shell bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir1 test bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles familyDir2 test echo "scan 'test'" | bin/hbase shell HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 0.94.8-SNAPSHOT, r, Thu May 16 08:43:22 EDT 2013 scan 'test' ROW COLUMN+CELL aaaa column=myfam:myqual, timestamp=1368157470713, value=oldVal
          Hide
          Jean-Marc Spaggiari added a comment -

          I just tried with a new client including this fix to bulkload to an existing cluster running 0.94.7 using the usecase Jonathan had provide above.

          bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs://node3:9000/user/hbase/familyDir1 test
          
           echo "scan 'test'" | bin/hbase shell
          HBase Shell; enter 'help<RETURN>' for list of supported commands.
          Type "exit<RETURN>" to leave the HBase Shell
          Version 0.94.8-SNAPSHOT, r, Thu May 16 08:43:22 EDT 2013
          
          scan 'test'
          ROW                                                   COLUMN+CELL                                                                                                                                                  
           aaaa                                                 column=myfam:myqual, timestamp=1368157470713, value=oldVal                                                                                                   
          1 row(s) in 0.7900 seconds
          
          bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs://node3:9000/user/hbase/familyDir2 test
          
          echo "scan 'test'" | bin/hbase shell
          HBase Shell; enter 'help<RETURN>' for list of supported commands.
          Type "exit<RETURN>" to leave the HBase Shell
          Version 0.94.8-SNAPSHOT, r, Thu May 16 08:43:22 EDT 2013
          
          scan 'test'
          ROW                                                   COLUMN+CELL                                                                                                                                                  
           aaaa                                                 column=myfam:myqual, timestamp=1368157470713, value=oldVal                                                                                                   
          1 row(s) in 0.6810 seconds
          

          Things are consistent and working as before, not any error is displayed.

          Full log:

          bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs://node3:9000/user/hbase/familyDir2 test
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:host.name=cloudera.distparser.com
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_45
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/jdk1.6.0_45/jre
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../conf:/usr/local/jdk1.6.0_45//lib/tools.jar:/home/jmspaggiari/.m2/repository/asm/asm/3.1/asm-3.1.jar:/home/jmspaggiari/.m2/repository/com/github/stephenc/high-scale-lib/high-scale-lib/1.1.1/high-scale-lib-1.1.1.jar:/home/jmspaggiari/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/jmspaggiari/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar:/home/jmspaggiari/.m2/repository/com/google/protobuf/protobuf-java/2.4.0a/protobuf-java-2.4.0a.jar:/home/jmspaggiari/.m2/repository/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar:/home/jmspaggiari/.m2/repository/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar:/home/jmspaggiari/.m2/repository/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar:/home/jmspaggiari/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/home/jmspaggiari/.m2/repository/com/yammer/metrics/metrics-core/2.1.2/metrics-core-2.1.2.jar:/home/jmspaggiari/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/home/jmspaggiari/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/home/jmspaggiari/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/jmspaggiari/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/home/jmspaggiari/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/home/jmspaggiari/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/home/jmspaggiari/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/home/jmspaggiari/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/home/jmspaggiari/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/jmspaggiari/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/home/jmspaggiari/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar:/home/jmspaggiari/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/home/jmspaggiari/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/home/jmspaggiari/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/home/jmspaggiari/.m2/repository/javax/xml/bind/jaxb-api/2.1/jaxb-api-2.1.jar:/home/jmspaggiari/.m2/repository/junit/junit/4.10-HBASE-1/junit-4.10-HBASE-1.jar:/home/jmspaggiari/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar:/home/jmspaggiari/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:/home/jmspaggiari/.m2/repository/org/apache/avro/avro-ipc/1.5.3/avro-ipc-1.5.3.jar:/home/jmspaggiari/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/home/jmspaggiari/.m2/repository/org/apache/ftpserver/ftplet-api/1.0.0/ftplet-api-1.0.0.jar:/home/jmspaggiari/.m2/repository/org/apache/ftpserver/ftpserver-core/1.0.0/ftpserver-core-1.0.0.jar:/home/jmspaggiari/.m2/repository/org/apache/ftpserver/ftpserver-deprecated/1.0.0-M2/ftpserver-deprecated-1.0.0-M2.jar:/home/jmspaggiari/.m2/repository/org/apache/hadoop/hadoop-core/1.0.4/hadoop-core-1.0.4.jar:/home/jmspaggiari/.m2/repository/org/apache/hadoop/hadoop-test/1.0.4/hadoop-test-1.0.4.jar:/home/jmspaggiari/.m2/repository/org/apache/httpcomponents/httpclient/4.1.2/httpclient-4.1.2.jar:/home/jmspaggiari/.m2/repository/org/apache/httpcomponents/httpcore/4.1.3/httpcore-4.1.3.jar:/home/jmspaggiari/.m2/repository/org/apache/mina/mina-core/2.0.0-M5/mina-core-2.0.0-M5.jar:/home/jmspaggiari/.m2/repository/org/apache/thrift/libthrift/0.8.0/libthrift-0.8.0.jar:/home/jmspaggiari/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar:/home/jmspaggiari/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.8/jackson-jaxrs-1.8.8.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.8/jackson-xc-1.8.8.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/home/jmspaggiari/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar:/home/jmspaggiari/.m2/repository/org/jamon/jamon-runtime/2.3.1/jamon-runtime-2.3.1.jar:/home/jmspaggiari/.m2/repository/org/jboss/netty/netty/3.2.4.Final/netty-3.2.4.Final.jar:/home/jmspaggiari/.m2/repository/org/jruby/jruby-complete/1.6.5/jruby-complete-1.6.5.jar:/home/jmspaggiari/.m2/repository/org/mockito/mockito-all/1.8.5/mockito-all-1.8.5.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:/home/jmspaggiari/.m2/repository/org/slf4j/slf4j-api/1.4.3/slf4j-api-1.4.3.jar:/home/jmspaggiari/.m2/repository/org/slf4j/slf4j-log4j12/1.4.3/slf4j-log4j12-1.4.3.jar:/home/jmspaggiari/.m2/repository/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar:/home/jmspaggiari/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/home/jmspaggiari/.m2/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar:/home/jmspaggiari/.m2/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar:/home/jmspaggiari/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../target/classes:/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../target/test-classes:/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../target:/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../lib/*.jar:
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/jdk1.6.0_45/jre/lib/amd64/server:/usr/local/jdk1.6.0_45/jre/lib/amd64:/usr/local/jdk1.6.0_45/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:os.version=3.2.0-4-amd64
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:user.name=jmspaggiari
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/jmspaggiari
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=latitude:2181,cube:2181,node3:2181 sessionTimeout=180000 watcher=hconnection
          13/05/16 14:38:15 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 16902@cloudera
          13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Opening socket connection to server latitude/192.168.23.4:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
          13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Socket connection established to latitude/192.168.23.4:2181, initiating session
          13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Session establishment complete on server latitude/192.168.23.4:2181, sessionid = 0x23eadc5ddeb008f, negotiated timeout = 40000
          13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=latitude:2181,cube:2181,node3:2181 sessionTimeout=180000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c
          13/05/16 14:38:15 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 16902@cloudera
          13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Opening socket connection to server latitude/192.168.23.4:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration)
          13/05/16 14:38:15 DEBUG catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@77ff92f5
          13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Socket connection established to latitude/192.168.23.4:2181, initiating session
          13/05/16 14:38:16 INFO zookeeper.ClientCnxn: Session establishment complete on server latitude/192.168.23.4:2181, sessionid = 0x23eadc5ddeb0090, negotiated timeout = 40000
          13/05/16 14:38:16 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c; serverName=node5,60020,1368669066647
          13/05/16 14:38:16 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is node6:60020
          13/05/16 14:38:16 DEBUG client.ClientScanner: Creating scanner over .META. starting at key 'test,,'
          13/05/16 14:38:16 DEBUG client.ClientScanner: Advancing internal scanner to startKey at 'test,,'
          13/05/16 14:38:16 DEBUG catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@77ff92f5
          13/05/16 14:38:16 INFO zookeeper.ZooKeeper: Session: 0x23eadc5ddeb0090 closed
          13/05/16 14:38:16 INFO zookeeper.ClientCnxn: EventThread shut down
          13/05/16 14:38:16 DEBUG client.MetaScanner: Scanning .META. starting at row=test,,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c
          13/05/16 14:38:16 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for test,,1368729286037.32038a4d760aa9303643e2985dcd29a5. is node2:60020
          13/05/16 14:38:16 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c
          13/05/16 14:38:17 DEBUG client.MetaScanner: Scanning .META. starting at row=test,,00000000000000 for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c
          13/05/16 14:38:17 INFO hfile.CacheConfig: Allocating LruBlockCache with maximum size 246.9m
          13/05/16 14:38:17 INFO util.ChecksumType: Checksum can use java.util.zip.CRC32
          13/05/16 14:38:17 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://node3:9000/user/hbase/familyDir2/myfam/hfile_1 first=aaaa last=aaaa
          13/05/16 14:38:18 DEBUG mapreduce.LoadIncrementalHFiles: Going to connect to server region=test,,1368729286037.32038a4d760aa9303643e2985dcd29a5., hostname=node2, port=60020 for row 
          
          Show
          Jean-Marc Spaggiari added a comment - I just tried with a new client including this fix to bulkload to an existing cluster running 0.94.7 using the usecase Jonathan had provide above. bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs: //node3:9000/user/hbase/familyDir1 test echo "scan 'test'" | bin/hbase shell HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 0.94.8-SNAPSHOT, r, Thu May 16 08:43:22 EDT 2013 scan 'test' ROW COLUMN+CELL aaaa column=myfam:myqual, timestamp=1368157470713, value=oldVal 1 row(s) in 0.7900 seconds bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs: //node3:9000/user/hbase/familyDir2 test echo "scan 'test'" | bin/hbase shell HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 0.94.8-SNAPSHOT, r, Thu May 16 08:43:22 EDT 2013 scan 'test' ROW COLUMN+CELL aaaa column=myfam:myqual, timestamp=1368157470713, value=oldVal 1 row(s) in 0.6810 seconds Things are consistent and working as before, not any error is displayed. Full log: bin/hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles hdfs: //node3:9000/user/hbase/familyDir2 test 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:host.name=cloudera.distparser.com 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_45 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/jdk1.6.0_45/jre 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../conf:/usr/local/jdk1.6.0_45 //lib/tools.jar:/home/jmspaggiari/.m2/repository/asm/asm/3.1/asm-3.1.jar:/home/jmspaggiari/.m2/repository/com/github/stephenc/high-scale-lib/high-scale-lib/1.1.1/high-scale-lib-1.1.1.jar:/home/jmspaggiari/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/jmspaggiari/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar:/home/jmspaggiari/.m2/repository/com/google/protobuf/protobuf-java/2.4.0a/protobuf-java-2.4.0a.jar:/home/jmspaggiari/.m2/repository/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar:/home/jmspaggiari/.m2/repository/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar:/home/jmspaggiari/.m2/repository/com/sun/jersey/jersey-server/1.8/jersey-server-1.8.jar:/home/jmspaggiari/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/home/jmspaggiari/.m2/repository/com/yammer/metrics/metrics-core/2.1.2/metrics-core-2.1.2.jar:/home/jmspaggiari/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/home/jmspaggiari/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/home/jmspaggiari/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/jmspaggiari/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/home/jmspaggiari/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/home/jmspaggiari/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/home/jmspaggiari/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/home/jmspaggiari/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/home/jmspaggiari/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/jmspaggiari/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/home/jmspaggiari/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar:/home/jmspaggiari/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/home/jmspaggiari/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:/home/jmspaggiari/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/home/jmspaggiari/.m2/repository/javax/xml/bind/jaxb-api/2.1/jaxb-api-2.1.jar:/home/jmspaggiari/.m2/repository/junit/junit/4.10-HBASE-1/junit-4.10-HBASE-1.jar:/home/jmspaggiari/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar:/home/jmspaggiari/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:/home/jmspaggiari/.m2/repository/org/apache/avro/avro-ipc/1.5.3/avro-ipc-1.5.3.jar:/home/jmspaggiari/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/home/jmspaggiari/.m2/repository/org/apache/ftpserver/ftplet-api/1.0.0/ftplet-api-1.0.0.jar:/home/jmspaggiari/.m2/repository/org/apache/ftpserver/ftpserver-core/1.0.0/ftpserver-core-1.0.0.jar:/home/jmspaggiari/.m2/repository/org/apache/ftpserver/ftpserver-deprecated/1.0.0-M2/ftpserver-deprecated-1.0.0-M2.jar:/home/jmspaggiari/.m2/repository/org/apache/hadoop/hadoop-core/1.0.4/hadoop-core-1.0.4.jar:/home/jmspaggiari/.m2/repository/org/apache/hadoop/hadoop-test/1.0.4/hadoop-test-1.0.4.jar:/home/jmspaggiari/.m2/repository/org/apache/httpcomponents/httpclient/4.1.2/httpclient-4.1.2.jar:/home/jmspaggiari/.m2/repository/org/apache/httpcomponents/httpcore/4.1.3/httpcore-4.1.3.jar:/home/jmspaggiari/.m2/repository/org/apache/mina/mina-core/2.0.0-M5/mina-core-2.0.0-M5.jar:/home/jmspaggiari/.m2/repository/org/apache/thrift/libthrift/0.8.0/libthrift-0.8.0.jar:/home/jmspaggiari/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar:/home/jmspaggiari/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.8/jackson-jaxrs-1.8.8.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.8/jackson-xc-1.8.8.jar:/home/jmspaggiari/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/home/jmspaggiari/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar:/home/jmspaggiari/.m2/repository/org/jamon/jamon-runtime/2.3.1/jamon-runtime-2.3.1.jar:/home/jmspaggiari/.m2/repository/org/jboss/netty/netty/3.2.4.Final/netty-3.2.4.Final.jar:/home/jmspaggiari/.m2/repository/org/jruby/jruby-complete/1.6.5/jruby-complete-1.6.5.jar:/home/jmspaggiari/.m2/repository/org/mockito/mockito-all/1.8.5/mockito-all-1.8.5.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:/home/jmspaggiari/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:/home/jmspaggiari/.m2/repository/org/slf4j/slf4j-api/1.4.3/slf4j-api-1.4.3.jar:/home/jmspaggiari/.m2/repository/org/slf4j/slf4j-log4j12/1.4.3/slf4j-log4j12-1.4.3.jar:/home/jmspaggiari/.m2/repository/org/xerial/snappy/snappy-java/1.0.3.2/snappy-java-1.0.3.2.jar:/home/jmspaggiari/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/home/jmspaggiari/.m2/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar:/home/jmspaggiari/.m2/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar:/home/jmspaggiari/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../target/classes:/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../target/test-classes:/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../target:/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT/bin/../lib/*.jar: 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/jdk1.6.0_45/jre/lib/amd64/server:/usr/local/jdk1.6.0_45/jre/lib/amd64:/usr/local/jdk1.6.0_45/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:os.version=3.2.0-4-amd64 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:user.name=jmspaggiari 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/jmspaggiari 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/jmspaggiari/workspace/hbase-0.94.8-SNAPSHOT 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=latitude:2181,cube:2181,node3:2181 sessionTimeout=180000 watcher=hconnection 13/05/16 14:38:15 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 16902@cloudera 13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Opening socket connection to server latitude/192.168.23.4:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) 13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Socket connection established to latitude/192.168.23.4:2181, initiating session 13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Session establishment complete on server latitude/192.168.23.4:2181, sessionid = 0x23eadc5ddeb008f, negotiated timeout = 40000 13/05/16 14:38:15 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=latitude:2181,cube:2181,node3:2181 sessionTimeout=180000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c 13/05/16 14:38:15 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 16902@cloudera 13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Opening socket connection to server latitude/192.168.23.4:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) 13/05/16 14:38:15 DEBUG catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@77ff92f5 13/05/16 14:38:15 INFO zookeeper.ClientCnxn: Socket connection established to latitude/192.168.23.4:2181, initiating session 13/05/16 14:38:16 INFO zookeeper.ClientCnxn: Session establishment complete on server latitude/192.168.23.4:2181, sessionid = 0x23eadc5ddeb0090, negotiated timeout = 40000 13/05/16 14:38:16 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c; serverName=node5,60020,1368669066647 13/05/16 14:38:16 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is node6:60020 13/05/16 14:38:16 DEBUG client.ClientScanner: Creating scanner over .META. starting at key 'test,,' 13/05/16 14:38:16 DEBUG client.ClientScanner: Advancing internal scanner to startKey at 'test,,' 13/05/16 14:38:16 DEBUG catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@77ff92f5 13/05/16 14:38:16 INFO zookeeper.ZooKeeper: Session: 0x23eadc5ddeb0090 closed 13/05/16 14:38:16 INFO zookeeper.ClientCnxn: EventThread shut down 13/05/16 14:38:16 DEBUG client.MetaScanner: Scanning .META. starting at row=test,,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c 13/05/16 14:38:16 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for test,,1368729286037.32038a4d760aa9303643e2985dcd29a5. is node2:60020 13/05/16 14:38:16 DEBUG client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c 13/05/16 14:38:17 DEBUG client.MetaScanner: Scanning .META. starting at row=test,,00000000000000 for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@79ee2c2c 13/05/16 14:38:17 INFO hfile.CacheConfig: Allocating LruBlockCache with maximum size 246.9m 13/05/16 14:38:17 INFO util.ChecksumType: Checksum can use java.util.zip.CRC32 13/05/16 14:38:17 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs: //node3:9000/user/hbase/familyDir2/myfam/hfile_1 first=aaaa last=aaaa 13/05/16 14:38:18 DEBUG mapreduce.LoadIncrementalHFiles: Going to connect to server region=test,,1368729286037.32038a4d760aa9303643e2985dcd29a5., hostname=node2, port=60020 for row
          Hide
          Jean-Marc Spaggiari added a comment -

          Any chance to have someone looking at this one?

          Show
          Jean-Marc Spaggiari added a comment - Any chance to have someone looking at this one?
          Hide
          Sergey Shelukhin added a comment -

          patch looks reasonable esp. given it's a backport.
          It has long lines.
          These ifs:

          +            if (assignSeqIds) {
          +              success = secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken, assignSeqIds);
          +            } else {
          +              success = secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken);
          +            }
          

          seem excessive, just the first call is enough, right?

          Show
          Sergey Shelukhin added a comment - patch looks reasonable esp. given it's a backport. It has long lines. These ifs: + if (assignSeqIds) { + success = secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken, assignSeqIds); + } else { + success = secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken); + } seem excessive, just the first call is enough, right?
          Hide
          Jean-Marc Spaggiari added a comment -

          secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken) calls secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken, false) so if assignSeqIds is false, secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken) and secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken, assignSeqIds) are going to be the same call, therefore the if is not required.

          Same for

                      if (assignSeqIds) {
                        success = secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken, assignSeqIds);
                      } else {
                        success = secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken);
                      }
          

          I will change that and re-base the patch since it's not working anymore with current 0.94 branch...

          Thanks for looking at it.

          Show
          Jean-Marc Spaggiari added a comment - secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken) calls secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken, false) so if assignSeqIds is false, secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken) and secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken, assignSeqIds) are going to be the same call, therefore the if is not required. Same for if (assignSeqIds) { success = secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken, assignSeqIds); } else { success = secureClient.bulkLoadHFiles(famPaths, userToken, bulkToken); } I will change that and re-base the patch since it's not working anymore with current 0.94 branch... Thanks for looking at it.
          Hide
          Jean-Marc Spaggiari added a comment -

          Rebased patch attached. Some modifications required because minSeqId got removed by HBASE-7210. Also applied Ted's comment. I'm not testing with the data attached to this ticket.

          Show
          Jean-Marc Spaggiari added a comment - Rebased patch attached. Some modifications required because minSeqId got removed by HBASE-7210 . Also applied Ted's comment. I'm not testing with the data attached to this ticket.
          Hide
          Jean-Daniel Cryans added a comment -

          This jira will need a title that accurately describes what's being done here.

          On the patch:

          • There are 11 "== Durability.USE_DEFAULT", can we just have a method somewhere (like in Mutation) that does it and is named "isDefaultDurability"?
          • LoadIncrementalHFiles.assignSeqIds should be final.
          • Since bulk loaded files can have sequence ids, we should print it out. StoreFile.toStringDetailed is a candidate for that change, there might be more
          • What's up with the commented out code in HRegion?
          • Is it passing all the unit tests? A trunk version would here to get some Hadoop QA love.

          I tested the patch to see if Jonathan's original use case is covered and it looks like it is. I also did some mixed workloads of normal Puts and bulk loaded files.

          Show
          Jean-Daniel Cryans added a comment - This jira will need a title that accurately describes what's being done here. On the patch: There are 11 "== Durability.USE_DEFAULT", can we just have a method somewhere (like in Mutation) that does it and is named "isDefaultDurability"? LoadIncrementalHFiles.assignSeqIds should be final. Since bulk loaded files can have sequence ids, we should print it out. StoreFile.toStringDetailed is a candidate for that change, there might be more What's up with the commented out code in HRegion? Is it passing all the unit tests? A trunk version would here to get some Hadoop QA love. I tested the patch to see if Jonathan's original use case is covered and it looks like it is. I also did some mixed workloads of normal Puts and bulk loaded files.
          Hide
          Jean-Marc Spaggiari added a comment -

          Hi Jean-Daniel Cryans, thanks for looking at it.

          There are 11 "== Durability.USE_DEFAULT", can we just have a method somewhere (like in Mutation) that does it and is named "isDefaultDurability"?

          This is because setWriteToWAL is now deprecated. If I set something locally to do the same thing as what setWriteToWAL was doing, then it's not very clean since we are just bypassing the deprecation warning. Other option is to keep setWriteToWAL calls and add @suppressWarning to avoid them and keep setWriteToWAL?

          LoadIncrementalHFiles.assignSeqIds should be final.

          I don't think we can. If I put it final, how are you going to modify it on the constructor? (assignSeqIds = conf.getBoolean(ASSIGN_SEQ_IDS, true))

          Since bulk loaded files can have sequence ids, we should print it out. StoreFile.toStringDetailed is a candidate for that change, there might be more

          StoreFile.toStringDetailed already display the sequenceId under certain conditions. I will remove the condition to make sure we always have this information.

          What's up with the commented out code in HRegion?

          Wow, this should not be there. At all! Removed!

          Is it passing all the unit tests? A trunk version would here to get some Hadoop QA love.

          Yes, it's passing unit tests. I will post the results before EOD. Also, there is no trunk version because it's a backport of HBASE-6630 which is already doing the same thing in trunk.

          So, waiting for your recommendation regarding Durability.USE_DEFAULT tests and I will post an updated version.

          Show
          Jean-Marc Spaggiari added a comment - Hi Jean-Daniel Cryans , thanks for looking at it. There are 11 "== Durability.USE_DEFAULT", can we just have a method somewhere (like in Mutation) that does it and is named "isDefaultDurability"? This is because setWriteToWAL is now deprecated. If I set something locally to do the same thing as what setWriteToWAL was doing, then it's not very clean since we are just bypassing the deprecation warning. Other option is to keep setWriteToWAL calls and add @suppressWarning to avoid them and keep setWriteToWAL? LoadIncrementalHFiles.assignSeqIds should be final. I don't think we can. If I put it final, how are you going to modify it on the constructor? (assignSeqIds = conf.getBoolean(ASSIGN_SEQ_IDS, true)) Since bulk loaded files can have sequence ids, we should print it out. StoreFile.toStringDetailed is a candidate for that change, there might be more StoreFile.toStringDetailed already display the sequenceId under certain conditions. I will remove the condition to make sure we always have this information. What's up with the commented out code in HRegion? Wow, this should not be there. At all! Removed! Is it passing all the unit tests? A trunk version would here to get some Hadoop QA love. Yes, it's passing unit tests. I will post the results before EOD. Also, there is no trunk version because it's a backport of HBASE-6630 which is already doing the same thing in trunk. So, waiting for your recommendation regarding Durability.USE_DEFAULT tests and I will post an updated version.
          Hide
          Jean-Marc Spaggiari added a comment -

          Update version attached.
          1) final added. Was not working on trunk, but working fine on 0.94.
          2) Removed the Durability modifications since they are not related to this patch. Was only deprecated calls cleanup.

          Should be nicer now.

          Show
          Jean-Marc Spaggiari added a comment - Update version attached. 1) final added. Was not working on trunk, but working fine on 0.94. 2) Removed the Durability modifications since they are not related to this patch. Was only deprecated calls cleanup. Should be nicer now.
          Hide
          Jean-Marc Spaggiari added a comment -

          Hum, seems that I pushed the wrong file...

          Show
          Jean-Marc Spaggiari added a comment - Hum, seems that I pushed the wrong file...
          Hide
          Jean-Marc Spaggiari added a comment -

          Removed commented code.

          Show
          Jean-Marc Spaggiari added a comment - Removed commented code.
          Hide
          Jean-Marc Spaggiari added a comment -

          Lars Hofhansl any chance to look at that? Want it into 0.94? This patch is already running on a production cluster.

          Show
          Jean-Marc Spaggiari added a comment - Lars Hofhansl any chance to look at that? Want it into 0.94? This patch is already running on a production cluster.
          Hide
          Lars Hofhansl added a comment -

          Still +0 on this.

          Any strong sponsors? If so, let's get it in.

          Show
          Lars Hofhansl added a comment - Still +0 on this. Any strong sponsors? If so, let's get it in.
          Hide
          Alexandre Normand added a comment -

          I'll jump in late to the discussion to add a personal story.

          We're very much relying on this patch since we bulk load everything and our use-case depends on this. We're running with this patch at the moment and we're hoping not to lose it when upgrading. We put automated testing of our feature that relies on this and we spent a lot of time looking at the internals (fun times with the Hfile tool) to check that all was as expected. While the test doesn't provide a full guarantee that the behavior is as expected as opposed to still being a combination of non-deterministic behavior and luck, this runs every day and each day that the test doesn't fail increases our confidence.

          I'm very much +1. If there's one patch we ever needed from hbase, this is it.

          Show
          Alexandre Normand added a comment - I'll jump in late to the discussion to add a personal story. We're very much relying on this patch since we bulk load everything and our use-case depends on this. We're running with this patch at the moment and we're hoping not to lose it when upgrading. We put automated testing of our feature that relies on this and we spent a lot of time looking at the internals (fun times with the Hfile tool) to check that all was as expected. While the test doesn't provide a full guarantee that the behavior is as expected as opposed to still being a combination of non-deterministic behavior and luck, this runs every day and each day that the test doesn't fail increases our confidence. I'm very much +1. If there's one patch we ever needed from hbase, this is it.
          Hide
          Lars Hofhansl added a comment -

          Thanks for chiming in Alexandre... I'll count that a "strong sponsor" as per above.
          If there are no objections, I will commit this to 0.94.13 tomorrow.

          Show
          Lars Hofhansl added a comment - Thanks for chiming in Alexandre... I'll count that a "strong sponsor" as per above. If there are no objections, I will commit this to 0.94.13 tomorrow.
          Hide
          Alexandre Normand added a comment -

          Thanks Lars, I appreciate it. I really do.

          Show
          Alexandre Normand added a comment - Thanks Lars, I appreciate it. I really do.
          Hide
          Jean-Marc Spaggiari added a comment -

          Cool, that's all good news too! Thanks guys. One less on my radar.

          Show
          Jean-Marc Spaggiari added a comment - Cool, that's all good news too! Thanks guys. One less on my radar.
          Hide
          Nick Dimiduk added a comment -

          Lars Hofhansl do you have time to cut a patch for 0.96 before the next RC? (cc stack)

          Show
          Nick Dimiduk added a comment - Lars Hofhansl do you have time to cut a patch for 0.96 before the next RC? (cc stack )
          Hide
          Nick Dimiduk added a comment -
          Show
          Nick Dimiduk added a comment - ^^^ or Jean-Marc Spaggiari
          Hide
          Jean-Marc Spaggiari added a comment -

          It's already there in HBASE-6630... Only 0.94 was missing it. So we should be good.

          Show
          Jean-Marc Spaggiari added a comment - It's already there in HBASE-6630 ... Only 0.94 was missing it. So we should be good.
          Hide
          Nick Dimiduk added a comment -

          ACK. Thanks JM.

          Show
          Nick Dimiduk added a comment - ACK. Thanks JM.
          Hide
          Lars Hofhansl added a comment -

          Committed to the 0.94 branch. Thanks for the patch.

          Show
          Lars Hofhansl added a comment - Committed to the 0.94 branch. Thanks for the patch.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94-security #307 (See https://builds.apache.org/job/HBase-0.94-security/307/)
          HBASE-8521 Cells cannot be overwritten with bulk loaded HFiles (Jean-Marc Spaggiari) (larsh: rev 1528592)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/coprocessor/SecureBulkLoadClient.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #307 (See https://builds.apache.org/job/HBase-0.94-security/307/ ) HBASE-8521 Cells cannot be overwritten with bulk loaded HFiles (Jean-Marc Spaggiari) (larsh: rev 1528592) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/coprocessor/SecureBulkLoadClient.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94 #1166 (See https://builds.apache.org/job/HBase-0.94/1166/)
          HBASE-8521 Cells cannot be overwritten with bulk loaded HFiles (Jean-Marc Spaggiari) (larsh: rev 1528592)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/coprocessor/SecureBulkLoadClient.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1166 (See https://builds.apache.org/job/HBase-0.94/1166/ ) HBASE-8521 Cells cannot be overwritten with bulk loaded HFiles (Jean-Marc Spaggiari) (larsh: rev 1528592) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/coprocessor/SecureBulkLoadClient.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
          Hide
          Lars Hofhansl added a comment -

          TestLoadIncrementalHFilesSplitRecovery.testBulkLoadPhaseFailure failed... Looks suspicious.

          Show
          Lars Hofhansl added a comment - TestLoadIncrementalHFilesSplitRecovery.testBulkLoadPhaseFailure failed... Looks suspicious.
          Hide
          Jean-Marc Spaggiari added a comment -

          I agree. Let me run that locally few multiple to validate...

          Show
          Jean-Marc Spaggiari added a comment - I agree. Let me run that locally few multiple to validate...
          Hide
          Lars Hofhansl added a comment -

          Also, looking at v4 now. It adds

          private void verifyAssignedSequenceNumber(String testName, byte hfileRanges, boolean nonZero)
          

          But it is never actually called from anywhere.

          Show
          Lars Hofhansl added a comment - Also, looking at v4 now. It adds private void verifyAssignedSequenceNumber( String testName, byte hfileRanges, boolean nonZero) But it is never actually called from anywhere.
          Hide
          Lars Hofhansl added a comment -

          Definitely passes without the patch and fail with it.

          Show
          Lars Hofhansl added a comment - Definitely passes without the patch and fail with it.
          Hide
          Jean-Marc Spaggiari added a comment -

          Failed also on 0.94.10+HBASE-8521 Something might have changed since the last time I merged it. Please revert, I will look at the issue.

          Show
          Jean-Marc Spaggiari added a comment - Failed also on 0.94.10+ HBASE-8521 Something might have changed since the last time I merged it. Please revert, I will look at the issue.
          Hide
          Jean-Marc Spaggiari added a comment -

          verifyAssignedSequenceNumber has been removed from trunk because it's never called. I took it from 0.89. I will remove it. Looking at the test failure now.

          Show
          Jean-Marc Spaggiari added a comment - verifyAssignedSequenceNumber has been removed from trunk because it's never called. I took it from 0.89. I will remove it. Looking at the test failure now.
          Hide
          Jean-Marc Spaggiari added a comment -

          Ok. Looking deeper, I'm not sure that it's really an issue.

          The test which failed:

            /**
             * Test that shows that exception thrown from the RS side will result in an
             * exception on the LIHFile client.
             */
          

          RS is not throwing an exception. But nothing related to this patch. But still, occurs only with this patch activated. So there might be some side effect. I will start to roll it back locally line by line to figure what the issue is...

          Show
          Jean-Marc Spaggiari added a comment - Ok. Looking deeper, I'm not sure that it's really an issue. The test which failed: /** * Test that shows that exception thrown from the RS side will result in an * exception on the LIHFile client. */ RS is not throwing an exception. But nothing related to this patch. But still, occurs only with this patch activated. So there might be some side effect. I will start to roll it back locally line by line to figure what the issue is...
          Hide
          Jean-Marc Spaggiari added a comment -

          Found it.

          Don't revert. It's related to the test, not to the code. Patch and detailed to come in a minute.

          Show
          Jean-Marc Spaggiari added a comment - Found it. Don't revert. It's related to the test, not to the code. Patch and detailed to come in a minute.
          Hide
          Lars Hofhansl added a comment -

          I knew I could wait a few minutes

          Show
          Lars Hofhansl added a comment - I knew I could wait a few minutes
          Hide
          Jean-Marc Spaggiari added a comment -

          So here is the culprit.

          There is now an extra parameter to the bulkLoadHFiles call where we need to specify the assignSeqNum boolean. However, the mocked connection was still configured to fail on the method call without this parameter (which is not the one being called).

          This addendum is to update the mocked connection configuration.

          Passed the test locally. Sorry for this.

          Show
          Jean-Marc Spaggiari added a comment - So here is the culprit. There is now an extra parameter to the bulkLoadHFiles call where we need to specify the assignSeqNum boolean. However, the mocked connection was still configured to fail on the method call without this parameter (which is not the one being called). This addendum is to update the mocked connection configuration. Passed the test locally. Sorry for this.
          Hide
          Lars Hofhansl added a comment -

          Committed. Thanks JM!

          Show
          Lars Hofhansl added a comment - Committed. Thanks JM!
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94-security #308 (See https://builds.apache.org/job/HBase-0.94-security/308/)
          HBASE-8521 Addendum - test fix (Jean-Marc Spaggiari) (larsh: rev 1528692)

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #308 (See https://builds.apache.org/job/HBase-0.94-security/308/ ) HBASE-8521 Addendum - test fix (Jean-Marc Spaggiari) (larsh: rev 1528692) /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94 #1167 (See https://builds.apache.org/job/HBase-0.94/1167/)
          HBASE-8521 Addendum - test fix (Jean-Marc Spaggiari) (larsh: rev 1528692)

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1167 (See https://builds.apache.org/job/HBase-0.94/1167/ ) HBASE-8521 Addendum - test fix (Jean-Marc Spaggiari) (larsh: rev 1528692) /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java

            People

            • Assignee:
              Jean-Marc Spaggiari
              Reporter:
              Jonathan Natkins
            • Votes:
              5 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development