HBase
  1. HBase
  2. HBASE-5885

Invalid HFile block magic on Local file System

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.94.0, 0.95.2
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      ERROR: java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=7, exceptions:
      Thu Apr 26 11:19:18 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: java.io.IOException: Could not iterate StoreFileScanner[HFileScanner for reader reader=file:/tmp/hbase-eclark/hbase/TestTable/e2d1c846363c75262cbfd85ea278b342/info/bae2681d63734066957b58fe791a0268, compression=none, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=0000000001/info:data/1335463981520/Put, lastKey=0002588100/info:data/1335463902296/Put, avgKeyLen=30, avgValueLen=1000, entries=1215085, length=1264354417, cur=0000000248/info:data/1335463994457/Put/vlen=1000/ts=0]
      at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:135)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
      at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:368)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
      Caused by: java.io.IOException: Invalid HFile block magic: \xEC\xD5\x9D\xB4\xC2bfo
      at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:153)
      at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:164)
      at org.apache.hadoop.hbase.io.hfile.HFileBlock.<init>(HFileBlock.java:254)
      at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1779)
      at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1637)
      at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:327)
      at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:555)
      at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:651)
      at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
      ... 12 more

      Thu Apr 26 11:19:19 PDT 2012, org.apache.hadoop.hbase.client.ScannerCallable@190a621a, java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1132)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1121)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2420)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
      at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)
      Caused by: java.lang.IllegalArgumentException
      at java.nio.Buffer.position(Buffer.java:216)
      at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:630)
      at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:130)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:95)
      at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:406)
      at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3323)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3279)
      at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3296)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2393)
      ... 5 more

      On latest 0.94 branch I spun up a new standalone hbase. Then I started a performance evaluation run hbase/bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred randomWrite 10
      after that completed I tried a scan of TestTable. The scan got the above error.

      1. 5885-trunk-v2.txt
        2 kB
        Ted Yu
      2. HBASE-5885-94-0.patch
        1 kB
        Elliott Clark
      3. HBASE-5885-94-1.patch
        2 kB
        Elliott Clark
      4. HBASE-5885-trunk-0.patch
        1 kB
        Elliott Clark
      5. HBASE-5885-trunk-1.patch
        0.7 kB
        Elliott Clark

        Issue Links

          Activity

          Hide
          Elliott Clark added a comment -

          Forgot to add that I also tried this with and without HBASE-5864

          Show
          Elliott Clark added a comment - Forgot to add that I also tried this with and without HBASE-5864
          Hide
          stack added a comment -

          What if you disable this setting? Does it still fail?

          +  /** 
          +   * If this parameter is set to true, then hbase will read
          +   * data and then verify checksums. Checksum verification 
          +   * inside hdfs will be switched off.  However, if the hbase-checksum 
          +   * verification fails, then it will switch back to using
          +   * hdfs checksums for verifiying data that is being read from storage.
          +   *
          +   * If this parameter is set to false, then hbase will not
          +   * verify any checksums, instead it will depend on checksum verification
          +   * being done in the hdfs client.
          +   */
          +  public static final String HBASE_CHECKSUM_VERIFICATION = 
          +      "hbase.regionserver.checksum.verify";
          
          Show
          stack added a comment - What if you disable this setting? Does it still fail? + /** + * If this parameter is set to true , then hbase will read + * data and then verify checksums. Checksum verification + * inside hdfs will be switched off. However, if the hbase-checksum + * verification fails, then it will switch back to using + * hdfs checksums for verifiying data that is being read from storage. + * + * If this parameter is set to false , then hbase will not + * verify any checksums, instead it will depend on checksum verification + * being done in the hdfs client. + */ + public static final String HBASE_CHECKSUM_VERIFICATION = + "hbase.regionserver.checksum.verify" ;
          Hide
          Elliott Clark added a comment -

          I don't get an exception when that is set to false.

          Show
          Elliott Clark added a comment - I don't get an exception when that is set to false.
          Hide
          Lars Hofhansl added a comment -

          So specifically this only happens in local mode (where HBase writes directly to the local file system)? If so, can we disable this feature in local mode (where it is pointless anyway)

          Show
          Lars Hofhansl added a comment - So specifically this only happens in local mode (where HBase writes directly to the local file system)? If so, can we disable this feature in local mode (where it is pointless anyway)
          Hide
          Lars Hofhansl added a comment -

          Are you working on this Elliot?

          Show
          Lars Hofhansl added a comment - Are you working on this Elliot?
          Hide
          Elliott Clark added a comment -

          Stack and I were looking into this; I'll pick it back up first thing tomorrow morning. Brain dump below so that I remember it all in the morning.

          Running a standalone instance with hbase.regionserver.checksum.verify to true will see an error when trying to read.
          Running a standalone instance on the same data with hbase.regionserver.checksum.verify set to false will not see this error and all data looks intact.

          The first read through an HFileReader seems to be fine no matter what. It's the second read that seems to be erroring out.

          Looking at data files they all start with 'DATABLK*' and there are several of those headers in the files. So data too much data is being written per block, or not enough is being read.

          Show
          Elliott Clark added a comment - Stack and I were looking into this; I'll pick it back up first thing tomorrow morning. Brain dump below so that I remember it all in the morning. Running a standalone instance with hbase.regionserver.checksum.verify to true will see an error when trying to read. Running a standalone instance on the same data with hbase.regionserver.checksum.verify set to false will not see this error and all data looks intact. The first read through an HFileReader seems to be fine no matter what. It's the second read that seems to be erroring out. Looking at data files they all start with 'DATABLK*' and there are several of those headers in the files. So data too much data is being written per block, or not enough is being read.
          Hide
          stack added a comment -

          With one region with one file only in it, I run into the issue. I can read the file outside of hbase using HFile tool fine. When I scan, the problem happens at a random seeming point each time; i.e. run it once and its at key 162, run it again its at key 246, etc. There does not seem to be correlation with index boundaries if I dump them out with:

          pynchon:0.94 stack$ ./bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -m -s -b -f /tmp/hbase-stack/hbase/TestTable/75c7e8e6612fdf1fa308f9a34104e847/info/2830d6bcf8e34767af9baaf94228559a 
          
          Show
          stack added a comment - With one region with one file only in it, I run into the issue. I can read the file outside of hbase using HFile tool fine. When I scan, the problem happens at a random seeming point each time; i.e. run it once and its at key 162, run it again its at key 246, etc. There does not seem to be correlation with index boundaries if I dump them out with: pynchon:0.94 stack$ ./bin/hbase org.apache.hadoop.hbase.io.hfile.HFile -m -s -b -f /tmp/hbase-stack/hbase/TestTable/75c7e8e6612fdf1fa308f9a34104e847/info/2830d6bcf8e34767af9baaf94228559a
          Hide
          Lars Hofhansl added a comment -

          Scary. Just to confirm, you still only see this in local mode and with HBase checksums enabled, right?

          Show
          Lars Hofhansl added a comment - Scary. Just to confirm, you still only see this in local mode and with HBase checksums enabled, right?
          Hide
          Elliott Clark added a comment -

          94 patch.

          Show
          Elliott Clark added a comment - 94 patch.
          Hide
          Elliott Clark added a comment -

          Trunk patch.
          org.apache.hadoop.LocalFileSystem's seek does not work as expected when setVerifyChecksum is set to false.

          This patch means that on the local machine all reads will use the underlying checksum.

          Show
          Elliott Clark added a comment - Trunk patch. org.apache.hadoop.LocalFileSystem's seek does not work as expected when setVerifyChecksum is set to false. This patch means that on the local machine all reads will use the underlying checksum.
          Hide
          Lars Hofhansl added a comment -

          Do you envision this as the final fix, or as a tempory solutions for now?
          Might still be nice to know what exactly causes this for the local fs.

          Show
          Lars Hofhansl added a comment - Do you envision this as the final fix, or as a tempory solutions for now? Might still be nice to know what exactly causes this for the local fs.
          Hide
          Lars Hofhansl added a comment -

          +1 on patch

          Show
          Lars Hofhansl added a comment - +1 on patch
          Hide
          stack added a comment -

          +1 on patch.

          Show
          stack added a comment - +1 on patch.
          Hide
          Elliott Clark added a comment -

          I would see this as the final solution. Anything else would either be much bigger and more invasive or would rely on hadoop changes which is unlikely.
          Inside local FS they seek to the next boundary of checksums. However when not reading checksums it gives the wrong data when a read is issued.

          Show
          Elliott Clark added a comment - I would see this as the final solution. Anything else would either be much bigger and more invasive or would rely on hadoop changes which is unlikely. Inside local FS they seek to the next boundary of checksums. However when not reading checksums it gives the wrong data when a read is issued.
          Hide
          Lars Hofhansl added a comment -

          @Elliot: fair enough

          Show
          Lars Hofhansl added a comment - @Elliot: fair enough
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12524942/HBASE-5885-trunk-0.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.TestReplication
          org.apache.hadoop.hbase.io.hfile.TestChecksum
          org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1675//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1675//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1675//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524942/HBASE-5885-trunk-0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.io.hfile.TestChecksum org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1675//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1675//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1675//console This message is automatically generated.
          Hide
          Elliott Clark added a comment -

          94 missed a couple asserts

          Show
          Elliott Clark added a comment - 94 missed a couple asserts
          Hide
          Elliott Clark added a comment -

          trunk

          Show
          Elliott Clark added a comment - trunk
          Hide
          Elliott Clark added a comment -

          Missed a couple of asserts. This should fix the TestChecksum test. TestForceCacheImportantBlocks passes for me and TestReplication doesn't run for me.

          I've got to run to catch a plane but I'll try and check on this later.

          Show
          Elliott Clark added a comment - Missed a couple of asserts. This should fix the TestChecksum test. TestForceCacheImportantBlocks passes for me and TestReplication doesn't run for me. I've got to run to catch a plane but I'll try and check on this later.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12524953/HBASE-5885-trunk-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1677//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524953/HBASE-5885-trunk-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1677//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          I would see this as the final solution. Anything else would either be much bigger and more invasive or would rely on hadoop changes which is unlikely.
          Inside local FS they seek to the next boundary of checksums. However when not reading checksums it gives the wrong data when a read is issued.

          Please file an upstream bug in the hadoop project for this as well - we should absolutely fix this. I agree a workaround in the near term makes sense, but if LocalFileSystem is broken, we should fix it!

          Show
          Todd Lipcon added a comment - I would see this as the final solution. Anything else would either be much bigger and more invasive or would rely on hadoop changes which is unlikely. Inside local FS they seek to the next boundary of checksums. However when not reading checksums it gives the wrong data when a read is issued. Please file an upstream bug in the hadoop project for this as well - we should absolutely fix this. I agree a workaround in the near term makes sense, but if LocalFileSystem is broken, we should fix it!
          Hide
          Ted Yu added a comment -

          Wrong patch was attached as trunk-1.

          Re-attaching patch for trunk.

          Show
          Ted Yu added a comment - Wrong patch was attached as trunk-1. Re-attaching patch for trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12524960/5885-trunk-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1678//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1678//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1678//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524960/5885-trunk-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1678//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1678//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1678//console This message is automatically generated.
          Hide
          stack added a comment -

          Committed 0.94 and trunk. Thanks for the patch Elliott (See Todd's comment above – add an old hadoop issue when you get a chance)

          Show
          stack added a comment - Committed 0.94 and trunk. Thanks for the patch Elliott (See Todd's comment above – add an old hadoop issue when you get a chance)
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #153 (See https://builds.apache.org/job/HBase-0.94/153/)
          HBASE-5885 Invalid HFile block magic on Local file System (Revision 1331676)

          Result = FAILURE
          stack :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #153 (See https://builds.apache.org/job/HBase-0.94/153/ ) HBASE-5885 Invalid HFile block magic on Local file System (Revision 1331676) Result = FAILURE stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2823 (See https://builds.apache.org/job/HBase-TRUNK/2823/)
          HBASE-5885 Invalid HFile block magic on Local file System (Revision 1331675)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2823 (See https://builds.apache.org/job/HBase-TRUNK/2823/ ) HBASE-5885 Invalid HFile block magic on Local file System (Revision 1331675) Result = FAILURE stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #187 (See https://builds.apache.org/job/HBase-TRUNK-security/187/)
          HBASE-5885 Invalid HFile block magic on Local file System (Revision 1331675)

          Result = SUCCESS
          stack :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #187 (See https://builds.apache.org/job/HBase-TRUNK-security/187/ ) HBASE-5885 Invalid HFile block magic on Local file System (Revision 1331675) Result = SUCCESS stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #24 (See https://builds.apache.org/job/HBase-0.94-security/24/)
          HBASE-5885 Invalid HFile block magic on Local file System (Revision 1331676)

          Result = FAILURE
          stack :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #24 (See https://builds.apache.org/job/HBase-0.94-security/24/ ) HBASE-5885 Invalid HFile block magic on Local file System (Revision 1331676) Result = FAILURE stack : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java
          Hide
          Ted Yu added a comment -

          Now that HBASE-5611 was taken out of 0.94, yet we still see the following test failure:

          https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.94/159/testReport/org.apache.hadoop.hbase.io.encoding/TestChangingEncoding/testFlippingEncodeOnDisk/
          

          I start to think that this JIRA is related to the failure above.

          Show
          Ted Yu added a comment - Now that HBASE-5611 was taken out of 0.94, yet we still see the following test failure: https: //builds.apache.org/view/G-L/view/HBase/job/HBase-0.94/159/testReport/org.apache.hadoop.hbase.io.encoding/TestChangingEncoding/testFlippingEncodeOnDisk/ I start to think that this JIRA is related to the failure above.
          Hide
          Lars Hofhansl added a comment -

          I don't see that failures in earlier test runs that include this patch.

          Show
          Lars Hofhansl added a comment - I don't see that failures in earlier test runs that include this patch.
          Show
          Ted Yu added a comment - Here is another one: https://builds.apache.org/job/HBase-0.94/157/testReport/org.apache.hadoop.hbase.io.encoding/TestChangingEncoding/testFlippingEncodeOnDisk/
          Hide
          stack added a comment -

          I don't think the TestChangingEncoding is related to this change. It enables verification of checksum in local filesystem. The TestChangingEncoding doesn't even use local filesystem. I opened HBASE-5906 to look into the TestChangingEncoding fails.

          Show
          stack added a comment - I don't think the TestChangingEncoding is related to this change. It enables verification of checksum in local filesystem. The TestChangingEncoding doesn't even use local filesystem. I opened HBASE-5906 to look into the TestChangingEncoding fails.
          Hide
          Lars Hofhansl added a comment -

          Did anybody file an upstream hadoop bug, yet? If not, I'll do so.

          Show
          Lars Hofhansl added a comment - Did anybody file an upstream hadoop bug, yet? If not, I'll do so.
          Hide
          Todd Lipcon added a comment -

          Elliott filed HADOOP-8336 but I can't seem to reproduce it there. Will comment there momentarily.

          Show
          Todd Lipcon added a comment - Elliott filed HADOOP-8336 but I can't seem to reproduce it there. Will comment there momentarily.

            People

            • Assignee:
              Elliott Clark
              Reporter:
              Elliott Clark
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development