Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.6.0
    • Component/s: Tests
    • Labels:
      None

      Description

      With HBASE-4746 iTest hbase tests are broken:

      4. ERROR in /root/work/bigtop/bigtop-tests/test-artifacts/hbase/src/main/groovy/org/apache/bigtop/itest/hbase/system/Putter.java (at line 114)
      HTable table = new HTable(tableName);
      ^^^^^^^^^^^^^^^^^^^^^
      The constructor HTable(String) is undefined
      ----------
      ----------
      5. ERROR in /root/work/bigtop/bigtop-tests/test-artifacts/hbase/src/main/groovy/org/apache/bigtop/itest/hbase/smoke/TestHFileOutputFormat.java (at line 129)
      HTable table = new HTable(TABLE_NAME);
      ^^^^^^^^^^^^^^^^^^^^^^
      The constructor HTable(byte[]) is undefined

      1. 856.txt
        9 kB
        Konstantin Boudnik
      2. 858.txt
        11 kB
        stack
      3. BIGTOP-858.patch
        12 kB
        Konstantin Boudnik
      4. BIGTOP-858.patch
        9 kB
        Konstantin Boudnik
      5. BIGTOP-858.patch
        5 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          stack added a comment -

          What would like us to do Konstantin Boudnik? Add back the removed constructors or would you like me change the bigtop test? Thanks. It looks like that change rolled out w/ 0.94.0

          Show
          stack added a comment - What would like us to do Konstantin Boudnik ? Add back the removed constructors or would you like me change the bigtop test? Thanks. It looks like that change rolled out w/ 0.94.0
          Hide
          Konstantin Boudnik added a comment -

          Stack, I think fixing the tests is more appropriate way to go. I have found out that at least another change affects the tests, e.g

          HFile.getWriterFactory(conf).createWriter(fs, path,
          ^^^^^^^^^^^^^^^^
          The method getWriterFactory(Configuration, CacheConfig) in the type HFile is not applicable for the arguments (Configuration)

          I am looking through the history to find where the change has happened to open another ticket.

          If you guys can fix the tests - that'd be great. If there're no spare cycles on your side - I will be working on it later in the day.

          Show
          Konstantin Boudnik added a comment - Stack , I think fixing the tests is more appropriate way to go. I have found out that at least another change affects the tests, e.g HFile.getWriterFactory(conf).createWriter(fs, path, ^^^^^^^^^^^^^^^^ The method getWriterFactory(Configuration, CacheConfig) in the type HFile is not applicable for the arguments (Configuration) I am looking through the history to find where the change has happened to open another ticket. If you guys can fix the tests - that'd be great. If there're no spare cycles on your side - I will be working on it later in the day.
          Hide
          stack added a comment -

          Suggested change. Compiles. Do not run tests.

          I tried to replace all places where a String was used creating an HTable instance and replaced them with byte []. byte [] will work with 0.92 and 0.94.

          Show
          stack added a comment - Suggested change. Compiles. Do not run tests. I tried to replace all places where a String was used creating an HTable instance and replaced them with byte []. byte [] will work with 0.92 and 0.94.
          Hide
          stack added a comment -

          Konstantin Boudnik File issue for the getWriterFactory and will try and get you suggested fix there – if that helps.

          Show
          stack added a comment - Konstantin Boudnik File issue for the getWriterFactory and will try and get you suggested fix there – if that helps.
          Hide
          Konstantin Boudnik added a comment - - edited

          Yeah, I just found the root cause - my knowledge of Hbase codebase is rusty - it seems that between 0.92.1 and 0.94.2 getWriterFactory(Configuration) has migrated to getWriterFactoryNoCache(Configuration) and createWriter(...) is now protected in HFile.WriterFactory

          I have opened HBASE-7973 (tracked by BIGTOP-860 on this side). Thanks

          Show
          Konstantin Boudnik added a comment - - edited Yeah, I just found the root cause - my knowledge of Hbase codebase is rusty - it seems that between 0.92.1 and 0.94.2 getWriterFactory(Configuration) has migrated to getWriterFactoryNoCache(Configuration) and createWriter(...) is now protected in HFile.WriterFactory I have opened HBASE-7973 (tracked by BIGTOP-860 on this side). Thanks
          Hide
          Konstantin Boudnik added a comment -

          This patch only get it this far, until BIGTOP-860 isn't fixes on HBase side

          Show
          Konstantin Boudnik added a comment - This patch only get it this far, until BIGTOP-860 isn't fixes on HBase side
          Hide
          Konstantin Boudnik added a comment -

          Stack,

          it looks like in 0.92.4
          {{ public HTable(Configuration conf, final byte [] tableName) }}
          so, your change doesn't work.

          Show
          Konstantin Boudnik added a comment - Stack, it looks like in 0.92.4 {{ public HTable(Configuration conf, final byte [] tableName) }} so, your change doesn't work.
          Hide
          Konstantin Boudnik added a comment -

          That seems to address the issues with API incompatibility (safe BIGTOP-860).
          Also, removed some blank and formatting changes from Stack's

          Show
          Konstantin Boudnik added a comment - That seems to address the issues with API incompatibility (safe BIGTOP-860 ). Also, removed some blank and formatting changes from Stack's
          Hide
          Konstantin Boudnik added a comment -

          This patch combines both Stack's suggestions and Ted's recommendation on how to work around BIGTOP-860.

          Show
          Konstantin Boudnik added a comment - This patch combines both Stack's suggestions and Ted's recommendation on how to work around BIGTOP-860 .
          Hide
          stack added a comment -

          Patch lgtm (if target is hbase 0.94). Thanks Konstantin Boudnik

          Show
          stack added a comment - Patch lgtm (if target is hbase 0.94). Thanks Konstantin Boudnik
          Hide
          Ted Yu added a comment -

          In HBase 0.94, we have this method in HFile.WriterFactory:

              protected abstract Writer createWriter(FileSystem fs, Path path,
                  FSDataOutputStream ostream, int blockSize,
                  Compression.Algorithm compress,
                  HFileDataBlockEncoder dataBlockEncoder,
                  KeyComparator comparator, ChecksumType checksumType,
                  int bytesPerChecksum) throws IOException;
          

          I don't see the 5 parameter createWriter() which is used by the patch.

          Show
          Ted Yu added a comment - In HBase 0.94, we have this method in HFile.WriterFactory: protected abstract Writer createWriter(FileSystem fs, Path path, FSDataOutputStream ostream, int blockSize, Compression.Algorithm compress, HFileDataBlockEncoder dataBlockEncoder, KeyComparator comparator, ChecksumType checksumType, int bytesPerChecksum) throws IOException; I don't see the 5 parameter createWriter() which is used by the patch.
          Hide
          Konstantin Boudnik added a comment - - edited

          Sorry, for the confusion, I have uploaded the same patch twice before. Now this is the correct one. Please check again.

          Show
          Konstantin Boudnik added a comment - - edited Sorry, for the confusion, I have uploaded the same patch twice before. Now this is the correct one. Please check again.
          Hide
          Ted Yu added a comment -

          Latest patch should be good

          I had problem checking out bigtop repo. Will try again tomorrow with patch.

          Show
          Ted Yu added a comment - Latest patch should be good I had problem checking out bigtop repo. Will try again tomorrow with patch.
          Hide
          Konstantin Boudnik added a comment -

          I can compile hbase smoke tests against HBase 0.94.2 with the patch in place.

          Show
          Konstantin Boudnik added a comment - I can compile hbase smoke tests against HBase 0.94.2 with the patch in place.
          Hide
          Roman Shaposhnik added a comment - - edited

          Cos, it seems that the target HBase (0.94.5) still has a few issues:

          $ cd bigtop-tests/test-artifacts/hbase
          $ mvn clean compile
          [INFO] Using Groovy-Eclipse compiler to compile both Java and Groovy files
          ----------
          1. ERROR in /home/rvs/src/bigtop/bigtop-tests/test-artifacts/hbase/src/main/groovy/org/apache/bigtop/itest/hbase/smoke/IncrementalPELoad.java (at line 108)
          	HTable table = new HTable(Bytes.toBytes(args[0]));
          	               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          The constructor HTable(byte[]) is undefined
          ----------
          ----------
          2. ERROR in /home/rvs/src/bigtop/bigtop-tests/test-artifacts/hbase/src/main/groovy/org/apache/bigtop/itest/hbase/smoke/TestHFileOutputFormat.java (at line 129)
          	HTable table = new HTable(TABLE_NAME);
          	               ^^^^^^^^^^^^^^^^^^^^^^
          The constructor HTable(byte[]) is undefined
          

          Could you, please, take a look?

          Show
          Roman Shaposhnik added a comment - - edited Cos, it seems that the target HBase (0.94.5) still has a few issues: $ cd bigtop-tests/test-artifacts/hbase $ mvn clean compile [INFO] Using Groovy-Eclipse compiler to compile both Java and Groovy files ---------- 1. ERROR in /home/rvs/src/bigtop/bigtop-tests/test-artifacts/hbase/src/main/groovy/org/apache/bigtop/itest/hbase/smoke/IncrementalPELoad.java (at line 108) HTable table = new HTable(Bytes.toBytes(args[0])); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The constructor HTable(byte[]) is undefined ---------- ---------- 2. ERROR in /home/rvs/src/bigtop/bigtop-tests/test-artifacts/hbase/src/main/groovy/org/apache/bigtop/itest/hbase/smoke/TestHFileOutputFormat.java (at line 129) HTable table = new HTable(TABLE_NAME); ^^^^^^^^^^^^^^^^^^^^^^ The constructor HTable(byte[]) is undefined Could you, please, take a look?
          Hide
          Ted Yu added a comment -

          A Configuration object is required:

            public HTable(Configuration conf, final byte [] tableName)
          
          Show
          Ted Yu added a comment - A Configuration object is required: public HTable(Configuration conf, final byte [] tableName)
          Hide
          Konstantin Boudnik added a comment -

          Fixing last two compilation issues.

          Show
          Konstantin Boudnik added a comment - Fixing last two compilation issues.
          Hide
          Roman Shaposhnik added a comment -

          +1 and committed!

          Show
          Roman Shaposhnik added a comment - +1 and committed!

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development