Hive
  1. Hive
  2. HIVE-6584

Add HiveHBaseTableSnapshotInputFormat

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: HBase Handler
    • Labels:
    • Release Note:
      Hide
      Hive can now execute queries against HBase table snapshots. This feature is available for any table defined using the HBaseStorageHandler. It requires at least HBase 0.98.3.

      To query against a snapshot instead of the online table, specify the snapshot name via hive.hbase.snapshot.name. The snapshot will be restored into a unique directory under /tmp. This location can be overridden by setting a path via hive.hbase.snapshot.restoredir.
      Show
      Hive can now execute queries against HBase table snapshots. This feature is available for any table defined using the HBaseStorageHandler. It requires at least HBase 0.98.3. To query against a snapshot instead of the online table, specify the snapshot name via hive.hbase.snapshot.name. The snapshot will be restored into a unique directory under /tmp. This location can be overridden by setting a path via hive.hbase.snapshot.restoredir.

      Description

      HBASE-8369 provided mapreduce support for reading from HBase table snapsopts. This allows a MR job to consume a stable, read-only view of an HBase table directly off of HDFS. Bypassing the online region server API provides a nice performance boost for the full scan. HBASE-10642 is backporting that feature to 0.94/0.96 and also adding a mapred implementation. Once that's available, we should add an input format. A follow-on patch could work out how to integrate this functionality into the StorageHandler, similar to how HIVE-6473 integrates the HFileOutputFormat into existing table definitions.

      1. HIVE-6584.14.patch
        50 kB
        Nick Dimiduk
      2. HIVE-6584.13.patch
        47 kB
        Nick Dimiduk
      3. HIVE-6584.12.patch
        48 kB
        Nick Dimiduk
      4. HIVE-6584.11.patch
        48 kB
        Nick Dimiduk
      5. HIVE-6584.10.patch
        48 kB
        Nick Dimiduk
      6. HIVE-6584.9.patch
        41 kB
        Nick Dimiduk
      7. HIVE-6584.8.patch
        58 kB
        Nick Dimiduk
      8. HIVE-6584.7.patch
        46 kB
        Teng Yutong
      9. HIVE-6584.6.patch
        26 kB
        Teng Yutong
      10. HIVE-6584.5.patch
        26 kB
        Teng Yutong
      11. HIVE-6584.4.patch
        25 kB
        Nick Dimiduk
      12. HIVE-6584.3.patch
        23 kB
        Nick Dimiduk
      13. HIVE-6584.2.patch
        22 kB
        Nick Dimiduk
      14. HIVE-6584.1.patch
        21 kB
        Teng Yutong
      15. HIVE-6584.0.patch
        21 kB
        Nick Dimiduk

        Issue Links

          Activity

          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          Nick Dimiduk added a comment -

          Restore location is optional. It defaults to /tmp. The restore process creates a uniquely named (random uuid) directory under this path for any give restore, so users who never set this value will not conflict with each other.

          It would be nice if hive had some kind of post-job hook that could be used to clean up the restoredir artifacts after the input format is finished with them.

          Show
          Nick Dimiduk added a comment - Restore location is optional. It defaults to /tmp. The restore process creates a uniquely named (random uuid) directory under this path for any give restore, so users who never set this value will not conflict with each other. It would be nice if hive had some kind of post-job hook that could be used to clean up the restoredir artifacts after the input format is finished with them.
          Hide
          Carter Shanklin added a comment -

          I can't edit my comment but due to some unwanted formatting let me clarify the example:

          Example:
          set hive.hbase.snapshot.name=snapshot_2014_08_03;
          set hive.hbase.snapshot.restoredir=/tmp/restore
          select count(*) from hbase_table;
          After the job is complete, /tmp/restore and its subdirectories can be deleted.
          
          Show
          Carter Shanklin added a comment - I can't edit my comment but due to some unwanted formatting let me clarify the example: Example: set hive.hbase.snapshot.name=snapshot_2014_08_03; set hive.hbase.snapshot.restoredir=/tmp/restore select count(*) from hbase_table; After the job is complete, /tmp/restore and its subdirectories can be deleted.
          Hide
          Carter Shanklin added a comment -

          I had to read the source code to get it to work, so I vote yes.

          Using Hive over HBase snapshots requires 2 variables to be set,
          hive.hbase.snapshot.name - The name of the HBase snapshot to be used when reading the HBase data.
          hive.hbase.snapshot.restoredir - A temporary directory into which the hbase snapshot is restored when queried using hive.hbase.snapshot.name. A number of directories and small files will be created under this directory, proportional to the number of regions in the HBase table. The table data itself will not be copied under this directory, only metadata. After query execution is complete, this directory can be removed.

          Example:
          set hive.hbase.snapshot.name=snapshot_2014_08_03;
          set hive.hbase.snapshot.restoredir=/tmp/restore
          select count from hbase_table;
          After the job is complete, /tmp/restore and its subdirectories can be deleted.

          Nick Dimiduk talked about making hive.hbase.snapshot.restoredir an optional setting, he can comment whether he implemented this or not.

          Show
          Carter Shanklin added a comment - I had to read the source code to get it to work, so I vote yes. Using Hive over HBase snapshots requires 2 variables to be set, hive.hbase.snapshot.name - The name of the HBase snapshot to be used when reading the HBase data. hive.hbase.snapshot.restoredir - A temporary directory into which the hbase snapshot is restored when queried using hive.hbase.snapshot.name. A number of directories and small files will be created under this directory, proportional to the number of regions in the HBase table. The table data itself will not be copied under this directory, only metadata. After query execution is complete, this directory can be removed. Example: set hive.hbase.snapshot.name=snapshot_2014_08_03; set hive.hbase.snapshot.restoredir=/tmp/restore select count from hbase_table; After the job is complete, /tmp/restore and its subdirectories can be deleted. Nick Dimiduk talked about making hive.hbase.snapshot.restoredir an optional setting, he can comment whether he implemented this or not.
          Hide
          Lefty Leverenz added a comment -

          Does this need to be documented in the wiki?

          Show
          Lefty Leverenz added a comment - Does this need to be documented in the wiki?
          Hide
          Sushanth Sowmyan added a comment -

          Committed to trunk. Thanks Nick, and Thanks Navis for the review as well.

          Show
          Sushanth Sowmyan added a comment - Committed to trunk. Thanks Nick, and Thanks Navis for the review as well.
          Hide
          Nick Dimiduk added a comment -

          Thanks folks! Any chance of getting a commit this week?

          Show
          Nick Dimiduk added a comment - Thanks folks! Any chance of getting a commit this week?
          Hide
          Navis added a comment -

          +1

          Show
          Navis added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12658980/HIVE-6584.14.patch

          ERROR: -1 due to 3 failed/errored test(s), 5861 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/125/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/125/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-125/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12658980

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12658980/HIVE-6584.14.patch ERROR: -1 due to 3 failed/errored test(s), 5861 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/125/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/125/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-125/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12658980
          Hide
          Sushanth Sowmyan added a comment -

          I like the changes made to .14.patch, definitely cleaner, and will make resolution of HIVE-7534 trivial. I also like the new error message, makes it more obvious for the end-user as to what they need.

          +1.

          Show
          Sushanth Sowmyan added a comment - I like the changes made to .14.patch, definitely cleaner, and will make resolution of HIVE-7534 trivial. I also like the new error message, makes it more obvious for the end-user as to what they need. +1.
          Hide
          Nick Dimiduk added a comment -

          I updated RB as well, the interesting addition is HBaseTableSnapshotInputFormatUtil.java and its use: https://reviews.apache.org/r/23824/diff/1-2/#7

          Show
          Nick Dimiduk added a comment - I updated RB as well, the interesting addition is HBaseTableSnapshotInputFormatUtil.java and its use: https://reviews.apache.org/r/23824/diff/1-2/#7
          Hide
          Nick Dimiduk added a comment -

          Once more, for old time's sake, eh Navis?

          Attaching and updated patch that isolates the snapshot classes from the StorageHandler. I tried this out against a local-mode HBase built against the tag 0.96.0RC5 (I didn't see a release tag, surprisingly...). Regular online operations work as expected. When I set hive.hbase.snapshot.name=foo;, I get a nice error message in my stacktrace:

          FAILED: RuntimeException This version of HBase does not support Hive over table snapshots. Please upgrade to at least HBase 0.98.3 or later. See HIVE-6584 for details.
          

          I hope this meets your requirement.

          Show
          Nick Dimiduk added a comment - Once more, for old time's sake, eh Navis ? Attaching and updated patch that isolates the snapshot classes from the StorageHandler. I tried this out against a local-mode HBase built against the tag 0.96.0RC5 (I didn't see a release tag, surprisingly...). Regular online operations work as expected. When I set hive.hbase.snapshot.name=foo; , I get a nice error message in my stacktrace: FAILED: RuntimeException This version of HBase does not support Hive over table snapshots. Please upgrade to at least HBase 0.98.3 or later. See HIVE-6584 for details. I hope this meets your requirement.
          Hide
          Navis added a comment -

          Nick Dimiduk Could you refactor-out new codes in HBaseStorageHandler to a new utility class? Something like,

          if (this.configureInputJobProps) {
            String snapshotName = HiveConf.getVar(jobConf, HiveConf.ConfVars.HIVE_HBASE_SNAPSHOT_NAME);
            if (snapshotName != null) {
              HBaseSnapshotUtil.configure(jobConf, hbaseConf, jobProperties);
            }
          }
          

          By doing this, we can use hbase-0.96.0 if snapshot is not configured.

          Show
          Navis added a comment - Nick Dimiduk Could you refactor-out new codes in HBaseStorageHandler to a new utility class? Something like, if ( this .configureInputJobProps) { String snapshotName = HiveConf.getVar(jobConf, HiveConf.ConfVars.HIVE_HBASE_SNAPSHOT_NAME); if (snapshotName != null ) { HBaseSnapshotUtil.configure(jobConf, hbaseConf, jobProperties); } } By doing this, we can use hbase-0.96.0 if snapshot is not configured.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12658238/HIVE-6584.13.patch

          ERROR: -1 due to 4 failed/errored test(s), 5786 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/82/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/82/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-82/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12658238

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12658238/HIVE-6584.13.patch ERROR: -1 due to 4 failed/errored test(s), 5786 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/82/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/82/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-82/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12658238
          Hide
          Sushanth Sowmyan added a comment -

          Aha, sounds good. And thanks for creating the new jira.

          +1 on .13.patch.

          Show
          Sushanth Sowmyan added a comment - Aha, sounds good. And thanks for creating the new jira. +1 on .13.patch.
          Hide
          Nick Dimiduk added a comment -

          Attaching luck v13: rebased onto trunk.

          Show
          Nick Dimiduk added a comment - Attaching luck v13: rebased onto trunk.
          Hide
          Nick Dimiduk added a comment -

          Thanks for having a look, Sushanth Sowmyan!

          The one thing I'd change before committing is a word-wrap for the ASF header in conf/hive-default.xml.template, to retain old newline behaviour there. But otherwise, looks good to me.

          I believe HIVE-7496 drops conf/hive-default.xml.template all together.

          We'll need to update those TODOs in a bit once we upgrade to a newer version of HBase (0.98.5+) to pick up HBASE-11555.

          I opened HIVE-7534 to track this.

          Show
          Nick Dimiduk added a comment - Thanks for having a look, Sushanth Sowmyan ! The one thing I'd change before committing is a word-wrap for the ASF header in conf/hive-default.xml.template, to retain old newline behaviour there. But otherwise, looks good to me. I believe HIVE-7496 drops conf/hive-default.xml.template all together. We'll need to update those TODOs in a bit once we upgrade to a newer version of HBase (0.98.5+) to pick up HBASE-11555 . I opened HIVE-7534 to track this.
          Hide
          Sushanth Sowmyan added a comment -

          +1 on the patch.

          The one thing I'd change before committing is a word-wrap for the ASF header in conf/hive-default.xml.template, to retain old newline behaviour there. But otherwise, looks good to me.

          We'll need to update those TODOs in a bit once we upgrade to a newer version of HBase (0.98.5+) to pick up HBASE-11555. I would have suggested doing that in this patch itself, given that you're already bumping version up to 0.98.3, except that I see that that got resolved only recently, and I don't want to drag this patch out any further. Could you please open another jira to track that TODO?

          Show
          Sushanth Sowmyan added a comment - +1 on the patch. The one thing I'd change before committing is a word-wrap for the ASF header in conf/hive-default.xml.template, to retain old newline behaviour there. But otherwise, looks good to me. We'll need to update those TODOs in a bit once we upgrade to a newer version of HBase (0.98.5+) to pick up HBASE-11555 . I would have suggested doing that in this patch itself, given that you're already bumping version up to 0.98.3, except that I see that that got resolved only recently, and I don't want to drag this patch out any further. Could you please open another jira to track that TODO?
          Hide
          Nick Dimiduk added a comment -

          Thanks for having a look, Navis. As it is, this patch requires HBASE-11137, which has not been pack-ported to 0.96. There's no technical reason not to back-port it, simply that 0.96 is in maintenance mode only and we're encouraging folks to upgrade from 0.96.2 to 0.98.x.

          Show
          Nick Dimiduk added a comment - Thanks for having a look, Navis . As it is, this patch requires HBASE-11137 , which has not been pack-ported to 0.96. There's no technical reason not to back-port it, simply that 0.96 is in maintenance mode only and we're encouraging folks to upgrade from 0.96.2 to 0.98.x.
          Hide
          Navis added a comment -

          LGTM, and one question. Can we use pre-snapshot-hbase versions (hbase-0.96.0, for example) with this patch applied? TableSnapshotInputFormatImpl is referenced from HBaseStorageHandler and seemed to throw exception loading this class.

          Show
          Navis added a comment - LGTM, and one question. Can we use pre-snapshot-hbase versions (hbase-0.96.0, for example) with this patch applied? TableSnapshotInputFormatImpl is referenced from HBaseStorageHandler and seemed to throw exception loading this class.
          Hide
          Carter Shanklin added a comment -

          I tested the .12 version of this patch on a 20 node cluster to see what sort of performance gains might be expected.

          I did a YCSB load of 180m rows and ran a few simple SQL queries in Hive while simultaneously running a YCSB 32-thread workload.

          TLDR the snapshot approach provides a nice performance boost of about 2.5x across different types of queries. The more fields I queried the better the performance was.

          Query Run Workload Snapshot Time (s) Direct Time (s) Time X Factor
          count 1 a 191.019 488.915 2.56x
          count 2 a 200.641 480.837 2.40x
          Aggregate 1 field 1 a 214.452 499.304 2.33x
          Aggregate 1 field 2 a 217.744 500.07 2.30x
          Aggregate 9 fields 1 a 281.514 802.799 2.85x
          Aggregate 9 fields 2 a 272.358 785.816 2.89x
          Aggregate 1 with GBY 1 a 248.874 558.143 2.24x
          Aggregate 1 with GBY 2 a 269.658 533.562 1.98x
          count 1 b 194.739 482.261 2.48x
          count 2 b 195.178 481.437 2.47x
          Aggregate 1 field 1 b 220.325 498.956 2.26x
          Aggregate 1 field 2 b 227.117 489.27 2.15x
          Aggregate 9 fields 1 b 276.939 817.118 2.95x
          Aggregate 9 fields 2 b 290.288 876.753 3.02x
          Aggregate 1 with GBY 1 b 244.025 563.884 2.31x
          Aggregate 1 with GBY 2 b 225.431 570.723 2.53x
          count 1 c 194.568 502.79 2.58x
          count 2 c 205.418 508.319 2.47x
          Aggregate 1 field 1 c 209.709 531.39 2.53x
          Aggregate 1 field 2 c 217.551 526.878 2.42x
          Aggregate 9 fields 1 c 267.93 756.476 2.82x
          Aggregate 9 fields 2 c 273.107 723.459 2.65x
          Aggregate 1 with GBY 1 c 240.991 526.053 2.18x
          Aggregate 1 with GBY 2 c 258.06 527.845 2.05x

          For those not familiar with YCSB it uses a table with 9 fields, each filled with random junk 100 characters long. It defines workloads A-F, of which I've used A-C.

          The main point to note is the more of the fields my query fetches, the better it works in snapshot mode.

          The other thing I measured was throughput as reported by the YCSB tool. For the most part, when running the query over a snapshot the throughput was much better.

          Workload Tput Snapshot Tput Direct Throughput Improvement (Snapshot)
          a 83443.11623 56267.34148 48.30%
          b 45709.15011 31224.30376 46.39%
          c 46634.58415 43224.86383 7.89%

          The throughput when using the snapshot seems to be close to the throughput when not scanning data, but I didn't run the baseline tests long enough to get anything conclusive here.

          In any event this looks like a good patch, especially considering its small size.

          The numbers quoted here are for reference only, YMMV, etc.

          Show
          Carter Shanklin added a comment - I tested the .12 version of this patch on a 20 node cluster to see what sort of performance gains might be expected. I did a YCSB load of 180m rows and ran a few simple SQL queries in Hive while simultaneously running a YCSB 32-thread workload. TLDR the snapshot approach provides a nice performance boost of about 2.5x across different types of queries. The more fields I queried the better the performance was. Query Run Workload Snapshot Time (s) Direct Time (s) Time X Factor count 1 a 191.019 488.915 2.56x count 2 a 200.641 480.837 2.40x Aggregate 1 field 1 a 214.452 499.304 2.33x Aggregate 1 field 2 a 217.744 500.07 2.30x Aggregate 9 fields 1 a 281.514 802.799 2.85x Aggregate 9 fields 2 a 272.358 785.816 2.89x Aggregate 1 with GBY 1 a 248.874 558.143 2.24x Aggregate 1 with GBY 2 a 269.658 533.562 1.98x count 1 b 194.739 482.261 2.48x count 2 b 195.178 481.437 2.47x Aggregate 1 field 1 b 220.325 498.956 2.26x Aggregate 1 field 2 b 227.117 489.27 2.15x Aggregate 9 fields 1 b 276.939 817.118 2.95x Aggregate 9 fields 2 b 290.288 876.753 3.02x Aggregate 1 with GBY 1 b 244.025 563.884 2.31x Aggregate 1 with GBY 2 b 225.431 570.723 2.53x count 1 c 194.568 502.79 2.58x count 2 c 205.418 508.319 2.47x Aggregate 1 field 1 c 209.709 531.39 2.53x Aggregate 1 field 2 c 217.551 526.878 2.42x Aggregate 9 fields 1 c 267.93 756.476 2.82x Aggregate 9 fields 2 c 273.107 723.459 2.65x Aggregate 1 with GBY 1 c 240.991 526.053 2.18x Aggregate 1 with GBY 2 c 258.06 527.845 2.05x For those not familiar with YCSB it uses a table with 9 fields, each filled with random junk 100 characters long. It defines workloads A-F, of which I've used A-C. The main point to note is the more of the fields my query fetches, the better it works in snapshot mode. The other thing I measured was throughput as reported by the YCSB tool. For the most part, when running the query over a snapshot the throughput was much better. Workload Tput Snapshot Tput Direct Throughput Improvement (Snapshot) a 83443.11623 56267.34148 48.30% b 45709.15011 31224.30376 46.39% c 46634.58415 43224.86383 7.89% The throughput when using the snapshot seems to be close to the throughput when not scanning data, but I didn't run the baseline tests long enough to get anything conclusive here. In any event this looks like a good patch, especially considering its small size. The numbers quoted here are for reference only, YMMV, etc.
          Hide
          Nick Dimiduk added a comment -

          I think these failed tests are unrelated.

          Show
          Nick Dimiduk added a comment - I think these failed tests are unrelated.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12657183/HIVE-6584.12.patch

          ERROR: -1 due to 6 failed/errored test(s), 5752 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_fail_8
          org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.jdbc.TestJdbcDriver2.testParentReferences
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/9/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/9/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-9/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12657183

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12657183/HIVE-6584.12.patch ERROR: -1 due to 6 failed/errored test(s), 5752 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_fail_8 org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes org.apache.hive.jdbc.TestJdbcDriver2.testParentReferences Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/9/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/9/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-9/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12657183
          Hide
          Nick Dimiduk added a comment -
          Show
          Nick Dimiduk added a comment - Link to RB: https://reviews.apache.org/r/23824/
          Hide
          Nick Dimiduk added a comment -

          Patch v12 should fix the two test failures. One comes from changes made in HBASE-11335. The other has to do with assumptions around default filesystem path that are unrelated to HBase.

          Show
          Nick Dimiduk added a comment - Patch v12 should fix the two test failures. One comes from changes made in HBASE-11335 . The other has to do with assumptions around default filesystem path that are unrelated to HBase.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12657009/HIVE-6584.11.patch

          ERROR: -1 due to 5 failed/errored test(s), 5752 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_snapshot
          org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_fail_8
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/893/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/893/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-893/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12657009

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12657009/HIVE-6584.11.patch ERROR: -1 due to 5 failed/errored test(s), 5752 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_snapshot org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_fail_8 org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/893/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/893/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-893/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12657009
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12656962/HIVE-6584.10.patch

          ERROR: -1 due to 6 failed/errored test(s), 5750 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_pipe
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_fail_8
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/889/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/889/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-889/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12656962

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12656962/HIVE-6584.10.patch ERROR: -1 due to 6 failed/errored test(s), 5750 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_pipe org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_fail_8 org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/889/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/889/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-889/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12656962
          Hide
          Nick Dimiduk added a comment -

          HBASE-11557 will remove the requirement of specifying high-scale-lib.jar in HADOOP_CLASSPATH.

          Show
          Nick Dimiduk added a comment - HBASE-11557 will remove the requirement of specifying high-scale-lib.jar in HADOOP_CLASSPATH.
          Hide
          Nick Dimiduk added a comment -

          Updated the patch once more. This has been tested on a distributed cluster as well, things are working correctly.

          HADOOP_CLASSPATH=/path/to/high-scale-lib-1.1.1.jar hive -e "set hive.hbase.snapshot.name=foo_snap; select count(*) from foo;"
          

          Optionally you can specify hive.hbase.snapshot.restoredir to something other than the default.

          I also opened HBASE-11555 so we can do away with the reflection stuff after a later release.

          Show
          Nick Dimiduk added a comment - Updated the patch once more. This has been tested on a distributed cluster as well, things are working correctly. HADOOP_CLASSPATH=/path/to/high-scale-lib-1.1.1.jar hive -e "set hive.hbase.snapshot.name=foo_snap; select count(*) from foo;" Optionally you can specify hive.hbase.snapshot.restoredir to something other than the default. I also opened HBASE-11555 so we can do away with the reflection stuff after a later release.
          Hide
          Nick Dimiduk added a comment -

          Teng Yutong:

          1. HBaseStorageHandler.getInputFormatClass(): i am afraid that the returned inputformat will always be HiveHBaseTabelInputFormat (at least according to my test)

          My patch has the logic necessary to perform the switch at runtime. It does indeed work with the latest patch.

          2. in the method HBaseStorageHandler.preCreateTable, hive will check whether the HBase table exist or not, regardless the external table that hive gonna create is based on actual table or a snapshot.

          I'm not sure about this. Anyway that's not related to this feature. HBaseStorageHandler has no means of creating/dropping table snapshots. If you're seeing some issue here with StorageHandler DDL operations, please file a separate JIRA.

          3. the TableSnapshotRegionSplit used in TableSnapshotInputFormat is a direct subclass of InputSplit, not a subclass of tablesplit

          Nor should it be. The TableSnapshotRegionSplit is tracking different information from TableSplit.

          4. there is no public setScan method in TableSnapshotInputFormat.RecordReader, instead it will translate a string into a scan instance by using mapreduce.TableMapReduceUitls.convertStringToScan.

          Indeed, there is disparity between the HBase's mapred and mapreduce implementations. I opened HBASE-11179 for some cleanup on the HBase side. convertStringToScan details are HBase-private API as of 0.96. I opened HBASE-11163 to make necessary scanner support available in mapred API, but it's not yet been implemented.

          Show
          Nick Dimiduk added a comment - Teng Yutong : 1. HBaseStorageHandler.getInputFormatClass(): i am afraid that the returned inputformat will always be HiveHBaseTabelInputFormat (at least according to my test) My patch has the logic necessary to perform the switch at runtime. It does indeed work with the latest patch. 2. in the method HBaseStorageHandler.preCreateTable, hive will check whether the HBase table exist or not, regardless the external table that hive gonna create is based on actual table or a snapshot. I'm not sure about this. Anyway that's not related to this feature. HBaseStorageHandler has no means of creating/dropping table snapshots. If you're seeing some issue here with StorageHandler DDL operations, please file a separate JIRA. 3. the TableSnapshotRegionSplit used in TableSnapshotInputFormat is a direct subclass of InputSplit, not a subclass of tablesplit Nor should it be. The TableSnapshotRegionSplit is tracking different information from TableSplit. 4. there is no public setScan method in TableSnapshotInputFormat.RecordReader, instead it will translate a string into a scan instance by using mapreduce.TableMapReduceUitls.convertStringToScan. Indeed, there is disparity between the HBase's mapred and mapreduce implementations. I opened HBASE-11179 for some cleanup on the HBase side. convertStringToScan details are HBase-private API as of 0.96. I opened HBASE-11163 to make necessary scanner support available in mapred API, but it's not yet been implemented.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12656322/HIVE-6584.9.patch

          ERROR: -1 due to 4 failed/errored test(s), 5726 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/836/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/836/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-836/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12656322

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12656322/HIVE-6584.9.patch ERROR: -1 due to 4 failed/errored test(s), 5726 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/836/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/836/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-836/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12656322
          Hide
          Nick Dimiduk added a comment -

          Ouch. Most of these tests run/pass for me locally. Will investigate further. I'm also curious why the explain commands in hbase_handler_snapshot.q are not including the Input/OutputFormats.

          Sushanth Sowmyan, Ashutosh Chauhan any ideas on this latter issue?

          Show
          Nick Dimiduk added a comment - Ouch. Most of these tests run/pass for me locally. Will investigate further. I'm also curious why the explain commands in hbase_handler_snapshot.q are not including the Input/OutputFormats. Sushanth Sowmyan , Ashutosh Chauhan any ideas on this latter issue?
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12655948/HIVE-6584.8.patch

          ERROR: -1 due to 17 failed/errored test(s), 5735 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_multiple
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_bulk
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_snapshot
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_single_sourced_multi_insert
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats2
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats3
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats_empty_partition
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
          org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
          org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_generatehfiles_require_family_path
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/803/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/803/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-803/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 17 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12655948

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12655948/HIVE-6584.8.patch ERROR: -1 due to 17 failed/errored test(s), 5735 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_multiple org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_bulk org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_snapshot org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_single_sourced_multi_insert org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats2 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats3 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats_empty_partition org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_generatehfiles_require_family_path org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/803/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/803/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-803/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 17 tests failed This message is automatically generated. ATTACHMENT ID: 12655948
          Hide
          Nick Dimiduk added a comment -

          Attaching my updated patch. It includes changes to the hbase test drivers so that there are snapshots available to testing from q files.

          Teng Yutong: I'll have a look over your patch tomorrow. Maybe we can put our stuff together and get a working new feature

          Show
          Nick Dimiduk added a comment - Attaching my updated patch. It includes changes to the hbase test drivers so that there are snapshots available to testing from q files. Teng Yutong : I'll have a look over your patch tomorrow. Maybe we can put our stuff together and get a working new feature
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12653797/HIVE-6584.7.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/672/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/672/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-672/

          Messages:

          **** This message was trimmed, see log for full details ****
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-contrib ---
          [INFO] Compiling 39 source files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/classes
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example/UDAFExampleMax.java: Some input files use or override a deprecated API.
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example/UDAFExampleMax.java: Recompile with -Xlint:deprecation for details.
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java: /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java uses unchecked or unsafe operations.
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java: Recompile with -Xlint:unchecked for details.
          [INFO] 
          [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-contrib ---
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/resources
          [INFO] Copying 3 resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-contrib ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-contrib ---
          [INFO] Compiling 2 source files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/test-classes
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java: /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java uses or overrides a deprecated API.
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java: Recompile with -Xlint:deprecation for details.
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-contrib ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-contrib ---
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/hive-contrib-0.14.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-contrib ---
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-contrib ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/hive-contrib-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-contrib/0.14.0-SNAPSHOT/hive-contrib-0.14.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/contrib/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-contrib/0.14.0-SNAPSHOT/hive-contrib-0.14.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive HBase Handler 0.14.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hbase-handler ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-hbase-handler ---
          [INFO] 
          [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-hbase-handler ---
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/main/resources
          [INFO] Copying 3 resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hbase-handler ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hbase-handler ---
          [INFO] Compiling 31 source files to /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/classes
          [INFO] -------------------------------------------------------------
          [WARNING] COMPILATION WARNING : 
          [INFO] -------------------------------------------------------------
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java: Some input files use or override a deprecated API.
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java: Recompile with -Xlint:deprecation for details.
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java: Some input files use unchecked or unsafe operations.
          [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java: Recompile with -Xlint:unchecked for details.
          [INFO] 4 warnings 
          [INFO] -------------------------------------------------------------
          [INFO] -------------------------------------------------------------
          [ERROR] COMPILATION ERROR : 
          [INFO] -------------------------------------------------------------
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol
            symbol:   class TableSnapshotInputFormat
            location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol
            symbol:   class TableSnapshotInputFormat
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[36,41] cannot find symbol
            symbol:   class TableSnapshotInputFormatImpl
            location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol
            symbol:   class TableSnapshotInputFormat
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[125,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[279,46] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[295,5] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[169,11] cannot find symbol
            symbol:   variable TableSnapshotInputFormatImpl
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [INFO] 14 errors 
          [INFO] -------------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Hive .............................................. SUCCESS [9.666s]
          [INFO] Hive Ant Utilities ................................ SUCCESS [5.737s]
          [INFO] Hive Shims Common ................................. SUCCESS [4.585s]
          [INFO] Hive Shims 0.20 ................................... SUCCESS [2.598s]
          [INFO] Hive Shims Secure Common .......................... SUCCESS [4.482s]
          [INFO] Hive Shims 0.20S .................................. SUCCESS [2.989s]
          [INFO] Hive Shims 0.23 ................................... SUCCESS [6.850s]
          [INFO] Hive Shims ........................................ SUCCESS [0.676s]
          [INFO] Hive Common ....................................... SUCCESS [20.287s]
          [INFO] Hive Serde ........................................ SUCCESS [13.145s]
          [INFO] Hive Metastore .................................... SUCCESS [33.448s]
          [INFO] Hive Query Language ............................... SUCCESS [1:29.659s]
          [INFO] Hive Service ...................................... SUCCESS [7.314s]
          [INFO] Hive JDBC ......................................... SUCCESS [15.917s]
          [INFO] Hive Beeline ...................................... SUCCESS [2.681s]
          [INFO] Hive CLI .......................................... SUCCESS [2.613s]
          [INFO] Hive Contrib ...................................... SUCCESS [3.214s]
          [INFO] Hive HBase Handler ................................ FAILURE [7.778s]
          [INFO] Hive HCatalog ..................................... SKIPPED
          [INFO] Hive HCatalog Core ................................ SKIPPED
          [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
          [INFO] Hive HCatalog Server Extensions ................... SKIPPED
          [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
          [INFO] Hive HCatalog Webhcat ............................. SKIPPED
          [INFO] Hive HCatalog Streaming ........................... SKIPPED
          [INFO] Hive HWI .......................................... SKIPPED
          [INFO] Hive ODBC ......................................... SKIPPED
          [INFO] Hive Shims Aggregator ............................. SKIPPED
          [INFO] Hive TestUtils .................................... SKIPPED
          [INFO] Hive Packaging .................................... SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 3:56.405s
          [INFO] Finished at: Thu Jul 03 10:22:22 EDT 2014
          [INFO] Final Memory: 120M/710M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-hbase-handler: Compilation failure: Compilation failure:
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormat
          [ERROR] location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormat
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[36,41] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormatImpl
          [ERROR] location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormat
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[125,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[279,46] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[295,5] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[169,11] cannot find symbol
          [ERROR] symbol:   variable TableSnapshotInputFormatImpl
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hive-hbase-handler
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12653797

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12653797/HIVE-6584.7.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/672/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/672/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-672/ Messages: **** This message was trimmed, see log for full details **** [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-contrib --- [INFO] Compiling 39 source files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/classes [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example/UDAFExampleMax.java: Some input files use or override a deprecated API. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example/UDAFExampleMax.java: Recompile with -Xlint:deprecation for details. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java: /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java uses unchecked or unsafe operations. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-contrib --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-contrib --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-contrib --- [INFO] Compiling 2 source files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/test-classes [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java: /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java uses or overrides a deprecated API. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-contrib --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-contrib --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/hive-contrib-0.14.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-contrib --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-contrib --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/hive-contrib-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-contrib/0.14.0-SNAPSHOT/hive-contrib-0.14.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/contrib/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-contrib/0.14.0-SNAPSHOT/hive-contrib-0.14.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive HBase Handler 0.14.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hbase-handler --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-hbase-handler --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-hbase-handler --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hbase-handler --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hbase-handler --- [INFO] Compiling 31 source files to /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/classes [INFO] ------------------------------------------------------------- [WARNING] COMPILATION WARNING : [INFO] ------------------------------------------------------------- [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java: Some input files use or override a deprecated API. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java: Recompile with -Xlint:deprecation for details. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java: Some input files use unchecked or unsafe operations. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java: Recompile with -Xlint:unchecked for details. [INFO] 4 warnings [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol symbol: class TableSnapshotInputFormat location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol symbol: class TableSnapshotInputFormat location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[36,41] cannot find symbol symbol: class TableSnapshotInputFormatImpl location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol symbol: class TableSnapshotInputFormat location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[125,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[279,46] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[295,5] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[169,11] cannot find symbol symbol: variable TableSnapshotInputFormatImpl location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [INFO] 14 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [9.666s] [INFO] Hive Ant Utilities ................................ SUCCESS [5.737s] [INFO] Hive Shims Common ................................. SUCCESS [4.585s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.598s] [INFO] Hive Shims Secure Common .......................... SUCCESS [4.482s] [INFO] Hive Shims 0.20S .................................. SUCCESS [2.989s] [INFO] Hive Shims 0.23 ................................... SUCCESS [6.850s] [INFO] Hive Shims ........................................ SUCCESS [0.676s] [INFO] Hive Common ....................................... SUCCESS [20.287s] [INFO] Hive Serde ........................................ SUCCESS [13.145s] [INFO] Hive Metastore .................................... SUCCESS [33.448s] [INFO] Hive Query Language ............................... SUCCESS [1:29.659s] [INFO] Hive Service ...................................... SUCCESS [7.314s] [INFO] Hive JDBC ......................................... SUCCESS [15.917s] [INFO] Hive Beeline ...................................... SUCCESS [2.681s] [INFO] Hive CLI .......................................... SUCCESS [2.613s] [INFO] Hive Contrib ...................................... SUCCESS [3.214s] [INFO] Hive HBase Handler ................................ FAILURE [7.778s] [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog Streaming ........................... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3:56.405s [INFO] Finished at: Thu Jul 03 10:22:22 EDT 2014 [INFO] Final Memory: 120M/710M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-hbase-handler: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormat [ERROR] location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormat [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[36,41] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormatImpl [ERROR] location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormat [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[125,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[279,46] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[295,5] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[169,11] cannot find symbol [ERROR] symbol: variable TableSnapshotInputFormatImpl [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-hbase-handler + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12653797
          Hide
          Teng Yutong added a comment -

          fix some bug..but still need changes on HBase side

          Show
          Teng Yutong added a comment - fix some bug..but still need changes on HBase side
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12652773/HIVE-6584.6.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/617/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/617/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-617/

          Messages:

          **** This message was trimmed, see log for full details ****
          [INFO] -------------------------------------------------------------
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol
            symbol:   class TableSnapshotInputFormat
            location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[33,47] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HBaseSerDe
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol
            symbol:   class TableSnapshotInputFormat
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[34,41] cannot find symbol
            symbol:   class TableSnapshotInputFormatImpl
            location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[37,47] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HBaseSerDe
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol
            symbol:   class TableSnapshotInputFormat
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[87,10] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[89,54] incompatible types
            required: java.util.List<ColumnMapping>
            found:    org.apache.hadoop.hive.hbase.ColumnMappings
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[108,9] cannot find symbol
            symbol:   variable HiveHBaseInputFormatUtil
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[118,7] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[119,10] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[134,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[235,46] cannot find symbol
            symbol:   method getCount()
            location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[236,86] cannot find symbol
            symbol:   method getCount()
            location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[287,46] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[303,5] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,12] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,74] incompatible types
            required: java.util.List<ColumnMapping>
            found:    org.apache.hadoop.hive.hbase.ColumnMappings
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[91,11] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[93,14] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[147,13] configureInputJobProps has private access in org.apache.hadoop.hive.hbase.HBaseStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[164,11] cannot find symbol
            symbol:   variable TableSnapshotInputFormatImpl
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [INFO] 28 errors 
          [INFO] -------------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Hive .............................................. SUCCESS [10.392s]
          [INFO] Hive Ant Utilities ................................ SUCCESS [5.910s]
          [INFO] Hive Shims Common ................................. SUCCESS [4.731s]
          [INFO] Hive Shims 0.20 ................................... SUCCESS [2.688s]
          [INFO] Hive Shims Secure Common .......................... SUCCESS [4.394s]
          [INFO] Hive Shims 0.20S .................................. SUCCESS [2.786s]
          [INFO] Hive Shims 0.23 ................................... SUCCESS [7.076s]
          [INFO] Hive Shims ........................................ SUCCESS [0.859s]
          [INFO] Hive Common ....................................... SUCCESS [19.823s]
          [INFO] Hive Serde ........................................ SUCCESS [12.804s]
          [INFO] Hive Metastore .................................... SUCCESS [31.857s]
          [INFO] Hive Query Language ............................... SUCCESS [1:28.518s]
          [INFO] Hive Service ...................................... SUCCESS [7.889s]
          [INFO] Hive JDBC ......................................... SUCCESS [16.035s]
          [INFO] Hive Beeline ...................................... SUCCESS [2.359s]
          [INFO] Hive CLI .......................................... SUCCESS [1.983s]
          [INFO] Hive Contrib ...................................... SUCCESS [2.592s]
          [INFO] Hive HBase Handler ................................ FAILURE [5.046s]
          [INFO] Hive HCatalog ..................................... SKIPPED
          [INFO] Hive HCatalog Core ................................ SKIPPED
          [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
          [INFO] Hive HCatalog Server Extensions ................... SKIPPED
          [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
          [INFO] Hive HCatalog Webhcat ............................. SKIPPED
          [INFO] Hive HCatalog Streaming ........................... SKIPPED
          [INFO] Hive HWI .......................................... SKIPPED
          [INFO] Hive ODBC ......................................... SKIPPED
          [INFO] Hive Shims Aggregator ............................. SKIPPED
          [INFO] Hive TestUtils .................................... SKIPPED
          [INFO] Hive Packaging .................................... SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 3:50.588s
          [INFO] Finished at: Fri Jun 27 06:42:23 EDT 2014
          [INFO] Final Memory: 113M/447M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-hbase-handler: Compilation failure: Compilation failure:
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormat
          [ERROR] location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[33,47] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSerDe
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormat
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[34,41] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormatImpl
          [ERROR] location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[37,47] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSerDe
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormat
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[87,10] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[89,54] incompatible types
          [ERROR] required: java.util.List<ColumnMapping>
          [ERROR] found:    org.apache.hadoop.hive.hbase.ColumnMappings
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[108,9] cannot find symbol
          [ERROR] symbol:   variable HiveHBaseInputFormatUtil
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[118,7] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[119,10] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[134,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[235,46] cannot find symbol
          [ERROR] symbol:   method getCount()
          [ERROR] location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[236,86] cannot find symbol
          [ERROR] symbol:   method getCount()
          [ERROR] location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[287,46] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[303,5] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,12] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,74] incompatible types
          [ERROR] required: java.util.List<ColumnMapping>
          [ERROR] found:    org.apache.hadoop.hive.hbase.ColumnMappings
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[91,11] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[93,14] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[147,13] configureInputJobProps has private access in org.apache.hadoop.hive.hbase.HBaseStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[164,11] cannot find symbol
          [ERROR] symbol:   variable TableSnapshotInputFormatImpl
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hive-hbase-handler
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12652773

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12652773/HIVE-6584.6.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/617/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/617/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-617/ Messages: **** This message was trimmed, see log for full details **** [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol symbol: class TableSnapshotInputFormat location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[33,47] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HBaseSerDe [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol symbol: class TableSnapshotInputFormat location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[34,41] cannot find symbol symbol: class TableSnapshotInputFormatImpl location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[37,47] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HBaseSerDe [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol symbol: class TableSnapshotInputFormat location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[87,10] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[89,54] incompatible types required: java.util.List<ColumnMapping> found: org.apache.hadoop.hive.hbase.ColumnMappings [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[108,9] cannot find symbol symbol: variable HiveHBaseInputFormatUtil location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[118,7] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[119,10] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[134,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[235,46] cannot find symbol symbol: method getCount() location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[236,86] cannot find symbol symbol: method getCount() location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[287,46] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[303,5] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,12] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,74] incompatible types required: java.util.List<ColumnMapping> found: org.apache.hadoop.hive.hbase.ColumnMappings [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[91,11] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[93,14] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[147,13] configureInputJobProps has private access in org.apache.hadoop.hive.hbase.HBaseStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[164,11] cannot find symbol symbol: variable TableSnapshotInputFormatImpl location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [INFO] 28 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [10.392s] [INFO] Hive Ant Utilities ................................ SUCCESS [5.910s] [INFO] Hive Shims Common ................................. SUCCESS [4.731s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.688s] [INFO] Hive Shims Secure Common .......................... SUCCESS [4.394s] [INFO] Hive Shims 0.20S .................................. SUCCESS [2.786s] [INFO] Hive Shims 0.23 ................................... SUCCESS [7.076s] [INFO] Hive Shims ........................................ SUCCESS [0.859s] [INFO] Hive Common ....................................... SUCCESS [19.823s] [INFO] Hive Serde ........................................ SUCCESS [12.804s] [INFO] Hive Metastore .................................... SUCCESS [31.857s] [INFO] Hive Query Language ............................... SUCCESS [1:28.518s] [INFO] Hive Service ...................................... SUCCESS [7.889s] [INFO] Hive JDBC ......................................... SUCCESS [16.035s] [INFO] Hive Beeline ...................................... SUCCESS [2.359s] [INFO] Hive CLI .......................................... SUCCESS [1.983s] [INFO] Hive Contrib ...................................... SUCCESS [2.592s] [INFO] Hive HBase Handler ................................ FAILURE [5.046s] [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog Streaming ........................... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3:50.588s [INFO] Finished at: Fri Jun 27 06:42:23 EDT 2014 [INFO] Final Memory: 113M/447M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-hbase-handler: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormat [ERROR] location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[33,47] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSerDe [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormat [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[34,41] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormatImpl [ERROR] location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[37,47] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSerDe [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormat [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[87,10] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[89,54] incompatible types [ERROR] required: java.util.List<ColumnMapping> [ERROR] found: org.apache.hadoop.hive.hbase.ColumnMappings [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[108,9] cannot find symbol [ERROR] symbol: variable HiveHBaseInputFormatUtil [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[118,7] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[119,10] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[134,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[235,46] cannot find symbol [ERROR] symbol: method getCount() [ERROR] location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[236,86] cannot find symbol [ERROR] symbol: method getCount() [ERROR] location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[287,46] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[303,5] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,12] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,74] incompatible types [ERROR] required: java.util.List<ColumnMapping> [ERROR] found: org.apache.hadoop.hive.hbase.ColumnMappings [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[91,11] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[93,14] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[147,13] configureInputJobProps has private access in org.apache.hadoop.hive.hbase.HBaseStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[164,11] cannot find symbol [ERROR] symbol: variable TableSnapshotInputFormatImpl [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-hbase-handler + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12652773
          Hide
          Teng Yutong added a comment -

          hi,

          sorry for the late reply...this is the regenerated patch. But It won't work unless HBase has been modified. Because we need HBase expose TableSnapshotRegionSplit and convertStringToScan.

          BR

          Show
          Teng Yutong added a comment - hi, sorry for the late reply...this is the regenerated patch. But It won't work unless HBase has been modified. Because we need HBase expose TableSnapshotRegionSplit and convertStringToScan. BR
          Hide
          Nick Dimiduk added a comment -

          Can you regenerate your patch, rooted in the trunk directory instead of above it? That's the reason this patch fails the buildbot.

          Show
          Nick Dimiduk added a comment - Can you regenerate your patch, rooted in the trunk directory instead of above it? That's the reason this patch fails the buildbot.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12651366/HIVE-6584.5.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/532/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/532/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-532/

          Messages:

          **** This message was trimmed, see log for full details ****
          [INFO] -------------------------------------------------------------
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol
            symbol:   class TableSnapshotInputFormat
            location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[33,47] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HBaseSerDe
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol
            symbol:   class TableSnapshotInputFormat
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[34,41] cannot find symbol
            symbol:   class TableSnapshotInputFormatImpl
            location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[37,47] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HBaseSerDe
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol
            symbol:   class TableSnapshotInputFormat
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[87,10] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[89,54] incompatible types
            required: java.util.List<ColumnMapping>
            found:    org.apache.hadoop.hive.hbase.ColumnMappings
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[108,9] cannot find symbol
            symbol:   variable HiveHBaseInputFormatUtil
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[118,7] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[119,10] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[134,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[235,46] cannot find symbol
            symbol:   method getCount()
            location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[236,86] cannot find symbol
            symbol:   method getCount()
            location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[287,46] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[303,5] cannot find symbol
            symbol:   class TableSnapshotRegionSplit
            location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,12] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,74] incompatible types
            required: java.util.List<ColumnMapping>
            found:    org.apache.hadoop.hive.hbase.ColumnMappings
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[91,11] cannot find symbol
            symbol:   class ColumnMapping
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[93,14] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[147,13] configureInputJobProps has private access in org.apache.hadoop.hive.hbase.HBaseStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[164,11] cannot find symbol
            symbol:   variable TableSnapshotInputFormatImpl
            location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [INFO] 28 errors 
          [INFO] -------------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Hive .............................................. SUCCESS [9.949s]
          [INFO] Hive Ant Utilities ................................ SUCCESS [5.805s]
          [INFO] Hive Shims Common ................................. SUCCESS [4.226s]
          [INFO] Hive Shims 0.20 ................................... SUCCESS [2.731s]
          [INFO] Hive Shims Secure Common .......................... SUCCESS [4.313s]
          [INFO] Hive Shims 0.20S .................................. SUCCESS [2.754s]
          [INFO] Hive Shims 0.23 ................................... SUCCESS [6.647s]
          [INFO] Hive Shims ........................................ SUCCESS [0.822s]
          [INFO] Hive Common ....................................... SUCCESS [15.288s]
          [INFO] Hive Serde ........................................ SUCCESS [12.681s]
          [INFO] Hive Metastore .................................... SUCCESS [30.807s]
          [INFO] Hive Query Language ............................... SUCCESS [1:30.590s]
          [INFO] Hive Service ...................................... SUCCESS [8.958s]
          [INFO] Hive JDBC ......................................... SUCCESS [15.328s]
          [INFO] Hive Beeline ...................................... SUCCESS [2.985s]
          [INFO] Hive CLI .......................................... SUCCESS [1.850s]
          [INFO] Hive Contrib ...................................... SUCCESS [2.599s]
          [INFO] Hive HBase Handler ................................ FAILURE [5.462s]
          [INFO] Hive HCatalog ..................................... SKIPPED
          [INFO] Hive HCatalog Core ................................ SKIPPED
          [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
          [INFO] Hive HCatalog Server Extensions ................... SKIPPED
          [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
          [INFO] Hive HCatalog Webhcat ............................. SKIPPED
          [INFO] Hive HCatalog Streaming ........................... SKIPPED
          [INFO] Hive HWI .......................................... SKIPPED
          [INFO] Hive ODBC ......................................... SKIPPED
          [INFO] Hive Shims Aggregator ............................. SKIPPED
          [INFO] Hive TestUtils .................................... SKIPPED
          [INFO] Hive Packaging .................................... SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 3:46.497s
          [INFO] Finished at: Fri Jun 20 16:50:16 EDT 2014
          [INFO] Final Memory: 100M/507M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-hbase-handler: Compilation failure: Compilation failure:
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormat
          [ERROR] location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[33,47] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSerDe
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormat
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[34,41] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormatImpl
          [ERROR] location: package org.apache.hadoop.hbase.mapreduce
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[37,47] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSerDe
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol
          [ERROR] symbol:   class TableSnapshotInputFormat
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[87,10] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[89,54] incompatible types
          [ERROR] required: java.util.List<ColumnMapping>
          [ERROR] found:    org.apache.hadoop.hive.hbase.ColumnMappings
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[108,9] cannot find symbol
          [ERROR] symbol:   variable HiveHBaseInputFormatUtil
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[118,7] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[119,10] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[134,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[235,46] cannot find symbol
          [ERROR] symbol:   method getCount()
          [ERROR] location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[236,86] cannot find symbol
          [ERROR] symbol:   method getCount()
          [ERROR] location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[287,46] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[303,5] cannot find symbol
          [ERROR] symbol:   class TableSnapshotRegionSplit
          [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,12] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,74] incompatible types
          [ERROR] required: java.util.List<ColumnMapping>
          [ERROR] found:    org.apache.hadoop.hive.hbase.ColumnMappings
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[91,11] cannot find symbol
          [ERROR] symbol:   class ColumnMapping
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[93,14] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[147,13] configureInputJobProps has private access in org.apache.hadoop.hive.hbase.HBaseStorageHandler
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[164,11] cannot find symbol
          [ERROR] symbol:   variable TableSnapshotInputFormatImpl
          [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler
          [ERROR] -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hive-hbase-handler
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12651366

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12651366/HIVE-6584.5.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/532/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/532/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-532/ Messages: **** This message was trimmed, see log for full details **** [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol symbol: class TableSnapshotInputFormat location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[33,47] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HBaseSerDe [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol symbol: class TableSnapshotInputFormat location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[34,41] cannot find symbol symbol: class TableSnapshotInputFormatImpl location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[37,47] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HBaseSerDe [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol symbol: class TableSnapshotInputFormat location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[87,10] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[89,54] incompatible types required: java.util.List<ColumnMapping> found: org.apache.hadoop.hive.hbase.ColumnMappings [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[108,9] cannot find symbol symbol: variable HiveHBaseInputFormatUtil location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[118,7] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[119,10] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[134,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[235,46] cannot find symbol symbol: method getCount() location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[236,86] cannot find symbol symbol: method getCount() location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[287,46] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[303,5] cannot find symbol symbol: class TableSnapshotRegionSplit location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,12] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,74] incompatible types required: java.util.List<ColumnMapping> found: org.apache.hadoop.hive.hbase.ColumnMappings [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[91,11] cannot find symbol symbol: class ColumnMapping location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[93,14] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[147,13] configureInputJobProps has private access in org.apache.hadoop.hive.hbase.HBaseStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[164,11] cannot find symbol symbol: variable TableSnapshotInputFormatImpl location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [INFO] 28 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [9.949s] [INFO] Hive Ant Utilities ................................ SUCCESS [5.805s] [INFO] Hive Shims Common ................................. SUCCESS [4.226s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.731s] [INFO] Hive Shims Secure Common .......................... SUCCESS [4.313s] [INFO] Hive Shims 0.20S .................................. SUCCESS [2.754s] [INFO] Hive Shims 0.23 ................................... SUCCESS [6.647s] [INFO] Hive Shims ........................................ SUCCESS [0.822s] [INFO] Hive Common ....................................... SUCCESS [15.288s] [INFO] Hive Serde ........................................ SUCCESS [12.681s] [INFO] Hive Metastore .................................... SUCCESS [30.807s] [INFO] Hive Query Language ............................... SUCCESS [1:30.590s] [INFO] Hive Service ...................................... SUCCESS [8.958s] [INFO] Hive JDBC ......................................... SUCCESS [15.328s] [INFO] Hive Beeline ...................................... SUCCESS [2.985s] [INFO] Hive CLI .......................................... SUCCESS [1.850s] [INFO] Hive Contrib ...................................... SUCCESS [2.599s] [INFO] Hive HBase Handler ................................ FAILURE [5.462s] [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog Streaming ........................... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3:46.497s [INFO] Finished at: Fri Jun 20 16:50:16 EDT 2014 [INFO] Final Memory: 100M/507M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-hbase-handler: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[10,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[17,17] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[24,29] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[29,10] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[28,41] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormat [ERROR] location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[31,66] package org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat does not exist [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[33,47] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSerDe [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,3] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormat [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[34,41] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormatImpl [ERROR] location: package org.apache.hadoop.hbase.mapreduce [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[37,47] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSerDe [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotSplit.java:[21,17] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotSplit [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[76,43] cannot find symbol [ERROR] symbol: class TableSnapshotInputFormat [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[87,10] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[89,54] incompatible types [ERROR] required: java.util.List<ColumnMapping> [ERROR] found: org.apache.hadoop.hive.hbase.ColumnMappings [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[108,9] cannot find symbol [ERROR] symbol: variable HiveHBaseInputFormatUtil [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[118,7] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[119,10] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[134,54] convertScanToString(org.apache.hadoop.hbase.client.Scan) is not public in org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; cannot be accessed from outside package [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[235,46] cannot find symbol [ERROR] symbol: method getCount() [ERROR] location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[236,86] cannot find symbol [ERROR] symbol: method getCount() [ERROR] location: variable serializeStream of type org.apache.hadoop.hive.serde2.ByteStream.Output [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[287,46] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableSnapshotInputFormat.java:[303,5] cannot find symbol [ERROR] symbol: class TableSnapshotRegionSplit [ERROR] location: class org.apache.hadoop.hive.hbase.HiveHBaseTableSnapshotInputFormat [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,12] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[73,74] incompatible types [ERROR] required: java.util.List<ColumnMapping> [ERROR] found: org.apache.hadoop.hive.hbase.ColumnMappings [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[91,11] cannot find symbol [ERROR] symbol: class ColumnMapping [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[93,14] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[147,13] configureInputJobProps has private access in org.apache.hadoop.hive.hbase.HBaseStorageHandler [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSnapshotStorageHandler.java:[164,11] cannot find symbol [ERROR] symbol: variable TableSnapshotInputFormatImpl [ERROR] location: class org.apache.hadoop.hive.hbase.HBaseSnapshotStorageHandler [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-hbase-handler + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12651366
          Hide
          Teng Yutong added a comment -

          hi,

          this patch is my current workaround when dealing with HBase snapshot.

          but in order to make this patch work, still some changes is needed on the HBase side (change the visible descriptor of mapreduce.TableMapReduceUitls.convertStringToScan and mapreduce.TableSnapshotInputFormat.TableSnapshotRegionSplit into public). Since there is no issue related to this in HBase JIRA, so i haven't create a patch for these changes.

          Show
          Teng Yutong added a comment - hi, this patch is my current workaround when dealing with HBase snapshot. but in order to make this patch work, still some changes is needed on the HBase side (change the visible descriptor of mapreduce.TableMapReduceUitls.convertStringToScan and mapreduce.TableSnapshotInputFormat.TableSnapshotRegionSplit into public). Since there is no issue related to this in HBase JIRA, so i haven't create a patch for these changes.
          Hide
          Sushanth Sowmyan added a comment -

          zjkyly/Teng Yutong : Could you please upload a patch for your proposal so we might compare/contrast?

          Show
          Sushanth Sowmyan added a comment - zjkyly / Teng Yutong : Could you please upload a patch for your proposal so we might compare/contrast?
          Hide
          zjkyly added a comment -

          hi Nick Dimiduk and Teng Yutong. what we can do at repsent is to scan all kv records of snapshot.
          we weren't able to solve the issue that hfile had different version for record, so we set default version of hbase table as 1, and run major compact before doing snapshot table.
          we don't know the open soure community, is it necessary to solve the problem of same kv version? shall we find all kv version ? or just find the latest one?
          We will try to solve the problem of multiple versions.

          Show
          zjkyly added a comment - hi Nick Dimiduk and Teng Yutong. what we can do at repsent is to scan all kv records of snapshot. we weren't able to solve the issue that hfile had different version for record, so we set default version of hbase table as 1, and run major compact before doing snapshot table. we don't know the open soure community, is it necessary to solve the problem of same kv version? shall we find all kv version ? or just find the latest one? We will try to solve the problem of multiple versions.
          Hide
          zjkyly added a comment -

          Teng YuTong and I are colleagues. we have a patch for HIVE-6584 and a patch for HBASE-11163 , and we modify
          org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat(line 93)
          from: static class TableSnapshotRegionSplit extends InputSplit implements Writable
          to: public static class TableSnapshotRegionSplit extends InputSplit implements Writable

          we can run mapred on snapshot. mapred (count(1)) result :

          2014-06-17 16:29:34,540 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2467.57 sec
          2014-06-17 16:29:35,578 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2468.35 sec
          MapReduce Total cumulative CPU time: 41 minutes 8 seconds 350 msec
          Ended Job = job_1402970116480_0015
          MapReduce Jobs Launched:
          Job 0: Map: 64 Reduce: 1 Cumulative CPU: 2468.35 sec HDFS Read: 18334 HDFS Write: 9 SUCCESS
          Total MapReduce CPU Time Spent: 41 minutes 8 seconds 350 msec
          OK
          65497163
          Time taken: 429.647 seconds, Fetched: 1 row(s)

          hbase count result:
          Current count: 65400000, row: user987684650651905350
          65497163 row(s) in 1446.2310 seconds
          => 65497163

          but hfile has different versions of the record. We can not solve this problem. So, we set the version of hbase table is 1, and run major compact before snapshot table.

          Show
          zjkyly added a comment - Teng YuTong and I are colleagues. we have a patch for HIVE-6584 and a patch for HBASE-11163 , and we modify org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat(line 93) from: static class TableSnapshotRegionSplit extends InputSplit implements Writable to: public static class TableSnapshotRegionSplit extends InputSplit implements Writable we can run mapred on snapshot. mapred (count(1)) result : 2014-06-17 16:29:34,540 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2467.57 sec 2014-06-17 16:29:35,578 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2468.35 sec MapReduce Total cumulative CPU time: 41 minutes 8 seconds 350 msec Ended Job = job_1402970116480_0015 MapReduce Jobs Launched: Job 0: Map: 64 Reduce: 1 Cumulative CPU: 2468.35 sec HDFS Read: 18334 HDFS Write: 9 SUCCESS Total MapReduce CPU Time Spent: 41 minutes 8 seconds 350 msec OK 65497163 Time taken: 429.647 seconds, Fetched: 1 row(s) hbase count result: Current count: 65400000, row: user987684650651905350 65497163 row(s) in 1446.2310 seconds => 65497163 but hfile has different versions of the record. We can not solve this problem. So, we set the version of hbase table is 1, and run major compact before snapshot table.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12650362/HIVE-6584.4.patch

          ERROR: -1 due to 6 failed/errored test(s), 5536 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_columnar
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/472/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/472/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-472/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12650362

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12650362/HIVE-6584.4.patch ERROR: -1 due to 6 failed/errored test(s), 5536 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_columnar org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/472/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/472/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-472/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12650362
          Hide
          Nick Dimiduk added a comment -

          Rebased onto trunk and fixed two broken hbase tests.

          Show
          Nick Dimiduk added a comment - Rebased onto trunk and fixed two broken hbase tests.
          Hide
          Sushanth Sowmyan added a comment -

          Teng, I'd be interested in how your patch winds up being.

          If you mean that at runtime, the HBaseStorageHandler decides to deputize a subclass of itself to do the work, then that might work. But if you mean that your approach would lead to the user having to create a separate table (kinda like a view) that associates with a snapshot, then speaking from the hive side, I think I would prefer having only one SH to deal with, and having it decide what to do with various set parameters as opposed to creating separate hive tables with a different SH in hive. That way, using the same hive table definition, a query could decide to use a snapshot or not.

          Show
          Sushanth Sowmyan added a comment - Teng, I'd be interested in how your patch winds up being. If you mean that at runtime, the HBaseStorageHandler decides to deputize a subclass of itself to do the work, then that might work. But if you mean that your approach would lead to the user having to create a separate table (kinda like a view) that associates with a snapshot, then speaking from the hive side, I think I would prefer having only one SH to deal with, and having it decide what to do with various set parameters as opposed to creating separate hive tables with a different SH in hive. That way, using the same hive table definition, a query could decide to use a snapshot or not.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12649918/HIVE-6584.3.patch

          ERROR: -1 due to 9 failed/errored test(s), 5610 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
          org.apache.hive.hcatalog.templeton.tool.TestTempletonUtils.testPropertiesParsing
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/446/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/446/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-446/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 9 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12649918

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12649918/HIVE-6584.3.patch ERROR: -1 due to 9 failed/errored test(s), 5610 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers org.apache.hive.hcatalog.templeton.tool.TestTempletonUtils.testPropertiesParsing Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/446/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/446/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-446/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed This message is automatically generated. ATTACHMENT ID: 12649918
          Hide
          Nick Dimiduk added a comment -

          Thanks for the insightful comments, Teng Yutong.

          1. HBaseStorageHandler.getInputFormatClass(): i am afraid that the returned inputformat will always be HiveHBaseTabelInputFormat (at least according to my test)

          I was afraid of this in my initial design thinking, but my experiments proved otherwise. Can you elaborate on your tests? I'd like to reproduce this issue if I'm able.

          2. in the method HBaseStorageHandler.preCreateTable, hive will check whether the HBase table exist or not, regardless the external table that hive gonna create is based on actual table or a snapshot.

          I haven't yet looked at the use-case of consuming a snapshot for which there is no table in HBase. I planned to approach this kind of feature in follow-on work; the goal here is to get jus the basics working.

          3, 4 [snip]

          These are both true.

          So I suggest adding a subclass of HBaseStorageHandler(and other necessary classes) ,say HBaseSnapshotStorageHandler, to deal with the hbase snapshot situation.

          A goal of this patch is to be able to query snapshots created from online tables already registered with Hive using the HBaseStorageHandler. Implementing HBaseSnapshotStorageHandler requires a separate table registration for the snapshot. I think that's undesirable. Regarding the "hbase snapshot situation", let's make it better on the HBase side. What do you recommend?

          Show
          Nick Dimiduk added a comment - Thanks for the insightful comments, Teng Yutong . 1. HBaseStorageHandler.getInputFormatClass(): i am afraid that the returned inputformat will always be HiveHBaseTabelInputFormat (at least according to my test) I was afraid of this in my initial design thinking, but my experiments proved otherwise. Can you elaborate on your tests? I'd like to reproduce this issue if I'm able. 2. in the method HBaseStorageHandler.preCreateTable, hive will check whether the HBase table exist or not, regardless the external table that hive gonna create is based on actual table or a snapshot. I haven't yet looked at the use-case of consuming a snapshot for which there is no table in HBase. I planned to approach this kind of feature in follow-on work; the goal here is to get jus the basics working. 3, 4 [snip] These are both true. So I suggest adding a subclass of HBaseStorageHandler(and other necessary classes) ,say HBaseSnapshotStorageHandler, to deal with the hbase snapshot situation. A goal of this patch is to be able to query snapshots created from online tables already registered with Hive using the HBaseStorageHandler. Implementing HBaseSnapshotStorageHandler requires a separate table registration for the snapshot. I think that's undesirable. Regarding the "hbase snapshot situation", let's make it better on the HBase side. What do you recommend?
          Hide
          Teng Yutong added a comment -

          hi nick,

          i have some concerns about these patches:
          1. HBaseStorageHandler.getInputFormatClass(): i am afraid that the returned inputformat will always be HiveHBaseTabelInputFormat (at least according to my test)
          2. in the method HBaseStorageHandler.preCreateTable, hive will check whether the HBase table exist or not, regardless the external table that hive gonna create is based on actual table or a snapshot.
          3. the TableSnapshotRegionSplit used in TableSnapshotInputFormat is a direct subclass of InputSplit, not a subclass of tablesplit
          4. there is no public setScan method in TableSnapshotInputFormat.RecordReader, instead it will translate a string into a scan instance by using mapreduce.TableMapReduceUitls.convertStringToScan.

          So I suggest adding a subclass of HBaseStorageHandler(and other necessary classes) ,say HBaseSnapshotStorageHandler, to deal with the hbase snapshot situation.

          In fact, I have already finished the necessary code changes and done some tests. The tests show that my modification works out.

          i will upload my patch soon

          Show
          Teng Yutong added a comment - hi nick, i have some concerns about these patches: 1. HBaseStorageHandler.getInputFormatClass(): i am afraid that the returned inputformat will always be HiveHBaseTabelInputFormat (at least according to my test) 2. in the method HBaseStorageHandler.preCreateTable, hive will check whether the HBase table exist or not, regardless the external table that hive gonna create is based on actual table or a snapshot. 3. the TableSnapshotRegionSplit used in TableSnapshotInputFormat is a direct subclass of InputSplit, not a subclass of tablesplit 4. there is no public setScan method in TableSnapshotInputFormat.RecordReader, instead it will translate a string into a scan instance by using mapreduce.TableMapReduceUitls.convertStringToScan. So I suggest adding a subclass of HBaseStorageHandler(and other necessary classes) ,say HBaseSnapshotStorageHandler, to deal with the hbase snapshot situation. In fact, I have already finished the necessary code changes and done some tests. The tests show that my modification works out. i will upload my patch soon
          Hide
          Nick Dimiduk added a comment -

          Ping. Rebased onto trunk.

          Show
          Nick Dimiduk added a comment - Ping. Rebased onto trunk.
          Hide
          Nick Dimiduk added a comment -

          Rebased onto trunk, mostly clean, though there were some changes to merge since HIVE-6411.

          Also updated pom.xml to hbase-0.98.3. This will be released by the end of the month and will include the dependency, HBASE-11137.

          I'm still looking for advice for adding tests.

          Please have a look, Sushanth Sowmyan, Ashutosh Chauhan, Sergey Shelukhin, Swarnim Kulkarni, Navis.

          Show
          Nick Dimiduk added a comment - Rebased onto trunk, mostly clean, though there were some changes to merge since HIVE-6411 . Also updated pom.xml to hbase-0.98.3. This will be released by the end of the month and will include the dependency, HBASE-11137 . I'm still looking for advice for adding tests. Please have a look, Sushanth Sowmyan , Ashutosh Chauhan , Sergey Shelukhin , Swarnim Kulkarni , Navis .
          Hide
          Nick Dimiduk added a comment -

          Thanks Teng Yutong. Any thoughts regarding how to test this?

          Show
          Nick Dimiduk added a comment - Thanks Teng Yutong . Any thoughts regarding how to test this?
          Hide
          Teng Yutong added a comment -

          this patch is based on the newest patch(HBASE-11137.02-0.98.patch) related HBASE-11137.

          Show
          Teng Yutong added a comment - this patch is based on the newest patch( HBASE-11137 .02-0.98.patch) related HBASE-11137 .
          Hide
          Nick Dimiduk added a comment -

          Attaching preliminary patch, based on the patch attached to HBASE-11137.

          In order to test this properly, I need an HBase table snapshot created. Short of exposing this through hive sql, how can I write a .q file test for this?

          Show
          Nick Dimiduk added a comment - Attaching preliminary patch, based on the patch attached to HBASE-11137 . In order to test this properly, I need an HBase table snapshot created. Short of exposing this through hive sql, how can I write a .q file test for this?
          Hide
          Swarnim Kulkarni added a comment -

          +1 on this one. This should be very nice addition to the existing integration.

          Show
          Swarnim Kulkarni added a comment - +1 on this one. This should be very nice addition to the existing integration.

            People

            • Assignee:
              Nick Dimiduk
              Reporter:
              Nick Dimiduk
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development