Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-2743

HivePhoenixHandler for big-big join with predicate push down

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.5.0, 4.6.0
    • Fix Version/s: 4.8.0
    • Environment:

      hive-1.2.1

      Description

      Phoenix support hash join & sort-merge join. But in case of big*big join does not process well.
      Therefore Need other method like Hive.
      I implemented hive-phoenix-handler that can access Apache Phoenix table on HBase using HiveQL.
      hive-phoenix-handler is very faster than hive-hbase-handler because of applying predicate push down.
      I am publishing source code to github for contribution and maybe will be completed by next week.
      https://github.com/mini666/hive-phoenix-handler

      please, review my proposal.

      1. PHOENIX-2743-1.patch
        258 kB
        Sergey Soldatov
      2. PHOENIX-2743-0.98.patch
        327 kB
        Sergey Soldatov
      3. hivephoenixhandler.jstack
        881 kB
        Josh Elser

        Issue Links

          Activity

          Hide
          ankit@apache.org Ankit Singhal added a comment -

          Bulk close of all issues that has been resolved in a released version.

          Show
          ankit@apache.org Ankit Singhal added a comment - Bulk close of all issues that has been resolved in a released version.
          Hide
          elserj Josh Elser added a comment -

          Applied to 4.x-HBase-0.98 as well. Thanks for the great work here Sergey Soldatov (and JeongMin Ju too!).

          Show
          elserj Josh Elser added a comment - Applied to 4.x-HBase-0.98 as well. Thanks for the great work here Sergey Soldatov (and JeongMin Ju too!).
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Phoenix-master #1196 (See https://builds.apache.org/job/Phoenix-master/1196/)
          PHOENIX-2743 HivePhoenixHandler for big-big join with predicate push (elserj: rev 537b90bef90ff650bb0f00fe0591e4fab6ec7391)

          • phoenix-hive/src/it/java/org/apache/phoenix/hive/HiveTestUtil.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixRowKey.java
          • phoenix-server/pom.xml
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/util/PhoenixStorageHandlerUtil.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixObjectInspectorFactory.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixCharObjectInspector.java
          • phoenix-hive/pom.xml
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixRecordWriter.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/IndexSearchCondition.java
          • pom.xml
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/ppd/PhoenixPredicateDecomposer.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixStringObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/constants/PhoenixStorageHandlerConstants.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixDecimalObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/query/PhoenixQueryBuilder.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixOutputFormat.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixDoubleObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixDateObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/util/PhoenixUtil.java
          • phoenix-hive/src/it/java/org/apache/phoenix/hive/HivePhoenixStoreIT.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixRecordReader.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixResultWritable.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixTimestampObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixRow.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixBinaryObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/ppd/PhoenixPredicateDecomposerManager.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixMetaHook.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixBooleanObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputSplit.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixByteObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixSerDe.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixFloatObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixRecordUpdater.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixIntObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/IndexPredicateAnalyzer.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixLongObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/util/PhoenixConnectionUtil.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixStorageHandler.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/AbstractPhoenixObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixShortObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/PredicateAnalyzerFactory.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixListObjectInspector.java
          • phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixSerializer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Phoenix-master #1196 (See https://builds.apache.org/job/Phoenix-master/1196/ ) PHOENIX-2743 HivePhoenixHandler for big-big join with predicate push (elserj: rev 537b90bef90ff650bb0f00fe0591e4fab6ec7391) phoenix-hive/src/it/java/org/apache/phoenix/hive/HiveTestUtil.java phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixRowKey.java phoenix-server/pom.xml phoenix-hive/src/main/java/org/apache/phoenix/hive/util/PhoenixStorageHandlerUtil.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixObjectInspectorFactory.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixCharObjectInspector.java phoenix-hive/pom.xml phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixRecordWriter.java phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/IndexSearchCondition.java pom.xml phoenix-hive/src/main/java/org/apache/phoenix/hive/ppd/PhoenixPredicateDecomposer.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixStringObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/constants/PhoenixStorageHandlerConstants.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixDecimalObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/query/PhoenixQueryBuilder.java phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixOutputFormat.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixDoubleObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixDateObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/util/PhoenixUtil.java phoenix-hive/src/it/java/org/apache/phoenix/hive/HivePhoenixStoreIT.java phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixRecordReader.java phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixResultWritable.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixTimestampObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixRow.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixBinaryObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/ppd/PhoenixPredicateDecomposerManager.java phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixMetaHook.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixBooleanObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputSplit.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixByteObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixSerDe.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixFloatObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixInputFormat.java phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixRecordUpdater.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixIntObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/IndexPredicateAnalyzer.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixLongObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/util/PhoenixConnectionUtil.java phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixStorageHandler.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/AbstractPhoenixObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixShortObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/PredicateAnalyzerFactory.java phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixListObjectInspector.java phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixSerializer.java
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Patch for 4.x-HBase-0.98

          Show
          sergey.soldatov Sergey Soldatov added a comment - Patch for 4.x-HBase-0.98
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Sure

          Show
          sergey.soldatov Sergey Soldatov added a comment - Sure
          Hide
          elserj Josh Elser added a comment -

          Alrighty, I just pushed this to master and 4.x-HBase-1.0. Sergey Soldatov, this had lots of errors trying to bring it back to 4.x-HBase-0.98. Can you look at a patch for that, please?

          Show
          elserj Josh Elser added a comment - Alrighty, I just pushed this to master and 4.x-HBase-1.0. Sergey Soldatov , this had lots of errors trying to bring it back to 4.x-HBase-0.98. Can you look at a patch for that, please?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/phoenix/pull/155

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/phoenix/pull/155
          Hide
          elserj Josh Elser added a comment -

          Could it be that you have core-site.xml in your path/classpath and it interfere with the test?

          I unset all of my HADOOP_* environment variables and that did the trick. It seems like I should be filing a bug against MiniMR now (goody).

          I'll see if I can come around with a quick workaround (or at least warning for others who might run into it) and then push these changes.

          Thanks for your help, Sergey!

          Show
          elserj Josh Elser added a comment - Could it be that you have core-site.xml in your path/classpath and it interfere with the test? I unset all of my HADOOP_* environment variables and that did the trick. It seems like I should be filing a bug against MiniMR now (goody). I'll see if I can come around with a quick workaround (or at least warning for others who might run into it) and then push these changes. Thanks for your help, Sergey!
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Josh Elser Nope, on my setup I had the only problem with number of open flies and it passed fine once I set ulimit -n. Could it be that you have core-site.xml in your path/classpath and it interfere with the test?

          Show
          sergey.soldatov Sergey Soldatov added a comment - Josh Elser Nope, on my setup I had the only problem with number of open flies and it passed fine once I set ulimit -n. Could it be that you have core-site.xml in your path/classpath and it interfere with the test?
          Hide
          elserj Josh Elser added a comment -

          I've gotten HivePhoenixStoreIT to pass in Eclipse, but I have still been unable to get it to pass on the commandline (same command as above). The MRAppMaster for (what I assume is) the hive job is failing to talk to HDFS, thinking that it should be at localhost:8020.

          2016-04-19 16:48:46,179 ERROR [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Failed while checking for/creating  history staging path: [hdfs://localhost:8020/tmp/hadoop-yarn/staging/jelser/.staging/job_1461098862737_0002]
          java.net.ConnectException: Call From myhostname/127.0.0.1 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
                  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
                  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                  at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
                  at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
                  at org.apache.hadoop.ipc.Client.call(Client.java:1480)
                  at org.apache.hadoop.ipc.Client.call(Client.java:1407)
                  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
                  at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
                  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:606)
                  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
                  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
                  at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
                  at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116)
                  at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
                  at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
                  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
                  at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
                  at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
                  at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.mkdir(JobHistoryEventHandler.java:267)
                  at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceInit(JobHistoryEventHandler.java:166)
                  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
                  at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
                  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:450)
                  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
                  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at javax.security.auth.Subject.doAs(Subject.java:415)
                  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
                  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
                  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
          Caused by: java.net.ConnectException: Connection refused
                  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
                  at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
                  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
                  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
                  at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:609)
                  at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:707)
                  at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
                  at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529)
                  at org.apache.hadoop.ipc.Client.call(Client.java:1446)
                  ... 29 more
          

          Maybe this is coming from my local installation somehow? I'm not sure at the moment (but localhost:8020 is where I run my namenode). Sergey Soldatov, do you run into the same problems?

          Show
          elserj Josh Elser added a comment - I've gotten HivePhoenixStoreIT to pass in Eclipse, but I have still been unable to get it to pass on the commandline (same command as above). The MRAppMaster for (what I assume is) the hive job is failing to talk to HDFS, thinking that it should be at localhost:8020. 2016-04-19 16:48:46,179 ERROR [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Failed while checking for/creating history staging path: [hdfs://localhost:8020/tmp/hadoop-yarn/staging/jelser/.staging/job_1461098862737_0002] java.net.ConnectException: Call From myhostname/127.0.0.1 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1407) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2116) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.mkdir(JobHistoryEventHandler.java:267) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceInit(JobHistoryEventHandler.java:166) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:450) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:609) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:707) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1446) ... 29 more Maybe this is coming from my local installation somehow? I'm not sure at the moment (but localhost:8020 is where I run my namenode). Sergey Soldatov , do you run into the same problems?
          Hide
          elserj Josh Elser added a comment -

          I can't get the HivePhoenixStoreIT to pass on the command line. I'm using `mvn clean verify -Dit.test=HivePhoenixStoreIT -DfailIfNoTests=false`. I've attached a copy of the jstack where it's hung. I'll continue to try to poke at it too.

          Show
          elserj Josh Elser added a comment - I can't get the HivePhoenixStoreIT to pass on the command line. I'm using `mvn clean verify -Dit.test=HivePhoenixStoreIT -DfailIfNoTests=false`. I've attached a copy of the jstack where it's hung. I'll continue to try to poke at it too.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ss77892 opened a pull request:

          https://github.com/apache/phoenix/pull/165

          PHOENIX-2743 HivePhoenixHandler for big-big join with predicate push …

          Rebased on the current master as a single patch.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/ss77892/phoenix PHOENIX-2743-2

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/phoenix/pull/165.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #165


          commit 684c127a936295a4bdd16f48ed1723dd35a40e6f
          Author: Sergey Soldatov <sergey.soldatov@gmail.com>
          Date: 2016-04-19T06:36:36Z

          PHOENIX-2743 HivePhoenixHandler for big-big join with predicate push down


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ss77892 opened a pull request: https://github.com/apache/phoenix/pull/165 PHOENIX-2743 HivePhoenixHandler for big-big join with predicate push … Rebased on the current master as a single patch. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ss77892/phoenix PHOENIX-2743 -2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/phoenix/pull/165.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #165 commit 684c127a936295a4bdd16f48ed1723dd35a40e6f Author: Sergey Soldatov <sergey.soldatov@gmail.com> Date: 2016-04-19T06:36:36Z PHOENIX-2743 HivePhoenixHandler for big-big join with predicate push down
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Yep, that's because of the recent commit about splits. Got it fixed, testing at the moment. I will make a new pull request rebased on the master.

          Show
          sergey.soldatov Sergey Soldatov added a comment - Yep, that's because of the recent commit about splits. Got it fixed, testing at the moment. I will make a new pull request rebased on the master.
          Hide
          elserj Josh Elser added a comment -

          Sergey Soldatov, some compilation issues on the latest?

          [INFO] -------------------------------------------------------------
          [ERROR] COMPILATION ERROR :
          [INFO] -------------------------------------------------------------
          [ERROR] /Users/jelser/projects/phoenix.git/phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixRecordReader.java:[116,65] no suitable constructor found for TableResultIterator(org.apache.phoenix.execute.MutationState,org.apache.phoenix.schema.TableRef,org.apache.hadoop.hbase.client.Scan,org.apache.phoenix.monitoring.CombinableMetric,long)
              constructor org.apache.phoenix.iterate.TableResultIterator.TableResultIterator(org.apache.phoenix.execute.MutationState,org.apache.hadoop.hbase.client.Scan,org.apache.phoenix.monitoring.CombinableMetric,long,org.apache.phoenix.iterate.PeekingResultIterator,org.apache.phoenix.compile.QueryPlan,org.apache.phoenix.iterate.ParallelScanGrouper) is not applicable
                (actual and formal argument lists differ in length)
              constructor org.apache.phoenix.iterate.TableResultIterator.TableResultIterator(org.apache.phoenix.execute.MutationState,org.apache.hadoop.hbase.client.Scan,org.apache.phoenix.monitoring.CombinableMetric,long,org.apache.phoenix.compile.QueryPlan,org.apache.phoenix.iterate.ParallelScanGrouper) is not applicable
                (actual and formal argument lists differ in length)
              constructor org.apache.phoenix.iterate.TableResultIterator.TableResultIterator() is not applicable
                (actual and formal argument lists differ in length)
          [INFO] 1 error
          
          Show
          elserj Josh Elser added a comment - Sergey Soldatov , some compilation issues on the latest? [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /Users/jelser/projects/phoenix.git/phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixRecordReader.java:[116,65] no suitable constructor found for TableResultIterator(org.apache.phoenix.execute.MutationState,org.apache.phoenix.schema.TableRef,org.apache.hadoop.hbase.client.Scan,org.apache.phoenix.monitoring.CombinableMetric,long) constructor org.apache.phoenix.iterate.TableResultIterator.TableResultIterator(org.apache.phoenix.execute.MutationState,org.apache.hadoop.hbase.client.Scan,org.apache.phoenix.monitoring.CombinableMetric,long,org.apache.phoenix.iterate.PeekingResultIterator,org.apache.phoenix.compile.QueryPlan,org.apache.phoenix.iterate.ParallelScanGrouper) is not applicable (actual and formal argument lists differ in length) constructor org.apache.phoenix.iterate.TableResultIterator.TableResultIterator(org.apache.phoenix.execute.MutationState,org.apache.hadoop.hbase.client.Scan,org.apache.phoenix.monitoring.CombinableMetric,long,org.apache.phoenix.compile.QueryPlan,org.apache.phoenix.iterate.ParallelScanGrouper) is not applicable (actual and formal argument lists differ in length) constructor org.apache.phoenix.iterate.TableResultIterator.TableResultIterator() is not applicable (actual and formal argument lists differ in length) [INFO] 1 error
          Hide
          elserj Josh Elser added a comment -

          Fantastic. I will do the same!

          Show
          elserj Josh Elser added a comment - Fantastic. I will do the same!
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Josh Elser Pushed. Was trying to check that changing to 2.7.1 doesn't cause any other IT failures. Actually it's still running, but no new failures so far. Will update if anything else fail.

          Show
          sergey.soldatov Sergey Soldatov added a comment - Josh Elser Pushed. Was trying to check that changing to 2.7.1 doesn't cause any other IT failures. Actually it's still running, but no new failures so far. Will update if anything else fail.
          Hide
          elserj Josh Elser added a comment -

          Got it. As for the exception - it comes that 2.5.1 hadoop-auth comes with hbase. Will fix those shortly.

          Just so it's not silent: "ok". Just ping me when you get this updated – I checked GH and didn't see any new commits. If it's helpful at all, I was using `mvn dependency:tree` on the command line to track this down myself.

          Show
          elserj Josh Elser added a comment - Got it. As for the exception - it comes that 2.5.1 hadoop-auth comes with hbase. Will fix those shortly. Just so it's not silent: "ok". Just ping me when you get this updated – I checked GH and didn't see any new commits. If it's helpful at all, I was using `mvn dependency:tree` on the command line to track this down myself.
          Hide
          elserj Josh Elser added a comment -

          This error usually happen when hadoop-auth version is different from hadoop-common

          Yeah, right you are again. hbase dependencies are pulling in hadoop-2.5.1 still.

          Show
          elserj Josh Elser added a comment - This error usually happen when hadoop-auth version is different from hadoop-common Yeah, right you are again. hbase dependencies are pulling in hadoop-2.5.1 still.
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Got it. As for the exception - it comes that 2.5.1 hadoop-auth comes with hbase. Will fix those shortly.

          Show
          sergey.soldatov Sergey Soldatov added a comment - Got it. As for the exception - it comes that 2.5.1 hadoop-auth comes with hbase. Will fix those shortly.
          Hide
          elserj Josh Elser added a comment -

          bq . I would assume that it's not necessary for any of them, but let me know if I'm wrong

          Actually I added that only for the packages that are not in the dependencies in the root pom.xml. Not sure whether it's good to include them there since they are used in this package only and the module has it's own artifact. If we want to make it as a part of phoenix-client.jar, those dependencies can be moved to the root pom.xml and specify the version there.

          Ah! I'm with you now. Convention states that we should have all dependencies declared in dependencyManagement and just refer to dependencies by groupId:artifactId in sub-modules. I will clean that up too (super easy to change). Thanks.

          Show
          elserj Josh Elser added a comment - bq . I would assume that it's not necessary for any of them, but let me know if I'm wrong Actually I added that only for the packages that are not in the dependencies in the root pom.xml. Not sure whether it's good to include them there since they are used in this package only and the module has it's own artifact. If we want to make it as a part of phoenix-client.jar, those dependencies can be moved to the root pom.xml and specify the version there. Ah! I'm with you now. Convention states that we should have all dependencies declared in dependencyManagement and just refer to dependencies by groupId:artifactId in sub-modules. I will clean that up too (super easy to change). Thanks.
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Interesting. This error usually happen when hadoop-auth version is different from hadoop-common. For last changes I run IT tests in the IDE, let me rerun it from maven.

          Show
          sergey.soldatov Sergey Soldatov added a comment - Interesting. This error usually happen when hadoop-auth version is different from hadoop-common. For last changes I run IT tests in the IDE, let me rerun it from maven.
          Hide
          elserj Josh Elser added a comment -
          java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.server.AuthenticationFilter.constructSecretProvider(Ljavax/servlet/ServletContext;Ljava/util/Properties;Z)Lorg/apache/hadoop/security/authentication/util/SignerSecretProvider;
          Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.server.AuthenticationFilter.constructSecretProvider(Ljavax/servlet/ServletContext;Ljava/util/Properties;Z)Lorg/apache/hadoop/security/authentication/util/SignerSecretProvider;
          

          Did you have all of the tests successfully running, Sergey Soldatov? It seems like there might be some lingering Hadoop version change issues. I'll look into it, but I just wanted to let you know

          Show
          elserj Josh Elser added a comment - java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.server.AuthenticationFilter.constructSecretProvider(Ljavax/servlet/ServletContext;Ljava/util/Properties;Z)Lorg/apache/hadoop/security/authentication/util/SignerSecretProvider; Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.server.AuthenticationFilter.constructSecretProvider(Ljavax/servlet/ServletContext;Ljava/util/Properties;Z)Lorg/apache/hadoop/security/authentication/util/SignerSecretProvider; Did you have all of the tests successfully running, Sergey Soldatov ? It seems like there might be some lingering Hadoop version change issues. I'll look into it, but I just wanted to let you know
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          . I would assume that it's not necessary for any of them, but let me know if I'm wrong

          Actually I added that only for the packages that are not in the dependencies in the root pom.xml. Not sure whether it's good to include them there since they are used in this package only and the module has it's own artifact. If we want to make it as a part of phoenix-client.jar, those dependencies can be moved to the root pom.xml and specify the version there.

          Show
          sergey.soldatov Sergey Soldatov added a comment - . I would assume that it's not necessary for any of them, but let me know if I'm wrong Actually I added that only for the packages that are not in the dependencies in the root pom.xml . Not sure whether it's good to include them there since they are used in this package only and the module has it's own artifact. If we want to make it as a part of phoenix-client.jar , those dependencies can be moved to the root pom.xml and specify the version there.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user JamesRTaylor commented on the pull request:

          https://github.com/apache/phoenix/pull/155#issuecomment-211456028

          It's in your most capable hands, @joshelser & @SergeySoldatov. Thanks so much for whipping this into shape!

          Show
          githubbot ASF GitHub Bot added a comment - Github user JamesRTaylor commented on the pull request: https://github.com/apache/phoenix/pull/155#issuecomment-211456028 It's in your most capable hands, @joshelser & @SergeySoldatov. Thanks so much for whipping this into shape!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on the pull request:

          https://github.com/apache/phoenix/pull/155#issuecomment-211454505

          @ss77892 one more nit-pick, it seems like you removed the version for some of the entries in the pom but then added `$

          {hadoop-two.version}

          ` for others. I would assume that it's not necessary for any of them, but let me know if I'm wrong. I can also just fix this while merging it in as long as @JamesRTaylor thinks this good to go too

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on the pull request: https://github.com/apache/phoenix/pull/155#issuecomment-211454505 @ss77892 one more nit-pick, it seems like you removed the version for some of the entries in the pom but then added `$ {hadoop-two.version} ` for others. I would assume that it's not necessary for any of them, but let me know if I'm wrong. I can also just fix this while merging it in as long as @JamesRTaylor thinks this good to go too
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on the pull request:

          https://github.com/apache/phoenix/pull/155#issuecomment-210678972

          Recent improvements look good! The new integration test is very nice. The ObjectInspector stuff isn't a huge deal to replace, but it's something that can be consolidated later. Left a few minor comments, but I this is good to go after those are fixed.

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on the pull request: https://github.com/apache/phoenix/pull/155#issuecomment-210678972 Recent improvements look good! The new integration test is very nice. The ObjectInspector stuff isn't a huge deal to replace, but it's something that can be consolidated later. Left a few minor comments, but I this is good to go after those are fixed.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59953321

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixMetaHook.java —
          @@ -6,9 +6,9 @@

          • to you under the Apache License, Version 2.0 (the
          • "License"); you may not use this file except in compliance
          • with the License. You may obtain a copy of the License at

          Adding in the paragraph tags might break the RAT check. Best to revert that.

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59953321 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixMetaHook.java — @@ -6,9 +6,9 @@ to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * + * <p> End diff – Adding in the paragraph tags might break the RAT check. Best to revert that.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59953160

          — Diff: phoenix-hive/pom.xml —
          @@ -117,66 +93,44 @@
          </exclusions>
          </dependency>
          <dependency>

          • <groupId>org.apache.hbase</groupId>
          • <artifactId>hbase-common</artifactId>
          • </dependency>
          • <dependency>
          • <groupId>org.apache.hbase</groupId>
          • <artifactId>hbase-protocol</artifactId>
          • </dependency>
          • <dependency>
          • <groupId>org.apache.hbase</groupId>
          • <artifactId>hbase-client</artifactId>
          • </dependency>
          • <dependency>
          • <groupId>org.apache.hbase</groupId>
          • <artifactId>hbase-hadoop-compat</artifactId>
          • <scope>test</scope>
          • </dependency>
          • <dependency>
          • <groupId>org.apache.hbase</groupId>
          • <artifactId>hbase-hadoop-compat</artifactId>
          • <type>test-jar</type>
          • <scope>test</scope>
          • </dependency>
          • <dependency>
          • <groupId>org.apache.hbase</groupId>
          • <artifactId>hbase-hadoop2-compat</artifactId>
            + <groupId>org.apache.hadoop</groupId>
            + <artifactId>hadoop-hdfs</artifactId>
            + <version>2.7.1</version>
            <scope>test</scope>
            </dependency>
            <dependency>
          • <groupId>org.apache.hbase</groupId>
          • <artifactId>hbase-hadoop2-compat</artifactId>
            + <groupId>org.apache.hadoop</groupId>
            + <artifactId>hadoop-hdfs</artifactId>
            + <version>2.7.1</version>
              • End diff –

          Given James' previous comment, I think you can safely change the hadoop version at the top-level pom. Then, you won't have to specify 2.7.1 down here in phoenix-hive.

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59953160 — Diff: phoenix-hive/pom.xml — @@ -117,66 +93,44 @@ </exclusions> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-common</artifactId> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-protocol</artifactId> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-hadoop-compat</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-hadoop-compat</artifactId> <type>test-jar</type> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-hadoop2-compat</artifactId> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-hdfs</artifactId> + <version>2.7.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-hadoop2-compat</artifactId> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-hdfs</artifactId> + <version>2.7.1</version> End diff – Given James' previous comment, I think you can safely change the hadoop version at the top-level pom. Then, you won't have to specify 2.7.1 down here in phoenix-hive.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59558697

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixResultWritable.java —
          @@ -0,0 +1,215 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive.mapreduce;
          +
          +import java.io.DataInput;
          +import java.io.DataOutput;
          +import java.io.IOException;
          +import java.sql.PreparedStatement;
          +import java.sql.ResultSet;
          +import java.sql.ResultSetMetaData;
          +import java.sql.SQLException;
          +import java.util.List;
          +import java.util.Map;
          +
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.conf.Configurable;
          +import org.apache.hadoop.conf.Configuration;
          +import org.apache.hadoop.io.Writable;
          +import org.apache.hadoop.mapreduce.lib.db.DBWritable;
          +import org.apache.phoenix.hive.PhoenixRowKey;
          +import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants;
          +import org.apache.phoenix.hive.util.PhoenixStorageHandlerUtil;
          +import org.apache.phoenix.hive.util.PhoenixUtil;
          +import org.apache.phoenix.util.ColumnInfo;
          +
          +import com.google.common.collect.Lists;
          +import com.google.common.collect.Maps;
          +
          +/**
          + * Serialized class for SerDe
          + *
          + */
          +public class PhoenixResultWritable implements Writable, DBWritable, Configurable {
          +
          + private static final Log LOG = LogFactory.getLog(PhoenixResultWritable.class);
          +
          + private List<ColumnInfo> columnMetadataList;
          + private List<Object> valueList; // for output
          + private Map<String, Object> rowMap = Maps.newHashMap(); // for input
          +
          + private int columnCount = -1;
          +
          + private Configuration config;
          + private boolean isTransactional;
          + private Map<String, Object> rowKeyMap = Maps.newLinkedHashMap();
          + private List<String> primaryKeyColumnList;
          +
          + public PhoenixResultWritable()

          { + }
          +
          + public PhoenixResultWritable(Configuration config) throws IOException { + setConf(config); + }
          +
          + public PhoenixResultWritable(Configuration config, List<ColumnInfo> columnMetadataList) throws IOException { + this(config); + this.columnMetadataList = columnMetadataList; + + valueList = Lists.newArrayListWithExpectedSize(columnMetadataList.size()); + }
          +
          + @Override
          + public void write(DataOutput out) throws IOException { + }

          — End diff –

          Maybe add `throws new UnsupportedOperationException()` then?

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59558697 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixResultWritable.java — @@ -0,0 +1,215 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive.mapreduce; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.util.List; +import java.util.Map; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configurable; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.Writable; +import org.apache.hadoop.mapreduce.lib.db.DBWritable; +import org.apache.phoenix.hive.PhoenixRowKey; +import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants; +import org.apache.phoenix.hive.util.PhoenixStorageHandlerUtil; +import org.apache.phoenix.hive.util.PhoenixUtil; +import org.apache.phoenix.util.ColumnInfo; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; + +/** + * Serialized class for SerDe + * + */ +public class PhoenixResultWritable implements Writable, DBWritable, Configurable { + + private static final Log LOG = LogFactory.getLog(PhoenixResultWritable.class); + + private List<ColumnInfo> columnMetadataList; + private List<Object> valueList; // for output + private Map<String, Object> rowMap = Maps.newHashMap(); // for input + + private int columnCount = -1; + + private Configuration config; + private boolean isTransactional; + private Map<String, Object> rowKeyMap = Maps.newLinkedHashMap(); + private List<String> primaryKeyColumnList; + + public PhoenixResultWritable() { + } + + public PhoenixResultWritable(Configuration config) throws IOException { + setConf(config); + } + + public PhoenixResultWritable(Configuration config, List<ColumnInfo> columnMetadataList) throws IOException { + this(config); + this.columnMetadataList = columnMetadataList; + + valueList = Lists.newArrayListWithExpectedSize(columnMetadataList.size()); + } + + @Override + public void write(DataOutput out) throws IOException { + } — End diff – Maybe add `throws new UnsupportedOperationException()` then?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59558405

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixObjectInspectorFactory.java —
          @@ -0,0 +1,150 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive.objectinspector;
          +
          +import java.util.ArrayList;
          +import java.util.List;
          +
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters;
          +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory;
          +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector;
          +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
          +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions;
          +import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
          +import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
          +import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
          +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
          +
          +public class PhoenixObjectInspectorFactory {
          — End diff –

          Haha, ok. I want to say that I think they are superfluous – Hive already provides implementations for the OI's.

          {HBase,Accumulo}

          StorageHandler should already have examples on how they work. It would reduce the amount of code to maintain here which is great.

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59558405 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixObjectInspectorFactory.java — @@ -0,0 +1,150 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive.objectinspector; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory; +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions; +import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; + +public class PhoenixObjectInspectorFactory { — End diff – Haha, ok. I want to say that I think they are superfluous – Hive already provides implementations for the OI's. {HBase,Accumulo} StorageHandler should already have examples on how they work. It would reduce the amount of code to maintain here which is great.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ss77892 commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59513925

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixObjectInspectorFactory.java —
          @@ -0,0 +1,150 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive.objectinspector;
          +
          +import java.util.ArrayList;
          +import java.util.List;
          +
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters;
          +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory;
          +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector;
          +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
          +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions;
          +import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
          +import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
          +import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
          +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
          +
          +public class PhoenixObjectInspectorFactory {
          — End diff –

          Honestly speaking I'm not sure That's how it comes from the guys who contribute this code.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ss77892 commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59513925 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixObjectInspectorFactory.java — @@ -0,0 +1,150 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive.objectinspector; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory; +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions; +import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; + +public class PhoenixObjectInspectorFactory { — End diff – Honestly speaking I'm not sure That's how it comes from the guys who contribute this code.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ss77892 commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59511702

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixResultWritable.java —
          @@ -0,0 +1,215 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive.mapreduce;
          +
          +import java.io.DataInput;
          +import java.io.DataOutput;
          +import java.io.IOException;
          +import java.sql.PreparedStatement;
          +import java.sql.ResultSet;
          +import java.sql.ResultSetMetaData;
          +import java.sql.SQLException;
          +import java.util.List;
          +import java.util.Map;
          +
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.conf.Configurable;
          +import org.apache.hadoop.conf.Configuration;
          +import org.apache.hadoop.io.Writable;
          +import org.apache.hadoop.mapreduce.lib.db.DBWritable;
          +import org.apache.phoenix.hive.PhoenixRowKey;
          +import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants;
          +import org.apache.phoenix.hive.util.PhoenixStorageHandlerUtil;
          +import org.apache.phoenix.hive.util.PhoenixUtil;
          +import org.apache.phoenix.util.ColumnInfo;
          +
          +import com.google.common.collect.Lists;
          +import com.google.common.collect.Maps;
          +
          +/**
          + * Serialized class for SerDe
          + *
          + */
          +public class PhoenixResultWritable implements Writable, DBWritable, Configurable {
          +
          + private static final Log LOG = LogFactory.getLog(PhoenixResultWritable.class);
          +
          + private List<ColumnInfo> columnMetadataList;
          + private List<Object> valueList; // for output
          + private Map<String, Object> rowMap = Maps.newHashMap(); // for input
          +
          + private int columnCount = -1;
          +
          + private Configuration config;
          + private boolean isTransactional;
          + private Map<String, Object> rowKeyMap = Maps.newLinkedHashMap();
          + private List<String> primaryKeyColumnList;
          +
          + public PhoenixResultWritable()

          { + }
          +
          + public PhoenixResultWritable(Configuration config) throws IOException { + setConf(config); + }
          +
          + public PhoenixResultWritable(Configuration config, List<ColumnInfo> columnMetadataList) throws IOException { + this(config); + this.columnMetadataList = columnMetadataList; + + valueList = Lists.newArrayListWithExpectedSize(columnMetadataList.size()); + }
          +
          + @Override
          + public void write(DataOutput out) throws IOException { + }

          — End diff –

          We are using it in our PhoenixRecordWriter and we call write only with statement.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ss77892 commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59511702 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixResultWritable.java — @@ -0,0 +1,215 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive.mapreduce; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.util.List; +import java.util.Map; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configurable; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.Writable; +import org.apache.hadoop.mapreduce.lib.db.DBWritable; +import org.apache.phoenix.hive.PhoenixRowKey; +import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants; +import org.apache.phoenix.hive.util.PhoenixStorageHandlerUtil; +import org.apache.phoenix.hive.util.PhoenixUtil; +import org.apache.phoenix.util.ColumnInfo; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; + +/** + * Serialized class for SerDe + * + */ +public class PhoenixResultWritable implements Writable, DBWritable, Configurable { + + private static final Log LOG = LogFactory.getLog(PhoenixResultWritable.class); + + private List<ColumnInfo> columnMetadataList; + private List<Object> valueList; // for output + private Map<String, Object> rowMap = Maps.newHashMap(); // for input + + private int columnCount = -1; + + private Configuration config; + private boolean isTransactional; + private Map<String, Object> rowKeyMap = Maps.newLinkedHashMap(); + private List<String> primaryKeyColumnList; + + public PhoenixResultWritable() { + } + + public PhoenixResultWritable(Configuration config) throws IOException { + setConf(config); + } + + public PhoenixResultWritable(Configuration config, List<ColumnInfo> columnMetadataList) throws IOException { + this(config); + this.columnMetadataList = columnMetadataList; + + valueList = Lists.newArrayListWithExpectedSize(columnMetadataList.size()); + } + + @Override + public void write(DataOutput out) throws IOException { + } — End diff – We are using it in our PhoenixRecordWriter and we call write only with statement.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ss77892 commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59511383

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/IndexPredicateAnalyzer.java —
          @@ -0,0 +1,486 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive.ql.index;
          +
          +import java.util.ArrayList;
          +import java.util.Arrays;
          +import java.util.HashMap;
          +import java.util.HashSet;
          +import java.util.LinkedHashMap;
          +import java.util.List;
          +import java.util.Map;
          +import java.util.Set;
          +import java.util.Stack;
          +
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.hive.ql.exec.FunctionRegistry;
          +import org.apache.hadoop.hive.ql.lib.DefaultGraphWalker;
          +import org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher;
          +import org.apache.hadoop.hive.ql.lib.Dispatcher;
          +import org.apache.hadoop.hive.ql.lib.GraphWalker;
          +import org.apache.hadoop.hive.ql.lib.Node;
          +import org.apache.hadoop.hive.ql.lib.NodeProcessor;
          +import org.apache.hadoop.hive.ql.lib.NodeProcessorCtx;
          +import org.apache.hadoop.hive.ql.lib.Rule;
          +import org.apache.hadoop.hive.ql.parse.SemanticException;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeDescUtils;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeFieldDesc;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBaseCompare;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFIn;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNot;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNotNull;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNull;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToBinary;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToChar;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDecimal;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUnixTimeStamp;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUtcTimestamp;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToVarchar;
          +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
          +
          +import com.google.common.collect.Lists;
          +
          +/**
          + * Clone of org.apache.hadoop.hive.ql.index.IndexPredicateAnalyzer with modifying
          + * analyzePredicate method.
          + *
          + *
          + */
          +public class IndexPredicateAnalyzer {
          +
          + private static final Log LOG = LogFactory.getLog(IndexPredicateAnalyzer.class);
          +
          + private final Set<String> udfNames;
          + private final Map<String, Set<String>> columnToUDFs;
          + private FieldValidator fieldValidator;
          +
          + private boolean acceptsFields;
          +
          + public IndexPredicateAnalyzer()

          { + udfNames = new HashSet<String>(); + columnToUDFs = new HashMap<String, Set<String>>(); + }

          +
          + public void setFieldValidator(FieldValidator fieldValidator)

          { + this.fieldValidator = fieldValidator; + }

          +
          + /**
          + * Registers a comparison operator as one which can be satisfied by an index
          + * search. Unless this is called, analyzePredicate will never find any
          + * indexable conditions.
          + *
          + * @param udfName name of comparison operator as returned by either
          + *

          {@link GenericUDFBridge#getUdfName}

          (for simple UDF's) or
          + * udf.getClass().getName() (for generic UDF's).
          + */
          + public void addComparisonOp(String udfName)

          { + udfNames.add(udfName); + }

          +
          + /**
          + * Clears the set of column names allowed in comparisons. (Initially, all
          + * column names are allowed.)
          + */
          + public void clearAllowedColumnNames()

          { + columnToUDFs.clear(); + }

          +
          + /**
          + * Adds a column name to the set of column names allowed.
          + *
          + * @param columnName name of column to be allowed
          + */
          + public void allowColumnName(String columnName)

          { + columnToUDFs.put(columnName, udfNames); + }

          +
          + /**
          + * add allowed functions per column
          + *
          + * @param columnName
          + * @param udfs
          + */
          + public void addComparisonOp(String columnName, String... udfs) {
          + Set<String> allowed = columnToUDFs.get(columnName);
          + if (allowed == null || allowed == udfNames)

          { + // override + columnToUDFs.put(columnName, new HashSet<String>(Arrays.asList(udfs))); + }

          else

          { + allowed.addAll(Arrays.asList(udfs)); + }

          + }
          +
          + /**
          + * Analyzes a predicate.
          + *
          + * @param predicate predicate to be analyzed
          + * @param searchConditions receives conditions produced by analysis
          + * @return residual predicate which could not be translated to
          + * searchConditions
          + */
          + public ExprNodeDesc analyzePredicate(ExprNodeDesc predicate, final List<IndexSearchCondition> searchConditions) {
          +
          + Map<Rule, NodeProcessor> opRules = new LinkedHashMap<Rule, NodeProcessor>();
          + NodeProcessor nodeProcessor = new NodeProcessor() {
          + @Override
          + public Object process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, Object... nodeOutputs) throws SemanticException {
          +
          + // We can only push down stuff which appears as part of
          + // a pure conjunction: reject OR, CASE, etc.
          + for (Node ancestor : stack) {
          + if (nd == ancestor)

          { + break; + }

          + if (!FunctionRegistry.isOpAnd((ExprNodeDesc) ancestor))

          { + return nd; + }

          + }
          +
          + return analyzeExpr((ExprNodeGenericFuncDesc) nd, searchConditions, nodeOutputs);
          + }
          + };
          +
          + Dispatcher disp = new DefaultRuleDispatcher(nodeProcessor, opRules, null);
          + GraphWalker ogw = new DefaultGraphWalker(disp);
          + ArrayList<Node> topNodes = new ArrayList<Node>();
          + topNodes.add(predicate);
          + HashMap<Node, Object> nodeOutput = new HashMap<Node, Object>();
          +
          + try

          { + ogw.startWalking(topNodes, nodeOutput); + }

          catch (SemanticException ex)

          { + throw new RuntimeException(ex); + }

          +
          + ExprNodeDesc residualPredicate = (ExprNodeDesc) nodeOutput.get(predicate);
          + return residualPredicate;
          + }
          +
          + // Check if ExprNodeColumnDesc is wrapped in expr.
          + // If so, peel off. Otherwise return itself.
          + private ExprNodeDesc getColumnExpr(ExprNodeDesc expr) {
          + if (expr instanceof ExprNodeColumnDesc)

          { + return expr; + }
          + ExprNodeGenericFuncDesc funcDesc = null;
          + if (expr instanceof ExprNodeGenericFuncDesc) { + funcDesc = (ExprNodeGenericFuncDesc) expr; + }
          + if (null == funcDesc) { + return expr; + }

          + GenericUDF udf = funcDesc.getGenericUDF();
          + // check if its a simple cast expression.
          + if ((udf instanceof GenericUDFBridge || udf instanceof GenericUDFToBinary || udf instanceof GenericUDFToChar
          + || udf instanceof GenericUDFToVarchar || udf instanceof GenericUDFToDecimal
          + || udf instanceof GenericUDFToDate || udf instanceof GenericUDFToUnixTimeStamp
          + || udf instanceof GenericUDFToUtcTimestamp) && funcDesc.getChildren().size() == 1
          + && funcDesc.getChildren().get(0) instanceof ExprNodeColumnDesc)

          { + return expr.getChildren().get(0); + }

          + return expr;
          + }
          +
          + private void processingBetweenOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) {
          + ExprNodeColumnDesc columnDesc = null;
          + String[] fields = null;
          +
          + if (nodeOutputs[1] instanceof ExprNodeFieldDesc)

          { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[1]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[1], (ExprNodeDesc) nodeOutputs[2]); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + }

          else if (nodeOutputs[0] instanceof ExprNodeGenericFuncDesc)

          { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[1]).getChildren().get(0); + }

          else

          { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[1]; + }

          +
          + String udfName = expr.getGenericUDF().getUdfName();
          + ExprNodeConstantDesc[] betweenConstants = new ExprNodeConstantDesc[]

          {(ExprNodeConstantDesc) nodeOutputs[2], (ExprNodeConstantDesc) nodeOutputs[3]}

          ;
          + boolean isNot = (Boolean) ((ExprNodeConstantDesc) nodeOutputs[0]).getValue();
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, betweenConstants, expr, fields, isNot));
          + }
          +
          + private void processingInOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, boolean isNot, Object... nodeOutputs) {
          + ExprNodeColumnDesc columnDesc = null;
          + String[] fields = null;
          +
          + if (LOG.isTraceEnabled())

          { + LOG.trace("Processing In Operator. nodeOutputs : " + Lists.newArrayList(nodeOutputs)); + }

          +
          + if (nodeOutputs[0] instanceof ExprNodeFieldDesc) {
          + // rowKey field
          + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0];
          + fields = ExprNodeDescUtils.extractFields(fieldDesc);
          +
          + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], (ExprNodeDesc) nodeOutputs[1]);
          +
          + if (extracted == null)

          { // adding for tez + return; + }
          +
          + if (LOG.isTraceEnabled()) { + LOG.trace("nodeOutputs[0] : " + nodeOutputs[0] + ", nodeOutputs[1] : " + + nodeOutputs[1] + " => " + Lists.newArrayList(extracted)); + }
          +
          + columnDesc = (ExprNodeColumnDesc) extracted[0];
          + } else if (nodeOutputs[0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + }
          +
          + String udfName = expr.getGenericUDF().getUdfName();
          + ExprNodeConstantDesc[] inConstantDescs = new ExprNodeConstantDesc[nodeOutputs.length - 1];
          +
          + for (int i = 0, limit = inConstantDescs.length; i < limit; i++) {
          + if (!(nodeOutputs[i + 1] instanceof ExprNodeConstantDesc)) { // adding for tez + return; + }

          +
          + inConstantDescs[i] = (ExprNodeConstantDesc) nodeOutputs[i + 1];
          + }
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, inConstantDescs, expr, fields, isNot));
          + }
          +
          + private void processingNullOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) {
          + ExprNodeColumnDesc columnDesc = null;
          + String[] fields = null;
          +
          + if (nodeOutputs[0] instanceof ExprNodeFieldDesc)

          { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], new ExprNodeConstantDesc()); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + } else if (nodeOutputs[0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + }
          +
          + String udfName = expr.getGenericUDF().getUdfName();
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, null, expr, fields, false));
          + }
          +
          + private void processingNotNullOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) {
          + ExprNodeColumnDesc columnDesc = null;
          + String[] fields = null;
          +
          + if (nodeOutputs[0] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], new ExprNodeConstantDesc()); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + }

          else if (nodeOutputs[0] instanceof ExprNodeGenericFuncDesc)

          { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + }

          else

          { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + }

          +
          + String udfName = expr.getGenericUDF().getUdfName();
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, null, expr, fields, true));
          + }
          +
          + private ExprNodeDesc analyzeExpr(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) throws SemanticException {
          +
          + if (FunctionRegistry.isOpAnd(expr)) {
          + assert (nodeOutputs.length == 2);
          + ExprNodeDesc residual1 = (ExprNodeDesc) nodeOutputs[0];
          + ExprNodeDesc residual2 = (ExprNodeDesc) nodeOutputs[1];
          + if (residual1 == null)

          { + return residual2; + }

          + if (residual2 == null)

          { + return residual1; + }

          + List<ExprNodeDesc> residuals = new ArrayList<ExprNodeDesc>();
          + residuals.add(residual1);
          + residuals.add(residual2);
          + return new ExprNodeGenericFuncDesc(TypeInfoFactory.booleanTypeInfo, FunctionRegistry.getGenericUDFForAnd(), residuals);
          + }
          +
          + GenericUDF genericUDF = expr.getGenericUDF();
          + if (!(genericUDF instanceof GenericUDFBaseCompare)) {
          + // 2015-10-22 Added by JeongMin Ju : Processing Between/In Operator
          + if (genericUDF instanceof GenericUDFBetween)

          { + // In case of not between, The value of first element of nodeOutputs is true. otherwise false. + processingBetweenOperator(expr, searchConditions, nodeOutputs); + return expr; + }

          else if (genericUDF instanceof GenericUDFIn)

          { + // In case of not in operator, in operator exist as child of not operator. + processingInOperator(expr, searchConditions, false, nodeOutputs); + return expr; + }

          else if (genericUDF instanceof GenericUDFOPNot &&
          + ((ExprNodeGenericFuncDesc) expr.getChildren().get(0)).getGenericUDF() instanceof GenericUDFIn)

          { + // In case of not in operator, in operator exist as child of not operator. + processingInOperator((ExprNodeGenericFuncDesc) expr.getChildren().get(0), + searchConditions, true, ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().toArray()); + return expr; + }

          else if (genericUDF instanceof GenericUDFOPNull)

          { + processingNullOperator(expr, searchConditions, nodeOutputs); + return expr; + }

          else if (genericUDF instanceof GenericUDFOPNotNull)

          { + processingNotNullOperator(expr, searchConditions, nodeOutputs); + return expr; + }

          else

          { + return expr; + }
          + }
          + ExprNodeDesc expr1 = (ExprNodeDesc) nodeOutputs[0];
          + ExprNodeDesc expr2 = (ExprNodeDesc) nodeOutputs[1];
          + // We may need to peel off the GenericUDFBridge that is added by CBO or
          + // user
          + if (expr1.getTypeInfo().equals(expr2.getTypeInfo())) { + expr1 = getColumnExpr(expr1); + expr2 = getColumnExpr(expr2); + }
          +
          + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair(expr1, expr2);
          + if (extracted == null || (extracted.length > 2 && !acceptsFields)) { + return expr; + }
          +
          + ExprNodeColumnDesc columnDesc;
          + ExprNodeConstantDesc constantDesc;
          + if (extracted[0] instanceof ExprNodeConstantDesc) { + genericUDF = genericUDF.flip(); + columnDesc = (ExprNodeColumnDesc) extracted[1]; + constantDesc = (ExprNodeConstantDesc) extracted[0]; + } else { + columnDesc = (ExprNodeColumnDesc) extracted[0]; + constantDesc = (ExprNodeConstantDesc) extracted[1]; + }
          +
          + Set<String> allowed = columnToUDFs.get(columnDesc.getColumn());
          + if (allowed == null) { + return expr; + }
          +
          + String udfName = genericUDF.getUdfName();
          + if (!allowed.contains(genericUDF.getUdfName())) { + return expr; + }
          +
          + String[] fields = null;
          + if (extracted.length > 2) {
          + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) extracted[2];
          + if (!isValidField(fieldDesc)) { + return expr; + }

          + fields = ExprNodeDescUtils.extractFields(fieldDesc);
          + }
          +
          + // We also need to update the expr so that the index query can be
          + // generated.
          + // Note that, hive does not support UDFToDouble etc in the query text.
          + List<ExprNodeDesc> list = new ArrayList<ExprNodeDesc>();
          + list.add(expr1);
          + list.add(expr2);
          + expr = new ExprNodeGenericFuncDesc(expr.getTypeInfo(), expr.getGenericUDF(), list);
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, constantDesc, expr, fields));
          +
          + // we converted the expression to a search condition, so
          + // remove it from the residual predicate
          + return fields == null ? null : expr;
          + }
          +
          + private boolean isValidField(ExprNodeFieldDesc field)

          { + return fieldValidator == null || fieldValidator.validate(field); + }

          +
          + /**
          + * Translates search conditions back to ExprNodeDesc form (as a left-deep
          + * conjunction).
          + *
          + * @param searchConditions (typically produced by analyzePredicate)
          + * @return ExprNodeGenericFuncDesc form of search conditions
          + */
          + public ExprNodeGenericFuncDesc translateSearchConditions(List<IndexSearchCondition> searchConditions) {
          +
          + ExprNodeGenericFuncDesc expr = null;
          +
          + for (IndexSearchCondition searchCondition : searchConditions) {
          + if (expr == null)

          { + expr = searchCondition.getComparisonExpr(); + continue; + }

          +
          + List<ExprNodeDesc> children = new ArrayList<ExprNodeDesc>();
          + children.add(expr);
          + children.add(searchCondition.getComparisonExpr());
          + expr = new ExprNodeGenericFuncDesc(TypeInfoFactory.booleanTypeInfo, FunctionRegistry.getGenericUDFForAnd(), children);
          + }
          +
          + return expr;
          + }
          +
          + public void setAcceptsFields(boolean acceptsFields)

          { + this.acceptsFields = acceptsFields; + }

          +
          + public static interface FieldValidator

          { + boolean validate(ExprNodeFieldDesc exprNodeDesc); + }

          +
          + public static IndexPredicateAnalyzer createAnalyzer(boolean equalOnly) {
          + IndexPredicateAnalyzer analyzer = new IndexPredicateAnalyzer();
          + analyzer.addComparisonOp("org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqual");
          +
          + if (equalOnly)

          { + return analyzer; + }

          +
          + analyzer.addComparisonOp("org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan");
          — End diff –

          Actually not, but it was done in the same way as it's done in Hive's IndexPredicateAnalyzer.java

          Show
          githubbot ASF GitHub Bot added a comment - Github user ss77892 commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59511383 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/IndexPredicateAnalyzer.java — @@ -0,0 +1,486 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive.ql.index; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.Stack; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hive.ql.exec.FunctionRegistry; +import org.apache.hadoop.hive.ql.lib.DefaultGraphWalker; +import org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher; +import org.apache.hadoop.hive.ql.lib.Dispatcher; +import org.apache.hadoop.hive.ql.lib.GraphWalker; +import org.apache.hadoop.hive.ql.lib.Node; +import org.apache.hadoop.hive.ql.lib.NodeProcessor; +import org.apache.hadoop.hive.ql.lib.NodeProcessorCtx; +import org.apache.hadoop.hive.ql.lib.Rule; +import org.apache.hadoop.hive.ql.parse.SemanticException; +import org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc; +import org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc; +import org.apache.hadoop.hive.ql.plan.ExprNodeDesc; +import org.apache.hadoop.hive.ql.plan.ExprNodeDescUtils; +import org.apache.hadoop.hive.ql.plan.ExprNodeFieldDesc; +import org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDF; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBaseCompare; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFIn; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNot; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNotNull; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNull; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToBinary; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToChar; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDecimal; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUnixTimeStamp; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUtcTimestamp; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToVarchar; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory; + +import com.google.common.collect.Lists; + +/** + * Clone of org.apache.hadoop.hive.ql.index.IndexPredicateAnalyzer with modifying + * analyzePredicate method. + * + * + */ +public class IndexPredicateAnalyzer { + + private static final Log LOG = LogFactory.getLog(IndexPredicateAnalyzer.class); + + private final Set<String> udfNames; + private final Map<String, Set<String>> columnToUDFs; + private FieldValidator fieldValidator; + + private boolean acceptsFields; + + public IndexPredicateAnalyzer() { + udfNames = new HashSet<String>(); + columnToUDFs = new HashMap<String, Set<String>>(); + } + + public void setFieldValidator(FieldValidator fieldValidator) { + this.fieldValidator = fieldValidator; + } + + /** + * Registers a comparison operator as one which can be satisfied by an index + * search. Unless this is called, analyzePredicate will never find any + * indexable conditions. + * + * @param udfName name of comparison operator as returned by either + * {@link GenericUDFBridge#getUdfName} (for simple UDF's) or + * udf.getClass().getName() (for generic UDF's). + */ + public void addComparisonOp(String udfName) { + udfNames.add(udfName); + } + + /** + * Clears the set of column names allowed in comparisons. (Initially, all + * column names are allowed.) + */ + public void clearAllowedColumnNames() { + columnToUDFs.clear(); + } + + /** + * Adds a column name to the set of column names allowed. + * + * @param columnName name of column to be allowed + */ + public void allowColumnName(String columnName) { + columnToUDFs.put(columnName, udfNames); + } + + /** + * add allowed functions per column + * + * @param columnName + * @param udfs + */ + public void addComparisonOp(String columnName, String... udfs) { + Set<String> allowed = columnToUDFs.get(columnName); + if (allowed == null || allowed == udfNames) { + // override + columnToUDFs.put(columnName, new HashSet<String>(Arrays.asList(udfs))); + } else { + allowed.addAll(Arrays.asList(udfs)); + } + } + + /** + * Analyzes a predicate. + * + * @param predicate predicate to be analyzed + * @param searchConditions receives conditions produced by analysis + * @return residual predicate which could not be translated to + * searchConditions + */ + public ExprNodeDesc analyzePredicate(ExprNodeDesc predicate, final List<IndexSearchCondition> searchConditions) { + + Map<Rule, NodeProcessor> opRules = new LinkedHashMap<Rule, NodeProcessor>(); + NodeProcessor nodeProcessor = new NodeProcessor() { + @Override + public Object process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, Object... nodeOutputs) throws SemanticException { + + // We can only push down stuff which appears as part of + // a pure conjunction: reject OR, CASE, etc. + for (Node ancestor : stack) { + if (nd == ancestor) { + break; + } + if (!FunctionRegistry.isOpAnd((ExprNodeDesc) ancestor)) { + return nd; + } + } + + return analyzeExpr((ExprNodeGenericFuncDesc) nd, searchConditions, nodeOutputs); + } + }; + + Dispatcher disp = new DefaultRuleDispatcher(nodeProcessor, opRules, null); + GraphWalker ogw = new DefaultGraphWalker(disp); + ArrayList<Node> topNodes = new ArrayList<Node>(); + topNodes.add(predicate); + HashMap<Node, Object> nodeOutput = new HashMap<Node, Object>(); + + try { + ogw.startWalking(topNodes, nodeOutput); + } catch (SemanticException ex) { + throw new RuntimeException(ex); + } + + ExprNodeDesc residualPredicate = (ExprNodeDesc) nodeOutput.get(predicate); + return residualPredicate; + } + + // Check if ExprNodeColumnDesc is wrapped in expr. + // If so, peel off. Otherwise return itself. + private ExprNodeDesc getColumnExpr(ExprNodeDesc expr) { + if (expr instanceof ExprNodeColumnDesc) { + return expr; + } + ExprNodeGenericFuncDesc funcDesc = null; + if (expr instanceof ExprNodeGenericFuncDesc) { + funcDesc = (ExprNodeGenericFuncDesc) expr; + } + if (null == funcDesc) { + return expr; + } + GenericUDF udf = funcDesc.getGenericUDF(); + // check if its a simple cast expression. + if ((udf instanceof GenericUDFBridge || udf instanceof GenericUDFToBinary || udf instanceof GenericUDFToChar + || udf instanceof GenericUDFToVarchar || udf instanceof GenericUDFToDecimal + || udf instanceof GenericUDFToDate || udf instanceof GenericUDFToUnixTimeStamp + || udf instanceof GenericUDFToUtcTimestamp) && funcDesc.getChildren().size() == 1 + && funcDesc.getChildren().get(0) instanceof ExprNodeColumnDesc) { + return expr.getChildren().get(0); + } + return expr; + } + + private void processingBetweenOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) { + ExprNodeColumnDesc columnDesc = null; + String[] fields = null; + + if (nodeOutputs [1] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[1]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[1], (ExprNodeDesc) nodeOutputs[2]); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + } else if (nodeOutputs [0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[1]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[1]; + } + + String udfName = expr.getGenericUDF().getUdfName(); + ExprNodeConstantDesc[] betweenConstants = new ExprNodeConstantDesc[] {(ExprNodeConstantDesc) nodeOutputs[2], (ExprNodeConstantDesc) nodeOutputs[3]} ; + boolean isNot = (Boolean) ((ExprNodeConstantDesc) nodeOutputs [0] ).getValue(); + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, betweenConstants, expr, fields, isNot)); + } + + private void processingInOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, boolean isNot, Object... nodeOutputs) { + ExprNodeColumnDesc columnDesc = null; + String[] fields = null; + + if (LOG.isTraceEnabled()) { + LOG.trace("Processing In Operator. nodeOutputs : " + Lists.newArrayList(nodeOutputs)); + } + + if (nodeOutputs [0] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs [0] ; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs [0] , (ExprNodeDesc) nodeOutputs [1] ); + + if (extracted == null) { // adding for tez + return; + } + + if (LOG.isTraceEnabled()) { + LOG.trace("nodeOutputs[0] : " + nodeOutputs[0] + ", nodeOutputs[1] : " + + nodeOutputs[1] + " => " + Lists.newArrayList(extracted)); + } + + columnDesc = (ExprNodeColumnDesc) extracted [0] ; + } else if (nodeOutputs [0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + } + + String udfName = expr.getGenericUDF().getUdfName(); + ExprNodeConstantDesc[] inConstantDescs = new ExprNodeConstantDesc [nodeOutputs.length - 1] ; + + for (int i = 0, limit = inConstantDescs.length; i < limit; i++) { + if (!(nodeOutputs [i + 1] instanceof ExprNodeConstantDesc)) { // adding for tez + return; + } + + inConstantDescs [i] = (ExprNodeConstantDesc) nodeOutputs [i + 1] ; + } + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, inConstantDescs, expr, fields, isNot)); + } + + private void processingNullOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) { + ExprNodeColumnDesc columnDesc = null; + String[] fields = null; + + if (nodeOutputs [0] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], new ExprNodeConstantDesc()); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + } else if (nodeOutputs [0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + } + + String udfName = expr.getGenericUDF().getUdfName(); + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, null, expr, fields, false)); + } + + private void processingNotNullOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) { + ExprNodeColumnDesc columnDesc = null; + String[] fields = null; + + if (nodeOutputs [0] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], new ExprNodeConstantDesc()); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + } else if (nodeOutputs [0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + } + + String udfName = expr.getGenericUDF().getUdfName(); + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, null, expr, fields, true)); + } + + private ExprNodeDesc analyzeExpr(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) throws SemanticException { + + if (FunctionRegistry.isOpAnd(expr)) { + assert (nodeOutputs.length == 2); + ExprNodeDesc residual1 = (ExprNodeDesc) nodeOutputs [0] ; + ExprNodeDesc residual2 = (ExprNodeDesc) nodeOutputs [1] ; + if (residual1 == null) { + return residual2; + } + if (residual2 == null) { + return residual1; + } + List<ExprNodeDesc> residuals = new ArrayList<ExprNodeDesc>(); + residuals.add(residual1); + residuals.add(residual2); + return new ExprNodeGenericFuncDesc(TypeInfoFactory.booleanTypeInfo, FunctionRegistry.getGenericUDFForAnd(), residuals); + } + + GenericUDF genericUDF = expr.getGenericUDF(); + if (!(genericUDF instanceof GenericUDFBaseCompare)) { + // 2015-10-22 Added by JeongMin Ju : Processing Between/In Operator + if (genericUDF instanceof GenericUDFBetween) { + // In case of not between, The value of first element of nodeOutputs is true. otherwise false. + processingBetweenOperator(expr, searchConditions, nodeOutputs); + return expr; + } else if (genericUDF instanceof GenericUDFIn) { + // In case of not in operator, in operator exist as child of not operator. + processingInOperator(expr, searchConditions, false, nodeOutputs); + return expr; + } else if (genericUDF instanceof GenericUDFOPNot && + ((ExprNodeGenericFuncDesc) expr.getChildren().get(0)).getGenericUDF() instanceof GenericUDFIn) { + // In case of not in operator, in operator exist as child of not operator. + processingInOperator((ExprNodeGenericFuncDesc) expr.getChildren().get(0), + searchConditions, true, ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().toArray()); + return expr; + } else if (genericUDF instanceof GenericUDFOPNull) { + processingNullOperator(expr, searchConditions, nodeOutputs); + return expr; + } else if (genericUDF instanceof GenericUDFOPNotNull) { + processingNotNullOperator(expr, searchConditions, nodeOutputs); + return expr; + } else { + return expr; + } + } + ExprNodeDesc expr1 = (ExprNodeDesc) nodeOutputs [0] ; + ExprNodeDesc expr2 = (ExprNodeDesc) nodeOutputs [1] ; + // We may need to peel off the GenericUDFBridge that is added by CBO or + // user + if (expr1.getTypeInfo().equals(expr2.getTypeInfo())) { + expr1 = getColumnExpr(expr1); + expr2 = getColumnExpr(expr2); + } + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair(expr1, expr2); + if (extracted == null || (extracted.length > 2 && !acceptsFields)) { + return expr; + } + + ExprNodeColumnDesc columnDesc; + ExprNodeConstantDesc constantDesc; + if (extracted [0] instanceof ExprNodeConstantDesc) { + genericUDF = genericUDF.flip(); + columnDesc = (ExprNodeColumnDesc) extracted[1]; + constantDesc = (ExprNodeConstantDesc) extracted[0]; + } else { + columnDesc = (ExprNodeColumnDesc) extracted[0]; + constantDesc = (ExprNodeConstantDesc) extracted[1]; + } + + Set<String> allowed = columnToUDFs.get(columnDesc.getColumn()); + if (allowed == null) { + return expr; + } + + String udfName = genericUDF.getUdfName(); + if (!allowed.contains(genericUDF.getUdfName())) { + return expr; + } + + String[] fields = null; + if (extracted.length > 2) { + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) extracted [2] ; + if (!isValidField(fieldDesc)) { + return expr; + } + fields = ExprNodeDescUtils.extractFields(fieldDesc); + } + + // We also need to update the expr so that the index query can be + // generated. + // Note that, hive does not support UDFToDouble etc in the query text. + List<ExprNodeDesc> list = new ArrayList<ExprNodeDesc>(); + list.add(expr1); + list.add(expr2); + expr = new ExprNodeGenericFuncDesc(expr.getTypeInfo(), expr.getGenericUDF(), list); + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, constantDesc, expr, fields)); + + // we converted the expression to a search condition, so + // remove it from the residual predicate + return fields == null ? null : expr; + } + + private boolean isValidField(ExprNodeFieldDesc field) { + return fieldValidator == null || fieldValidator.validate(field); + } + + /** + * Translates search conditions back to ExprNodeDesc form (as a left-deep + * conjunction). + * + * @param searchConditions (typically produced by analyzePredicate) + * @return ExprNodeGenericFuncDesc form of search conditions + */ + public ExprNodeGenericFuncDesc translateSearchConditions(List<IndexSearchCondition> searchConditions) { + + ExprNodeGenericFuncDesc expr = null; + + for (IndexSearchCondition searchCondition : searchConditions) { + if (expr == null) { + expr = searchCondition.getComparisonExpr(); + continue; + } + + List<ExprNodeDesc> children = new ArrayList<ExprNodeDesc>(); + children.add(expr); + children.add(searchCondition.getComparisonExpr()); + expr = new ExprNodeGenericFuncDesc(TypeInfoFactory.booleanTypeInfo, FunctionRegistry.getGenericUDFForAnd(), children); + } + + return expr; + } + + public void setAcceptsFields(boolean acceptsFields) { + this.acceptsFields = acceptsFields; + } + + public static interface FieldValidator { + boolean validate(ExprNodeFieldDesc exprNodeDesc); + } + + public static IndexPredicateAnalyzer createAnalyzer(boolean equalOnly) { + IndexPredicateAnalyzer analyzer = new IndexPredicateAnalyzer(); + analyzer.addComparisonOp("org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqual"); + + if (equalOnly) { + return analyzer; + } + + analyzer.addComparisonOp("org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan"); — End diff – Actually not, but it was done in the same way as it's done in Hive's IndexPredicateAnalyzer.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user JamesRTaylor commented on the pull request:

          https://github.com/apache/phoenix/pull/155#issuecomment-208640594

          I think it's ok to change Phoenix to specify Hadoop 2.7.1. There's no particular reason it's set to 2.5.1 (other than no one has changed it in a while). See PHOENIX-2761.

          Show
          githubbot ASF GitHub Bot added a comment - Github user JamesRTaylor commented on the pull request: https://github.com/apache/phoenix/pull/155#issuecomment-208640594 I think it's ok to change Phoenix to specify Hadoop 2.7.1. There's no particular reason it's set to 2.5.1 (other than no one has changed it in a while). See PHOENIX-2761 .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ss77892 commented on the pull request:

          https://github.com/apache/phoenix/pull/155#issuecomment-208638163

          Yep, I know. The problem is that hbase-it is built against 2.5.1 and if I set hadoop version 2.7.1 it will fail to start DFS minicluster for hbase.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ss77892 commented on the pull request: https://github.com/apache/phoenix/pull/155#issuecomment-208638163 Yep, I know. The problem is that hbase-it is built against 2.5.1 and if I set hadoop version 2.7.1 it will fail to start DFS minicluster for hbase.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on the pull request:

          https://github.com/apache/phoenix/pull/155#issuecomment-208626268

          > it's hard to get all those miniclusters run together. I still hope to get it resolved. That's why the pom contains dependencies on calcite, hbase-test and others.

          If it's just at the test scope, you might be able to circumvent the issue by just overriding the hbase and hadoop versions to compatible versions only at the test scope.

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on the pull request: https://github.com/apache/phoenix/pull/155#issuecomment-208626268 > it's hard to get all those miniclusters run together. I still hope to get it resolved. That's why the pom contains dependencies on calcite, hbase-test and others. If it's just at the test scope, you might be able to circumvent the issue by just overriding the hbase and hadoop versions to compatible versions only at the test scope.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ss77892 commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59285747

          — Diff: phoenix-hive/pom.xml —
          @@ -0,0 +1,224 @@
          +<?xml version='1.0'?>
          +<!--
          +
          + Licensed to the Apache Software Foundation (ASF) under one
          + or more contributor license agreements. See the NOTICE file
          + distributed with this work for additional information
          + regarding copyright ownership. The ASF licenses this file
          + to you under the Apache License, Version 2.0 (the
          + "License"); you may not use this file except in compliance
          + with the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          + Unless required by applicable law or agreed to in writing,
          + software distributed under the License is distributed on an
          + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
          + KIND, either express or implied. See the License for the
          + specific language governing permissions and limitations
          + under the License.
          +
          +-->
          +
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <modelVersion>4.0.0</modelVersion>
          + <parent>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix</artifactId>
          + <version>4.8.0-HBase-1.1-SNAPSHOT</version>
          + </parent>
          + <artifactId>phoenix-hive</artifactId>
          + <name>Phoenix - Hive</name>
          +
          + <dependencies>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + <classifier>tests</classifier>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>joda-time</groupId>
          + <artifactId>joda-time</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-exec</artifactId>
          + <version>$

          {hive.version}</version>
          + <exclusions>
          + <exclusion>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>*</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-core</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-avatica</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-common</artifactId>
          + <version>${hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-cli</artifactId>
          + <version>$

          {hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>jline</groupId>
          + <artifactId>jline</artifactId>
          + <version>0.9.94</version>
          + </dependency>
          + <dependency>
          + <groupId>commons-lang</groupId>
          + <artifactId>commons-lang</artifactId>
          + <version>$

          {commons-lang.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>commons-logging</groupId>
          + <artifactId>commons-logging</artifactId>
          + <version>$

          {commons-logging.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-testing-util</artifactId>
          + <scope>test</scope>
          + <optional>true</optional>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-it</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-protocol</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-client</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-annotations</artifactId>
          — End diff –

          It will be there as soon as I get miniclusters working for ITs.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ss77892 commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59285747 — Diff: phoenix-hive/pom.xml — @@ -0,0 +1,224 @@ +<?xml version='1.0'?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix</artifactId> + <version>4.8.0-HBase-1.1-SNAPSHOT</version> + </parent> + <artifactId>phoenix-hive</artifactId> + <name>Phoenix - Hive</name> + + <dependencies> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + <dependency> + <groupId>joda-time</groupId> + <artifactId>joda-time</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <version>$ {hive.version}</version> + <exclusions> + <exclusion> + <groupId>org.apache.calcite</groupId> + <artifactId>*</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-core</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-avatica</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-common</artifactId> + <version>${hive.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-cli</artifactId> + <version>$ {hive.version} </version> + </dependency> + <dependency> + <groupId>jline</groupId> + <artifactId>jline</artifactId> + <version>0.9.94</version> + </dependency> + <dependency> + <groupId>commons-lang</groupId> + <artifactId>commons-lang</artifactId> + <version>$ {commons-lang.version} </version> + </dependency> + <dependency> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + <version>$ {commons-logging.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-testing-util</artifactId> + <scope>test</scope> + <optional>true</optional> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-it</artifactId> + <type>test-jar</type> + <scope>test</scope> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-protocol</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-client</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-annotations</artifactId> — End diff – It will be there as soon as I get miniclusters working for ITs.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ss77892 commented on the pull request:

          https://github.com/apache/phoenix/pull/155#issuecomment-208572335

          Actually I want to get some integration tests in the similar way as it was done in PHOENIX-331, but there is a clash between versions. This implementation uses Hive 1.2.1, so to run Hive minicluster it's require hadoop version >= 2.6, but hbase artifacts are still using 2.5, so it's hard to get all those miniclusters run together. I still hope to get it resolved. That's why the pom contains dependencies on calcite, hbase-test and others.
          I will add javadocs and clean commented code where it's possible.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ss77892 commented on the pull request: https://github.com/apache/phoenix/pull/155#issuecomment-208572335 Actually I want to get some integration tests in the similar way as it was done in PHOENIX-331 , but there is a clash between versions. This implementation uses Hive 1.2.1, so to run Hive minicluster it's require hadoop version >= 2.6, but hbase artifacts are still using 2.5, so it's hard to get all those miniclusters run together. I still hope to get it resolved. That's why the pom contains dependencies on calcite, hbase-test and others. I will add javadocs and clean commented code where it's possible.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on the pull request:

          https://github.com/apache/phoenix/pull/155#issuecomment-208542873

          Some general thoughts (I stopped leaving them inline everytime I saw them). I'm guessing you "inherited" some of these from JeongMin's original work.

          • Dbl-check indentations
          • Try to remove commented out code
          • Some class-level javadoc comments would be amazing
          • Not a single unit test?

          Other things that I remember biting me previously:

          • Make sure you try to run with Tez as well. Both in the "uber" (local job) mode and a normal tez task. There are.. subtleties between them, sadly (as sadly, I don't remember the specifics anymore).

          Other general thoughts:

          • The RecordUpdater implementation looks pretty cool. Didn't know they made this available for StorageHandlers.
          • Hive has a decent suite for running Hive tests as a part of their build (which includes tests for StorageHandlers) with this qtest/itest modules. You might be able to take some inspiration from these for testing.

          Looks good so far. It will be a nice bridge between Phoenix and Hive (as we work towards a common-core of Calcite).

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on the pull request: https://github.com/apache/phoenix/pull/155#issuecomment-208542873 Some general thoughts (I stopped leaving them inline everytime I saw them). I'm guessing you "inherited" some of these from JeongMin's original work. Dbl-check indentations Try to remove commented out code Some class-level javadoc comments would be amazing Not a single unit test? Other things that I remember biting me previously: Make sure you try to run with Tez as well. Both in the "uber" (local job) mode and a normal tez task. There are.. subtleties between them, sadly (as sadly, I don't remember the specifics anymore). Other general thoughts: The RecordUpdater implementation looks pretty cool. Didn't know they made this available for StorageHandlers. Hive has a decent suite for running Hive tests as a part of their build (which includes tests for StorageHandlers) with this qtest/itest modules. You might be able to take some inspiration from these for testing. Looks good so far. It will be a nice bridge between Phoenix and Hive (as we work towards a common-core of Calcite).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59273812

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/IndexPredicateAnalyzer.java —
          @@ -0,0 +1,486 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive.ql.index;
          +
          +import java.util.ArrayList;
          +import java.util.Arrays;
          +import java.util.HashMap;
          +import java.util.HashSet;
          +import java.util.LinkedHashMap;
          +import java.util.List;
          +import java.util.Map;
          +import java.util.Set;
          +import java.util.Stack;
          +
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.hive.ql.exec.FunctionRegistry;
          +import org.apache.hadoop.hive.ql.lib.DefaultGraphWalker;
          +import org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher;
          +import org.apache.hadoop.hive.ql.lib.Dispatcher;
          +import org.apache.hadoop.hive.ql.lib.GraphWalker;
          +import org.apache.hadoop.hive.ql.lib.Node;
          +import org.apache.hadoop.hive.ql.lib.NodeProcessor;
          +import org.apache.hadoop.hive.ql.lib.NodeProcessorCtx;
          +import org.apache.hadoop.hive.ql.lib.Rule;
          +import org.apache.hadoop.hive.ql.parse.SemanticException;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeDescUtils;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeFieldDesc;
          +import org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBaseCompare;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFIn;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNot;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNotNull;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNull;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToBinary;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToChar;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDecimal;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUnixTimeStamp;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUtcTimestamp;
          +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToVarchar;
          +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
          +
          +import com.google.common.collect.Lists;
          +
          +/**
          + * Clone of org.apache.hadoop.hive.ql.index.IndexPredicateAnalyzer with modifying
          + * analyzePredicate method.
          + *
          + *
          + */
          +public class IndexPredicateAnalyzer {
          +
          + private static final Log LOG = LogFactory.getLog(IndexPredicateAnalyzer.class);
          +
          + private final Set<String> udfNames;
          + private final Map<String, Set<String>> columnToUDFs;
          + private FieldValidator fieldValidator;
          +
          + private boolean acceptsFields;
          +
          + public IndexPredicateAnalyzer()

          { + udfNames = new HashSet<String>(); + columnToUDFs = new HashMap<String, Set<String>>(); + }

          +
          + public void setFieldValidator(FieldValidator fieldValidator)

          { + this.fieldValidator = fieldValidator; + }

          +
          + /**
          + * Registers a comparison operator as one which can be satisfied by an index
          + * search. Unless this is called, analyzePredicate will never find any
          + * indexable conditions.
          + *
          + * @param udfName name of comparison operator as returned by either
          + *

          {@link GenericUDFBridge#getUdfName}

          (for simple UDF's) or
          + * udf.getClass().getName() (for generic UDF's).
          + */
          + public void addComparisonOp(String udfName)

          { + udfNames.add(udfName); + }

          +
          + /**
          + * Clears the set of column names allowed in comparisons. (Initially, all
          + * column names are allowed.)
          + */
          + public void clearAllowedColumnNames()

          { + columnToUDFs.clear(); + }

          +
          + /**
          + * Adds a column name to the set of column names allowed.
          + *
          + * @param columnName name of column to be allowed
          + */
          + public void allowColumnName(String columnName)

          { + columnToUDFs.put(columnName, udfNames); + }

          +
          + /**
          + * add allowed functions per column
          + *
          + * @param columnName
          + * @param udfs
          + */
          + public void addComparisonOp(String columnName, String... udfs) {
          + Set<String> allowed = columnToUDFs.get(columnName);
          + if (allowed == null || allowed == udfNames)

          { + // override + columnToUDFs.put(columnName, new HashSet<String>(Arrays.asList(udfs))); + }

          else

          { + allowed.addAll(Arrays.asList(udfs)); + }

          + }
          +
          + /**
          + * Analyzes a predicate.
          + *
          + * @param predicate predicate to be analyzed
          + * @param searchConditions receives conditions produced by analysis
          + * @return residual predicate which could not be translated to
          + * searchConditions
          + */
          + public ExprNodeDesc analyzePredicate(ExprNodeDesc predicate, final List<IndexSearchCondition> searchConditions) {
          +
          + Map<Rule, NodeProcessor> opRules = new LinkedHashMap<Rule, NodeProcessor>();
          + NodeProcessor nodeProcessor = new NodeProcessor() {
          + @Override
          + public Object process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, Object... nodeOutputs) throws SemanticException {
          +
          + // We can only push down stuff which appears as part of
          + // a pure conjunction: reject OR, CASE, etc.
          + for (Node ancestor : stack) {
          + if (nd == ancestor)

          { + break; + }

          + if (!FunctionRegistry.isOpAnd((ExprNodeDesc) ancestor))

          { + return nd; + }

          + }
          +
          + return analyzeExpr((ExprNodeGenericFuncDesc) nd, searchConditions, nodeOutputs);
          + }
          + };
          +
          + Dispatcher disp = new DefaultRuleDispatcher(nodeProcessor, opRules, null);
          + GraphWalker ogw = new DefaultGraphWalker(disp);
          + ArrayList<Node> topNodes = new ArrayList<Node>();
          + topNodes.add(predicate);
          + HashMap<Node, Object> nodeOutput = new HashMap<Node, Object>();
          +
          + try

          { + ogw.startWalking(topNodes, nodeOutput); + }

          catch (SemanticException ex)

          { + throw new RuntimeException(ex); + }

          +
          + ExprNodeDesc residualPredicate = (ExprNodeDesc) nodeOutput.get(predicate);
          + return residualPredicate;
          + }
          +
          + // Check if ExprNodeColumnDesc is wrapped in expr.
          + // If so, peel off. Otherwise return itself.
          + private ExprNodeDesc getColumnExpr(ExprNodeDesc expr) {
          + if (expr instanceof ExprNodeColumnDesc)

          { + return expr; + }
          + ExprNodeGenericFuncDesc funcDesc = null;
          + if (expr instanceof ExprNodeGenericFuncDesc) { + funcDesc = (ExprNodeGenericFuncDesc) expr; + }
          + if (null == funcDesc) { + return expr; + }

          + GenericUDF udf = funcDesc.getGenericUDF();
          + // check if its a simple cast expression.
          + if ((udf instanceof GenericUDFBridge || udf instanceof GenericUDFToBinary || udf instanceof GenericUDFToChar
          + || udf instanceof GenericUDFToVarchar || udf instanceof GenericUDFToDecimal
          + || udf instanceof GenericUDFToDate || udf instanceof GenericUDFToUnixTimeStamp
          + || udf instanceof GenericUDFToUtcTimestamp) && funcDesc.getChildren().size() == 1
          + && funcDesc.getChildren().get(0) instanceof ExprNodeColumnDesc)

          { + return expr.getChildren().get(0); + }

          + return expr;
          + }
          +
          + private void processingBetweenOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) {
          + ExprNodeColumnDesc columnDesc = null;
          + String[] fields = null;
          +
          + if (nodeOutputs[1] instanceof ExprNodeFieldDesc)

          { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[1]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[1], (ExprNodeDesc) nodeOutputs[2]); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + }

          else if (nodeOutputs[0] instanceof ExprNodeGenericFuncDesc)

          { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[1]).getChildren().get(0); + }

          else

          { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[1]; + }

          +
          + String udfName = expr.getGenericUDF().getUdfName();
          + ExprNodeConstantDesc[] betweenConstants = new ExprNodeConstantDesc[]

          {(ExprNodeConstantDesc) nodeOutputs[2], (ExprNodeConstantDesc) nodeOutputs[3]}

          ;
          + boolean isNot = (Boolean) ((ExprNodeConstantDesc) nodeOutputs[0]).getValue();
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, betweenConstants, expr, fields, isNot));
          + }
          +
          + private void processingInOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, boolean isNot, Object... nodeOutputs) {
          + ExprNodeColumnDesc columnDesc = null;
          + String[] fields = null;
          +
          + if (LOG.isTraceEnabled())

          { + LOG.trace("Processing In Operator. nodeOutputs : " + Lists.newArrayList(nodeOutputs)); + }

          +
          + if (nodeOutputs[0] instanceof ExprNodeFieldDesc) {
          + // rowKey field
          + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0];
          + fields = ExprNodeDescUtils.extractFields(fieldDesc);
          +
          + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], (ExprNodeDesc) nodeOutputs[1]);
          +
          + if (extracted == null)

          { // adding for tez + return; + }
          +
          + if (LOG.isTraceEnabled()) { + LOG.trace("nodeOutputs[0] : " + nodeOutputs[0] + ", nodeOutputs[1] : " + + nodeOutputs[1] + " => " + Lists.newArrayList(extracted)); + }
          +
          + columnDesc = (ExprNodeColumnDesc) extracted[0];
          + } else if (nodeOutputs[0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + }
          +
          + String udfName = expr.getGenericUDF().getUdfName();
          + ExprNodeConstantDesc[] inConstantDescs = new ExprNodeConstantDesc[nodeOutputs.length - 1];
          +
          + for (int i = 0, limit = inConstantDescs.length; i < limit; i++) {
          + if (!(nodeOutputs[i + 1] instanceof ExprNodeConstantDesc)) { // adding for tez + return; + }

          +
          + inConstantDescs[i] = (ExprNodeConstantDesc) nodeOutputs[i + 1];
          + }
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, inConstantDescs, expr, fields, isNot));
          + }
          +
          + private void processingNullOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) {
          + ExprNodeColumnDesc columnDesc = null;
          + String[] fields = null;
          +
          + if (nodeOutputs[0] instanceof ExprNodeFieldDesc)

          { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], new ExprNodeConstantDesc()); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + } else if (nodeOutputs[0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + }
          +
          + String udfName = expr.getGenericUDF().getUdfName();
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, null, expr, fields, false));
          + }
          +
          + private void processingNotNullOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) {
          + ExprNodeColumnDesc columnDesc = null;
          + String[] fields = null;
          +
          + if (nodeOutputs[0] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], new ExprNodeConstantDesc()); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + }

          else if (nodeOutputs[0] instanceof ExprNodeGenericFuncDesc)

          { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + }

          else

          { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + }

          +
          + String udfName = expr.getGenericUDF().getUdfName();
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, null, expr, fields, true));
          + }
          +
          + private ExprNodeDesc analyzeExpr(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) throws SemanticException {
          +
          + if (FunctionRegistry.isOpAnd(expr)) {
          + assert (nodeOutputs.length == 2);
          + ExprNodeDesc residual1 = (ExprNodeDesc) nodeOutputs[0];
          + ExprNodeDesc residual2 = (ExprNodeDesc) nodeOutputs[1];
          + if (residual1 == null)

          { + return residual2; + }

          + if (residual2 == null)

          { + return residual1; + }

          + List<ExprNodeDesc> residuals = new ArrayList<ExprNodeDesc>();
          + residuals.add(residual1);
          + residuals.add(residual2);
          + return new ExprNodeGenericFuncDesc(TypeInfoFactory.booleanTypeInfo, FunctionRegistry.getGenericUDFForAnd(), residuals);
          + }
          +
          + GenericUDF genericUDF = expr.getGenericUDF();
          + if (!(genericUDF instanceof GenericUDFBaseCompare)) {
          + // 2015-10-22 Added by JeongMin Ju : Processing Between/In Operator
          + if (genericUDF instanceof GenericUDFBetween)

          { + // In case of not between, The value of first element of nodeOutputs is true. otherwise false. + processingBetweenOperator(expr, searchConditions, nodeOutputs); + return expr; + }

          else if (genericUDF instanceof GenericUDFIn)

          { + // In case of not in operator, in operator exist as child of not operator. + processingInOperator(expr, searchConditions, false, nodeOutputs); + return expr; + }

          else if (genericUDF instanceof GenericUDFOPNot &&
          + ((ExprNodeGenericFuncDesc) expr.getChildren().get(0)).getGenericUDF() instanceof GenericUDFIn)

          { + // In case of not in operator, in operator exist as child of not operator. + processingInOperator((ExprNodeGenericFuncDesc) expr.getChildren().get(0), + searchConditions, true, ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().toArray()); + return expr; + }

          else if (genericUDF instanceof GenericUDFOPNull)

          { + processingNullOperator(expr, searchConditions, nodeOutputs); + return expr; + }

          else if (genericUDF instanceof GenericUDFOPNotNull)

          { + processingNotNullOperator(expr, searchConditions, nodeOutputs); + return expr; + }

          else

          { + return expr; + }
          + }
          + ExprNodeDesc expr1 = (ExprNodeDesc) nodeOutputs[0];
          + ExprNodeDesc expr2 = (ExprNodeDesc) nodeOutputs[1];
          + // We may need to peel off the GenericUDFBridge that is added by CBO or
          + // user
          + if (expr1.getTypeInfo().equals(expr2.getTypeInfo())) { + expr1 = getColumnExpr(expr1); + expr2 = getColumnExpr(expr2); + }
          +
          + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair(expr1, expr2);
          + if (extracted == null || (extracted.length > 2 && !acceptsFields)) { + return expr; + }
          +
          + ExprNodeColumnDesc columnDesc;
          + ExprNodeConstantDesc constantDesc;
          + if (extracted[0] instanceof ExprNodeConstantDesc) { + genericUDF = genericUDF.flip(); + columnDesc = (ExprNodeColumnDesc) extracted[1]; + constantDesc = (ExprNodeConstantDesc) extracted[0]; + } else { + columnDesc = (ExprNodeColumnDesc) extracted[0]; + constantDesc = (ExprNodeConstantDesc) extracted[1]; + }
          +
          + Set<String> allowed = columnToUDFs.get(columnDesc.getColumn());
          + if (allowed == null) { + return expr; + }
          +
          + String udfName = genericUDF.getUdfName();
          + if (!allowed.contains(genericUDF.getUdfName())) { + return expr; + }
          +
          + String[] fields = null;
          + if (extracted.length > 2) {
          + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) extracted[2];
          + if (!isValidField(fieldDesc)) { + return expr; + }

          + fields = ExprNodeDescUtils.extractFields(fieldDesc);
          + }
          +
          + // We also need to update the expr so that the index query can be
          + // generated.
          + // Note that, hive does not support UDFToDouble etc in the query text.
          + List<ExprNodeDesc> list = new ArrayList<ExprNodeDesc>();
          + list.add(expr1);
          + list.add(expr2);
          + expr = new ExprNodeGenericFuncDesc(expr.getTypeInfo(), expr.getGenericUDF(), list);
          +
          + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, constantDesc, expr, fields));
          +
          + // we converted the expression to a search condition, so
          + // remove it from the residual predicate
          + return fields == null ? null : expr;
          + }
          +
          + private boolean isValidField(ExprNodeFieldDesc field)

          { + return fieldValidator == null || fieldValidator.validate(field); + }

          +
          + /**
          + * Translates search conditions back to ExprNodeDesc form (as a left-deep
          + * conjunction).
          + *
          + * @param searchConditions (typically produced by analyzePredicate)
          + * @return ExprNodeGenericFuncDesc form of search conditions
          + */
          + public ExprNodeGenericFuncDesc translateSearchConditions(List<IndexSearchCondition> searchConditions) {
          +
          + ExprNodeGenericFuncDesc expr = null;
          +
          + for (IndexSearchCondition searchCondition : searchConditions) {
          + if (expr == null)

          { + expr = searchCondition.getComparisonExpr(); + continue; + }

          +
          + List<ExprNodeDesc> children = new ArrayList<ExprNodeDesc>();
          + children.add(expr);
          + children.add(searchCondition.getComparisonExpr());
          + expr = new ExprNodeGenericFuncDesc(TypeInfoFactory.booleanTypeInfo, FunctionRegistry.getGenericUDFForAnd(), children);
          + }
          +
          + return expr;
          + }
          +
          + public void setAcceptsFields(boolean acceptsFields)

          { + this.acceptsFields = acceptsFields; + }

          +
          + public static interface FieldValidator

          { + boolean validate(ExprNodeFieldDesc exprNodeDesc); + }

          +
          + public static IndexPredicateAnalyzer createAnalyzer(boolean equalOnly) {
          + IndexPredicateAnalyzer analyzer = new IndexPredicateAnalyzer();
          + analyzer.addComparisonOp("org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqual");
          +
          + if (equalOnly)

          { + return analyzer; + }

          +
          + analyzer.addComparisonOp("org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan");
          — End diff –

          Is there a reason to use hard-coded strings and not Class.getName()?

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59273812 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/ql/index/IndexPredicateAnalyzer.java — @@ -0,0 +1,486 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive.ql.index; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.Stack; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hive.ql.exec.FunctionRegistry; +import org.apache.hadoop.hive.ql.lib.DefaultGraphWalker; +import org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher; +import org.apache.hadoop.hive.ql.lib.Dispatcher; +import org.apache.hadoop.hive.ql.lib.GraphWalker; +import org.apache.hadoop.hive.ql.lib.Node; +import org.apache.hadoop.hive.ql.lib.NodeProcessor; +import org.apache.hadoop.hive.ql.lib.NodeProcessorCtx; +import org.apache.hadoop.hive.ql.lib.Rule; +import org.apache.hadoop.hive.ql.parse.SemanticException; +import org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc; +import org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc; +import org.apache.hadoop.hive.ql.plan.ExprNodeDesc; +import org.apache.hadoop.hive.ql.plan.ExprNodeDescUtils; +import org.apache.hadoop.hive.ql.plan.ExprNodeFieldDesc; +import org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDF; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBaseCompare; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFIn; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNot; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNotNull; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPNull; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToBinary; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToChar; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDate; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToDecimal; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUnixTimeStamp; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToUtcTimestamp; +import org.apache.hadoop.hive.ql.udf.generic.GenericUDFToVarchar; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory; + +import com.google.common.collect.Lists; + +/** + * Clone of org.apache.hadoop.hive.ql.index.IndexPredicateAnalyzer with modifying + * analyzePredicate method. + * + * + */ +public class IndexPredicateAnalyzer { + + private static final Log LOG = LogFactory.getLog(IndexPredicateAnalyzer.class); + + private final Set<String> udfNames; + private final Map<String, Set<String>> columnToUDFs; + private FieldValidator fieldValidator; + + private boolean acceptsFields; + + public IndexPredicateAnalyzer() { + udfNames = new HashSet<String>(); + columnToUDFs = new HashMap<String, Set<String>>(); + } + + public void setFieldValidator(FieldValidator fieldValidator) { + this.fieldValidator = fieldValidator; + } + + /** + * Registers a comparison operator as one which can be satisfied by an index + * search. Unless this is called, analyzePredicate will never find any + * indexable conditions. + * + * @param udfName name of comparison operator as returned by either + * {@link GenericUDFBridge#getUdfName} (for simple UDF's) or + * udf.getClass().getName() (for generic UDF's). + */ + public void addComparisonOp(String udfName) { + udfNames.add(udfName); + } + + /** + * Clears the set of column names allowed in comparisons. (Initially, all + * column names are allowed.) + */ + public void clearAllowedColumnNames() { + columnToUDFs.clear(); + } + + /** + * Adds a column name to the set of column names allowed. + * + * @param columnName name of column to be allowed + */ + public void allowColumnName(String columnName) { + columnToUDFs.put(columnName, udfNames); + } + + /** + * add allowed functions per column + * + * @param columnName + * @param udfs + */ + public void addComparisonOp(String columnName, String... udfs) { + Set<String> allowed = columnToUDFs.get(columnName); + if (allowed == null || allowed == udfNames) { + // override + columnToUDFs.put(columnName, new HashSet<String>(Arrays.asList(udfs))); + } else { + allowed.addAll(Arrays.asList(udfs)); + } + } + + /** + * Analyzes a predicate. + * + * @param predicate predicate to be analyzed + * @param searchConditions receives conditions produced by analysis + * @return residual predicate which could not be translated to + * searchConditions + */ + public ExprNodeDesc analyzePredicate(ExprNodeDesc predicate, final List<IndexSearchCondition> searchConditions) { + + Map<Rule, NodeProcessor> opRules = new LinkedHashMap<Rule, NodeProcessor>(); + NodeProcessor nodeProcessor = new NodeProcessor() { + @Override + public Object process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, Object... nodeOutputs) throws SemanticException { + + // We can only push down stuff which appears as part of + // a pure conjunction: reject OR, CASE, etc. + for (Node ancestor : stack) { + if (nd == ancestor) { + break; + } + if (!FunctionRegistry.isOpAnd((ExprNodeDesc) ancestor)) { + return nd; + } + } + + return analyzeExpr((ExprNodeGenericFuncDesc) nd, searchConditions, nodeOutputs); + } + }; + + Dispatcher disp = new DefaultRuleDispatcher(nodeProcessor, opRules, null); + GraphWalker ogw = new DefaultGraphWalker(disp); + ArrayList<Node> topNodes = new ArrayList<Node>(); + topNodes.add(predicate); + HashMap<Node, Object> nodeOutput = new HashMap<Node, Object>(); + + try { + ogw.startWalking(topNodes, nodeOutput); + } catch (SemanticException ex) { + throw new RuntimeException(ex); + } + + ExprNodeDesc residualPredicate = (ExprNodeDesc) nodeOutput.get(predicate); + return residualPredicate; + } + + // Check if ExprNodeColumnDesc is wrapped in expr. + // If so, peel off. Otherwise return itself. + private ExprNodeDesc getColumnExpr(ExprNodeDesc expr) { + if (expr instanceof ExprNodeColumnDesc) { + return expr; + } + ExprNodeGenericFuncDesc funcDesc = null; + if (expr instanceof ExprNodeGenericFuncDesc) { + funcDesc = (ExprNodeGenericFuncDesc) expr; + } + if (null == funcDesc) { + return expr; + } + GenericUDF udf = funcDesc.getGenericUDF(); + // check if its a simple cast expression. + if ((udf instanceof GenericUDFBridge || udf instanceof GenericUDFToBinary || udf instanceof GenericUDFToChar + || udf instanceof GenericUDFToVarchar || udf instanceof GenericUDFToDecimal + || udf instanceof GenericUDFToDate || udf instanceof GenericUDFToUnixTimeStamp + || udf instanceof GenericUDFToUtcTimestamp) && funcDesc.getChildren().size() == 1 + && funcDesc.getChildren().get(0) instanceof ExprNodeColumnDesc) { + return expr.getChildren().get(0); + } + return expr; + } + + private void processingBetweenOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) { + ExprNodeColumnDesc columnDesc = null; + String[] fields = null; + + if (nodeOutputs [1] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[1]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[1], (ExprNodeDesc) nodeOutputs[2]); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + } else if (nodeOutputs [0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[1]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[1]; + } + + String udfName = expr.getGenericUDF().getUdfName(); + ExprNodeConstantDesc[] betweenConstants = new ExprNodeConstantDesc[] {(ExprNodeConstantDesc) nodeOutputs[2], (ExprNodeConstantDesc) nodeOutputs[3]} ; + boolean isNot = (Boolean) ((ExprNodeConstantDesc) nodeOutputs [0] ).getValue(); + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, betweenConstants, expr, fields, isNot)); + } + + private void processingInOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, boolean isNot, Object... nodeOutputs) { + ExprNodeColumnDesc columnDesc = null; + String[] fields = null; + + if (LOG.isTraceEnabled()) { + LOG.trace("Processing In Operator. nodeOutputs : " + Lists.newArrayList(nodeOutputs)); + } + + if (nodeOutputs [0] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs [0] ; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs [0] , (ExprNodeDesc) nodeOutputs [1] ); + + if (extracted == null) { // adding for tez + return; + } + + if (LOG.isTraceEnabled()) { + LOG.trace("nodeOutputs[0] : " + nodeOutputs[0] + ", nodeOutputs[1] : " + + nodeOutputs[1] + " => " + Lists.newArrayList(extracted)); + } + + columnDesc = (ExprNodeColumnDesc) extracted [0] ; + } else if (nodeOutputs [0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + } + + String udfName = expr.getGenericUDF().getUdfName(); + ExprNodeConstantDesc[] inConstantDescs = new ExprNodeConstantDesc [nodeOutputs.length - 1] ; + + for (int i = 0, limit = inConstantDescs.length; i < limit; i++) { + if (!(nodeOutputs [i + 1] instanceof ExprNodeConstantDesc)) { // adding for tez + return; + } + + inConstantDescs [i] = (ExprNodeConstantDesc) nodeOutputs [i + 1] ; + } + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, inConstantDescs, expr, fields, isNot)); + } + + private void processingNullOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) { + ExprNodeColumnDesc columnDesc = null; + String[] fields = null; + + if (nodeOutputs [0] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], new ExprNodeConstantDesc()); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + } else if (nodeOutputs [0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + } + + String udfName = expr.getGenericUDF().getUdfName(); + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, null, expr, fields, false)); + } + + private void processingNotNullOperator(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) { + ExprNodeColumnDesc columnDesc = null; + String[] fields = null; + + if (nodeOutputs [0] instanceof ExprNodeFieldDesc) { + // rowKey field + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) nodeOutputs[0]; + fields = ExprNodeDescUtils.extractFields(fieldDesc); + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair((ExprNodeDesc) nodeOutputs[0], new ExprNodeConstantDesc()); + columnDesc = (ExprNodeColumnDesc) extracted[0]; + } else if (nodeOutputs [0] instanceof ExprNodeGenericFuncDesc) { + columnDesc = (ExprNodeColumnDesc) ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().get(0); + } else { + columnDesc = (ExprNodeColumnDesc) nodeOutputs[0]; + } + + String udfName = expr.getGenericUDF().getUdfName(); + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, null, expr, fields, true)); + } + + private ExprNodeDesc analyzeExpr(ExprNodeGenericFuncDesc expr, List<IndexSearchCondition> searchConditions, Object... nodeOutputs) throws SemanticException { + + if (FunctionRegistry.isOpAnd(expr)) { + assert (nodeOutputs.length == 2); + ExprNodeDesc residual1 = (ExprNodeDesc) nodeOutputs [0] ; + ExprNodeDesc residual2 = (ExprNodeDesc) nodeOutputs [1] ; + if (residual1 == null) { + return residual2; + } + if (residual2 == null) { + return residual1; + } + List<ExprNodeDesc> residuals = new ArrayList<ExprNodeDesc>(); + residuals.add(residual1); + residuals.add(residual2); + return new ExprNodeGenericFuncDesc(TypeInfoFactory.booleanTypeInfo, FunctionRegistry.getGenericUDFForAnd(), residuals); + } + + GenericUDF genericUDF = expr.getGenericUDF(); + if (!(genericUDF instanceof GenericUDFBaseCompare)) { + // 2015-10-22 Added by JeongMin Ju : Processing Between/In Operator + if (genericUDF instanceof GenericUDFBetween) { + // In case of not between, The value of first element of nodeOutputs is true. otherwise false. + processingBetweenOperator(expr, searchConditions, nodeOutputs); + return expr; + } else if (genericUDF instanceof GenericUDFIn) { + // In case of not in operator, in operator exist as child of not operator. + processingInOperator(expr, searchConditions, false, nodeOutputs); + return expr; + } else if (genericUDF instanceof GenericUDFOPNot && + ((ExprNodeGenericFuncDesc) expr.getChildren().get(0)).getGenericUDF() instanceof GenericUDFIn) { + // In case of not in operator, in operator exist as child of not operator. + processingInOperator((ExprNodeGenericFuncDesc) expr.getChildren().get(0), + searchConditions, true, ((ExprNodeGenericFuncDesc) nodeOutputs[0]).getChildren().toArray()); + return expr; + } else if (genericUDF instanceof GenericUDFOPNull) { + processingNullOperator(expr, searchConditions, nodeOutputs); + return expr; + } else if (genericUDF instanceof GenericUDFOPNotNull) { + processingNotNullOperator(expr, searchConditions, nodeOutputs); + return expr; + } else { + return expr; + } + } + ExprNodeDesc expr1 = (ExprNodeDesc) nodeOutputs [0] ; + ExprNodeDesc expr2 = (ExprNodeDesc) nodeOutputs [1] ; + // We may need to peel off the GenericUDFBridge that is added by CBO or + // user + if (expr1.getTypeInfo().equals(expr2.getTypeInfo())) { + expr1 = getColumnExpr(expr1); + expr2 = getColumnExpr(expr2); + } + + ExprNodeDesc[] extracted = ExprNodeDescUtils.extractComparePair(expr1, expr2); + if (extracted == null || (extracted.length > 2 && !acceptsFields)) { + return expr; + } + + ExprNodeColumnDesc columnDesc; + ExprNodeConstantDesc constantDesc; + if (extracted [0] instanceof ExprNodeConstantDesc) { + genericUDF = genericUDF.flip(); + columnDesc = (ExprNodeColumnDesc) extracted[1]; + constantDesc = (ExprNodeConstantDesc) extracted[0]; + } else { + columnDesc = (ExprNodeColumnDesc) extracted[0]; + constantDesc = (ExprNodeConstantDesc) extracted[1]; + } + + Set<String> allowed = columnToUDFs.get(columnDesc.getColumn()); + if (allowed == null) { + return expr; + } + + String udfName = genericUDF.getUdfName(); + if (!allowed.contains(genericUDF.getUdfName())) { + return expr; + } + + String[] fields = null; + if (extracted.length > 2) { + ExprNodeFieldDesc fieldDesc = (ExprNodeFieldDesc) extracted [2] ; + if (!isValidField(fieldDesc)) { + return expr; + } + fields = ExprNodeDescUtils.extractFields(fieldDesc); + } + + // We also need to update the expr so that the index query can be + // generated. + // Note that, hive does not support UDFToDouble etc in the query text. + List<ExprNodeDesc> list = new ArrayList<ExprNodeDesc>(); + list.add(expr1); + list.add(expr2); + expr = new ExprNodeGenericFuncDesc(expr.getTypeInfo(), expr.getGenericUDF(), list); + + searchConditions.add(new IndexSearchCondition(columnDesc, udfName, constantDesc, expr, fields)); + + // we converted the expression to a search condition, so + // remove it from the residual predicate + return fields == null ? null : expr; + } + + private boolean isValidField(ExprNodeFieldDesc field) { + return fieldValidator == null || fieldValidator.validate(field); + } + + /** + * Translates search conditions back to ExprNodeDesc form (as a left-deep + * conjunction). + * + * @param searchConditions (typically produced by analyzePredicate) + * @return ExprNodeGenericFuncDesc form of search conditions + */ + public ExprNodeGenericFuncDesc translateSearchConditions(List<IndexSearchCondition> searchConditions) { + + ExprNodeGenericFuncDesc expr = null; + + for (IndexSearchCondition searchCondition : searchConditions) { + if (expr == null) { + expr = searchCondition.getComparisonExpr(); + continue; + } + + List<ExprNodeDesc> children = new ArrayList<ExprNodeDesc>(); + children.add(expr); + children.add(searchCondition.getComparisonExpr()); + expr = new ExprNodeGenericFuncDesc(TypeInfoFactory.booleanTypeInfo, FunctionRegistry.getGenericUDFForAnd(), children); + } + + return expr; + } + + public void setAcceptsFields(boolean acceptsFields) { + this.acceptsFields = acceptsFields; + } + + public static interface FieldValidator { + boolean validate(ExprNodeFieldDesc exprNodeDesc); + } + + public static IndexPredicateAnalyzer createAnalyzer(boolean equalOnly) { + IndexPredicateAnalyzer analyzer = new IndexPredicateAnalyzer(); + analyzer.addComparisonOp("org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqual"); + + if (equalOnly) { + return analyzer; + } + + analyzer.addComparisonOp("org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan"); — End diff – Is there a reason to use hard-coded strings and not Class .getName()?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59273613

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixResultWritable.java —
          @@ -0,0 +1,215 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive.mapreduce;
          +
          +import java.io.DataInput;
          +import java.io.DataOutput;
          +import java.io.IOException;
          +import java.sql.PreparedStatement;
          +import java.sql.ResultSet;
          +import java.sql.ResultSetMetaData;
          +import java.sql.SQLException;
          +import java.util.List;
          +import java.util.Map;
          +
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.conf.Configurable;
          +import org.apache.hadoop.conf.Configuration;
          +import org.apache.hadoop.io.Writable;
          +import org.apache.hadoop.mapreduce.lib.db.DBWritable;
          +import org.apache.phoenix.hive.PhoenixRowKey;
          +import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants;
          +import org.apache.phoenix.hive.util.PhoenixStorageHandlerUtil;
          +import org.apache.phoenix.hive.util.PhoenixUtil;
          +import org.apache.phoenix.util.ColumnInfo;
          +
          +import com.google.common.collect.Lists;
          +import com.google.common.collect.Maps;
          +
          +/**
          + * Serialized class for SerDe
          + *
          + */
          +public class PhoenixResultWritable implements Writable, DBWritable, Configurable {
          +
          + private static final Log LOG = LogFactory.getLog(PhoenixResultWritable.class);
          +
          + private List<ColumnInfo> columnMetadataList;
          + private List<Object> valueList; // for output
          + private Map<String, Object> rowMap = Maps.newHashMap(); // for input
          +
          + private int columnCount = -1;
          +
          + private Configuration config;
          + private boolean isTransactional;
          + private Map<String, Object> rowKeyMap = Maps.newLinkedHashMap();
          + private List<String> primaryKeyColumnList;
          +
          + public PhoenixResultWritable()

          { + }
          +
          + public PhoenixResultWritable(Configuration config) throws IOException { + setConf(config); + }
          +
          + public PhoenixResultWritable(Configuration config, List<ColumnInfo> columnMetadataList) throws IOException { + this(config); + this.columnMetadataList = columnMetadataList; + + valueList = Lists.newArrayListWithExpectedSize(columnMetadataList.size()); + }
          +
          + @Override
          + public void write(DataOutput out) throws IOException { + }

          — End diff –

          No need to implement these?

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59273613 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/mapreduce/PhoenixResultWritable.java — @@ -0,0 +1,215 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive.mapreduce; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.util.List; +import java.util.Map; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configurable; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.Writable; +import org.apache.hadoop.mapreduce.lib.db.DBWritable; +import org.apache.phoenix.hive.PhoenixRowKey; +import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants; +import org.apache.phoenix.hive.util.PhoenixStorageHandlerUtil; +import org.apache.phoenix.hive.util.PhoenixUtil; +import org.apache.phoenix.util.ColumnInfo; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; + +/** + * Serialized class for SerDe + * + */ +public class PhoenixResultWritable implements Writable, DBWritable, Configurable { + + private static final Log LOG = LogFactory.getLog(PhoenixResultWritable.class); + + private List<ColumnInfo> columnMetadataList; + private List<Object> valueList; // for output + private Map<String, Object> rowMap = Maps.newHashMap(); // for input + + private int columnCount = -1; + + private Configuration config; + private boolean isTransactional; + private Map<String, Object> rowKeyMap = Maps.newLinkedHashMap(); + private List<String> primaryKeyColumnList; + + public PhoenixResultWritable() { + } + + public PhoenixResultWritable(Configuration config) throws IOException { + setConf(config); + } + + public PhoenixResultWritable(Configuration config, List<ColumnInfo> columnMetadataList) throws IOException { + this(config); + this.columnMetadataList = columnMetadataList; + + valueList = Lists.newArrayListWithExpectedSize(columnMetadataList.size()); + } + + @Override + public void write(DataOutput out) throws IOException { + } — End diff – No need to implement these?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59272080

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixObjectInspectorFactory.java —
          @@ -0,0 +1,150 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive.objectinspector;
          +
          +import java.util.ArrayList;
          +import java.util.List;
          +
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters;
          +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory;
          +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector;
          +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
          +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions;
          +import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo;
          +import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo;
          +import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
          +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
          +
          +public class PhoenixObjectInspectorFactory {
          — End diff –

          I'm wondering why you need this factory (and the accompanying OI implementations for the different types) and you can't reuse the Hive ObjectInspector implementations. Something to do with strongly-typed values from Phoenix (as opposed to HBase/Accumulo's implementations)?

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59272080 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/objectinspector/PhoenixObjectInspectorFactory.java — @@ -0,0 +1,150 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive.objectinspector; + +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory; +import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions; +import org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; + +public class PhoenixObjectInspectorFactory { — End diff – I'm wondering why you need this factory (and the accompanying OI implementations for the different types) and you can't reuse the Hive ObjectInspector implementations. Something to do with strongly-typed values from Phoenix (as opposed to HBase/Accumulo's implementations)?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59271033

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/constants/PhoenixStorageHandlerConstants.java —
          @@ -0,0 +1,101 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive.constants;
          +
          +import java.util.List;
          +
          +import org.apache.hadoop.io.IntWritable;
          +
          +import com.google.common.collect.Lists;
          +
          +/**
          + * Constants using for Hive Storage Handler implementation
          + */
          +public class PhoenixStorageHandlerConstants {
          +
          + public static final String HBASE_INPUT_FORMAT_CLASS = "phoenix.input.format.class";
          +
          + public static final String PHOENIX_TABLE_NAME = "phoenix.table.name";
          +
          + public static final String DEFAULT_PHOENIX_INPUT_CLASS = "com.lgcns.bigdata.platform.hive.hbase.phoenix.mapreduce.PhoenixResultWritable";
          +
          + public static final String ZOOKEEPER_QUORUM = "phoenix.zookeeper.quorum";
          + public static final String ZOOKEEPER_PORT = "phoenix.zookeeper.client.port";
          + public static final String ZOOKEEPER_PARENT = "phoenix.zookeeper.znode.parent";
          + public static final String DEFAULT_ZOOKEEPER_QUORUM = "localhost";
          + public static final int DEFAULT_ZOOKEEPER_PORT = 2181;
          + public static final String DEFAULT_ZOOKEEPER_PARENT = "/hbase";
          +
          + public static final String PHOENIX_ROWKEYS = "phoenix.rowkeys";
          + public static final String PHOENIX_COLUMN_MAPPING = "phoenix.column.mapping";
          + public static final String PHOENIX_TABLE_OPTIONS = "phoenix.table.options";
          +
          + public static final String PHOENIX_TABLE_QUERY_HINT = ".query.hint";
          + public static final String PHOENIX_REDUCER_NUMBER = ".reducer.count";
          + public static final String DISABLE_WAL = ".disable.wal";
          + public static final String BATCH_MODE = "batch.mode";
          + public static final String AUTO_FLUSH = ".auto.flush";
          +
          + public static final String COLON = ":";
          + public static final String COMMA = ",";
          + public static final String EMPTY_STRING = "";
          + public static final String SPACE = " ";
          + public static final String LEFT_ROUND_BRACKET = "(";
          + public static final String RIGHT_ROUND_BRACKET = ")";
          + public static final String QUOTATION_MARK = "'";
          + public static final String EQUAL = "=";
          + public static final String IS = "is";
          + public static final String QUESTION = "?";
          +
          + public static final String SPLIT_BY_STATS = "split.by.stats";
          + public static final String HBASE_SCAN_CACHE = "hbase.scan.cache";
          + public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock";
          + public static final String HBASE_DATE_FORMAT = "hbase.date.format";
          + public static final String HBASE_TIMESTAMP_FORMAT = "hbase.timestamp.format";
          + public static final String DEFAULT_DATE_FORMAT = "yyyy-MM-dd";
          + public static final String DEFAULT_TIMESTAMP_FORMAT = "yyyy-MM-dd HH:mm:ss.SSS";
          +
          + public static final String IN_OUT_WORK = "in.out.work";
          + public static final String IN_WORK = "input";
          + public static final String OUT_WORK = "output";
          +
          + public static final String MR = "mr";
          + public static final String TEZ = "tez";
          + public static final String SPARK = "spark";
          +
          + public static final String DATE_TYPE = "date";
          + public static final String TIMESTAMP_TYPE = "timestamp";
          + public static final String BETWEEN_COMPARATOR = "between";
          + public static final String IN_COMPARATOR = "in";
          + public static final List<String> COMMON_COMPARATOR = Lists.newArrayList("=", "<", ">", "<=", ">=");
          +
          + // date/timestamp 타입 컬럼의 쿼리 조건 변환
          — End diff –

          Remove the non-english characters

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59271033 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/constants/PhoenixStorageHandlerConstants.java — @@ -0,0 +1,101 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive.constants; + +import java.util.List; + +import org.apache.hadoop.io.IntWritable; + +import com.google.common.collect.Lists; + +/** + * Constants using for Hive Storage Handler implementation + */ +public class PhoenixStorageHandlerConstants { + + public static final String HBASE_INPUT_FORMAT_CLASS = "phoenix.input.format.class"; + + public static final String PHOENIX_TABLE_NAME = "phoenix.table.name"; + + public static final String DEFAULT_PHOENIX_INPUT_CLASS = "com.lgcns.bigdata.platform.hive.hbase.phoenix.mapreduce.PhoenixResultWritable"; + + public static final String ZOOKEEPER_QUORUM = "phoenix.zookeeper.quorum"; + public static final String ZOOKEEPER_PORT = "phoenix.zookeeper.client.port"; + public static final String ZOOKEEPER_PARENT = "phoenix.zookeeper.znode.parent"; + public static final String DEFAULT_ZOOKEEPER_QUORUM = "localhost"; + public static final int DEFAULT_ZOOKEEPER_PORT = 2181; + public static final String DEFAULT_ZOOKEEPER_PARENT = "/hbase"; + + public static final String PHOENIX_ROWKEYS = "phoenix.rowkeys"; + public static final String PHOENIX_COLUMN_MAPPING = "phoenix.column.mapping"; + public static final String PHOENIX_TABLE_OPTIONS = "phoenix.table.options"; + + public static final String PHOENIX_TABLE_QUERY_HINT = ".query.hint"; + public static final String PHOENIX_REDUCER_NUMBER = ".reducer.count"; + public static final String DISABLE_WAL = ".disable.wal"; + public static final String BATCH_MODE = "batch.mode"; + public static final String AUTO_FLUSH = ".auto.flush"; + + public static final String COLON = ":"; + public static final String COMMA = ","; + public static final String EMPTY_STRING = ""; + public static final String SPACE = " "; + public static final String LEFT_ROUND_BRACKET = "("; + public static final String RIGHT_ROUND_BRACKET = ")"; + public static final String QUOTATION_MARK = "'"; + public static final String EQUAL = "="; + public static final String IS = "is"; + public static final String QUESTION = "?"; + + public static final String SPLIT_BY_STATS = "split.by.stats"; + public static final String HBASE_SCAN_CACHE = "hbase.scan.cache"; + public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; + public static final String HBASE_DATE_FORMAT = "hbase.date.format"; + public static final String HBASE_TIMESTAMP_FORMAT = "hbase.timestamp.format"; + public static final String DEFAULT_DATE_FORMAT = "yyyy-MM-dd"; + public static final String DEFAULT_TIMESTAMP_FORMAT = "yyyy-MM-dd HH:mm:ss.SSS"; + + public static final String IN_OUT_WORK = "in.out.work"; + public static final String IN_WORK = "input"; + public static final String OUT_WORK = "output"; + + public static final String MR = "mr"; + public static final String TEZ = "tez"; + public static final String SPARK = "spark"; + + public static final String DATE_TYPE = "date"; + public static final String TIMESTAMP_TYPE = "timestamp"; + public static final String BETWEEN_COMPARATOR = "between"; + public static final String IN_COMPARATOR = "in"; + public static final List<String> COMMON_COMPARATOR = Lists.newArrayList("=", "<", ">", "<=", ">="); + + // date/timestamp 타입 컬럼의 쿼리 조건 변환 — End diff – Remove the non-english characters
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59270355

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixRowKey.java —
          @@ -0,0 +1,69 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive;
          +
          +import com.google.common.collect.Maps;
          +import org.apache.hadoop.hive.ql.io.RecordIdentifier;
          +
          +import java.io.DataInput;
          +import java.io.DataOutput;
          +import java.io.IOException;
          +import java.io.InputStream;
          +import java.io.ObjectInputStream;
          +import java.io.ObjectOutputStream;
          +import java.io.OutputStream;
          +import java.util.Map;
          +
          +public class PhoenixRowKey extends RecordIdentifier {
          +
          + private Map<String, Object> rowKeyMap = Maps.newHashMap();
          +
          + public PhoenixRowKey()

          { + + }

          +
          + // public void add(String columnName, Object value) {
          — End diff –

          Dead code

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59270355 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixRowKey.java — @@ -0,0 +1,69 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive; + +import com.google.common.collect.Maps; +import org.apache.hadoop.hive.ql.io.RecordIdentifier; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import java.io.InputStream; +import java.io.ObjectInputStream; +import java.io.ObjectOutputStream; +import java.io.OutputStream; +import java.util.Map; + +public class PhoenixRowKey extends RecordIdentifier { + + private Map<String, Object> rowKeyMap = Maps.newHashMap(); + + public PhoenixRowKey() { + + } + + // public void add(String columnName, Object value) { — End diff – Dead code
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59269847

          — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixMetaHook.java —
          @@ -0,0 +1,245 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.phoenix.hive;
          +
          +import java.sql.Connection;
          +import java.sql.SQLException;
          +import java.util.Collections;
          +import java.util.List;
          +import java.util.Map;
          +
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.hive.metastore.HiveMetaHook;
          +import org.apache.hadoop.hive.metastore.TableType;
          +import org.apache.hadoop.hive.metastore.api.FieldSchema;
          +import org.apache.hadoop.hive.metastore.api.MetaException;
          +import org.apache.hadoop.hive.metastore.api.Table;
          +import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants;
          +import org.apache.phoenix.hive.util.PhoenixConnectionUtil;
          +import org.apache.phoenix.hive.util.PhoenixStorageHandlerUtil;
          +import org.apache.phoenix.hive.util.PhoenixUtil;
          +
          +import com.google.common.base.CharMatcher;
          +import com.google.common.base.Splitter;
          +import com.google.common.collect.Lists;
          +
          +public class PhoenixMetaHook implements HiveMetaHook {
          +
          + private static final Log LOG = LogFactory.getLog(PhoenixMetaHook.class);
          +
          + @Override
          + public void preCreateTable(Table table) throws MetaException {
          + if (LOG.isDebugEnabled())

          { + LOG.debug("Precreate table : " + table.getTableName()); + }

          +
          + try (Connection conn = PhoenixConnectionUtil.getConnection(table)) {
          + String tableType = table.getTableType();
          + String tableName = PhoenixStorageHandlerUtil.getTargetTableName(table);
          +
          + if (TableType.EXTERNAL_TABLE.name().equals(tableType)) {
          + // Check whether phoenix table exists.
          + if (!PhoenixUtil.existTable(conn, tableName))

          { + // Error if phoenix table not exist. + throw new MetaException("Phoenix table " + tableName + " doesn't exist"); + }

          + } else if (TableType.MANAGED_TABLE.name().equals(tableType)) {
          + // Check whether phoenix table exists.
          + if (PhoenixUtil.existTable(conn, tableName))

          { + // Error if phoenix table already exist. + throw new MetaException("Phoenix table " + tableName + " already exist."); + }

          +
          + PhoenixUtil.createTable(conn, createTableStatement(table));
          + } else

          { + throw new MetaException("Unsupported table Type: " + table.getTableType()); + }

          +
          + if (LOG.isDebugEnabled())

          { + LOG.debug("Phoenix table " + tableName + " was created"); + }

          + } catch (SQLException e)

          { + throw new MetaException(e.getMessage()); + }

          + }
          +
          + private String createTableStatement(Table table) throws MetaException {
          + Map<String, String> tableParameterMap = table.getParameters();
          +
          + String tableName = PhoenixStorageHandlerUtil.getTargetTableName(table);
          + StringBuilder ddl = new StringBuilder("create table ").append(tableName).append(" (\n");
          +
          + String phoenixRowKeys = tableParameterMap.get(PhoenixStorageHandlerConstants
          + .PHOENIX_ROWKEYS);
          + StringBuilder realRowKeys = new StringBuilder();
          + List<String> phoenixRowKeyList = Lists.newArrayList(Splitter.on
          + (PhoenixStorageHandlerConstants.COMMA).trimResults().split(phoenixRowKeys));
          + Map<String, String> columnMappingMap = getColumnMappingMap(tableParameterMap.get
          + (PhoenixStorageHandlerConstants.PHOENIX_COLUMN_MAPPING));
          +
          + List<FieldSchema> fieldSchemaList = table.getSd().getCols();
          + for (int i = 0, limit = fieldSchemaList.size(); i < limit; i++) {
          + FieldSchema fieldSchema = fieldSchemaList.get;
          + String fieldName = fieldSchema.getName();
          + String fieldType = fieldSchema.getType();
          + String columnType = PhoenixUtil.getPhoenixType(fieldType);
          +
          + String rowKeyName = getRowKeyMapping(fieldName, phoenixRowKeyList);
          + if (rowKeyName != null) {
          + // In case of RowKey
          + if ("binary".equals(columnType))

          { + // Phoenix must define max length of binary when type definition. Obtaining + // information from the column mapping. ex) phoenix.rowkeys = "r1, r2(100), ..." + List<String> tokenList = Lists.newArrayList(Splitter.on(CharMatcher.is('(') + .or(CharMatcher.is(')'))).trimResults().split(rowKeyName)); + columnType = columnType + "(" + tokenList.get(1) + ")"; + rowKeyName = tokenList.get(0); + }

          +
          + ddl.append(" ").append(rowKeyName).append(" ").append(columnType).append(" not " +
          + "null,\n");
          + realRowKeys.append(rowKeyName).append(",");
          + } else {
          + // In case of Column
          + String columnName = columnMappingMap.get(fieldName);
          +
          + if (columnName == null) {
          + // Use field definition.
          + columnName = fieldName;
          +// throw new MetaException("<<<<<<<<<< " + fieldName + " column mapping not exist
          — End diff –

          Remove this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59269847 — Diff: phoenix-hive/src/main/java/org/apache/phoenix/hive/PhoenixMetaHook.java — @@ -0,0 +1,245 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.phoenix.hive; + +import java.sql.Connection; +import java.sql.SQLException; +import java.util.Collections; +import java.util.List; +import java.util.Map; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hive.metastore.HiveMetaHook; +import org.apache.hadoop.hive.metastore.TableType; +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.MetaException; +import org.apache.hadoop.hive.metastore.api.Table; +import org.apache.phoenix.hive.constants.PhoenixStorageHandlerConstants; +import org.apache.phoenix.hive.util.PhoenixConnectionUtil; +import org.apache.phoenix.hive.util.PhoenixStorageHandlerUtil; +import org.apache.phoenix.hive.util.PhoenixUtil; + +import com.google.common.base.CharMatcher; +import com.google.common.base.Splitter; +import com.google.common.collect.Lists; + +public class PhoenixMetaHook implements HiveMetaHook { + + private static final Log LOG = LogFactory.getLog(PhoenixMetaHook.class); + + @Override + public void preCreateTable(Table table) throws MetaException { + if (LOG.isDebugEnabled()) { + LOG.debug("Precreate table : " + table.getTableName()); + } + + try (Connection conn = PhoenixConnectionUtil.getConnection(table)) { + String tableType = table.getTableType(); + String tableName = PhoenixStorageHandlerUtil.getTargetTableName(table); + + if (TableType.EXTERNAL_TABLE.name().equals(tableType)) { + // Check whether phoenix table exists. + if (!PhoenixUtil.existTable(conn, tableName)) { + // Error if phoenix table not exist. + throw new MetaException("Phoenix table " + tableName + " doesn't exist"); + } + } else if (TableType.MANAGED_TABLE.name().equals(tableType)) { + // Check whether phoenix table exists. + if (PhoenixUtil.existTable(conn, tableName)) { + // Error if phoenix table already exist. + throw new MetaException("Phoenix table " + tableName + " already exist."); + } + + PhoenixUtil.createTable(conn, createTableStatement(table)); + } else { + throw new MetaException("Unsupported table Type: " + table.getTableType()); + } + + if (LOG.isDebugEnabled()) { + LOG.debug("Phoenix table " + tableName + " was created"); + } + } catch (SQLException e) { + throw new MetaException(e.getMessage()); + } + } + + private String createTableStatement(Table table) throws MetaException { + Map<String, String> tableParameterMap = table.getParameters(); + + String tableName = PhoenixStorageHandlerUtil.getTargetTableName(table); + StringBuilder ddl = new StringBuilder("create table ").append(tableName).append(" (\n"); + + String phoenixRowKeys = tableParameterMap.get(PhoenixStorageHandlerConstants + .PHOENIX_ROWKEYS); + StringBuilder realRowKeys = new StringBuilder(); + List<String> phoenixRowKeyList = Lists.newArrayList(Splitter.on + (PhoenixStorageHandlerConstants.COMMA).trimResults().split(phoenixRowKeys)); + Map<String, String> columnMappingMap = getColumnMappingMap(tableParameterMap.get + (PhoenixStorageHandlerConstants.PHOENIX_COLUMN_MAPPING)); + + List<FieldSchema> fieldSchemaList = table.getSd().getCols(); + for (int i = 0, limit = fieldSchemaList.size(); i < limit; i++) { + FieldSchema fieldSchema = fieldSchemaList.get ; + String fieldName = fieldSchema.getName(); + String fieldType = fieldSchema.getType(); + String columnType = PhoenixUtil.getPhoenixType(fieldType); + + String rowKeyName = getRowKeyMapping(fieldName, phoenixRowKeyList); + if (rowKeyName != null) { + // In case of RowKey + if ("binary".equals(columnType)) { + // Phoenix must define max length of binary when type definition. Obtaining + // information from the column mapping. ex) phoenix.rowkeys = "r1, r2(100), ..." + List<String> tokenList = Lists.newArrayList(Splitter.on(CharMatcher.is('(') + .or(CharMatcher.is(')'))).trimResults().split(rowKeyName)); + columnType = columnType + "(" + tokenList.get(1) + ")"; + rowKeyName = tokenList.get(0); + } + + ddl.append(" ").append(rowKeyName).append(" ").append(columnType).append(" not " + + "null,\n"); + realRowKeys.append(rowKeyName).append(","); + } else { + // In case of Column + String columnName = columnMappingMap.get(fieldName); + + if (columnName == null) { + // Use field definition. + columnName = fieldName; +// throw new MetaException("<<<<<<<<<< " + fieldName + " column mapping not exist — End diff – Remove this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59269604

          — Diff: phoenix-hive/pom.xml —
          @@ -0,0 +1,224 @@
          +<?xml version='1.0'?>
          +<!--
          +
          + Licensed to the Apache Software Foundation (ASF) under one
          + or more contributor license agreements. See the NOTICE file
          + distributed with this work for additional information
          + regarding copyright ownership. The ASF licenses this file
          + to you under the Apache License, Version 2.0 (the
          + "License"); you may not use this file except in compliance
          + with the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          + Unless required by applicable law or agreed to in writing,
          + software distributed under the License is distributed on an
          + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
          + KIND, either express or implied. See the License for the
          + specific language governing permissions and limitations
          + under the License.
          +
          +-->
          +
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <modelVersion>4.0.0</modelVersion>
          + <parent>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix</artifactId>
          + <version>4.8.0-HBase-1.1-SNAPSHOT</version>
          + </parent>
          + <artifactId>phoenix-hive</artifactId>
          + <name>Phoenix - Hive</name>
          +
          + <dependencies>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + <classifier>tests</classifier>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>joda-time</groupId>
          + <artifactId>joda-time</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-exec</artifactId>
          + <version>$

          {hive.version}

          </version>
          + <exclusions>
          + <exclusion>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>*</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-core</artifactId>
          — End diff –

          Same here, unused?

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59269604 — Diff: phoenix-hive/pom.xml — @@ -0,0 +1,224 @@ +<?xml version='1.0'?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix</artifactId> + <version>4.8.0-HBase-1.1-SNAPSHOT</version> + </parent> + <artifactId>phoenix-hive</artifactId> + <name>Phoenix - Hive</name> + + <dependencies> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + <dependency> + <groupId>joda-time</groupId> + <artifactId>joda-time</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <version>$ {hive.version} </version> + <exclusions> + <exclusion> + <groupId>org.apache.calcite</groupId> + <artifactId>*</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-core</artifactId> — End diff – Same here, unused?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59269559

          — Diff: phoenix-hive/pom.xml —
          @@ -0,0 +1,224 @@
          +<?xml version='1.0'?>
          +<!--
          +
          + Licensed to the Apache Software Foundation (ASF) under one
          + or more contributor license agreements. See the NOTICE file
          + distributed with this work for additional information
          + regarding copyright ownership. The ASF licenses this file
          + to you under the Apache License, Version 2.0 (the
          + "License"); you may not use this file except in compliance
          + with the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          + Unless required by applicable law or agreed to in writing,
          + software distributed under the License is distributed on an
          + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
          + KIND, either express or implied. See the License for the
          + specific language governing permissions and limitations
          + under the License.
          +
          +-->
          +
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <modelVersion>4.0.0</modelVersion>
          + <parent>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix</artifactId>
          + <version>4.8.0-HBase-1.1-SNAPSHOT</version>
          + </parent>
          + <artifactId>phoenix-hive</artifactId>
          + <name>Phoenix - Hive</name>
          +
          + <dependencies>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + <classifier>tests</classifier>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>joda-time</groupId>
          + <artifactId>joda-time</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-exec</artifactId>
          + <version>$

          {hive.version}

          </version>
          + <exclusions>
          + <exclusion>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>*</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-core</artifactId>
          + <version>$

          {calcite.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-avatica</artifactId>
          — End diff –

          Is this used?

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59269559 — Diff: phoenix-hive/pom.xml — @@ -0,0 +1,224 @@ +<?xml version='1.0'?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix</artifactId> + <version>4.8.0-HBase-1.1-SNAPSHOT</version> + </parent> + <artifactId>phoenix-hive</artifactId> + <name>Phoenix - Hive</name> + + <dependencies> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + <dependency> + <groupId>joda-time</groupId> + <artifactId>joda-time</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <version>$ {hive.version} </version> + <exclusions> + <exclusion> + <groupId>org.apache.calcite</groupId> + <artifactId>*</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-core</artifactId> + <version>$ {calcite.version} </version> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-avatica</artifactId> — End diff – Is this used?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59269404

          — Diff: phoenix-hive/pom.xml —
          @@ -0,0 +1,224 @@
          +<?xml version='1.0'?>
          +<!--
          +
          + Licensed to the Apache Software Foundation (ASF) under one
          + or more contributor license agreements. See the NOTICE file
          + distributed with this work for additional information
          + regarding copyright ownership. The ASF licenses this file
          + to you under the Apache License, Version 2.0 (the
          + "License"); you may not use this file except in compliance
          + with the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          + Unless required by applicable law or agreed to in writing,
          + software distributed under the License is distributed on an
          + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
          + KIND, either express or implied. See the License for the
          + specific language governing permissions and limitations
          + under the License.
          +
          +-->
          +
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <modelVersion>4.0.0</modelVersion>
          + <parent>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix</artifactId>
          + <version>4.8.0-HBase-1.1-SNAPSHOT</version>
          + </parent>
          + <artifactId>phoenix-hive</artifactId>
          + <name>Phoenix - Hive</name>
          +
          + <dependencies>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + <classifier>tests</classifier>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>joda-time</groupId>
          + <artifactId>joda-time</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-exec</artifactId>
          + <version>$

          {hive.version}</version>
          + <exclusions>
          + <exclusion>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>*</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-core</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-avatica</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-common</artifactId>
          + <version>${hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-cli</artifactId>
          + <version>$

          {hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>jline</groupId>
          + <artifactId>jline</artifactId>
          + <version>0.9.94</version>
          + </dependency>
          + <dependency>
          + <groupId>commons-lang</groupId>
          + <artifactId>commons-lang</artifactId>
          + <version>$

          {commons-lang.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>commons-logging</groupId>
          + <artifactId>commons-logging</artifactId>
          + <version>$

          {commons-logging.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-testing-util</artifactId>
          + <scope>test</scope>
          + <optional>true</optional>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-it</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-protocol</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-client</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-annotations</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-mapreduce-client-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-minicluster</artifactId>
          + </dependency>
          + <!-- Test Dependencies -->
          + <dependency>
          + <groupId>org.mockito</groupId>
          + <artifactId>mockito-all</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>junit</groupId>
          + <artifactId>junit</artifactId>
          + <scope>test</scope>
          + </dependency>
          + </dependencies>
          +
          + <build>
          + <plugins>
          + <plugin>
          + <groupId>org.codehaus.mojo</groupId>
          + <artifactId>build-helper-maven-plugin</artifactId>
          + </plugin>
          + <plugin>
          + <groupId>org.apache.maven.plugins</groupId>
          + <artifactId>maven-failsafe-plugin</artifactId>
          + </plugin>
          + <plugin>
          + <artifactId>maven-dependency-plugin</artifactId>
          + <version>$

          {maven-dependency-plugin.version}

          </version>
          + <executions>
          — End diff –

          Wonky indentation here. Tabs maybe?

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59269404 — Diff: phoenix-hive/pom.xml — @@ -0,0 +1,224 @@ +<?xml version='1.0'?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix</artifactId> + <version>4.8.0-HBase-1.1-SNAPSHOT</version> + </parent> + <artifactId>phoenix-hive</artifactId> + <name>Phoenix - Hive</name> + + <dependencies> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + <dependency> + <groupId>joda-time</groupId> + <artifactId>joda-time</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <version>$ {hive.version}</version> + <exclusions> + <exclusion> + <groupId>org.apache.calcite</groupId> + <artifactId>*</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-core</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-avatica</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-common</artifactId> + <version>${hive.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-cli</artifactId> + <version>$ {hive.version} </version> + </dependency> + <dependency> + <groupId>jline</groupId> + <artifactId>jline</artifactId> + <version>0.9.94</version> + </dependency> + <dependency> + <groupId>commons-lang</groupId> + <artifactId>commons-lang</artifactId> + <version>$ {commons-lang.version} </version> + </dependency> + <dependency> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + <version>$ {commons-logging.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-testing-util</artifactId> + <scope>test</scope> + <optional>true</optional> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-it</artifactId> + <type>test-jar</type> + <scope>test</scope> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-protocol</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-client</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-annotations</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-minicluster</artifactId> + </dependency> + <!-- Test Dependencies --> + <dependency> + <groupId>org.mockito</groupId> + <artifactId>mockito-all</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + </dependencies> + + <build> + <plugins> + <plugin> + <groupId>org.codehaus.mojo</groupId> + <artifactId>build-helper-maven-plugin</artifactId> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-failsafe-plugin</artifactId> + </plugin> + <plugin> + <artifactId>maven-dependency-plugin</artifactId> + <version>$ {maven-dependency-plugin.version} </version> + <executions> — End diff – Wonky indentation here. Tabs maybe?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59269345

          — Diff: phoenix-hive/pom.xml —
          @@ -0,0 +1,224 @@
          +<?xml version='1.0'?>
          +<!--
          +
          + Licensed to the Apache Software Foundation (ASF) under one
          + or more contributor license agreements. See the NOTICE file
          + distributed with this work for additional information
          + regarding copyright ownership. The ASF licenses this file
          + to you under the Apache License, Version 2.0 (the
          + "License"); you may not use this file except in compliance
          + with the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          + Unless required by applicable law or agreed to in writing,
          + software distributed under the License is distributed on an
          + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
          + KIND, either express or implied. See the License for the
          + specific language governing permissions and limitations
          + under the License.
          +
          +-->
          +
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <modelVersion>4.0.0</modelVersion>
          + <parent>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix</artifactId>
          + <version>4.8.0-HBase-1.1-SNAPSHOT</version>
          + </parent>
          + <artifactId>phoenix-hive</artifactId>
          + <name>Phoenix - Hive</name>
          +
          + <dependencies>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + <classifier>tests</classifier>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>joda-time</groupId>
          + <artifactId>joda-time</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-exec</artifactId>
          + <version>$

          {hive.version}</version>
          + <exclusions>
          + <exclusion>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>*</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-core</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-avatica</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-common</artifactId>
          + <version>${hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-cli</artifactId>
          + <version>$

          {hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>jline</groupId>
          + <artifactId>jline</artifactId>
          + <version>0.9.94</version>
          + </dependency>
          + <dependency>
          + <groupId>commons-lang</groupId>
          + <artifactId>commons-lang</artifactId>
          + <version>$

          {commons-lang.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>commons-logging</groupId>
          + <artifactId>commons-logging</artifactId>
          + <version>$

          {commons-logging.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-testing-util</artifactId>
          + <scope>test</scope>
          + <optional>true</optional>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-it</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-protocol</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-client</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-annotations</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-mapreduce-client-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-minicluster</artifactId>
          + </dependency>
          + <!-- Test Dependencies -->
          + <dependency>
          + <groupId>org.mockito</groupId>
          + <artifactId>mockito-all</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>junit</groupId>
          + <artifactId>junit</artifactId>
          + <scope>test</scope>
          + </dependency>
          + </dependencies>
          +
          + <build>
          + <plugins>
          + <plugin>
          + <groupId>org.codehaus.mojo</groupId>
          + <artifactId>build-helper-maven-plugin</artifactId>
          + </plugin>
          + <plugin>
          + <groupId>org.apache.maven.plugins</groupId>
          + <artifactId>maven-failsafe-plugin</artifactId>
          + </plugin>
          + <plugin>
          + <artifactId>maven-dependency-plugin</artifactId>
          + <version>$

          {maven-dependency-plugin.version}

          </version>
          — End diff –

          Likely unnecessary version specification

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59269345 — Diff: phoenix-hive/pom.xml — @@ -0,0 +1,224 @@ +<?xml version='1.0'?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix</artifactId> + <version>4.8.0-HBase-1.1-SNAPSHOT</version> + </parent> + <artifactId>phoenix-hive</artifactId> + <name>Phoenix - Hive</name> + + <dependencies> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + <dependency> + <groupId>joda-time</groupId> + <artifactId>joda-time</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <version>$ {hive.version}</version> + <exclusions> + <exclusion> + <groupId>org.apache.calcite</groupId> + <artifactId>*</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-core</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-avatica</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-common</artifactId> + <version>${hive.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-cli</artifactId> + <version>$ {hive.version} </version> + </dependency> + <dependency> + <groupId>jline</groupId> + <artifactId>jline</artifactId> + <version>0.9.94</version> + </dependency> + <dependency> + <groupId>commons-lang</groupId> + <artifactId>commons-lang</artifactId> + <version>$ {commons-lang.version} </version> + </dependency> + <dependency> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + <version>$ {commons-logging.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-testing-util</artifactId> + <scope>test</scope> + <optional>true</optional> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-it</artifactId> + <type>test-jar</type> + <scope>test</scope> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-protocol</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-client</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-annotations</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-minicluster</artifactId> + </dependency> + <!-- Test Dependencies --> + <dependency> + <groupId>org.mockito</groupId> + <artifactId>mockito-all</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + </dependencies> + + <build> + <plugins> + <plugin> + <groupId>org.codehaus.mojo</groupId> + <artifactId>build-helper-maven-plugin</artifactId> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-failsafe-plugin</artifactId> + </plugin> + <plugin> + <artifactId>maven-dependency-plugin</artifactId> + <version>$ {maven-dependency-plugin.version} </version> — End diff – Likely unnecessary version specification
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59269299

          — Diff: phoenix-hive/pom.xml —
          @@ -0,0 +1,224 @@
          +<?xml version='1.0'?>
          +<!--
          +
          + Licensed to the Apache Software Foundation (ASF) under one
          + or more contributor license agreements. See the NOTICE file
          + distributed with this work for additional information
          + regarding copyright ownership. The ASF licenses this file
          + to you under the Apache License, Version 2.0 (the
          + "License"); you may not use this file except in compliance
          + with the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          + Unless required by applicable law or agreed to in writing,
          + software distributed under the License is distributed on an
          + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
          + KIND, either express or implied. See the License for the
          + specific language governing permissions and limitations
          + under the License.
          +
          +-->
          +
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <modelVersion>4.0.0</modelVersion>
          + <parent>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix</artifactId>
          + <version>4.8.0-HBase-1.1-SNAPSHOT</version>
          + </parent>
          + <artifactId>phoenix-hive</artifactId>
          + <name>Phoenix - Hive</name>
          +
          + <dependencies>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + <classifier>tests</classifier>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>joda-time</groupId>
          + <artifactId>joda-time</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-exec</artifactId>
          + <version>$

          {hive.version}</version>
          + <exclusions>
          + <exclusion>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>*</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-core</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-avatica</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-common</artifactId>
          + <version>${hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-cli</artifactId>
          + <version>$

          {hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>jline</groupId>
          + <artifactId>jline</artifactId>
          + <version>0.9.94</version>
          + </dependency>
          + <dependency>
          + <groupId>commons-lang</groupId>
          + <artifactId>commons-lang</artifactId>
          + <version>$

          {commons-lang.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>commons-logging</groupId>
          + <artifactId>commons-logging</artifactId>
          + <version>$

          {commons-logging.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-testing-util</artifactId>
          + <scope>test</scope>
          + <optional>true</optional>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-it</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-protocol</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-client</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-annotations</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-mapreduce-client-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-minicluster</artifactId>
          — End diff –

          Should this be test scope?

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59269299 — Diff: phoenix-hive/pom.xml — @@ -0,0 +1,224 @@ +<?xml version='1.0'?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix</artifactId> + <version>4.8.0-HBase-1.1-SNAPSHOT</version> + </parent> + <artifactId>phoenix-hive</artifactId> + <name>Phoenix - Hive</name> + + <dependencies> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + <dependency> + <groupId>joda-time</groupId> + <artifactId>joda-time</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <version>$ {hive.version}</version> + <exclusions> + <exclusion> + <groupId>org.apache.calcite</groupId> + <artifactId>*</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-core</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-avatica</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-common</artifactId> + <version>${hive.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-cli</artifactId> + <version>$ {hive.version} </version> + </dependency> + <dependency> + <groupId>jline</groupId> + <artifactId>jline</artifactId> + <version>0.9.94</version> + </dependency> + <dependency> + <groupId>commons-lang</groupId> + <artifactId>commons-lang</artifactId> + <version>$ {commons-lang.version} </version> + </dependency> + <dependency> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + <version>$ {commons-logging.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-testing-util</artifactId> + <scope>test</scope> + <optional>true</optional> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-it</artifactId> + <type>test-jar</type> + <scope>test</scope> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-protocol</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-client</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-annotations</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-minicluster</artifactId> — End diff – Should this be test scope?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user joshelser commented on a diff in the pull request:

          https://github.com/apache/phoenix/pull/155#discussion_r59269276

          — Diff: phoenix-hive/pom.xml —
          @@ -0,0 +1,224 @@
          +<?xml version='1.0'?>
          +<!--
          +
          + Licensed to the Apache Software Foundation (ASF) under one
          + or more contributor license agreements. See the NOTICE file
          + distributed with this work for additional information
          + regarding copyright ownership. The ASF licenses this file
          + to you under the Apache License, Version 2.0 (the
          + "License"); you may not use this file except in compliance
          + with the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          + Unless required by applicable law or agreed to in writing,
          + software distributed under the License is distributed on an
          + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
          + KIND, either express or implied. See the License for the
          + specific language governing permissions and limitations
          + under the License.
          +
          +-->
          +
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <modelVersion>4.0.0</modelVersion>
          + <parent>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix</artifactId>
          + <version>4.8.0-HBase-1.1-SNAPSHOT</version>
          + </parent>
          + <artifactId>phoenix-hive</artifactId>
          + <name>Phoenix - Hive</name>
          +
          + <dependencies>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.phoenix</groupId>
          + <artifactId>phoenix-core</artifactId>
          + <classifier>tests</classifier>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>joda-time</groupId>
          + <artifactId>joda-time</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-exec</artifactId>
          + <version>$

          {hive.version}</version>
          + <exclusions>
          + <exclusion>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>*</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-core</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.calcite</groupId>
          + <artifactId>calcite-avatica</artifactId>
          + <version>${calcite.version}</version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-common</artifactId>
          + <version>${hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hive</groupId>
          + <artifactId>hive-cli</artifactId>
          + <version>$

          {hive.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>jline</groupId>
          + <artifactId>jline</artifactId>
          + <version>0.9.94</version>
          + </dependency>
          + <dependency>
          + <groupId>commons-lang</groupId>
          + <artifactId>commons-lang</artifactId>
          + <version>$

          {commons-lang.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>commons-logging</groupId>
          + <artifactId>commons-logging</artifactId>
          + <version>$

          {commons-logging.version}

          </version>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-testing-util</artifactId>
          + <scope>test</scope>
          + <optional>true</optional>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-it</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + <exclusions>
          + <exclusion>
          + <groupId>org.jruby</groupId>
          + <artifactId>jruby-complete</artifactId>
          + </exclusion>
          + </exclusions>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-protocol</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-client</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hbase</groupId>
          + <artifactId>hbase-hadoop2-compat</artifactId>
          + <type>test-jar</type>
          + <scope>test</scope>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-common</artifactId>
          + </dependency>
          + <dependency>
          + <groupId>org.apache.hadoop</groupId>
          + <artifactId>hadoop-annotations</artifactId>
          — End diff –

          I don't see this actively referenced in your changes.

          Show
          githubbot ASF GitHub Bot added a comment - Github user joshelser commented on a diff in the pull request: https://github.com/apache/phoenix/pull/155#discussion_r59269276 — Diff: phoenix-hive/pom.xml — @@ -0,0 +1,224 @@ +<?xml version='1.0'?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix</artifactId> + <version>4.8.0-HBase-1.1-SNAPSHOT</version> + </parent> + <artifactId>phoenix-hive</artifactId> + <name>Phoenix - Hive</name> + + <dependencies> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + </dependency> + <dependency> + <groupId>org.apache.phoenix</groupId> + <artifactId>phoenix-core</artifactId> + <classifier>tests</classifier> + <scope>test</scope> + </dependency> + <dependency> + <groupId>joda-time</groupId> + <artifactId>joda-time</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-exec</artifactId> + <version>$ {hive.version}</version> + <exclusions> + <exclusion> + <groupId>org.apache.calcite</groupId> + <artifactId>*</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-core</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.calcite</groupId> + <artifactId>calcite-avatica</artifactId> + <version>${calcite.version}</version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-common</artifactId> + <version>${hive.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hive</groupId> + <artifactId>hive-cli</artifactId> + <version>$ {hive.version} </version> + </dependency> + <dependency> + <groupId>jline</groupId> + <artifactId>jline</artifactId> + <version>0.9.94</version> + </dependency> + <dependency> + <groupId>commons-lang</groupId> + <artifactId>commons-lang</artifactId> + <version>$ {commons-lang.version} </version> + </dependency> + <dependency> + <groupId>commons-logging</groupId> + <artifactId>commons-logging</artifactId> + <version>$ {commons-logging.version} </version> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-testing-util</artifactId> + <scope>test</scope> + <optional>true</optional> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-it</artifactId> + <type>test-jar</type> + <scope>test</scope> + <exclusions> + <exclusion> + <groupId>org.jruby</groupId> + <artifactId>jruby-complete</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-protocol</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-client</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hbase</groupId> + <artifactId>hbase-hadoop2-compat</artifactId> + <type>test-jar</type> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-common</artifactId> + </dependency> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-annotations</artifactId> — End diff – I don't see this actively referenced in your changes.
          Hide
          jamestaylor James Taylor added a comment - - edited

          Is this ready to go, Sergey Soldatov? How's the test coverage look? I'm not seeing any - is that an oversight, JeongMin Ju. Should we dup out PHOENIX-331 or is that complimentary? Mind taking a look at the PR, ravi?

          Show
          jamestaylor James Taylor added a comment - - edited Is this ready to go, Sergey Soldatov ? How's the test coverage look? I'm not seeing any - is that an oversight, JeongMin Ju . Should we dup out PHOENIX-331 or is that complimentary? Mind taking a look at the PR, ravi ?
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ss77892 opened a pull request:

          https://github.com/apache/phoenix/pull/155

          PHOENIX-2743 Hive Storage support

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/ss77892/phoenix PHOENIX-2743

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/phoenix/pull/155.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #155


          commit 2cb80a71520fd896e5530ac77a2c3c4b17d6f8e2
          Author: Sergey Soldatov <sergey.soldatov@gmail.com>
          Date: 2016-03-28T05:42:17Z

          PHOENIX-2743 Hive Storage support


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ss77892 opened a pull request: https://github.com/apache/phoenix/pull/155 PHOENIX-2743 Hive Storage support You can merge this pull request into a Git repository by running: $ git pull https://github.com/ss77892/phoenix PHOENIX-2743 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/phoenix/pull/155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #155 commit 2cb80a71520fd896e5530ac77a2c3c4b17d6f8e2 Author: Sergey Soldatov <sergey.soldatov@gmail.com> Date: 2016-03-28T05:42:17Z PHOENIX-2743 Hive Storage support
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          I integrated it into phoenix workspace in the same way as it was done in PHOENIX-331. Didn't include modification for MutationState.java since I'm not sure whether it's still required in 4.7.

          Show
          sergey.soldatov Sergey Soldatov added a comment - I integrated it into phoenix workspace in the same way as it was done in PHOENIX-331 . Didn't include modification for MutationState.java since I'm not sure whether it's still required in 4.7.
          Hide
          jamestaylor James Taylor added a comment -

          YoungWoo Kim - yes, it's a dup, but both have implementations against them. Hopefully they'll talk and we'll get the best of both worlds.

          Show
          jamestaylor James Taylor added a comment - YoungWoo Kim - yes, it's a dup, but both have implementations against them. Hopefully they'll talk and we'll get the best of both worlds.
          Hide
          warwithin YoungWoo Kim added a comment -

          Dup of PHOENIX-331?

          Show
          warwithin YoungWoo Kim added a comment - Dup of PHOENIX-331 ?
          Hide
          jamestaylor James Taylor added a comment -

          JeongMin Ju - with our 4.7 release, the validation check done by MutationState is only done once per call to commit. You can also prevent the RPC we do to check if the client has the current metadata through the UPDATE_CACHE_FREQUENCY property (see https://phoenix.apache.org/language/index.html#options). Please let me know if there's still an issue with 4.7.0 release - otherwise let's come up with some other solution to prevents having to change MutationState.

          Show
          jamestaylor James Taylor added a comment - JeongMin Ju - with our 4.7 release, the validation check done by MutationState is only done once per call to commit. You can also prevent the RPC we do to check if the client has the current metadata through the UPDATE_CACHE_FREQUENCY property (see https://phoenix.apache.org/language/index.html#options ). Please let me know if there's still an issue with 4.7.0 release - otherwise let's come up with some other solution to prevents having to change MutationState.
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          It would be really nice since there are a lot of people who are still using older versions of Hive and who may be interested in this functionality.

          Show
          sergey.soldatov Sergey Soldatov added a comment - It would be really nice since there are a lot of people who are still using older versions of Hive and who may be interested in this functionality.
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          It would be really nice since there are a lot of people who are still using older versions of Hive and who may be interested in this functionality.

          Show
          sergey.soldatov Sergey Soldatov added a comment - It would be really nice since there are a lot of people who are still using older versions of Hive and who may be interested in this functionality.
          Hide
          mini666 JeongMin Ju added a comment -

          Oh, you right. my mistake.
          LazyObjectInspectorParameters, LazySerDeParameters classes were created from Hive 1.2.
          For now, it is supported above Hive 1.2.
          To enable from a previous version additional work is required, if necessary.

          Show
          mini666 JeongMin Ju added a comment - Oh, you right. my mistake. LazyObjectInspectorParameters, LazySerDeParameters classes were created from Hive 1.2. For now, it is supported above Hive 1.2. To enable from a previous version additional work is required, if necessary.
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Hi JeongMin,

          Yep, I took a look at the sources, tried to run it with old version of Hive (1.0.0) and got number of ClassNotFound exceptions.
          1. LazyObjectInspectorParameters as well as the rest classes I mentioned were introduced in Hive 1.2.0. How it can work with previous versions? That's exactly I was asking about. Just try to compile only phoenix related classes with older hive.
          2. Thanks.
          3. Oops. I just wanted to say that it would be nice to have comments that can be understood by others.

          Show
          sergey.soldatov Sergey Soldatov added a comment - Hi JeongMin, Yep, I took a look at the sources, tried to run it with old version of Hive (1.0.0) and got number of ClassNotFound exceptions. 1. LazyObjectInspectorParameters as well as the rest classes I mentioned were introduced in Hive 1.2.0. How it can work with previous versions? That's exactly I was asking about. Just try to compile only phoenix related classes with older hive. 2. Thanks. 3. Oops. I just wanted to say that it would be nice to have comments that can be understood by others.
          Hide
          mini666 JeongMin Ju added a comment - - edited

          Hello Sergey.
          Do you examine and execute what i posted on github?
          The support functions mentioned there are all true. and it is applied to my company system.

          1. Yes, it can. I checked Hive from 0.13 to 2.0 only eye check not execution. I found that it is possible to apply until 0.14 in 2.0.
          Hive 0.13 will require more changes. Even from Hive 0.14 to 2.0 should reflect only the changed code, rather than overwrite files.
          I will soon real test for the Hive 2.0. and I'll comment here the test iis finished.
          Can do to avoid being in a class Explain level I can not understand the question. What does it mean?
          LazyObjectInspectorParameters, LazySerDeParameters class is used.

          2. Thank you for your suggestion. I hope more people will be using Phoenix with this function.
          I will open the Hive Jira follow your advice.

          3. It's a joke? There is no doubt that you will not know the Chinese language. It is a Korean language.
          Does it mean you could not test because of comment written in korean.
          OK, I am willing to change it to english.

          Show
          mini666 JeongMin Ju added a comment - - edited Hello Sergey. Do you examine and execute what i posted on github? The support functions mentioned there are all true. and it is applied to my company system. 1. Yes, it can. I checked Hive from 0.13 to 2.0 only eye check not execution. I found that it is possible to apply until 0.14 in 2.0. Hive 0.13 will require more changes. Even from Hive 0.14 to 2.0 should reflect only the changed code, rather than overwrite files. I will soon real test for the Hive 2.0. and I'll comment here the test iis finished. Can do to avoid being in a class Explain level I can not understand the question. What does it mean? LazyObjectInspectorParameters, LazySerDeParameters class is used. 2. Thank you for your suggestion. I hope more people will be using Phoenix with this function. I will open the Hive Jira follow your advice. 3. It's a joke? There is no doubt that you will not know the Chinese language. It is a Korean language. Does it mean you could not test because of comment written in korean. OK, I am willing to change it to english.
          Hide
          sergey.soldatov Sergey Soldatov added a comment -

          Well, I have some concerns about it.
          1. Can it support Hive versions prior 1.2 ? So, is it possible to avoid using of LazyObjectInspectorParameters, LazySerDeParameters, Explain level, ReflectionUtil and similar recently added classes?
          2. It will be nice to have it as a module in Phoenix. That means no hiding hive classes at all. If hive misses some feature, that supposed to be contributed as a separate JIRA into Hive.
          3. Logging, comments in Chinese, no tests

          Show
          sergey.soldatov Sergey Soldatov added a comment - Well, I have some concerns about it. 1. Can it support Hive versions prior 1.2 ? So, is it possible to avoid using of LazyObjectInspectorParameters, LazySerDeParameters, Explain level, ReflectionUtil and similar recently added classes? 2. It will be nice to have it as a module in Phoenix. That means no hiding hive classes at all. If hive misses some feature, that supposed to be contributed as a separate JIRA into Hive. 3. Logging, comments in Chinese, no tests
          Hide
          mini666 JeongMin Ju added a comment -

          I completed publish source code and usage to github.

          https://github.com/mini666/hive-phoenix-handler

          Show
          mini666 JeongMin Ju added a comment - I completed publish source code and usage to github. https://github.com/mini666/hive-phoenix-handler
          Hide
          mini666 JeongMin Ju added a comment -

          Thange for reply.
          The change of MutationState is to disable validation check for batch upsert.
          The current code of MutationState.validate is very inefficient because of checking every row.
          If turn off the validation when batch upsert. Performance is improved.

          Show
          mini666 JeongMin Ju added a comment - Thange for reply. The change of MutationState is to disable validation check for batch upsert. The current code of MutationState.validate is very inefficient because of checking every row. If turn off the validation when batch upsert. Performance is improved.
          Hide
          devaraj Devaraj Das added a comment -

          Definitely is related .. FYI nicolas maillard

          Show
          devaraj Devaraj Das added a comment - Definitely is related .. FYI nicolas maillard
          Hide
          jamestaylor James Taylor added a comment - - edited

          JeongMin Ju - thank you so much for letting us know about this very interesting work. You should collaborate with Sergey Soldatov on PHOENIX-331. How does this differ? Your's is more about being able to use HiveQL?

          What changes did you need to make to MutationState? Would be interesting to see how our 4.7.0 release will impact your work, as we now support transactions.

          FYI, Devaraj Das, Enis Soztutar, ravi, Gabriel Reid, Nick Dimiduk.

          Show
          jamestaylor James Taylor added a comment - - edited JeongMin Ju - thank you so much for letting us know about this very interesting work. You should collaborate with Sergey Soldatov on PHOENIX-331 . How does this differ? Your's is more about being able to use HiveQL? What changes did you need to make to MutationState? Would be interesting to see how our 4.7.0 release will impact your work, as we now support transactions. FYI, Devaraj Das , Enis Soztutar , ravi , Gabriel Reid , Nick Dimiduk .

            People

            • Assignee:
              sergey.soldatov Sergey Soldatov
              Reporter:
              mini666 JeongMin Ju
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development