Hive
  1. Hive
  2. HIVE-7762

Enhancement while getting partitions via webhcat client

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: WebHCat
    • Labels:
      None

      Description

      Hcatalog creates partitions in lower case, whereas getting partitions from hcatalog via webhcat client doesn't handle this. So the client starts throwing exceptions.
      Ex:
      CREATE EXTERNAL TABLE in_table (word STRING, cnt INT) PARTITIONED BY (Year STRING, Month STRING, Date STRING, Hour STRING, Minute STRING) STORED AS TEXTFILE LOCATION '/user/suhas/hcat-data/in/';

      Then i try to get partitions by:

              String inputTableName = "in_table";
              String database = "default";
      
              Map<String, String> partitionSpec = new HashMap<String, String>();
              partitionSpec.put("Year", "2014");
              partitionSpec.put("Month", "08");
              partitionSpec.put("Date", "11");
              partitionSpec.put("Hour", "00");
              partitionSpec.put("Minute", "00");
      
              HCatClient client = get(catalogUrl);
              HCatPartition hCatPartition = client.getPartition(database, inputTableName, partitionSpec);
      

      This throws up saying:

      Exception in thread "main" org.apache.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : Invalid partition-key specified: year
      	at org.apache.hcatalog.api.HCatClientHMSImpl.getPartition(HCatClientHMSImpl.java:366)
      	at com.inmobi.demo.HcatPartitions.main(HcatPartitions.java:34)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      

      The same code works if i do

              partitionSpec.put("year", "2014");
              partitionSpec.put("month", "08");
              partitionSpec.put("date", "11");
              partitionSpec.put("hour", "00");
              partitionSpec.put("minute", "00");
      
      1. HIVE-7762.patch
        2 kB
        Suhas Vasu
      2. HIVE-7762.2.patch
        2 kB
        Suhas Vasu
      3. HIVE-7762.3.patch
        11 kB
        Suhas Vasu
      4. HIVE-7762.4.patch
        11 kB
        Suhas Vasu

        Activity

        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12670381/HIVE-7762.3.patch

        ERROR: -1 due to 1 failed/errored test(s), 6301 tests executed
        Failed tests:

        org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/920/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/920/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-920/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12670381

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12670381/HIVE-7762.3.patch ERROR: -1 due to 1 failed/errored test(s), 6301 tests executed Failed tests: org.apache.hadoop.hive.ql.parse.TestParse.testParse_union Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/920/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/920/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-920/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12670381
        Hide
        Suhas Vasu added a comment -

        Mithun Radhakrishnan - HCatClient.getPartitions() behaves as expected in this case. Its only HCatClient.getPartition() that breaks for the above mentioned scenario.

        Anyway I have added test cases for HCatClient.getPartitions() and HCatClient.getPartition().

        Thanks for looking into this.

        Show
        Suhas Vasu added a comment - Mithun Radhakrishnan - HCatClient.getPartitions() behaves as expected in this case. Its only HCatClient.getPartition() that breaks for the above mentioned scenario. Anyway I have added test cases for HCatClient.getPartitions() and HCatClient.getPartition() . Thanks for looking into this.
        Hide
        Suhas Vasu added a comment -

        Sure, i'' add the test-cases. makes sense to have them.

        Guess

        HCatClient.getPartitions()

        behaves as expected but i'll check it once.

        I'll update the jira with the findings

        Show
        Suhas Vasu added a comment - Sure, i'' add the test-cases. makes sense to have them. Guess HCatClient.getPartitions() behaves as expected but i'll check it once. I'll update the jira with the findings
        Hide
        Mithun Radhakrishnan added a comment -

        Hello, Suhas. Thanks for working on fixing this inconsistency. Generally, this is a good fix.

        I'd encourage you to add a test-case to TestHCatClient, to create a table with uppercase partition columns, and then querying it with a lowercase partition-spec. (Essentially, what you've included in your description.) Adding one won't be hard; you could just use one of the other tests for reference.
        Also, could I please bother you to verify the behaviour of HCatClient.getPartitions(), for case insensitivity? If it's broken too, I'd rather we fixed both here. I expect that this should be alright, since it goes through the listPartitionsByFilter() API, but it would be good to have confirmation.

        Mithun

        Show
        Mithun Radhakrishnan added a comment - Hello, Suhas. Thanks for working on fixing this inconsistency. Generally, this is a good fix. I'd encourage you to add a test-case to TestHCatClient, to create a table with uppercase partition columns, and then querying it with a lowercase partition-spec. (Essentially, what you've included in your description.) Adding one won't be hard; you could just use one of the other tests for reference. Also, could I please bother you to verify the behaviour of HCatClient.getPartitions() , for case insensitivity? If it's broken too, I'd rather we fixed both here. I expect that this should be alright, since it goes through the listPartitionsByFilter() API, but it would be good to have confirmation. Mithun
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12662754/HIVE-7762.2.patch

        SUCCESS: +1 5819 tests passed

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/402/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/402/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-402/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12662754

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662754/HIVE-7762.2.patch SUCCESS: +1 5819 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/402/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/402/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-402/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12662754
        Hide
        Suhas Vasu added a comment -

        Rebasing the patch

        Show
        Suhas Vasu added a comment - Rebasing the patch
        Hide
        Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12662470/HIVE-7762.patch

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/382/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/382/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-382/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
        + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-382/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ svn = \s\v\n ]]
        + [[ -n '' ]]
        + [[ -d apache-svn-trunk-source ]]
        + [[ ! -d apache-svn-trunk-source/.svn ]]
        + [[ ! -d apache-svn-trunk-source ]]
        + cd apache-svn-trunk-source
        + svn revert -R .
        Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java'
        Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
        Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
        Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java'
        Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java'
        Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java'
        Reverted 'service/src/java/org/apache/hive/service/cli/CLIService.java'
        Reverted 'service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java'
        Reverted 'service/src/java/org/apache/hive/service/cli/session/SessionManager.java'
        Reverted 'service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java'
        Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
        ++ egrep -v '^X|^Performing status on external'
        ++ awk '{print $2}'
        ++ svn status --no-ignore
        + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hwi/target common/target common/src/gen service/target service/src/java/org/apache/hive/service/server/ThreadWithGarbageCleanup.java service/src/java/org/apache/hive/service/server/ThreadFactoryWithGarbageCleanup.java contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
        + svn update
        U    ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java
        U    testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestParser.java
        
        Fetching external item into 'hcatalog/src/test/e2e/harness'
        Updated external to revision 1618663.
        
        Updated to revision 1618663.
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12662470

        Show
        Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662470/HIVE-7762.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/382/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/382/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-382/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-382/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java' Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java' Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java' Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java' Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java' Reverted 'service/src/java/org/apache/hive/service/cli/CLIService.java' Reverted 'service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java' Reverted 'service/src/java/org/apache/hive/service/cli/session/SessionManager.java' Reverted 'service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hwi/target common/target common/src/gen service/target service/src/java/org/apache/hive/service/server/ThreadWithGarbageCleanup.java service/src/java/org/apache/hive/service/server/ThreadFactoryWithGarbageCleanup.java contrib/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update U ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java U testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestParser.java Fetching external item into 'hcatalog/src/test/e2e/harness' Updated external to revision 1618663. Updated to revision 1618663. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12662470

          People

          • Assignee:
            Unassigned
            Reporter:
            Suhas Vasu
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development