Hive
  1. Hive
  2. HIVE-5564

Need to accomodate table decimal columns that were defined prior to HIVE-3976

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: Types
    • Labels:
      None

      Description

      With HIVE-3976, decimal columns are stored with precision/scale, such as decimal(17,5), as the type name. However, such columns defined in hive prior to HIVE-3976 have a name as "decimal". Those columns need to continue to work with a precision/scale as (10,0), per the functional doc. With patch in HIVE-3976, we may get the following error message in such case:

      0: jdbc:hive2://localhost:10000> desc dec;
      Error: Error while processing statement: FAILED: RuntimeException Decimal type is specified without length: decimal:int (state=42000,code=40000)
      

      This issue will be addressed in this JIRA as a follow-up task.

      1. HIVE-5564.1.patch
        6 kB
        Xuefu Zhang
      2. HIVE-5564.2.patch
        12 kB
        Xuefu Zhang
      3. HIVE-5564.3.patch
        12 kB
        Xuefu Zhang
      4. HIVE-5564.4.patch
        13 kB
        Xuefu Zhang
      5. HIVE-5564.patch
        6 kB
        Xuefu Zhang

        Issue Links

          Activity

          Hide
          Xuefu Zhang added a comment -

          Patch #1 rebased with latest trunk.

          Show
          Xuefu Zhang added a comment - Patch #1 rebased with latest trunk.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12610951/HIVE-5564.1.patch

          ERROR: -1 due to 23 failed/errored test(s), 4545 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_literal_decimal
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_decimal
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_serde_regex
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf7
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_pmod
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_multipartitioning
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_navfn
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_ntile
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_rank
          org.apache.hadoop.hive.jdbc.TestJdbcDriver.testDataTypes
          org.apache.hadoop.hive.serde2.binarysortable.TestBinarySortableSerDe.testBinarySortableSerDe
          org.apache.hadoop.hive.serde2.lazybinary.TestLazyBinarySerDe.testLazyBinarySerDe
          org.apache.hive.jdbc.TestJdbcDriver2.testDataTypes
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/84/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/84/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 23 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12610951/HIVE-5564.1.patch ERROR: -1 due to 23 failed/errored test(s), 4545 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_literal_decimal org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_decimal org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_serde_regex org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf7 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_pmod org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_multipartitioning org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_navfn org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_ntile org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_rank org.apache.hadoop.hive.jdbc.TestJdbcDriver.testDataTypes org.apache.hadoop.hive.serde2.binarysortable.TestBinarySortableSerDe.testBinarySortableSerDe org.apache.hadoop.hive.serde2.lazybinary.TestLazyBinarySerDe.testLazyBinarySerDe org.apache.hive.jdbc.TestJdbcDriver2.testDataTypes Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/84/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/84/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 23 tests failed This message is automatically generated.
          Hide
          Xuefu Zhang added a comment -

          Cancelled patch due to test failures.

          Show
          Xuefu Zhang added a comment - Cancelled patch due to test failures.
          Hide
          Xuefu Zhang added a comment -

          Previous test run result is no long accessible. Submit the patch for another run.

          Show
          Xuefu Zhang added a comment - Previous test run result is no long accessible. Submit the patch for another run.
          Hide
          Xuefu Zhang added a comment -

          Patch #2 updated to fix the failed test cases.

          Show
          Xuefu Zhang added a comment - Patch #2 updated to fix the failed test cases.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12612457/HIVE-5564.2.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/162/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/162/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-162/source-prep.txt
          + [[ true == \t\r\u\e ]]
          + rm -rf ivy maven
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          Reverted 'shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java'
          Reverted 'shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java'
          Reverted 'jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java'
          Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
          Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
          Reverted 'service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java'
          Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java'
          ++ awk '{print $2}'
          ++ egrep -v '^X|^Performing status on external'
          ++ svn status --no-ignore
          + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java.orig metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestSSL.java itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2 itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1539612.
          
          At revision 1539612.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12612457

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12612457/HIVE-5564.2.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/162/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/162/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-162/source-prep.txt + [[ true == \t\r\u\e ]] + rm -rf ivy maven + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java' Reverted 'shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java' Reverted 'jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java' Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java' Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' Reverted 'service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java' Reverted 'service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java' ++ awk '{print $2}' ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java.orig metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestSSL.java itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2 itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1539612. At revision 1539612. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12612457
          Hide
          Xuefu Zhang added a comment -

          Patch #3 rebased with latest trunk.

          Show
          Xuefu Zhang added a comment - Patch #3 rebased with latest trunk.
          Hide
          Xuefu Zhang added a comment -
          Show
          Xuefu Zhang added a comment - RB: https://reviews.apache.org/r/15308/
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12612624/HIVE-5564.3.patch

          SUCCESS: +1 4587 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/182/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/182/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12612624

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12612624/HIVE-5564.3.patch SUCCESS: +1 4587 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/182/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/182/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12612624
          Hide
          Xuefu Zhang added a comment -

          Patch #4 updated based on rb comments.

          Show
          Xuefu Zhang added a comment - Patch #4 updated based on rb comments.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12612688/HIVE-5564.4.patch

          SUCCESS: +1 4595 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/211/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/211/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12612688

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12612688/HIVE-5564.4.patch SUCCESS: +1 4595 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/211/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/211/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12612688
          Hide
          Xuefu Zhang added a comment -

          Reload the same patch to kick off test.

          Show
          Xuefu Zhang added a comment - Reload the same patch to kick off test.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12613439/HIVE-5564.4.patch

          SUCCESS: +1 4602 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/251/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/251/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12613439

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12613439/HIVE-5564.4.patch SUCCESS: +1 4602 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/251/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/251/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12613439
          Hide
          Jason Dere added a comment -

          +1 (non-binding). Will be good to get this one in so that users can use existing decimal tables (though they will likely need to change the decimal column definitions to set precision/scale).

          Show
          Jason Dere added a comment - +1 (non-binding). Will be good to get this one in so that users can use existing decimal tables (though they will likely need to change the decimal column definitions to set precision/scale).
          Hide
          Brock Noland added a comment -

          +1

          Show
          Brock Noland added a comment - +1
          Hide
          Xuefu Zhang added a comment -

          Patch HIVE-5564.4.patch committed to trunk. Thank Brock for the review.

          Show
          Xuefu Zhang added a comment - Patch HIVE-5564 .4.patch committed to trunk. Thank Brock for the review.

            People

            • Assignee:
              Xuefu Zhang
              Reporter:
              Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development