Hive
  1. Hive
  2. HIVE-5872

Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: Types, UDF
    • Labels:
      None

      Description

      Currently UDAFs are still reporting system default precision/scale (38, 18) for decimal results. Not only this is coarse, but also this can cause problems in subsequent operators such as division, where the result is dependent on the precision/scale of the input, which can go out of bound (38,38). Thus, these UDAFs should correctly report the precision/scale of the result.

      1. HIVE-5872.patch
        5 kB
        Xuefu Zhang
      2. HIVE-5872.4.patch
        17 kB
        Xuefu Zhang
      3. HIVE-5872.3.patch
        18 kB
        Xuefu Zhang
      4. HIVE-5872.2.patch
        524 kB
        Xuefu Zhang
      5. HIVE-5872.1.patch
        45 kB
        Xuefu Zhang

        Issue Links

          Activity

          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12615466/HIVE-5872.patch

          ERROR: -1 due to 3 failed/errored test(s), 4684 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/418/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/418/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12615466

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12615466/HIVE-5872.patch ERROR: -1 due to 3 failed/errored test(s), 4684 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/418/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/418/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12615466
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12615494/HIVE-5872.1.patch

          ERROR: -1 due to 2 failed/errored test(s), 4685 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testDecimalDivideDecimal2
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/426/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/426/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12615494

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12615494/HIVE-5872.1.patch ERROR: -1 due to 2 failed/errored test(s), 4685 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testDecimalDivideDecimal2 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/426/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/426/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12615494
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12615494/HIVE-5872.1.patch

          ERROR: -1 due to 2 failed/errored test(s), 4685 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
          org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testDecimalDivideDecimal2
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/427/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/427/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12615494

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12615494/HIVE-5872.1.patch ERROR: -1 due to 2 failed/errored test(s), 4685 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFOPDivide.testDecimalDivideDecimal2 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/427/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/427/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12615494
          Hide
          Xuefu Zhang added a comment -

          Patch #2 is updated, which also includes patch for HIVE-5866.

          Show
          Xuefu Zhang added a comment - Patch #2 is updated, which also includes patch for HIVE-5866 .
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12616853/HIVE-5872.2.patch

          ERROR: -1 due to 1 failed/errored test(s), 4451 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/500/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/500/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12616853

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12616853/HIVE-5872.2.patch ERROR: -1 due to 1 failed/errored test(s), 4451 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/500/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/500/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12616853
          Hide
          Xuefu Zhang added a comment -

          Patch #3 removed the patch for HIVE-5866 since the patch was in trunk already.

          Show
          Xuefu Zhang added a comment - Patch #3 removed the patch for HIVE-5866 since the patch was in trunk already.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12616931/HIVE-5872.3.patch

          SUCCESS: +1 4455 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/510/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/510/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12616931

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12616931/HIVE-5872.3.patch SUCCESS: +1 4455 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/510/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/510/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12616931
          Hide
          Xuefu Zhang added a comment -
          Show
          Xuefu Zhang added a comment - RB: https://reviews.apache.org/r/16008/
          Hide
          Xuefu Zhang added a comment -

          Patch #4 rebased with trunk.

          Show
          Xuefu Zhang added a comment - Patch #4 rebased with trunk.
          Hide
          Sergey Shelukhin added a comment -

          +1

          Show
          Sergey Shelukhin added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12617263/HIVE-5872.4.patch

          ERROR: -1 due to 1 failed/errored test(s), 4458 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_smb_mapjoin_8
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/535/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/535/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12617263

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12617263/HIVE-5872.4.patch ERROR: -1 due to 1 failed/errored test(s), 4458 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_smb_mapjoin_8 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/535/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/535/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12617263
          Hide
          Xuefu Zhang added a comment -

          The above test failure seems transient/unrelated. Reload the same patch to kick off another test run.

          Show
          Xuefu Zhang added a comment - The above test failure seems transient/unrelated. Reload the same patch to kick off another test run.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12617321/HIVE-5872.4.patch

          SUCCESS: +1 4458 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/537/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/537/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12617321

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12617321/HIVE-5872.4.patch SUCCESS: +1 4458 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/537/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/537/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12617321
          Hide
          Xuefu Zhang added a comment -

          Patch committed to trunk. Thanks go to Sergey Shelukhin for the review.

          Show
          Xuefu Zhang added a comment - Patch committed to trunk. Thanks go to Sergey Shelukhin for the review.
          Hide
          Thejas M Nair added a comment -

          Xuefu Zhang Sergey is not a committer. The policy is to get a +1 from a committer who is not the author of the patch before committing it. See https://cwiki.apache.org/confluence/display/Hive/Bylaws#Bylaws-

          Show
          Thejas M Nair added a comment - Xuefu Zhang Sergey is not a committer. The policy is to get a +1 from a committer who is not the author of the patch before committing it. See https://cwiki.apache.org/confluence/display/Hive/Bylaws#Bylaws-
          Hide
          Thejas M Nair added a comment -

          I guess Sergey might be doing a good job at "If you want to become a committer, behave like a committer." (from https://cwiki.apache.org/confluence/display/Hive/BecomingACommitter)

          Show
          Thejas M Nair added a comment - I guess Sergey might be doing a good job at "If you want to become a committer, behave like a committer." (from https://cwiki.apache.org/confluence/display/Hive/BecomingACommitter )
          Hide
          Xuefu Zhang added a comment -

          Thanks for pointing this out, Thejas M Nair. That's exactly my impression. I'll ask a committer to review the patch. If changes are due, I will create followup jira to accommodate.

          Show
          Xuefu Zhang added a comment - Thanks for pointing this out, Thejas M Nair . That's exactly my impression. I'll ask a committer to review the patch. If changes are due, I will create followup jira to accommodate.
          Hide
          Prasad Mujumdar added a comment -

          Xuefu Zhang I will take a look shortly.

          Show
          Prasad Mujumdar added a comment - Xuefu Zhang I will take a look shortly.
          Hide
          Thejas M Nair added a comment -

          Xuefu Zhang Sounds good.

          Show
          Thejas M Nair added a comment - Xuefu Zhang Sounds good.
          Hide
          Prasad Mujumdar added a comment -

          Looks fine to me.
          +1

          In general, it looks like we need more exception logic for handling decimals in UDAF (HIVE-5872, HIVE-5866). It might be useful to add a note in the dev guide for future work ..

          Show
          Prasad Mujumdar added a comment - Looks fine to me. +1 In general, it looks like we need more exception logic for handling decimals in UDAF ( HIVE-5872 , HIVE-5866 ). It might be useful to add a note in the dev guide for future work ..
          Hide
          Xuefu Zhang added a comment -

          Thanks for the review, Prasad.

          In general, it looks like we need more exception logic for handling decimals in UDAF (HIVE-5872, HIVE-5866). It might be useful to add a note in the dev guide for future work ..

          I assume you are referring the following code snippet:

                  if (t == null) {
                    return warnedOnceNullMapKey;
                  }
          

          I agree with your assessment. Currently Hive emits null as the only error handling option. Thus, null check is (or is missed) everywhere in the code, not specific to decimal. For a long run, I agree we need to have a better exception handling, especially when we introduce different error handling (HIVE-5438).

          Show
          Xuefu Zhang added a comment - Thanks for the review, Prasad. In general, it looks like we need more exception logic for handling decimals in UDAF ( HIVE-5872 , HIVE-5866 ). It might be useful to add a note in the dev guide for future work .. I assume you are referring the following code snippet: if (t == null ) { return warnedOnceNullMapKey; } I agree with your assessment. Currently Hive emits null as the only error handling option. Thus, null check is (or is missed) everywhere in the code, not specific to decimal. For a long run, I agree we need to have a better exception handling, especially when we introduce different error handling ( HIVE-5438 ).

            People

            • Assignee:
              Xuefu Zhang
              Reporter:
              Xuefu Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development