Hive
  1. Hive
  2. HIVE-4779

Enhance coverage of package org.apache.hadoop.hive.ql.udf

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      Enhance coverage of package org.apache.hadoop.hive.ql.udf up to 80%.

      1. HIVE-4779.patch
        65 kB
        Ashutosh Chauhan
      2. HIVE-4779-trunk--N1.patch
        64 kB
        Ivan A. Veselovsky

        Activity

        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Ashutosh Chauhan added a comment -

        This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

        Show
        Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #135 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/135/)
        HIVE-4779 : Enhance coverage of package org.apache.hadoop.hive.ql.udf (Ivan Veselovsky via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515946)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBaseCompare.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDate.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java
        • /hive/trunk/ql/src/test/queries/clientpositive/create_udaf.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf4.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_pmod.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_boolean.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_byte.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_double.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_float.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_long.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_short.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_string.q
        • /hive/trunk/ql/src/test/results/clientpositive/create_udaf.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf4.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_pmod.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_boolean.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_byte.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_double.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_float.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_long.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_short.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_string.q.out
        Show
        Hudson added a comment - SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #135 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/135/ ) HIVE-4779 : Enhance coverage of package org.apache.hadoop.hive.ql.udf (Ivan Veselovsky via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515946 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBaseCompare.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDate.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java /hive/trunk/ql/src/test/queries/clientpositive/create_udaf.q /hive/trunk/ql/src/test/queries/clientpositive/udf4.q /hive/trunk/ql/src/test/queries/clientpositive/udf_pmod.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_boolean.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_byte.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_double.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_float.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_long.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_short.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_string.q /hive/trunk/ql/src/test/results/clientpositive/create_udaf.q.out /hive/trunk/ql/src/test/results/clientpositive/udf4.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_pmod.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_boolean.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_byte.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_double.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_float.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_long.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_short.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_string.q.out
        Hide
        Hudson added a comment -

        ABORTED: Integrated in Hive-trunk-hadoop2 #373 (See https://builds.apache.org/job/Hive-trunk-hadoop2/373/)
        HIVE-4779 : Enhance coverage of package org.apache.hadoop.hive.ql.udf (Ivan Veselovsky via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515946)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBaseCompare.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDate.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java
        • /hive/trunk/ql/src/test/queries/clientpositive/create_udaf.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf4.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_pmod.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_boolean.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_byte.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_double.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_float.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_long.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_short.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_string.q
        • /hive/trunk/ql/src/test/results/clientpositive/create_udaf.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf4.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_pmod.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_boolean.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_byte.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_double.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_float.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_long.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_short.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_string.q.out
        Show
        Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #373 (See https://builds.apache.org/job/Hive-trunk-hadoop2/373/ ) HIVE-4779 : Enhance coverage of package org.apache.hadoop.hive.ql.udf (Ivan Veselovsky via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515946 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBaseCompare.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDate.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java /hive/trunk/ql/src/test/queries/clientpositive/create_udaf.q /hive/trunk/ql/src/test/queries/clientpositive/udf4.q /hive/trunk/ql/src/test/queries/clientpositive/udf_pmod.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_boolean.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_byte.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_double.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_float.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_long.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_short.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_string.q /hive/trunk/ql/src/test/results/clientpositive/create_udaf.q.out /hive/trunk/ql/src/test/results/clientpositive/udf4.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_pmod.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_boolean.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_byte.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_double.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_float.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_long.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_short.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_string.q.out
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2-ptest #66 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/66/)
        HIVE-4779 : Enhance coverage of package org.apache.hadoop.hive.ql.udf (Ivan Veselovsky via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515946)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBaseCompare.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDate.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java
        • /hive/trunk/ql/src/test/queries/clientpositive/create_udaf.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf4.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_pmod.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_boolean.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_byte.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_double.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_float.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_long.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_short.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_to_string.q
        • /hive/trunk/ql/src/test/results/clientpositive/create_udaf.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf4.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_pmod.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_boolean.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_byte.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_double.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_float.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_long.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_short.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_to_string.q.out
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #66 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/66/ ) HIVE-4779 : Enhance coverage of package org.apache.hadoop.hive.ql.udf (Ivan Veselovsky via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515946 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFBaseCompare.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDate.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java /hive/trunk/ql/src/test/queries/clientpositive/create_udaf.q /hive/trunk/ql/src/test/queries/clientpositive/udf4.q /hive/trunk/ql/src/test/queries/clientpositive/udf_pmod.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_boolean.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_byte.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_double.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_float.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_long.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_short.q /hive/trunk/ql/src/test/queries/clientpositive/udf_to_string.q /hive/trunk/ql/src/test/results/clientpositive/create_udaf.q.out /hive/trunk/ql/src/test/results/clientpositive/udf4.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_pmod.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_boolean.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_byte.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_double.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_float.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_long.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_short.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_to_string.q.out
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.12.0 [ 12324312 ]
        Resolution Fixed [ 1 ]
        Hide
        Ashutosh Chauhan added a comment -

        Ran bucket5.q, didnt fail for me. Patch checked in trunk. Thanks, Ivan for the cleanup and tests.!

        Show
        Ashutosh Chauhan added a comment - Ran bucket5.q, didnt fail for me. Patch checked in trunk. Thanks, Ivan for the cleanup and tests.!
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12598909/HIVE-4779.patch

        ERROR: -1 due to 1 failed/errored test(s), 2893 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/488/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/488/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests failed with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12598909/HIVE-4779.patch ERROR: -1 due to 1 failed/errored test(s), 2893 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/488/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/488/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
        Hide
        Ivan A. Veselovsky added a comment -

        I see you rebased the patch, thanks a lot.

        Show
        Ivan A. Veselovsky added a comment - I see you rebased the patch, thanks a lot.
        Ashutosh Chauhan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Ashutosh Chauhan made changes -
        Attachment HIVE-4779.patch [ 12598909 ]
        Hide
        Ashutosh Chauhan added a comment -

        Rebased the patch on latest trunk.

        Show
        Ashutosh Chauhan added a comment - Rebased the patch on latest trunk.
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Ashutosh Chauhan added a comment -

        Ivan A. Veselovsky Sorry for dropping ball on this one. Your explaination make sense. But unfortunately patch is not applying anymore. Can you rebase it and upload it in proper format so that HIVE QA can run on it?

        Show
        Ashutosh Chauhan added a comment - Ivan A. Veselovsky Sorry for dropping ball on this one. Your explaination make sense. But unfortunately patch is not applying anymore. Can you rebase it and upload it in proper format so that HIVE QA can run on it?
        Hide
        Edward Capriolo added a comment -

        +1 less is more.

        Show
        Edward Capriolo added a comment - +1 less is more.
        Ivan A. Veselovsky made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Ivan A. Veselovsky added a comment -

        Hi, Ashotush, Edward, thanks for the comments, answering regarding the deleted UDFs.

        1) UDFToDate was a function converting numbers (surprisingly not Writable-s) to java.sql.Date. It was not registered in FunctionRegistry, and not used anyhow else (neither in Java code nor .q scripts). In function registry we have registerUDF("to_date", UDFDate.class, false); (line 271). But its purpose slightly different: it extracts date part from Text or Timestamp writables. Conversion of an arbitrary type to Timestamp is done using org.apache.hadoop.hive.ql.udf.generic.GenericUDFTimestamp.

        2) UDFUnixTimeStamp was also not used anyhow, and, as far as I understand, is superseded by the following functions:
        registerGenericUDF("unix_timestamp", GenericUDFUnixTimeStamp.class);
        registerGenericUDF("to_unix_timestamp", GenericUDFToUnixTimeStamp.class);
        These functions are covered well by .q tests.

        3) UDFBaseCompare was simply not used. I suppose that all the generic comparison logic was transferred to org.apache.hadoop.hive.ql.udf.generic.GenericUDFBaseCompare and related classes.

        If some of these removed classes are still needed, of course, I will change the patch to return them back.

        Show
        Ivan A. Veselovsky added a comment - Hi, Ashotush, Edward, thanks for the comments, answering regarding the deleted UDFs. 1) UDFToDate was a function converting numbers (surprisingly not Writable-s) to java.sql.Date. It was not registered in FunctionRegistry, and not used anyhow else (neither in Java code nor .q scripts). In function registry we have registerUDF("to_date", UDFDate.class, false); (line 271). But its purpose slightly different: it extracts date part from Text or Timestamp writables. Conversion of an arbitrary type to Timestamp is done using org.apache.hadoop.hive.ql.udf.generic.GenericUDFTimestamp. 2) UDFUnixTimeStamp was also not used anyhow, and, as far as I understand, is superseded by the following functions: registerGenericUDF("unix_timestamp", GenericUDFUnixTimeStamp.class); registerGenericUDF("to_unix_timestamp", GenericUDFToUnixTimeStamp.class); These functions are covered well by .q tests. 3) UDFBaseCompare was simply not used. I suppose that all the generic comparison logic was transferred to org.apache.hadoop.hive.ql.udf.generic.GenericUDFBaseCompare and related classes. If some of these removed classes are still needed, of course, I will change the patch to return them back.
        Hide
        Edward Capriolo added a comment -

        What I was saying is lets ensure that some UDF is still responsible for unix_timestamp. The function registry should tell.

        Show
        Edward Capriolo added a comment - What I was saying is lets ensure that some UDF is still responsible for unix_timestamp. The function registry should tell.
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Ashutosh Chauhan added a comment -

        Ivan A. Veselovsky For completeness, can you point out whats supersedes UDFToDate and UDFUnixTimeStamp.
        In case, some one stumbles upon this jira, they will know which udf they should look for in lieu of these.

        Show
        Ashutosh Chauhan added a comment - Ivan A. Veselovsky For completeness, can you point out whats supersedes UDFToDate and UDFUnixTimeStamp. In case, some one stumbles upon this jira, they will know which udf they should look for in lieu of these.
        Hide
        Edward Capriolo added a comment -

        As long as something else implements I am cool. "unix_timestamp"

        Show
        Edward Capriolo added a comment - As long as something else implements I am cool. "unix_timestamp"
        Ivan A. Veselovsky made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Ivan A. Veselovsky added a comment -

        Hi, Ashutosh,
        yes, the UDFs UDFBaseCompare, UDFToDate, UDFUnixTimeStamp were intentionally removed because
        they either superseded by other UDFs or completely unused (e.g. UDFBaseCompare).

        Show
        Ivan A. Veselovsky added a comment - Hi, Ashutosh, yes, the UDFs UDFBaseCompare, UDFToDate, UDFUnixTimeStamp were intentionally removed because they either superseded by other UDFs or completely unused (e.g. UDFBaseCompare).
        Ashutosh Chauhan made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Ashutosh Chauhan added a comment -

        You removed couple of udfs in the patch. Was that accidental or intentional?

        Show
        Ashutosh Chauhan added a comment - You removed couple of udfs in the patch. Was that accidental or intentional?
        Ivan A. Veselovsky made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.12.0 [ 12324312 ]
        Ivan A. Veselovsky made changes -
        Field Original Value New Value
        Attachment HIVE-4779-trunk--N1.patch [ 12589866 ]
        Hide
        Ivan A. Veselovsky added a comment -

        the patch elevates coverage of package o.a.h.hive.ql.udf from 66% to 81.9%.

        Show
        Ivan A. Veselovsky added a comment - the patch elevates coverage of package o.a.h.hive.ql.udf from 66% to 81.9%.
        Ivan A. Veselovsky created issue -

          People

          • Assignee:
            Ivan A. Veselovsky
            Reporter:
            Ivan A. Veselovsky
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development