Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: UDF
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Issue Links

        Activity

        Hide
        Carl Steinbach added a comment -

        This UDF would be generally nice to have, but may also be a requirement if it turns out that the unordered output of functions like map_keys() and map_values() is non-deterministic.

        Show
        Carl Steinbach added a comment - This UDF would be generally nice to have, but may also be a requirement if it turns out that the unordered output of functions like map_keys() and map_values() is non-deterministic.
        Hide
        Phabricator added a comment -

        cwsteinbach requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
        Reviewers: JIRA

        HIVE-2279. Implement sort(array) UDF

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1059

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
        ql/src/test/queries/clientpositive/udf_sort_array.q
        ql/src/test/results/clientpositive/show_functions.q.out
        ql/src/test/results/clientpositive/udf_sort_array.q.out

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/2241/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - cwsteinbach requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 . Implement sort(array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1059 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2241/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        zhenxiao requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
        Reviewers: JIRA

        HIVE-2279: resolve Trailing Whitespace

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1101

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/2319/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : resolve Trailing Whitespace TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1101 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2319/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        zhenxiao requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
        Reviewers: JIRA

        HIVE-2279: Implement sort(array) UDF

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1107

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
        ql/src/test/queries/clientpositive/udf_sort_array.q
        ql/src/test/results/clientpositive/show_functions.q.out
        ql/src/test/results/clientpositive/udf_sort_array.q.out

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/2325/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : Implement sort(array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1107 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2325/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        zhenxiao has commented on the revision "HIVE-2279 [jira] Implement sort(array) UDF".

        Incomplete patch. D1107 is the correct patch for review.

        REVISION DETAIL
        https://reviews.facebook.net/D1101

        Show
        Phabricator added a comment - zhenxiao has commented on the revision " HIVE-2279 [jira] Implement sort(array) UDF". Incomplete patch. D1107 is the correct patch for review. REVISION DETAIL https://reviews.facebook.net/D1101
        Hide
        Phabricator added a comment -

        cwsteinbach has requested changes to the revision "HIVE-2279 [jira] Implement sort(array) UDF".

        INLINE COMMENTS
        ql/src/test/queries/clientpositive/udf_sort_array.q:8 Please add EXPLAIN queries.
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:82 Fix indentation.
        ql/src/test/results/clientpositive/udf_sort_array.q.out:13 "sort_array(sort_array(...))"?
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:45 "Sorts the input array in ascending order according to the natural ordering of the array elements."
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:55 Please add a negative testcase that exercises these code paths.
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:60 What happens if I try to sort an array of arrays, or array of structs?
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:67 The return type should be another ARRAY, not a string.
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:85 This method needs to return another ARRAY, not a string containing the concatenated, sorted input elements. Take a look at the array() UDF for hints on how to return an array from a UDF.
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:97 Indentation.

        REVISION DETAIL
        https://reviews.facebook.net/D1107

        Show
        Phabricator added a comment - cwsteinbach has requested changes to the revision " HIVE-2279 [jira] Implement sort(array) UDF". INLINE COMMENTS ql/src/test/queries/clientpositive/udf_sort_array.q:8 Please add EXPLAIN queries. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:82 Fix indentation. ql/src/test/results/clientpositive/udf_sort_array.q.out:13 "sort_array(sort_array(...))"? ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:45 "Sorts the input array in ascending order according to the natural ordering of the array elements." ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:55 Please add a negative testcase that exercises these code paths. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:60 What happens if I try to sort an array of arrays, or array of structs? ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:67 The return type should be another ARRAY, not a string. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:85 This method needs to return another ARRAY, not a string containing the concatenated, sorted input elements. Take a look at the array() UDF for hints on how to return an array from a UDF. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:97 Indentation. REVISION DETAIL https://reviews.facebook.net/D1107
        Hide
        Phabricator added a comment -

        zhenxiao requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
        Reviewers: JIRA

        HIVE-2279: Implement Sort(Array) UDF

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1125

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
        ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
        ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
        ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
        ql/src/test/queries/clientpositive/udf_sort_array.q
        ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
        ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
        ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
        ql/src/test/results/clientpositive/show_functions.q.out
        ql/src/test/results/clientpositive/udf_sort_array.q.out

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/2355/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : Implement Sort(Array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1125 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2355/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        cwsteinbach has commented on the revision "HIVE-2279 [jira] Implement sort(array) UDF".

        INLINE COMMENTS
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:47 This should be
        value="FUNC(array)"

        Otherwise, the output of "describe function sort_array" looks like this:

        sort_array(sort_array(obj1, obj2,...)) - Sorts the input...
        ^^^^^^^^
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:51 Please change the example to something simpler that doesn't involve corporate names, e.g. "array(4, 1, 3, 2)"
        ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:449 I think the name should be changed to "sort" and "GenericUDFSort". In the future we may want to extend it so that it can also support sorting the elements in an input string. In a similar manner we may want to implement a "reverse" UDF that reverse the elements of an array as well as the characters in a string.
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:74 "Argument 1" instead of "Argument " + 1 + ...
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:88 There's a single argument, right?

        REVISION DETAIL
        https://reviews.facebook.net/D1125

        Show
        Phabricator added a comment - cwsteinbach has commented on the revision " HIVE-2279 [jira] Implement sort(array) UDF". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:47 This should be value=" FUNC (array)" Otherwise, the output of "describe function sort_array" looks like this: sort_array(sort_array(obj1, obj2,...)) - Sorts the input... ^^^^^^^^ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:51 Please change the example to something simpler that doesn't involve corporate names, e.g. "array(4, 1, 3, 2)" ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:449 I think the name should be changed to "sort" and "GenericUDFSort". In the future we may want to extend it so that it can also support sorting the elements in an input string. In a similar manner we may want to implement a "reverse" UDF that reverse the elements of an array as well as the characters in a string. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:74 "Argument 1" instead of "Argument " + 1 + ... ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:88 There's a single argument, right? REVISION DETAIL https://reviews.facebook.net/D1125
        Hide
        Phabricator added a comment -

        cwsteinbach has requested changes to the revision "HIVE-2279 [jira] Implement sort(array) UDF".

        REVISION DETAIL
        https://reviews.facebook.net/D1125

        Show
        Phabricator added a comment - cwsteinbach has requested changes to the revision " HIVE-2279 [jira] Implement sort(array) UDF". REVISION DETAIL https://reviews.facebook.net/D1125
        Hide
        Phabricator added a comment -

        zhenxiao has commented on the revision "HIVE-2279 [jira] Implement sort(array) UDF".

        sort() is a better name for sort_array(), while, seems currently the parser/semantic analyzer has some problem taking a reserved keyword as UDF function name.

        I tried the following changes in HIve.g:

        [~/Code/hive]git diff ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g b/ql/src/java/org/apache/hadoop/h
        index 888bf47..ec256de 100644
        — a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        +++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        @@ -1816,7 +1816,7 @@ functionName
        @init

        { msgs.push("function name"); }

        @after

        { msgs.pop(); }

        : // Keyword IF is also a function name

        • Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE
          + Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE | KW_SORT
          ;

        castExpression
        @@ -2091,6 +2091,7 @@ sysFuncNames

        KW_MAP
        KW_STRUCT
        KW_UNIONTYPE
        +
        KW_SORT
        EQUAL
        NOTEQUAL
        LESSTHANOREQUALTO

        While, the testcase always fails during semantic analysis on argument length:

        – Evaluate function against STRING valued keys
        EXPLAIN
        SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1
        2012-01-09 11:31:55,134 INFO parse.ParseDriver (ParseDriver.java:parse(426)) - Parsing command:

        – Evaluate function against STRING valued keys
        EXPLAIN
        SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1
        2012-01-09 11:31:55,146 INFO parse.ParseDriver (ParseDriver.java:parse(443)) - Parse Completed
        2012-01-09 11:31:55,147 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7445)) - Starting Semantic Analysis
        2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7475)) - Completed phase 1 of Semantic Analysis
        2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(942)) - Get metadata for source tables
        2012-01-09 11:31:55,149 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(528)) - 0: get_table : db=default tbl=src
        2012-01-09 11:31:55,200 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src

        { string key, string value}
        2012-01-09 11:31:55,200 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[key, value] columnTypes=[string, string] separator=[[B@3bb20e65] nullstring=\N lastColumnTakesRest=false
        2012-01-09 11:31:55,200 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1021)) - Get metadata for subqueries
        2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1035)) - Get metadata for destination tables
        2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7478)) - Completed getting MetaData in Semantic Analysis
        2012-01-09 11:31:55,203 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src { string key, string value}

        2012-01-09 11:31:55,203 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[key, value] columnTypes=[string, string] separator=[[B@12e84396] nullstring=\N lastColumnTakesRest=false
        2012-01-09 11:31:55,222 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genTablePlan(6598)) - Created Table Plan for src org.apache.hadoop.hive.ql.exec.TableScanOperator@5e9ea579
        2012-01-09 11:31:55,223 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2117)) - tree: (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION sort (TOK_FUNCTION array "b" "d" "c" "a"))))
        2012-01-09 11:31:55,225 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2222)) - genSelectPlan: input = src

        {(key,key: string)(value,value: string)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)}

        2012-01-09 11:31:55,234 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Error in semantic analysis: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument.
        org.apache.hadoop.hive.ql.parse.SemanticException: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument.
        at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:810)
        at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
        at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
        at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:161)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:7708)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2301

        The same thing happens when I was doing format().

        REVISION DETAIL
        https://reviews.facebook.net/D1125

        Show
        Phabricator added a comment - zhenxiao has commented on the revision " HIVE-2279 [jira] Implement sort(array) UDF". sort() is a better name for sort_array(), while, seems currently the parser/semantic analyzer has some problem taking a reserved keyword as UDF function name. I tried the following changes in HIve.g: [~/Code/hive] git diff ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g b/ql/src/java/org/apache/hadoop/h index 888bf47..ec256de 100644 — a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g +++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g @@ -1816,7 +1816,7 @@ functionName @init { msgs.push("function name"); } @after { msgs.pop(); } : // Keyword IF is also a function name Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE + Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE | KW_SORT ; castExpression @@ -2091,6 +2091,7 @@ sysFuncNames KW_MAP KW_STRUCT KW_UNIONTYPE + KW_SORT EQUAL NOTEQUAL LESSTHANOREQUALTO While, the testcase always fails during semantic analysis on argument length: – Evaluate function against STRING valued keys EXPLAIN SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1 2012-01-09 11:31:55,134 INFO parse.ParseDriver (ParseDriver.java:parse(426)) - Parsing command: – Evaluate function against STRING valued keys EXPLAIN SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1 2012-01-09 11:31:55,146 INFO parse.ParseDriver (ParseDriver.java:parse(443)) - Parse Completed 2012-01-09 11:31:55,147 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7445)) - Starting Semantic Analysis 2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7475)) - Completed phase 1 of Semantic Analysis 2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(942)) - Get metadata for source tables 2012-01-09 11:31:55,149 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(528)) - 0: get_table : db=default tbl=src 2012-01-09 11:31:55,200 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src { string key, string value} 2012-01-09 11:31:55,200 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames= [key, value] columnTypes= [string, string] separator=[ [B@3bb20e65] nullstring=\N lastColumnTakesRest=false 2012-01-09 11:31:55,200 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1021)) - Get metadata for subqueries 2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1035)) - Get metadata for destination tables 2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7478)) - Completed getting MetaData in Semantic Analysis 2012-01-09 11:31:55,203 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src { string key, string value} 2012-01-09 11:31:55,203 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames= [key, value] columnTypes= [string, string] separator=[ [B@12e84396] nullstring=\N lastColumnTakesRest=false 2012-01-09 11:31:55,222 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genTablePlan(6598)) - Created Table Plan for src org.apache.hadoop.hive.ql.exec.TableScanOperator@5e9ea579 2012-01-09 11:31:55,223 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2117)) - tree: (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION sort (TOK_FUNCTION array "b" "d" "c" "a")))) 2012-01-09 11:31:55,225 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2222)) - genSelectPlan: input = src {(key,key: string)(value,value: string)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)} 2012-01-09 11:31:55,234 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Error in semantic analysis: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument. org.apache.hadoop.hive.ql.parse.SemanticException: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument. at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:810) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102) at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:161) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:7708) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2301 The same thing happens when I was doing format(). REVISION DETAIL https://reviews.facebook.net/D1125
        Hide
        Phabricator added a comment -

        zhenxiao requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
        Reviewers: JIRA

        HIVE-2279: Implement sort(array) UDF

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D1143

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
        ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
        ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
        ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
        ql/src/test/queries/clientpositive/udf_sort_array.q
        ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
        ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
        ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
        ql/src/test/results/clientpositive/show_functions.q.out
        ql/src/test/results/clientpositive/udf_sort_array.q.out

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/2415/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : Implement sort(array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1143 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2415/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Carl Steinbach added a comment -

        @Zhenxiao: Can you please open a JIRA for the parser problem? Thanks!

        Show
        Carl Steinbach added a comment - @Zhenxiao: Can you please open a JIRA for the parser problem? Thanks!
        Hide
        Phabricator added a comment -

        cwsteinbach has accepted the revision "HIVE-2279 [jira] Implement sort(array) UDF".

        +1. Will commit if tests pass.

        REVISION DETAIL
        https://reviews.facebook.net/D1143

        Show
        Phabricator added a comment - cwsteinbach has accepted the revision " HIVE-2279 [jira] Implement sort(array) UDF". +1. Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D1143
        Hide
        Carl Steinbach added a comment -

        @Zhenxiao: Please attach a copy of your patch (D1143) and click the box that gives license rights to Apache. This is a prerequisite for getting any patch committed. Thanks.

        Show
        Carl Steinbach added a comment - @Zhenxiao: Please attach a copy of your patch (D1143) and click the box that gives license rights to Apache. This is a prerequisite for getting any patch committed. Thanks.
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Zhenxiao!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Zhenxiao!
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1211 (See https://builds.apache.org/job/Hive-trunk-h0.21/1211/)
        HIVE-2279. Implement sort(array) UDF (Zhenxiao Luo via cws)

        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
        • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
        • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
        • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q
        • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1211 (See https://builds.apache.org/job/Hive-trunk-h0.21/1211/ ) HIVE-2279 . Implement sort(array) UDF (Zhenxiao Luo via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out
        Hide
        Ashutosh Chauhan added a comment -

        This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

        Show
        Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
        Hide
        Phabricator added a comment -

        zhenxiao has abandoned the revision "HIVE-2279 [jira] Implement sort(array) UDF".

        REVISION DETAIL
        https://reviews.facebook.net/D1101

        To: JIRA, zhenxiao
        Cc: zhenxiao

        Show
        Phabricator added a comment - zhenxiao has abandoned the revision " HIVE-2279 [jira] Implement sort(array) UDF". REVISION DETAIL https://reviews.facebook.net/D1101 To: JIRA, zhenxiao Cc: zhenxiao
        Hide
        Phabricator added a comment -

        zhenxiao has closed the revision "HIVE-2279 [jira] Implement sort(array) UDF".

        REVISION DETAIL
        https://reviews.facebook.net/D1143

        To: JIRA, cwsteinbach, zhenxiao
        Cc: cwsteinbach

        Show
        Phabricator added a comment - zhenxiao has closed the revision " HIVE-2279 [jira] Implement sort(array) UDF". REVISION DETAIL https://reviews.facebook.net/D1143 To: JIRA, cwsteinbach, zhenxiao Cc: cwsteinbach
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-2279. Implement sort(array) UDF (Zhenxiao Luo via cws) (Revision 1234146)

        Result = ABORTED
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
        • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
        • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
        • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
        • /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q
        • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
        • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
        • /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2279 . Implement sort(array) UDF (Zhenxiao Luo via cws) (Revision 1234146) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out

          People

          • Assignee:
            Zhenxiao Luo
            Reporter:
            Carl Steinbach
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development