Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: UDF
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Activity

      Hide
      Hudson added a comment -

      Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
      HIVE-2279. Implement sort(array) UDF (Zhenxiao Luo via cws) (Revision 1234146)

      Result = ABORTED
      cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146
      Files :

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
      • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
      • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
      • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
      • /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q
      • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
      • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
      • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
      • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
      • /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out
      Show
      Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2279 . Implement sort(array) UDF (Zhenxiao Luo via cws) (Revision 1234146) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out
      Hide
      Phabricator added a comment -

      zhenxiao has closed the revision "HIVE-2279 [jira] Implement sort(array) UDF".

      REVISION DETAIL
      https://reviews.facebook.net/D1143

      To: JIRA, cwsteinbach, zhenxiao
      Cc: cwsteinbach

      Show
      Phabricator added a comment - zhenxiao has closed the revision " HIVE-2279 [jira] Implement sort(array) UDF". REVISION DETAIL https://reviews.facebook.net/D1143 To: JIRA, cwsteinbach, zhenxiao Cc: cwsteinbach
      Hide
      Phabricator added a comment -

      zhenxiao has abandoned the revision "HIVE-2279 [jira] Implement sort(array) UDF".

      REVISION DETAIL
      https://reviews.facebook.net/D1101

      To: JIRA, zhenxiao
      Cc: zhenxiao

      Show
      Phabricator added a comment - zhenxiao has abandoned the revision " HIVE-2279 [jira] Implement sort(array) UDF". REVISION DETAIL https://reviews.facebook.net/D1101 To: JIRA, zhenxiao Cc: zhenxiao
      Hide
      Ashutosh Chauhan added a comment -

      This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

      Show
      Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
      Hide
      Hudson added a comment -

      Integrated in Hive-trunk-h0.21 #1211 (See https://builds.apache.org/job/Hive-trunk-h0.21/1211/)
      HIVE-2279. Implement sort(array) UDF (Zhenxiao Luo via cws)

      cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146
      Files :

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
      • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
      • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
      • /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
      • /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q
      • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
      • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
      • /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
      • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
      • /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out
      Show
      Hudson added a comment - Integrated in Hive-trunk-h0.21 #1211 (See https://builds.apache.org/job/Hive-trunk-h0.21/1211/ ) HIVE-2279 . Implement sort(array) UDF (Zhenxiao Luo via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out
      Hide
      Carl Steinbach added a comment -

      Committed to trunk. Thanks Zhenxiao!

      Show
      Carl Steinbach added a comment - Committed to trunk. Thanks Zhenxiao!
      Hide
      Carl Steinbach added a comment -

      @Zhenxiao: Please attach a copy of your patch (D1143) and click the box that gives license rights to Apache. This is a prerequisite for getting any patch committed. Thanks.

      Show
      Carl Steinbach added a comment - @Zhenxiao: Please attach a copy of your patch (D1143) and click the box that gives license rights to Apache. This is a prerequisite for getting any patch committed. Thanks.
      Hide
      Phabricator added a comment -

      cwsteinbach has accepted the revision "HIVE-2279 [jira] Implement sort(array) UDF".

      +1. Will commit if tests pass.

      REVISION DETAIL
      https://reviews.facebook.net/D1143

      Show
      Phabricator added a comment - cwsteinbach has accepted the revision " HIVE-2279 [jira] Implement sort(array) UDF". +1. Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D1143
      Hide
      Carl Steinbach added a comment -

      @Zhenxiao: Can you please open a JIRA for the parser problem? Thanks!

      Show
      Carl Steinbach added a comment - @Zhenxiao: Can you please open a JIRA for the parser problem? Thanks!
      Hide
      Phabricator added a comment -

      zhenxiao requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
      Reviewers: JIRA

      HIVE-2279: Implement sort(array) UDF

      TEST PLAN
      EMPTY

      REVISION DETAIL
      https://reviews.facebook.net/D1143

      AFFECTED FILES
      ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
      ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
      ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
      ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
      ql/src/test/queries/clientpositive/udf_sort_array.q
      ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
      ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
      ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
      ql/src/test/results/clientpositive/show_functions.q.out
      ql/src/test/results/clientpositive/udf_sort_array.q.out

      MANAGE HERALD DIFFERENTIAL RULES
      https://reviews.facebook.net/herald/view/differential/

      WHY DID I GET THIS EMAIL?
      https://reviews.facebook.net/herald/transcript/2415/

      Tip: use the X-Herald-Rules header to filter Herald messages in your client.

      Show
      Phabricator added a comment - zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : Implement sort(array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1143 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2415/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
      Hide
      Phabricator added a comment -

      zhenxiao has commented on the revision "HIVE-2279 [jira] Implement sort(array) UDF".

      sort() is a better name for sort_array(), while, seems currently the parser/semantic analyzer has some problem taking a reserved keyword as UDF function name.

      I tried the following changes in HIve.g:

      [~/Code/hive]git diff ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
      diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g b/ql/src/java/org/apache/hadoop/h
      index 888bf47..ec256de 100644
      — a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
      +++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
      @@ -1816,7 +1816,7 @@ functionName
      @init

      { msgs.push("function name"); }

      @after

      { msgs.pop(); }

      : // Keyword IF is also a function name

      • Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE
        + Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE | KW_SORT
        ;

      castExpression
      @@ -2091,6 +2091,7 @@ sysFuncNames

      KW_MAP
      KW_STRUCT
      KW_UNIONTYPE
      +
      KW_SORT
      EQUAL
      NOTEQUAL
      LESSTHANOREQUALTO

      While, the testcase always fails during semantic analysis on argument length:

      – Evaluate function against STRING valued keys
      EXPLAIN
      SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1
      2012-01-09 11:31:55,134 INFO parse.ParseDriver (ParseDriver.java:parse(426)) - Parsing command:

      – Evaluate function against STRING valued keys
      EXPLAIN
      SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1
      2012-01-09 11:31:55,146 INFO parse.ParseDriver (ParseDriver.java:parse(443)) - Parse Completed
      2012-01-09 11:31:55,147 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7445)) - Starting Semantic Analysis
      2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7475)) - Completed phase 1 of Semantic Analysis
      2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(942)) - Get metadata for source tables
      2012-01-09 11:31:55,149 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(528)) - 0: get_table : db=default tbl=src
      2012-01-09 11:31:55,200 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src

      { string key, string value}
      2012-01-09 11:31:55,200 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[key, value] columnTypes=[string, string] separator=[[B@3bb20e65] nullstring=\N lastColumnTakesRest=false
      2012-01-09 11:31:55,200 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1021)) - Get metadata for subqueries
      2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1035)) - Get metadata for destination tables
      2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7478)) - Completed getting MetaData in Semantic Analysis
      2012-01-09 11:31:55,203 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src { string key, string value}

      2012-01-09 11:31:55,203 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[key, value] columnTypes=[string, string] separator=[[B@12e84396] nullstring=\N lastColumnTakesRest=false
      2012-01-09 11:31:55,222 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genTablePlan(6598)) - Created Table Plan for src org.apache.hadoop.hive.ql.exec.TableScanOperator@5e9ea579
      2012-01-09 11:31:55,223 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2117)) - tree: (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION sort (TOK_FUNCTION array "b" "d" "c" "a"))))
      2012-01-09 11:31:55,225 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2222)) - genSelectPlan: input = src

      {(key,key: string)(value,value: string)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)}

      2012-01-09 11:31:55,234 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Error in semantic analysis: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument.
      org.apache.hadoop.hive.ql.parse.SemanticException: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument.
      at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:810)
      at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
      at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
      at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:161)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:7708)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2301

      The same thing happens when I was doing format().

      REVISION DETAIL
      https://reviews.facebook.net/D1125

      Show
      Phabricator added a comment - zhenxiao has commented on the revision " HIVE-2279 [jira] Implement sort(array) UDF". sort() is a better name for sort_array(), while, seems currently the parser/semantic analyzer has some problem taking a reserved keyword as UDF function name. I tried the following changes in HIve.g: [~/Code/hive] git diff ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g b/ql/src/java/org/apache/hadoop/h index 888bf47..ec256de 100644 — a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g +++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g @@ -1816,7 +1816,7 @@ functionName @init { msgs.push("function name"); } @after { msgs.pop(); } : // Keyword IF is also a function name Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE + Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE | KW_SORT ; castExpression @@ -2091,6 +2091,7 @@ sysFuncNames KW_MAP KW_STRUCT KW_UNIONTYPE + KW_SORT EQUAL NOTEQUAL LESSTHANOREQUALTO While, the testcase always fails during semantic analysis on argument length: – Evaluate function against STRING valued keys EXPLAIN SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1 2012-01-09 11:31:55,134 INFO parse.ParseDriver (ParseDriver.java:parse(426)) - Parsing command: – Evaluate function against STRING valued keys EXPLAIN SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1 2012-01-09 11:31:55,146 INFO parse.ParseDriver (ParseDriver.java:parse(443)) - Parse Completed 2012-01-09 11:31:55,147 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7445)) - Starting Semantic Analysis 2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7475)) - Completed phase 1 of Semantic Analysis 2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(942)) - Get metadata for source tables 2012-01-09 11:31:55,149 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(528)) - 0: get_table : db=default tbl=src 2012-01-09 11:31:55,200 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src { string key, string value} 2012-01-09 11:31:55,200 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames= [key, value] columnTypes= [string, string] separator=[ [B@3bb20e65] nullstring=\N lastColumnTakesRest=false 2012-01-09 11:31:55,200 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1021)) - Get metadata for subqueries 2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1035)) - Get metadata for destination tables 2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7478)) - Completed getting MetaData in Semantic Analysis 2012-01-09 11:31:55,203 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src { string key, string value} 2012-01-09 11:31:55,203 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames= [key, value] columnTypes= [string, string] separator=[ [B@12e84396] nullstring=\N lastColumnTakesRest=false 2012-01-09 11:31:55,222 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genTablePlan(6598)) - Created Table Plan for src org.apache.hadoop.hive.ql.exec.TableScanOperator@5e9ea579 2012-01-09 11:31:55,223 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2117)) - tree: (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION sort (TOK_FUNCTION array "b" "d" "c" "a")))) 2012-01-09 11:31:55,225 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2222)) - genSelectPlan: input = src {(key,key: string)(value,value: string)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)} 2012-01-09 11:31:55,234 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Error in semantic analysis: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument. org.apache.hadoop.hive.ql.parse.SemanticException: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument. at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:810) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102) at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:161) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:7708) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2301 The same thing happens when I was doing format(). REVISION DETAIL https://reviews.facebook.net/D1125
      Hide
      Phabricator added a comment -

      cwsteinbach has requested changes to the revision "HIVE-2279 [jira] Implement sort(array) UDF".

      REVISION DETAIL
      https://reviews.facebook.net/D1125

      Show
      Phabricator added a comment - cwsteinbach has requested changes to the revision " HIVE-2279 [jira] Implement sort(array) UDF". REVISION DETAIL https://reviews.facebook.net/D1125
      Hide
      Phabricator added a comment -

      cwsteinbach has commented on the revision "HIVE-2279 [jira] Implement sort(array) UDF".

      INLINE COMMENTS
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:47 This should be
      value="FUNC(array)"

      Otherwise, the output of "describe function sort_array" looks like this:

      sort_array(sort_array(obj1, obj2,...)) - Sorts the input...
      ^^^^^^^^
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:51 Please change the example to something simpler that doesn't involve corporate names, e.g. "array(4, 1, 3, 2)"
      ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:449 I think the name should be changed to "sort" and "GenericUDFSort". In the future we may want to extend it so that it can also support sorting the elements in an input string. In a similar manner we may want to implement a "reverse" UDF that reverse the elements of an array as well as the characters in a string.
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:74 "Argument 1" instead of "Argument " + 1 + ...
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:88 There's a single argument, right?

      REVISION DETAIL
      https://reviews.facebook.net/D1125

      Show
      Phabricator added a comment - cwsteinbach has commented on the revision " HIVE-2279 [jira] Implement sort(array) UDF". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:47 This should be value=" FUNC (array)" Otherwise, the output of "describe function sort_array" looks like this: sort_array(sort_array(obj1, obj2,...)) - Sorts the input... ^^^^^^^^ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:51 Please change the example to something simpler that doesn't involve corporate names, e.g. "array(4, 1, 3, 2)" ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:449 I think the name should be changed to "sort" and "GenericUDFSort". In the future we may want to extend it so that it can also support sorting the elements in an input string. In a similar manner we may want to implement a "reverse" UDF that reverse the elements of an array as well as the characters in a string. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:74 "Argument 1" instead of "Argument " + 1 + ... ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:88 There's a single argument, right? REVISION DETAIL https://reviews.facebook.net/D1125
      Hide
      Phabricator added a comment -

      zhenxiao requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
      Reviewers: JIRA

      HIVE-2279: Implement Sort(Array) UDF

      TEST PLAN
      EMPTY

      REVISION DETAIL
      https://reviews.facebook.net/D1125

      AFFECTED FILES
      ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
      ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
      ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
      ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
      ql/src/test/queries/clientpositive/udf_sort_array.q
      ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
      ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
      ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
      ql/src/test/results/clientpositive/show_functions.q.out
      ql/src/test/results/clientpositive/udf_sort_array.q.out

      MANAGE HERALD DIFFERENTIAL RULES
      https://reviews.facebook.net/herald/view/differential/

      WHY DID I GET THIS EMAIL?
      https://reviews.facebook.net/herald/transcript/2355/

      Tip: use the X-Herald-Rules header to filter Herald messages in your client.

      Show
      Phabricator added a comment - zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : Implement Sort(Array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1125 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2355/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
      Hide
      Phabricator added a comment -

      cwsteinbach has requested changes to the revision "HIVE-2279 [jira] Implement sort(array) UDF".

      INLINE COMMENTS
      ql/src/test/queries/clientpositive/udf_sort_array.q:8 Please add EXPLAIN queries.
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:82 Fix indentation.
      ql/src/test/results/clientpositive/udf_sort_array.q.out:13 "sort_array(sort_array(...))"?
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:45 "Sorts the input array in ascending order according to the natural ordering of the array elements."
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:55 Please add a negative testcase that exercises these code paths.
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:60 What happens if I try to sort an array of arrays, or array of structs?
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:67 The return type should be another ARRAY, not a string.
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:85 This method needs to return another ARRAY, not a string containing the concatenated, sorted input elements. Take a look at the array() UDF for hints on how to return an array from a UDF.
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:97 Indentation.

      REVISION DETAIL
      https://reviews.facebook.net/D1107

      Show
      Phabricator added a comment - cwsteinbach has requested changes to the revision " HIVE-2279 [jira] Implement sort(array) UDF". INLINE COMMENTS ql/src/test/queries/clientpositive/udf_sort_array.q:8 Please add EXPLAIN queries. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:82 Fix indentation. ql/src/test/results/clientpositive/udf_sort_array.q.out:13 "sort_array(sort_array(...))"? ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:45 "Sorts the input array in ascending order according to the natural ordering of the array elements." ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:55 Please add a negative testcase that exercises these code paths. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:60 What happens if I try to sort an array of arrays, or array of structs? ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:67 The return type should be another ARRAY, not a string. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:85 This method needs to return another ARRAY, not a string containing the concatenated, sorted input elements. Take a look at the array() UDF for hints on how to return an array from a UDF. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:97 Indentation. REVISION DETAIL https://reviews.facebook.net/D1107
      Hide
      Phabricator added a comment -

      zhenxiao has commented on the revision "HIVE-2279 [jira] Implement sort(array) UDF".

      Incomplete patch. D1107 is the correct patch for review.

      REVISION DETAIL
      https://reviews.facebook.net/D1101

      Show
      Phabricator added a comment - zhenxiao has commented on the revision " HIVE-2279 [jira] Implement sort(array) UDF". Incomplete patch. D1107 is the correct patch for review. REVISION DETAIL https://reviews.facebook.net/D1101
      Hide
      Phabricator added a comment -

      zhenxiao requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
      Reviewers: JIRA

      HIVE-2279: Implement sort(array) UDF

      TEST PLAN
      EMPTY

      REVISION DETAIL
      https://reviews.facebook.net/D1107

      AFFECTED FILES
      ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
      ql/src/test/queries/clientpositive/udf_sort_array.q
      ql/src/test/results/clientpositive/show_functions.q.out
      ql/src/test/results/clientpositive/udf_sort_array.q.out

      MANAGE HERALD DIFFERENTIAL RULES
      https://reviews.facebook.net/herald/view/differential/

      WHY DID I GET THIS EMAIL?
      https://reviews.facebook.net/herald/transcript/2325/

      Tip: use the X-Herald-Rules header to filter Herald messages in your client.

      Show
      Phabricator added a comment - zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : Implement sort(array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1107 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2325/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
      Hide
      Phabricator added a comment -

      zhenxiao requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
      Reviewers: JIRA

      HIVE-2279: resolve Trailing Whitespace

      TEST PLAN
      EMPTY

      REVISION DETAIL
      https://reviews.facebook.net/D1101

      AFFECTED FILES
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java

      MANAGE HERALD DIFFERENTIAL RULES
      https://reviews.facebook.net/herald/view/differential/

      WHY DID I GET THIS EMAIL?
      https://reviews.facebook.net/herald/transcript/2319/

      Tip: use the X-Herald-Rules header to filter Herald messages in your client.

      Show
      Phabricator added a comment - zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : resolve Trailing Whitespace TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1101 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2319/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
      Hide
      Phabricator added a comment -

      cwsteinbach requested code review of "HIVE-2279 [jira] Implement sort(array) UDF".
      Reviewers: JIRA

      HIVE-2279. Implement sort(array) UDF

      TEST PLAN
      EMPTY

      REVISION DETAIL
      https://reviews.facebook.net/D1059

      AFFECTED FILES
      ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
      ql/src/test/queries/clientpositive/udf_sort_array.q
      ql/src/test/results/clientpositive/show_functions.q.out
      ql/src/test/results/clientpositive/udf_sort_array.q.out

      MANAGE HERALD DIFFERENTIAL RULES
      https://reviews.facebook.net/herald/view/differential/

      WHY DID I GET THIS EMAIL?
      https://reviews.facebook.net/herald/transcript/2241/

      Tip: use the X-Herald-Rules header to filter Herald messages in your client.

      Show
      Phabricator added a comment - cwsteinbach requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 . Implement sort(array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1059 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2241/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
      Hide
      Carl Steinbach added a comment -

      This UDF would be generally nice to have, but may also be a requirement if it turns out that the unordered output of functions like map_keys() and map_values() is non-deterministic.

      Show
      Carl Steinbach added a comment - This UDF would be generally nice to have, but may also be a requirement if it turns out that the unordered output of functions like map_keys() and map_values() is non-deterministic.

        People

        • Assignee:
          Zhenxiao Luo
          Reporter:
          Carl Steinbach
        • Votes:
          0 Vote for this issue
          Watchers:
          4 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development