[HIVE-2279] Implement sort_array UDF - ASF JIRA

Hide

Permalink

Carl Steinbach added a comment - 12/Jul/11 00:48

This UDF would be generally nice to have, but may also be a requirement if it turns out that the unordered output of functions like map_keys() and map_values() is non-deterministic.

Show

Carl Steinbach added a comment - 12/Jul/11 00:48 This UDF would be generally nice to have, but may also be a requirement if it turns out that the unordered output of functions like map_keys() and map_values() is non-deterministic.

Hide

Permalink

Phabricator added a comment - 30/Dec/11 06:13

cwsteinbach requested code review of "~~HIVE-2279~~ [jira] Implement sort(array) UDF".
Reviewers: JIRA

~~HIVE-2279~~. Implement sort(array) UDF

TEST PLAN
EMPTY

REVISION DETAIL
https://reviews.facebook.net/D1059

AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
ql/src/test/queries/clientpositive/udf_sort_array.q
ql/src/test/results/clientpositive/show_functions.q.out
ql/src/test/results/clientpositive/udf_sort_array.q.out

MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/2241/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Show

Phabricator added a comment - 30/Dec/11 06:13 cwsteinbach requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 . Implement sort(array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1059 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2241/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Hide

Permalink

Phabricator added a comment - 04/Jan/12 01:54

zhenxiao requested code review of "~~HIVE-2279~~ [jira] Implement sort(array) UDF".
Reviewers: JIRA

~~HIVE-2279~~: resolve Trailing Whitespace

TEST PLAN
EMPTY

REVISION DETAIL
https://reviews.facebook.net/D1101

AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java

MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/2319/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Show

Phabricator added a comment - 04/Jan/12 01:54 zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : resolve Trailing Whitespace TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1101 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2319/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Hide

Permalink

Phabricator added a comment - 04/Jan/12 02:01

zhenxiao requested code review of "~~HIVE-2279~~ [jira] Implement sort(array) UDF".
Reviewers: JIRA

~~HIVE-2279~~: Implement sort(array) UDF

TEST PLAN
EMPTY

REVISION DETAIL
https://reviews.facebook.net/D1107

AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
ql/src/test/queries/clientpositive/udf_sort_array.q
ql/src/test/results/clientpositive/show_functions.q.out
ql/src/test/results/clientpositive/udf_sort_array.q.out

MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/2325/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Show

Phabricator added a comment - 04/Jan/12 02:01 zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : Implement sort(array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1107 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2325/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Hide

Permalink

Phabricator added a comment - 04/Jan/12 02:04

zhenxiao has commented on the revision "~~HIVE-2279~~ [jira] Implement sort(array) UDF".

Incomplete patch. D1107 is the correct patch for review.

REVISION DETAIL
https://reviews.facebook.net/D1101

Show

Phabricator added a comment - 04/Jan/12 02:04 zhenxiao has commented on the revision " HIVE-2279 [jira] Implement sort(array) UDF". Incomplete patch. D1107 is the correct patch for review. REVISION DETAIL https://reviews.facebook.net/D1101

Hide

Permalink

Phabricator added a comment - 04/Jan/12 02:28

cwsteinbach has requested changes to the revision "~~HIVE-2279~~ [jira] Implement sort(array) UDF".

INLINE COMMENTS
ql/src/test/queries/clientpositive/udf_sort_array.q:8 Please add EXPLAIN queries.
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:82 Fix indentation.
ql/src/test/results/clientpositive/udf_sort_array.q.out:13 "sort_array(sort_array(...))"?
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:45 "Sorts the input array in ascending order according to the natural ordering of the array elements."
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:55 Please add a negative testcase that exercises these code paths.
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:60 What happens if I try to sort an array of arrays, or array of structs?
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:67 The return type should be another ARRAY, not a string.
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:85 This method needs to return another ARRAY, not a string containing the concatenated, sorted input elements. Take a look at the array() UDF for hints on how to return an array from a UDF.
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:97 Indentation.

REVISION DETAIL
https://reviews.facebook.net/D1107

Show

Phabricator added a comment - 04/Jan/12 02:28 cwsteinbach has requested changes to the revision " HIVE-2279 [jira] Implement sort(array) UDF". INLINE COMMENTS ql/src/test/queries/clientpositive/udf_sort_array.q:8 Please add EXPLAIN queries. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:82 Fix indentation. ql/src/test/results/clientpositive/udf_sort_array.q.out:13 "sort_array(sort_array(...))"? ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:45 "Sorts the input array in ascending order according to the natural ordering of the array elements." ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:55 Please add a negative testcase that exercises these code paths. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:60 What happens if I try to sort an array of arrays, or array of structs? ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:67 The return type should be another ARRAY, not a string. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:85 This method needs to return another ARRAY, not a string containing the concatenated, sorted input elements. Take a look at the array() UDF for hints on how to return an array from a UDF. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:97 Indentation. REVISION DETAIL https://reviews.facebook.net/D1107

Hide

Permalink

Phabricator added a comment - 04/Jan/12 23:44

zhenxiao requested code review of "~~HIVE-2279~~ [jira] Implement sort(array) UDF".
Reviewers: JIRA

~~HIVE-2279~~: Implement Sort(Array) UDF

TEST PLAN
EMPTY

REVISION DETAIL
https://reviews.facebook.net/D1125

AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
ql/src/test/queries/clientpositive/udf_sort_array.q
ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
ql/src/test/results/clientpositive/show_functions.q.out
ql/src/test/results/clientpositive/udf_sort_array.q.out

MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/2355/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Show

Phabricator added a comment - 04/Jan/12 23:44 zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : Implement Sort(Array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1125 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2355/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Hide

Permalink

Phabricator added a comment - 09/Jan/12 09:08

cwsteinbach has commented on the revision "~~HIVE-2279~~ [jira] Implement sort(array) UDF".

INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:47 This should be
value="FUNC(array)"

Otherwise, the output of "describe function sort_array" looks like this:

sort_array(sort_array(obj1, obj2,...)) - Sorts the input...
^^^^^^^^
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:51 Please change the example to something simpler that doesn't involve corporate names, e.g. "array(4, 1, 3, 2)"
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:449 I think the name should be changed to "sort" and "GenericUDFSort". In the future we may want to extend it so that it can also support sorting the elements in an input string. In a similar manner we may want to implement a "reverse" UDF that reverse the elements of an array as well as the characters in a string.
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:74 "Argument 1" instead of "Argument " + 1 + ...
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:88 There's a single argument, right?

REVISION DETAIL
https://reviews.facebook.net/D1125

Show

Phabricator added a comment - 09/Jan/12 09:08 cwsteinbach has commented on the revision " HIVE-2279 [jira] Implement sort(array) UDF". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:47 This should be value=" FUNC (array)" Otherwise, the output of "describe function sort_array" looks like this: sort_array(sort_array(obj1, obj2,...)) - Sorts the input... ^^^^^^^^ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:51 Please change the example to something simpler that doesn't involve corporate names, e.g. "array(4, 1, 3, 2)" ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:449 I think the name should be changed to "sort" and "GenericUDFSort". In the future we may want to extend it so that it can also support sorting the elements in an input string. In a similar manner we may want to implement a "reverse" UDF that reverse the elements of an array as well as the characters in a string. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:74 "Argument 1" instead of "Argument " + 1 + ... ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java:88 There's a single argument, right? REVISION DETAIL https://reviews.facebook.net/D1125

Hide

Permalink

Phabricator added a comment - 09/Jan/12 09:24

cwsteinbach has requested changes to the revision "~~HIVE-2279~~ [jira] Implement sort(array) UDF".

REVISION DETAIL
https://reviews.facebook.net/D1125

Show

Phabricator added a comment - 09/Jan/12 09:24 cwsteinbach has requested changes to the revision " HIVE-2279 [jira] Implement sort(array) UDF". REVISION DETAIL https://reviews.facebook.net/D1125

Hide

Permalink

Phabricator added a comment - 09/Jan/12 21:45

zhenxiao has commented on the revision "~~HIVE-2279~~ [jira] Implement sort(array) UDF".

sort() is a better name for sort_array(), while, seems currently the parser/semantic analyzer has some problem taking a reserved keyword as UDF function name.

I tried the following changes in HIve.g:

[~/Code/hive]git diff ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g b/ql/src/java/org/apache/hadoop/h
index 888bf47..ec256de 100644
— a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
@@ -1816,7 +1816,7 @@ functionName
@init

{ msgs.push("function name"); }

@after

{ msgs.pop(); }

: // Keyword IF is also a function name

Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE
+ Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE | KW_SORT
;

castExpression
@@ -2091,6 +2091,7 @@ sysFuncNames

KW_MAP
KW_STRUCT
KW_UNIONTYPE +	KW_SORT
EQUAL
NOTEQUAL
LESSTHANOREQUALTO

While, the testcase always fails during semantic analysis on argument length:

– Evaluate function against STRING valued keys
EXPLAIN
SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1
2012-01-09 11:31:55,134 INFO parse.ParseDriver (ParseDriver.java:parse(426)) - Parsing command:

– Evaluate function against STRING valued keys
EXPLAIN
SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1
2012-01-09 11:31:55,146 INFO parse.ParseDriver (ParseDriver.java:parse(443)) - Parse Completed
2012-01-09 11:31:55,147 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7445)) - Starting Semantic Analysis
2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7475)) - Completed phase 1 of Semantic Analysis
2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(942)) - Get metadata for source tables
2012-01-09 11:31:55,149 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(528)) - 0: get_table : db=default tbl=src
2012-01-09 11:31:55,200 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src

{ string key, string value}
2012-01-09 11:31:55,200 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[key, value] columnTypes=[string, string] separator=[[B@3bb20e65] nullstring=\N lastColumnTakesRest=false
2012-01-09 11:31:55,200 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1021)) - Get metadata for subqueries
2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1035)) - Get metadata for destination tables
2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7478)) - Completed getting MetaData in Semantic Analysis
2012-01-09 11:31:55,203 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src { string key, string value}

2012-01-09 11:31:55,203 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[key, value] columnTypes=[string, string] separator=[[B@12e84396] nullstring=\N lastColumnTakesRest=false
2012-01-09 11:31:55,222 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genTablePlan(6598)) - Created Table Plan for src org.apache.hadoop.hive.ql.exec.TableScanOperator@5e9ea579
2012-01-09 11:31:55,223 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2117)) - tree: (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION sort (TOK_FUNCTION array "b" "d" "c" "a"))))
2012-01-09 11:31:55,225 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2222)) - genSelectPlan: input = src

{(key,key: string)(value,value: string)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)}

2012-01-09 11:31:55,234 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Error in semantic analysis: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument.
org.apache.hadoop.hive.ql.parse.SemanticException: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument.
at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:810)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102)
at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:161)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:7708)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2301

The same thing happens when I was doing format().

REVISION DETAIL
https://reviews.facebook.net/D1125

Show

Phabricator added a comment - 09/Jan/12 21:45 zhenxiao has commented on the revision " HIVE-2279 [jira] Implement sort(array) UDF". sort() is a better name for sort_array(), while, seems currently the parser/semantic analyzer has some problem taking a reserved keyword as UDF function name. I tried the following changes in HIve.g: [~/Code/hive] git diff ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g b/ql/src/java/org/apache/hadoop/h index 888bf47..ec256de 100644 — a/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g +++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g @@ -1816,7 +1816,7 @@ functionName @init { msgs.push("function name"); } @after { msgs.pop(); } : // Keyword IF is also a function name Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE + Identifier | KW_IF | KW_ARRAY | KW_MAP | KW_STRUCT | KW_UNIONTYPE | KW_SORT ; castExpression @@ -2091,6 +2091,7 @@ sysFuncNames KW_MAP KW_STRUCT KW_UNIONTYPE + KW_SORT EQUAL NOTEQUAL LESSTHANOREQUALTO While, the testcase always fails during semantic analysis on argument length: – Evaluate function against STRING valued keys EXPLAIN SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1 2012-01-09 11:31:55,134 INFO parse.ParseDriver (ParseDriver.java:parse(426)) - Parsing command: – Evaluate function against STRING valued keys EXPLAIN SELECT sort(array("b", "d", "c", "a")) FROM src LIMIT 1 2012-01-09 11:31:55,146 INFO parse.ParseDriver (ParseDriver.java:parse(443)) - Parse Completed 2012-01-09 11:31:55,147 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7445)) - Starting Semantic Analysis 2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7475)) - Completed phase 1 of Semantic Analysis 2012-01-09 11:31:55,148 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(942)) - Get metadata for source tables 2012-01-09 11:31:55,149 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(528)) - 0: get_table : db=default tbl=src 2012-01-09 11:31:55,200 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src { string key, string value} 2012-01-09 11:31:55,200 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames= [key, value] columnTypes= [string, string] separator=[ [B@3bb20e65] nullstring=\N lastColumnTakesRest=false 2012-01-09 11:31:55,200 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1021)) - Get metadata for subqueries 2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1035)) - Get metadata for destination tables 2012-01-09 11:31:55,201 INFO parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(7478)) - Completed getting MetaData in Semantic Analysis 2012-01-09 11:31:55,203 INFO hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(457)) - DDL: struct src { string key, string value} 2012-01-09 11:31:55,203 DEBUG lazy.LazySimpleSerDe (LazySimpleSerDe.java:initialize(195)) - org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames= [key, value] columnTypes= [string, string] separator=[ [B@12e84396] nullstring=\N lastColumnTakesRest=false 2012-01-09 11:31:55,222 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genTablePlan(6598)) - Created Table Plan for src org.apache.hadoop.hive.ql.exec.TableScanOperator@5e9ea579 2012-01-09 11:31:55,223 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2117)) - tree: (TOK_SELECT (TOK_SELEXPR (TOK_FUNCTION sort (TOK_FUNCTION array "b" "d" "c" "a")))) 2012-01-09 11:31:55,225 DEBUG parse.SemanticAnalyzer (SemanticAnalyzer.java:genSelectPlan(2222)) - genSelectPlan: input = src {(key,key: string)(value,value: string)(block__offset__inside__file,BLOCK__OFFSET__INSIDE__FILE: bigint)(input__file__name,INPUT__FILE__NAME: string)} 2012-01-09 11:31:55,234 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Error in semantic analysis: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument. org.apache.hadoop.hive.ql.parse.SemanticException: Line 5:7 Arguments length mismatch 'sort': The function SORT(array(obj1, obj2,...)) needs one argument. at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:810) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:88) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:125) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:102) at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:161) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:7708) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2301 The same thing happens when I was doing format(). REVISION DETAIL https://reviews.facebook.net/D1125

Hide

Permalink

Phabricator added a comment - 09/Jan/12 22:15

zhenxiao requested code review of "~~HIVE-2279~~ [jira] Implement sort(array) UDF".
Reviewers: JIRA

~~HIVE-2279~~: Implement sort(array) UDF

TEST PLAN
EMPTY

REVISION DETAIL
https://reviews.facebook.net/D1143

AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
ql/src/test/queries/clientpositive/udf_sort_array.q
ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
ql/src/test/results/clientpositive/show_functions.q.out
ql/src/test/results/clientpositive/udf_sort_array.q.out

MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/2415/

Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Show

Phabricator added a comment - 09/Jan/12 22:15 zhenxiao requested code review of " HIVE-2279 [jira] Implement sort(array) UDF". Reviewers: JIRA HIVE-2279 : Implement sort(array) UDF TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1143 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q ql/src/test/queries/clientpositive/udf_sort_array.q ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out ql/src/test/results/clientpositive/show_functions.q.out ql/src/test/results/clientpositive/udf_sort_array.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2415/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

Hide

Permalink

Carl Steinbach added a comment - 12/Jan/12 04:50

@Zhenxiao: Can you please open a JIRA for the parser problem? Thanks!

Show

Carl Steinbach added a comment - 12/Jan/12 04:50 @Zhenxiao: Can you please open a JIRA for the parser problem? Thanks!

Hide

Permalink

Phabricator added a comment - 12/Jan/12 04:53

cwsteinbach has accepted the revision "~~HIVE-2279~~ [jira] Implement sort(array) UDF".

+1. Will commit if tests pass.

REVISION DETAIL
https://reviews.facebook.net/D1143

Show

Phabricator added a comment - 12/Jan/12 04:53 cwsteinbach has accepted the revision " HIVE-2279 [jira] Implement sort(array) UDF". +1. Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D1143

Hide

Permalink

Carl Steinbach added a comment - 16/Jan/12 09:49

@Zhenxiao: Please attach a copy of your patch (D1143) and click the box that gives license rights to Apache. This is a prerequisite for getting any patch committed. Thanks.

Show

Carl Steinbach added a comment - 16/Jan/12 09:49 @Zhenxiao: Please attach a copy of your patch (D1143) and click the box that gives license rights to Apache. This is a prerequisite for getting any patch committed. Thanks.

Hide

Permalink

Carl Steinbach added a comment - 20/Jan/12 21:19

Committed to trunk. Thanks Zhenxiao!

Show

Carl Steinbach added a comment - 20/Jan/12 21:19 Committed to trunk. Thanks Zhenxiao!

Hide

Permalink

Hudson added a comment - 21/Jan/12 06:03

Integrated in Hive-trunk-h0.21 #1211 (See https://builds.apache.org/job/Hive-trunk-h0.21/1211/)
~~HIVE-2279~~. Implement sort(array) UDF (Zhenxiao Luo via cws)

cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146
Files :

/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
/hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
/hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
/hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
/hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q
/hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
/hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
/hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
/hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
/hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out

Show

Hudson added a comment - 21/Jan/12 06:03 Integrated in Hive-trunk-h0.21 #1211 (See https://builds.apache.org/job/Hive-trunk-h0.21/1211/ ) HIVE-2279 . Implement sort(array) UDF (Zhenxiao Luo via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out

Hide

Permalink

Ashutosh Chauhan added a comment - 30/Apr/12 21:11

This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

Show

Ashutosh Chauhan added a comment - 30/Apr/12 21:11 This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

Hide

Permalink

Phabricator added a comment - 11/Dec/12 00:27

zhenxiao has abandoned the revision "~~HIVE-2279~~ [jira] Implement sort(array) UDF".

REVISION DETAIL
https://reviews.facebook.net/D1101

To: JIRA, zhenxiao
Cc: zhenxiao

Show

Phabricator added a comment - 11/Dec/12 00:27 zhenxiao has abandoned the revision " HIVE-2279 [jira] Implement sort(array) UDF". REVISION DETAIL https://reviews.facebook.net/D1101 To: JIRA, zhenxiao Cc: zhenxiao

Hide

Permalink

Phabricator added a comment - 11/Dec/12 00:53

zhenxiao has closed the revision "~~HIVE-2279~~ [jira] Implement sort(array) UDF".

REVISION DETAIL
https://reviews.facebook.net/D1143

To: JIRA, cwsteinbach, zhenxiao
Cc: cwsteinbach

Show

Phabricator added a comment - 11/Dec/12 00:53 zhenxiao has closed the revision " HIVE-2279 [jira] Implement sort(array) UDF". REVISION DETAIL https://reviews.facebook.net/D1143 To: JIRA, cwsteinbach, zhenxiao Cc: cwsteinbach

Hide

Permalink

Hudson added a comment - 09/Jan/13 10:23

Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
~~HIVE-2279~~. Implement sort(array) UDF (Zhenxiao Luo via cws) (Revision 1234146)

Result = ABORTED
cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146
Files :

/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java
/hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q
/hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q
/hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q
/hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q
/hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out
/hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out
/hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out
/hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
/hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out

Show

Hudson added a comment - 09/Jan/13 10:23 Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2279 . Implement sort(array) UDF (Zhenxiao Luo via cws) (Revision 1234146) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1234146 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong1.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong2.q /hive/trunk/ql/src/test/queries/clientnegative/udf_sort_array_wrong3.q /hive/trunk/ql/src/test/queries/clientpositive/udf_sort_array.q /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong1.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong2.q.out /hive/trunk/ql/src/test/results/clientnegative/udf_sort_array_wrong3.q.out /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udf_sort_array.q.out

Implement sort_array UDF

Details

Attachments

Issue Links

Activity

People

Dates

Development

Agile