[DRILL-2083] order by on large dataset returns wrong results

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Component/s: Execution - Data Types, Execution - Relational Operators
• Assignee: Chun Chang
• Reporter: Chun Chang
• Created: 27/Jan/15 19:02
• Updated: 29/Apr/15 00:07
• Resolved: 28/Apr/15 03:03

Description

#Mon Jan 26 14:10:51 PST 2015
git.commit.id.abbrev=3c6d0ef

Test data has 1 million rows and can be accessed at

http://apache-drill.s3.amazonaws.com/files/complex.json.gz

0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select count (t.id) from `complex.json` t;
+------------+
|   EXPR$0   |
+------------+
| 1000000    |
+------------+

But order by returned 30 more rows.

0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select t.id from `complex.json` t order by t.id;
....
| 999997     |
| 999998     |
| 999999     |
| 1000000    |
+------------+
1,000,030 rows selected (19.449 seconds)

physical plan

0: jdbc:drill:schema=dfs.drillTestDirComplexJ> explain plan for select t.id from `complex.json` t order by t.id;
+------------+------------+
|    text    |    json    |
+------------+------------+
| 00-00    Screen
00-01      SingleMergeExchange(sort0=[0 ASC])
01-01        SelectionVectorRemover
01-02          Sort(sort0=[$0], dir0=[ASC])
01-03            HashToRandomExchange(dist0=[[$0]])
02-01              Scan(groupscan=[EasyGroupScan [selectionRoot=/drill/testdata/complex_type/json/complex.json, numFiles=1, columns=[`id`], files=[maprfs:/drill/testdata/complex_type/json/complex.json]]])

Issue Links

• is duplicated by: DRILL-2801 ORDER BY produces extra records
• is related to: DRILL-2167 Order by on a repeated index from the output of a flatten on large no of records results in incorrect results