[HIVE-12359] Hive ORC table reports different counts between select * and count(*)

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.1
• Component/s: CBO, HiveServer2, ORC, Statistics

Environment

HDP 2.3 + Kerberos

• Assignee: Unassigned
• Reporter: Hari Sekhon
• Created: 06/Nov/15 15:39
• Updated: 05/May/16 22:25

Description

I have an ORC table which is giving different figures between select count( * ) and select *:

> select count(*) from myTable;
+--------+--+
|  _c0   |
+--------+--+
| 56471  |
+--------+--+

> select * from myTable;
...
109,295 rows selected (62.993 seconds)

At first I thought this was obvious just "analyze table ... compute statistics" and it'll correct itself, however I've tried that as well as adding "for columns" but the results remain the same. The select count( * ) is very fast so it must be using the pre-computed stats.

When I transform the table to text or to another orc table the count star on that new tables returns the correct number.

I've even tried disabling stats, CBO, the works, restart, same result, with very fast return each time for select count( * ), indicating it's using either pre-computed stats stored in Metastore or ORC stats in file format, but I'm not sure how ORC could store the wrong count, especially as doing a CTAS to another ORC table returns the correct count when I select count( * ) that new ORC table.