[HIVE-21167] Bucketing: Bucketing version 1 is incorrectly partitioning data

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Deepak Jaiswal
• Reporter: Vaibhav Gumashta
• Created: 25/Jan/19 18:24
• Updated: 01/Dec/22 08:58
• Resolved: 23/Feb/19 00:23

Description

Using murmur hash for bucketing columns was introduced in HIVE-18910, following which 'bucketing_version'='1' stands for the old behaviour (where for example integer columns were partitioned based on mod values). Looks like we have a bug in the old bucketing scheme now. I could repro it when modified the existing schema using an alter table add column and adding new data. Repro:

0: jdbc:hive2://localhost:10010> create transactional table acid_ptn_bucket1 (a int, b int) partitioned by(ds string) clustered by (a) into 2 buckets stored as ORC TBLPROPERTIES('bucketing_version'='1', 'transactional'='true', 'transactional_properties'='default');

No rows affected (0.418 seconds)

0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds) values(1,2,'today'),(1,3,'today'),(1,4,'yesterday'),(2,2,'yesterday'),(2,3,'today'),(2,4,'today');
6 rows affected (3.695 seconds)

Data from ORC file (data as expected):

/apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000001_0000001_0000/bucket_00000
{"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 0, "currentTransaction": 1, "row": {"a": 2, "b": 4}}
{"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 1, "currentTransaction": 1, "row": {"a": 2, "b": 3}}


/apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000001_0000001_0000/bucket_00001
{"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 0, "currentTransaction": 1, "row": {"a": 1, "b": 3}}
{"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 1, "currentTransaction": 1, "row": {"a": 1, "b": 2}}

Modifying table schema and inserting new data:

0: jdbc:hive2://localhost:10010> alter table acid_ptn_bucket1 add columns(c int);

No rows affected (0.541 seconds)

0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds) values(3,2,1000,'yesterday'),(3,3,1001,'today'),(3,4,1002,'yesterday'),(4,2,1003,'today'), (4,3,1004,'yesterday'),(4,4,1005,'today');
6 rows affected (3.699 seconds)

Data from ORC file (wrong partitioning):

/apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000003_0000003_0000/bucket_00000
{"operation": 0, "originalTransaction": 3, "bucket": 536870912, "rowId": 0, "currentTransaction": 3, "row": {"a": 3, "b": 3, "c": 1001}}

/apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000003_0000003_0000/bucket_00001
{"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 0, "currentTransaction": 3, "row": {"a": 4, "b": 4, "c": 1005}}
{"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 1, "currentTransaction": 3, "row": {"a": 4, "b": 2, "c": 1003}}

As seen above, the expected behaviour is that new data with column 'a' being 3 should go to bucket1 and column 'a' being 4 should go to bucket0, but the partitioning is wrong.

Issue Links

• Is contained by: HIVE-26751 Bug Fixes and Improvements for 3.2.0 release
• links to: RB