[HUDI-3818] hudi doesn't support bytes column as primary key

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: hive
• Labels: pull-request-available
• Story Points: 0.5
• Assignee: rex xiong
• Reporter: rex xiong

Reviewers

Shiyan Xu

• Created: 07/Apr/22 09:10
• Updated: 23/May/23 04:26
• Resolved: 29/Mar/23 10:46

Description

 when use bytes column as primary key, hudi will generate fixed hoodie key, then upserts will only insert one row. 

scala> sql("desc extended binary_test1").show()
+--------------------+--------------------+-------+
|            col_name|           data_type|comment|
+--------------------+--------------------+-------+
| _hoodie_commit_time|              string|   null|
|_hoodie_commit_seqno|              string|   null|
|  _hoodie_record_key|              string|   null|
|_hoodie_partition...|              string|   null|
|   _hoodie_file_name|              string|   null|
|                  id|              binary|   null|
|                name|              string|   null|
|                  dt|              string|   null|
|                    |                    |       |
|# Detailed Table ...|                    |       |
|            Database|             default|       |
|               Table|        binary_test1|       |
|               Owner|                root|       |
|        Created Time|Sat Apr 02 13:28:...|       |
|         Last Access|             UNKNOWN|       |
|          Created By|         Spark 3.2.0|       |
|                Type|             MANAGED|       |
|            Provider|                hudi|       |
|    Table Properties|[last_commit_time...|       |
|          Statistics|        435194 bytes|       |
+--------------------+--------------------+-------+

scala> sql("select * from binary_test1").show()
+-------------------+--------------------+--------------------+----------------------+--------------------+--------------------+---------+--------+
|_hoodie_commit_time|_hoodie_commit_seqno|  _hoodie_record_key|_hoodie_partition_path|   _hoodie_file_name|                  id|     name|      dt|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------------------+---------+--------+
|  20220402132927590|20220402132927590...|id:java.nio.HeapB...|                      |1a06106e-5e7a-4e6...|[03 45 6A 00 00 0...|Mary Jane|20220401|
+-------------------+--------------------+--------------------+----------------------+--------------------+--------------------+---------+--------+

Issue Links

• links to: GitHub Pull Request #5264