[HUDI-6201] Timeline server sometimes does not send bootstrap base path for a skeleton file

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: bootstrap, timeline-server
• Labels: pull-request-available
• Assignee: Jonathan Vexler
• Reporter: Jonathan Vexler
• Created: 11/May/23 05:10
• Updated: 26/May/23 17:37
• Resolved: 24/May/23 18:08

Description

TestBootstrapRead.java In the attached file, enable the timeline server 

'hoodie.embed.timeline.server'. It will occasionally fail in metadata or mixed mode because some records will be null besides the metadata columns:

+-------------------+---------------------+------------------------------------+-------------------------+------------------------------------------------------------------------+------------------+------------------------------------+------------------+-------------------+-------------+------------+----------+------------------+----------+-------------------+-------------------+------------------------+-------------------+--------+----------+--------------+---------+--------------------+---------+--------------------------+----------+|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key                  |_hoodie_partition_path   |_hoodie_file_name                                                       |_hoodie_is_deleted|_row_key                            |begin_lat         |begin_lon          |city_to_state|current_date|current_ts|distance_in_meters|driver    |end_lat            |end_lon            |fare                    |height             |nation  |partition |partition_path|rider    |seconds_since_epoch |timestamp|tip_history               |weight    |+-------------------+---------------------+------------------------------------+-------------------------+------------------------------------------------------------------------+------------------+------------------------------------+------------------+-------------------+-------------+------------+----------+------------------+----------+-------------------+-------------------+------------------------+-------------------+--------+----------+--------------+---------+--------------------+---------+--------------------------+----------+|00000000000001     |00000000000001_4_0   |876743b0-f5e7-4289-b13b-1a0404d94380|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_1   |00923d1a-58fc-4d42-8953-4a47b47d738f|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||20230510125841762  |20230510125841762_1_2|b318c482-8e43-4614-bdab-80946d5a9f53|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|false             |b318c482-8e43-4614-bdab-80946d5a9f53|0.5285807377766387|0.12835359814395741|[CA]         |12          |1047178778|521899450         |driver-001|0.41394620067559684|0.08532822423986208|[42.25978252084417, USD]|[0, 0, 7, -91, -36]|[Canada]|2015-03-16|2015-03-16    |rider-001|-2845295541651788027|0        |[[32.10533813167099, USD]]|0.59076524||00000000000001     |00000000000001_4_3   |4dcc72d7-0878-41c2-a85d-3e6374b88bb8|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_4   |cfa79530-fc9f-42de-a181-34c06e79d9c5|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_5   |4d2a4755-83a7-4201-9b65-0148752d55b7|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_6   |76429084-f78e-4c6d-a70d-088cb5d955aa|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_7   |8436456b-9858-45cb-8a07-bdae536a2d17|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_8   |b6ca00ed-a4f4-4db5-a269-6be250b64caa|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      |+-------------------+---------------------+------------------------------------+-------------------------+------------------------------------------------------------------------+------------------+------------------------------------+------------------+-------------------+-------------+------------+----------+------------------+----------+-------------------+-------------------+------------------------+-------------------+--------+----------+--------------+---------+--------------------+---------+--------------------------+----------++-------------------+--------------------+------------------------------------+-------------------------+-------------------------------------------------------------------+ 

This is due to the bootstrap base file not being set, so HoodieMergeHelper does not process the file as a bootstrap skeleton file

Issue Links

• relates to: HUDI-6249 Make maps in HoodieTableFileSystemView concurrent maps
• links to: GitHub Pull Request #8782