Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-6201

Timeline server sometimes does not send bootstrap base path for a skeleton file

    XMLWordPrintableJSON

Details

    Description

      TestBootstrapRead.java In the attached file, enable the timeline serverĀ 

      'hoodie.embed.timeline.server'. It will occasionally fail in metadata or mixed mode because some records will be null besides the metadata columns:

      +-------------------+---------------------+------------------------------------+-------------------------+------------------------------------------------------------------------+------------------+------------------------------------+------------------+-------------------+-------------+------------+----------+------------------+----------+-------------------+-------------------+------------------------+-------------------+--------+----------+--------------+---------+--------------------+---------+--------------------------+----------+|_hoodie_commit_time|_hoodie_commit_seqno |_hoodie_record_key                  |_hoodie_partition_path   |_hoodie_file_name                                                       |_hoodie_is_deleted|_row_key                            |begin_lat         |begin_lon          |city_to_state|current_date|current_ts|distance_in_meters|driver    |end_lat            |end_lon            |fare                    |height             |nation  |partition |partition_path|rider    |seconds_since_epoch |timestamp|tip_history               |weight    |+-------------------+---------------------+------------------------------------+-------------------------+------------------------------------------------------------------------+------------------+------------------------------------+------------------+-------------------+-------------+------------+----------+------------------+----------+-------------------+-------------------+------------------------+-------------------+--------+----------+--------------+---------+--------------------+---------+--------------------------+----------+|00000000000001     |00000000000001_4_0   |876743b0-f5e7-4289-b13b-1a0404d94380|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_1   |00923d1a-58fc-4d42-8953-4a47b47d738f|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||20230510125841762  |20230510125841762_1_2|b318c482-8e43-4614-bdab-80946d5a9f53|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|false             |b318c482-8e43-4614-bdab-80946d5a9f53|0.5285807377766387|0.12835359814395741|[CA]         |12          |1047178778|521899450         |driver-001|0.41394620067559684|0.08532822423986208|[42.25978252084417, USD]|[0, 0, 7, -91, -36]|[Canada]|2015-03-16|2015-03-16    |rider-001|-2845295541651788027|0        |[[32.10533813167099, USD]]|0.59076524||00000000000001     |00000000000001_4_3   |4dcc72d7-0878-41c2-a85d-3e6374b88bb8|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_4   |cfa79530-fc9f-42de-a181-34c06e79d9c5|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_5   |4d2a4755-83a7-4201-9b65-0148752d55b7|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_6   |76429084-f78e-4c6d-a70d-088cb5d955aa|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_7   |8436456b-9858-45cb-8a07-bdae536a2d17|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      ||00000000000001     |00000000000001_4_8   |b6ca00ed-a4f4-4db5-a269-6be250b64caa|partition_path=2015-03-16|7e1dea56-e88c-4072-be61-f4ae01feaaa3_1-138-381_20230510125841762.parquet|null              |null                                |null              |null               |null         |null        |null      |null              |null      |null               |null               |null                    |null               |null    |null      |null          |null     |null                |null     |null                      |null      |+-------------------+---------------------+------------------------------------+-------------------------+------------------------------------------------------------------------+------------------+------------------------------------+------------------+-------------------+-------------+------------+----------+------------------+----------+-------------------+-------------------+------------------------+-------------------+--------+----------+--------------+---------+--------------------+---------+--------------------------+----------++-------------------+--------------------+------------------------------------+-------------------------+-------------------------------------------------------------------+ 

      This is due to the bootstrap base file not being set, so HoodieMergeHelper does not process the file as a bootstrap skeleton file

      Attachments

        1. TestBootstrapRead.java
          11 kB
          Jonathan Vexler
        2. base001.tar.gz
          10 kB
          Jonathan Vexler
        3. base.tar.gz
          42 kB
          Jonathan Vexler

        Issue Links

          Activity

            People

              jonvex Jonathan Vexler
              jonvex Jonathan Vexler
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: