[HUDI-3322] Rollback Plan for Delta Commits constructed incorrectly - ASF JIRA

Details

Type: Bug
Status: Closed
Priority: Blocker
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.11.0
Component/s: metadata
Labels:
- pull-request-available

Story Points:
0
Epic Link:
Read-Path-Rewrite

Description

Diving deeper into the issue of ~~HUDI-3279~~, i've realized that the root-cause of the problem is actually a Rollback Plan for Delta Commits is composed incorrectly for MOR tables. Consider the case below (we will continue to rely on test of TestHoodieSparkMergeOnReadTableRollback#testMORTableRestore):

Hoodie Timeline:

alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -la .hoodie
total 400
drwxr-xr-x  52 alexey.kudinkin  staff  1664 Jan 25 13:08 .
drwx------   5 alexey.kudinkin  staff   160 Jan 25 12:56 ..
-rw-r--r--   1 alexey.kudinkin  staff    48 Jan 25 12:56 .001.deltacommit.crc
-rw-r--r--   1 alexey.kudinkin  staff    28 Jan 25 12:56 .001.deltacommit.inflight.crc
-rw-r--r--   1 alexey.kudinkin  staff     8 Jan 25 12:56 .001.deltacommit.requested.crc
-rw-r--r--   1 alexey.kudinkin  staff    52 Jan 25 12:56 .002.deltacommit.crc
-rw-r--r--   1 alexey.kudinkin  staff    48 Jan 25 12:56 .002.deltacommit.inflight.crc
-rw-r--r--   1 alexey.kudinkin  staff     8 Jan 25 12:56 .002.deltacommit.requested.crc
-rw-r--r--   1 alexey.kudinkin  staff    56 Jan 25 12:57 .003.deltacommit.crc
-rw-r--r--   1 alexey.kudinkin  staff    48 Jan 25 12:57 .003.deltacommit.inflight.crc
-rw-r--r--   1 alexey.kudinkin  staff     8 Jan 25 12:56 .003.deltacommit.requested.crc
-rw-r--r--   1 alexey.kudinkin  staff    56 Jan 25 12:57 .004.deltacommit.crc
-rw-r--r--   1 alexey.kudinkin  staff    48 Jan 25 12:57 .004.deltacommit.inflight.crc
-rw-r--r--   1 alexey.kudinkin  staff     8 Jan 25 12:57 .004.deltacommit.requested.crc
-rw-r--r--   1 alexey.kudinkin  staff    48 Jan 25 12:57 .005.commit.crc
-rw-r--r--   1 alexey.kudinkin  staff     8 Jan 25 12:57 .005.compaction.inflight.crc
-rw-r--r--   1 alexey.kudinkin  staff    28 Jan 25 12:57 .005.compaction.requested.crc
-rw-r--r--   1 alexey.kudinkin  staff    52 Jan 25 12:57 .006.deltacommit.crc
-rw-r--r--   1 alexey.kudinkin  staff    48 Jan 25 12:57 .006.deltacommit.inflight.crc
-rw-r--r--   1 alexey.kudinkin  staff     8 Jan 25 12:57 .006.deltacommit.requested.crc
-rw-r--r--   1 alexey.kudinkin  staff    52 Jan 25 12:57 .007.deltacommit.crc
-rw-r--r--   1 alexey.kudinkin  staff    48 Jan 25 12:57 .007.deltacommit.inflight.crc
-rw-r--r--   1 alexey.kudinkin  staff     8 Jan 25 12:57 .007.deltacommit.requested.crc
-rw-r--r--   1 alexey.kudinkin  staff     8 Jan 25 13:08 .20220125130818473.restore.inflight.crc
drwxr-xr-x   5 alexey.kudinkin  staff   160 Jan 25 12:57 .aux
-rw-r--r--   1 alexey.kudinkin  staff    12 Jan 25 12:56 .hoodie.properties.crc
drwxr-xr-x   2 alexey.kudinkin  staff    64 Jan 25 12:57 .temp
-rw-r--r--   1 alexey.kudinkin  staff  4822 Jan 25 12:56 001.deltacommit
-rw-r--r--   1 alexey.kudinkin  staff  2499 Jan 25 12:56 001.deltacommit.inflight
-rw-r--r--   1 alexey.kudinkin  staff     0 Jan 25 12:56 001.deltacommit.requested
-rw-r--r--   1 alexey.kudinkin  staff  5451 Jan 25 12:56 002.deltacommit
-rw-r--r--   1 alexey.kudinkin  staff  4620 Jan 25 12:56 002.deltacommit.inflight
-rw-r--r--   1 alexey.kudinkin  staff     0 Jan 25 12:56 002.deltacommit.requested
-rw-r--r--   1 alexey.kudinkin  staff  5646 Jan 25 12:57 003.deltacommit
-rw-r--r--   1 alexey.kudinkin  staff  4620 Jan 25 12:57 003.deltacommit.inflight
-rw-r--r--   1 alexey.kudinkin  staff     0 Jan 25 12:56 003.deltacommit.requested
-rw-r--r--   1 alexey.kudinkin  staff  5835 Jan 25 12:57 004.deltacommit
-rw-r--r--   1 alexey.kudinkin  staff  4620 Jan 25 12:57 004.deltacommit.inflight
-rw-r--r--   1 alexey.kudinkin  staff     0 Jan 25 12:57 004.deltacommit.requested
-rw-r--r--   1 alexey.kudinkin  staff  4756 Jan 25 12:57 005.commit
-rw-r--r--   1 alexey.kudinkin  staff     0 Jan 25 12:57 005.compaction.inflight
-rw-r--r--   1 alexey.kudinkin  staff  2507 Jan 25 12:57 005.compaction.requested
-rw-r--r--   1 alexey.kudinkin  staff  5362 Jan 25 12:57 006.deltacommit
-rw-r--r--   1 alexey.kudinkin  staff  4620 Jan 25 12:57 006.deltacommit.inflight
-rw-r--r--   1 alexey.kudinkin  staff     0 Jan 25 12:57 006.deltacommit.requested
-rw-r--r--   1 alexey.kudinkin  staff  5551 Jan 25 12:57 007.deltacommit
-rw-r--r--   1 alexey.kudinkin  staff  4620 Jan 25 12:57 007.deltacommit.inflight
-rw-r--r--   1 alexey.kudinkin  staff     0 Jan 25 12:57 007.deltacommit.requested
-rw-r--r--   1 alexey.kudinkin  staff     0 Jan 25 13:08 20220125130818473.restore.inflight
drwxr-xr-x   2 alexey.kudinkin  staff    64 Jan 25 12:56 archived
-rw-r--r--   1 alexey.kudinkin  staff   347 Jan 25 12:56 hoodie.properties
drwxr-xr-x   4 alexey.kudinkin  staff   128 Jan 25 12:56 metadata

Data Folder layout:

alexey.kudinkin@alexeys-mbp junit5494198038159268501 % ls -al **/**
-rw-r--r--  1 alexey.kudinkin  staff  448853 Jan 25 12:57 2015/03/16/0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-200-322_005.parquet
-rw-r--r--  1 alexey.kudinkin  staff  448895 Jan 25 12:56 2015/03/16/0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-26-43_001.parquet
-rw-r--r--  1 alexey.kudinkin  staff  449458 Jan 25 12:57 2015/03/17/52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_0-200-321_005.parquet
-rw-r--r--  1 alexey.kudinkin  staff  449507 Jan 25 12:56 2015/03/17/52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_2-26-44_001.parquet
-rw-r--r--  1 alexey.kudinkin  staff  447405 Jan 25 12:56 2016/03/15/a685909e-14e8-4adf-b6a2-e0b669e89e61-0_0-26-42_001.parquet
-rw-r--r--  1 alexey.kudinkin  staff  447384 Jan 25 12:57 2016/03/15/a685909e-14e8-4adf-b6a2-e0b669e89e61-0_2-200-323_005.parquet2015:
total 0
drwxr-xr-x  3 alexey.kudinkin  staff   96 Jan 25 12:56 .
drwx------  5 alexey.kudinkin  staff  160 Jan 25 12:56 ..
drwxr-xr-x  4 alexey.kudinkin  staff  128 Jan 25 12:56 032015/03:
total 0
drwxr-xr-x   4 alexey.kudinkin  staff  128 Jan 25 12:56 .
drwxr-xr-x   3 alexey.kudinkin  staff   96 Jan 25 12:56 ..
drwxr-xr-x  18 alexey.kudinkin  staff  576 Jan 25 12:57 16
drwxr-xr-x  18 alexey.kudinkin  staff  576 Jan 25 12:57 172015/03/16:
total 2072
drwxr-xr-x  18 alexey.kudinkin  staff     576 Jan 25 12:57 .
drwxr-xr-x   4 alexey.kudinkin  staff     128 Jan 25 12:56 ..
-rw-r--r--   1 alexey.kudinkin  staff     176 Jan 25 12:56 ..0ac88bb7-ebd4-4384-94f1-593e6827a868-0_001.log.1_1-89-152.crc
-rw-r--r--   1 alexey.kudinkin  staff     176 Jan 25 12:57 ..0ac88bb7-ebd4-4384-94f1-593e6827a868-0_001.log.2_1-134-223.crc
-rw-r--r--   1 alexey.kudinkin  staff     176 Jan 25 12:57 ..0ac88bb7-ebd4-4384-94f1-593e6827a868-0_001.log.3_1-181-296.crc
-rw-r--r--   1 alexey.kudinkin  staff     176 Jan 25 12:57 ..0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399.crc
-rw-r--r--   1 alexey.kudinkin  staff     176 Jan 25 12:57 ..0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472.crc
-rw-r--r--   1 alexey.kudinkin  staff      12 Jan 25 12:56 ..hoodie_partition_metadata.crc
-rw-r--r--   1 alexey.kudinkin  staff   21415 Jan 25 12:56 .0ac88bb7-ebd4-4384-94f1-593e6827a868-0_001.log.1_1-89-152
-rw-r--r--   1 alexey.kudinkin  staff   21462 Jan 25 12:57 .0ac88bb7-ebd4-4384-94f1-593e6827a868-0_001.log.2_1-134-223
-rw-r--r--   1 alexey.kudinkin  staff   21468 Jan 25 12:57 .0ac88bb7-ebd4-4384-94f1-593e6827a868-0_001.log.3_1-181-296
-rw-r--r--   1 alexey.kudinkin  staff   21464 Jan 25 12:57 .0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399
-rw-r--r--   1 alexey.kudinkin  staff   21457 Jan 25 12:57 .0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472
-rw-r--r--   1 alexey.kudinkin  staff    3516 Jan 25 12:57 .0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-200-322_005.parquet.crc
-rw-r--r--   1 alexey.kudinkin  staff    3516 Jan 25 12:56 .0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-26-43_001.parquet.crc
-rw-r--r--   1 alexey.kudinkin  staff      82 Jan 25 12:56 .hoodie_partition_metadata
-rw-r--r--   1 alexey.kudinkin  staff  448853 Jan 25 12:57 0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-200-322_005.parquet
-rw-r--r--   1 alexey.kudinkin  staff  448895 Jan 25 12:56 0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-26-43_001.parquet2015/03/17:
total 2072
drwxr-xr-x  18 alexey.kudinkin  staff     576 Jan 25 12:57 .
drwxr-xr-x   4 alexey.kudinkin  staff     128 Jan 25 12:56 ..
-rw-r--r--   1 alexey.kudinkin  staff     188 Jan 25 12:56 ..52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_001.log.1_2-89-153.crc
-rw-r--r--   1 alexey.kudinkin  staff     188 Jan 25 12:57 ..52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_001.log.2_2-134-224.crc
-rw-r--r--   1 alexey.kudinkin  staff     188 Jan 25 12:57 ..52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_001.log.3_2-181-297.crc
-rw-r--r--   1 alexey.kudinkin  staff     188 Jan 25 12:57 ..52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400.crc
-rw-r--r--   1 alexey.kudinkin  staff     188 Jan 25 12:57 ..52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473.crc
-rw-r--r--   1 alexey.kudinkin  staff      12 Jan 25 12:56 ..hoodie_partition_metadata.crc
-rw-r--r--   1 alexey.kudinkin  staff    3520 Jan 25 12:57 .52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_0-200-321_005.parquet.crc
-rw-r--r--   1 alexey.kudinkin  staff   22857 Jan 25 12:56 .52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_001.log.1_2-89-153
-rw-r--r--   1 alexey.kudinkin  staff   22877 Jan 25 12:57 .52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_001.log.2_2-134-224
-rw-r--r--   1 alexey.kudinkin  staff   22892 Jan 25 12:57 .52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_001.log.3_2-181-297
-rw-r--r--   1 alexey.kudinkin  staff   22881 Jan 25 12:57 .52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400
-rw-r--r--   1 alexey.kudinkin  staff   22889 Jan 25 12:57 .52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473
-rw-r--r--   1 alexey.kudinkin  staff    3520 Jan 25 12:56 .52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_2-26-44_001.parquet.crc
-rw-r--r--   1 alexey.kudinkin  staff      82 Jan 25 12:56 .hoodie_partition_metadata
-rw-r--r--   1 alexey.kudinkin  staff  449458 Jan 25 12:57 52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_0-200-321_005.parquet
-rw-r--r--   1 alexey.kudinkin  staff  449507 Jan 25 12:56 52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_2-26-44_001.parquet2016:
total 0
drwxr-xr-x  3 alexey.kudinkin  staff   96 Jan 25 12:56 .
drwx------  5 alexey.kudinkin  staff  160 Jan 25 12:56 ..
drwxr-xr-x  3 alexey.kudinkin  staff   96 Jan 25 12:56 032016/03:
total 0
drwxr-xr-x   3 alexey.kudinkin  staff   96 Jan 25 12:56 .
drwxr-xr-x   3 alexey.kudinkin  staff   96 Jan 25 12:56 ..
drwxr-xr-x  18 alexey.kudinkin  staff  576 Jan 25 12:57 152016/03/15:
total 2032
drwxr-xr-x  18 alexey.kudinkin  staff     576 Jan 25 12:57 .
drwxr-xr-x   3 alexey.kudinkin  staff      96 Jan 25 12:56 ..
-rw-r--r--   1 alexey.kudinkin  staff     152 Jan 25 12:56 ..a685909e-14e8-4adf-b6a2-e0b669e89e61-0_001.log.1_0-89-151.crc
-rw-r--r--   1 alexey.kudinkin  staff     152 Jan 25 12:57 ..a685909e-14e8-4adf-b6a2-e0b669e89e61-0_001.log.2_0-134-222.crc
-rw-r--r--   1 alexey.kudinkin  staff     152 Jan 25 12:57 ..a685909e-14e8-4adf-b6a2-e0b669e89e61-0_001.log.3_0-181-295.crc
-rw-r--r--   1 alexey.kudinkin  staff     152 Jan 25 12:57 ..a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398.crc
-rw-r--r--   1 alexey.kudinkin  staff     152 Jan 25 12:57 ..a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471.crc
-rw-r--r--   1 alexey.kudinkin  staff      12 Jan 25 12:56 ..hoodie_partition_metadata.crc
-rw-r--r--   1 alexey.kudinkin  staff    3504 Jan 25 12:56 .a685909e-14e8-4adf-b6a2-e0b669e89e61-0_0-26-42_001.parquet.crc
-rw-r--r--   1 alexey.kudinkin  staff   17995 Jan 25 12:56 .a685909e-14e8-4adf-b6a2-e0b669e89e61-0_001.log.1_0-89-151
-rw-r--r--   1 alexey.kudinkin  staff   18021 Jan 25 12:57 .a685909e-14e8-4adf-b6a2-e0b669e89e61-0_001.log.2_0-134-222
-rw-r--r--   1 alexey.kudinkin  staff   18017 Jan 25 12:57 .a685909e-14e8-4adf-b6a2-e0b669e89e61-0_001.log.3_0-181-295
-rw-r--r--   1 alexey.kudinkin  staff   18022 Jan 25 12:57 .a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398
-rw-r--r--   1 alexey.kudinkin  staff   18054 Jan 25 12:57 .a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471
-rw-r--r--   1 alexey.kudinkin  staff    3504 Jan 25 12:57 .a685909e-14e8-4adf-b6a2-e0b669e89e61-0_2-200-323_005.parquet.crc
-rw-r--r--   1 alexey.kudinkin  staff      82 Jan 25 12:56 .hoodie_partition_metadata
-rw-r--r--   1 alexey.kudinkin  staff  447405 Jan 25 12:56 a685909e-14e8-4adf-b6a2-e0b669e89e61-0_0-26-42_001.parquet
-rw-r--r--   1 alexey.kudinkin  staff  447384 Jan 25 12:57 a685909e-14e8-4adf-b6a2-e0b669e89e61-0_2-200-323_005.parquet

Commit "007" metadata:

alexey.kudinkin@alexeys-mbp junit5494198038159268501 % cat .hoodie/007.deltacommit
{
  "partitionToWriteStats" : {
    "2016/03/15" : [ {
      "fileId" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
      "path" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471",
      "prevCommit" : "005",
      "numWrites" : 57,
      "numDeletes" : 0,
      "numUpdateWrites" : 57,
      "numInserts" : 0,
      "totalWriteBytes" : 18054,
      "totalWriteErrors" : 0,
      "tempPath" : null,
      "partitionPath" : "2016/03/15",
      "totalLogRecords" : 0,
      "totalLogFilesCompacted" : 0,
      "totalLogSizeCompacted" : 0,
      "totalUpdatedRecordsCompacted" : 0,
      "totalLogBlocks" : 0,
      "totalCorruptLogBlock" : 0,
      "totalRollbackBlocks" : 0,
      "fileSizeInBytes" : 18054,
      "minEventTime" : null,
      "maxEventTime" : null,
      "logVersion" : 2,
      "logOffset" : 0,
      "baseFile" : "a685909e-14e8-4adf-b6a2-e0b669e89e61-0_2-200-323_005.parquet",
      "logFiles" : [ ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398", ".a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471" ]
    } ],
    "2015/03/16" : [ {
      "fileId" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
      "path" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
      "prevCommit" : "005",
      "numWrites" : 69,
      "numDeletes" : 0,
      "numUpdateWrites" : 69,
      "numInserts" : 0,
      "totalWriteBytes" : 21457,
      "totalWriteErrors" : 0,
      "tempPath" : null,
      "partitionPath" : "2015/03/16",
      "totalLogRecords" : 0,
      "totalLogFilesCompacted" : 0,
      "totalLogSizeCompacted" : 0,
      "totalUpdatedRecordsCompacted" : 0,
      "totalLogBlocks" : 0,
      "totalCorruptLogBlock" : 0,
      "totalRollbackBlocks" : 0,
      "fileSizeInBytes" : 21457,
      "minEventTime" : null,
      "maxEventTime" : null,
      "logVersion" : 2,
      "logOffset" : 0,
      "baseFile" : "0ac88bb7-ebd4-4384-94f1-593e6827a868-0_1-200-322_005.parquet",
      "logFiles" : [ ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399", ".0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472" ]
    } ],
    "2015/03/17" : [ {
      "fileId" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
      "path" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
      "prevCommit" : "005",
      "numWrites" : 74,
      "numDeletes" : 0,
      "numUpdateWrites" : 74,
      "numInserts" : 0,
      "totalWriteBytes" : 22889,
      "totalWriteErrors" : 0,
      "tempPath" : null,
      "partitionPath" : "2015/03/17",
      "totalLogRecords" : 0,
      "totalLogFilesCompacted" : 0,
      "totalLogSizeCompacted" : 0,
      "totalUpdatedRecordsCompacted" : 0,
      "totalLogBlocks" : 0,
      "totalCorruptLogBlock" : 0,
      "totalRollbackBlocks" : 0,
      "fileSizeInBytes" : 22889,
      "minEventTime" : null,
      "maxEventTime" : null,
      "logVersion" : 2,
      "logOffset" : 0,
      "baseFile" : "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_0-200-321_005.parquet",
      "logFiles" : [ ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400", ".52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473" ]
    } ]
  },
  "compacted" : false,
  "extraMetadata" : {
    "schema" : "{\"type\":\"record\",\"name\":\"triprec\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\"},{\"name\":\"_row_key\",\"type\":\"string\"},{\"name\":\"partition_path\",\"type\":\"string\"},{\"name\":\"rider\",\"type\":\"string\"},{\"name\":\"driver\",\"type\":\"string\"},{\"name\":\"begin_lat\",\"type\":\"double\"},{\"name\":\"begin_lon\",\"type\":\"double\"},{\"name\":\"end_lat\",\"type\":\"double\"},{\"name\":\"end_lon\",\"type\":\"double\"},{\"name\":\"distance_in_meters\",\"type\":\"int\"},{\"name\":\"seconds_since_epoch\",\"type\":\"long\"},{\"name\":\"weight\",\"type\":\"float\"},{\"name\":\"nation\",\"type\":\"bytes\"},{\"name\":\"current_date\",\"type\":{\"type\":\"int\",\"logicalType\":\"date\"}},{\"name\":\"current_ts\",\"type\":\"long\"},{\"name\":\"height\",\"type\":{\"type\":\"fixed\",\"name\":\"abc\",\"size\":5,\"logicalType\":\"decimal\",\"precision\":10,\"scale\":6}},{\"name\":\"city_to_state\",\"type\":{\"type\":\"map\",\"values\":\"string\"}},{\"name\":\"fare\",\"type\":{\"type\":\"record\",\"name\":\"fare\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}]}},{\"name\":\"tip_history\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"tip_history\",\"fields\":[{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\"}],\"default\":null},\"default\":[]},\"default\":[]},{\"name\":\"_hoodie_is_deleted\",\"type\":\"boolean\",\"default\":false}]}"
  },
  "operationType" : "UPSERT",
  "totalLogFilesCompacted" : 0,
  "fileIdAndRelativePaths" : {
    "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0" : "2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473",
    "0ac88bb7-ebd4-4384-94f1-593e6827a868-0" : "2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472",
    "a685909e-14e8-4adf-b6a2-e0b669e89e61-0" : "2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471"
  },
  "totalLogRecordsCompacted" : 0,
  "totalCompactedRecordsUpdated" : 0,
  "totalRecordsDeleted" : 0,
  "totalLogFilesSize" : 0,
  "totalScanTime" : 0,
  "totalCreateTime" : 0,
  "totalUpsertTime" : 139,
  "minAndMaxEventTime" : {
    "Optional.empty" : {
      "val" : null,
      "present" : false
    }
  },
  "writePartitionPaths" : [ "2016/03/15", "2015/03/16", "2015/03/17" ]
}%

From the metadata it's not hard to see that only 3 files have changed as part of "007" commit:

"2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473" 
"2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472"
"2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471"

However, rollback of the "007" instant produces following plan:

{
  "instantToRollback": {
    "commitTime": "007",
    "action": "deltacommit"
  },
  "RollbackRequests": [
    {
      "partitionPath": "2015/03/16",
      "fileId": "",
      "latestBaseInstant": "",
      "filesToBeDeleted": [],
      "logBlocksToBeDeleted": {}
    },
    {
      "partitionPath": "2015/03/16",
      "fileId": "0ac88bb7-ebd4-4384-94f1-593e6827a868-0",
      "latestBaseInstant": "005",
      "filesToBeDeleted": [],
      "logBlocksToBeDeleted": {
        "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.1_1-247-399": 21464,
        "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/16/.0ac88bb7-ebd4-4384-94f1-593e6827a868-0_005.log.2_1-294-472": 21457
      }
    },
    {
      "partitionPath": "2015/03/17",
      "fileId": "",
      "latestBaseInstant": "",
      "filesToBeDeleted": [],
      "logBlocksToBeDeleted": {}
    },
    {
      "partitionPath": "2015/03/17",
      "fileId": "52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0",
      "latestBaseInstant": "005",
      "filesToBeDeleted": [],
      "logBlocksToBeDeleted": {
        "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.2_2-294-473": 22889,
        "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2015/03/17/.52b88ea6-09eb-4ab9-9f76-bfbf7cd5362f-0_005.log.1_2-247-400": 22881
      }
    },
    {
      "partitionPath": "2016/03/15",
      "fileId": "",
      "latestBaseInstant": "",
      "filesToBeDeleted": [],
      "logBlocksToBeDeleted": {}
    },
    {
      "partitionPath": "2016/03/15",
      "fileId": "a685909e-14e8-4adf-b6a2-e0b669e89e61-0",
      "latestBaseInstant": "005",
      "filesToBeDeleted": [],
      "logBlocksToBeDeleted": {
        "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.1_0-247-398": 18022,
        "file:/var/folders/kb/cnff55vj041g2nnlzs5ylqk00000gn/T/junit5494198038159268501/2016/03/15/.a685909e-14e8-4adf-b6a2-e0b669e89e61-0_005.log.2_0-294-471": 18054
      }
    }
  ],
  "version": 1
}

Which contains files that have not been changed in the "007" instant at all.

Attachments

Issue Links

causes

HUDI-3279 Metadata table stores incorrect file sizes after Restore

Closed

relates to

HUDI-3462 List of fixes to Metadata table after 0.10.1

Closed

Testing discovered

HUDI-3342 MOR Delta Block Rollbacks not applied if Lazy Block reading is disabled

Open

HUDI-3343 Metadata Table includes Uncommitted Log Files during Bootstrap

Closed

links to

GitHub Pull Request #4716

Rollback Plan for Delta Commits constructed incorrectly

Details

Description

Attachments

Issue Links

Activity

People

Dates