[IMPALA-12342] Erasure coding build fails on loading iceberg_lineitem_multiblock

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.2.0
• Fix Version/s: Impala 4.3.0
• Component/s: Infrastructure
• Labels: test-failure
• Environment: ERASURE_CODING=true
• Epic Color: ghx-label-2
• Assignee: Zoltán Borók-Nagy
• Reporter: Peter Rozsa
• Created: 07/Aug/23 13:15
• Updated: 26/Mar/24 22:49
• Resolved: 26/Mar/24 04:43

Description

Erasure coding build fails at statement generation:

 

Traceback (most recent call last): 
 File "/home/impdev/impala/testdata/bin/generate-schema-statements.py", line 935, in <module> 
   convert_orc_to_full_acid) 
 File "/home/impdev/impala/testdata/bin/generate-schema-statements.py", line 678, in generate_statements 
   insert = eval_section(section['DEPENDENT_LOAD']) 
 File "/home/impdev/impala/testdata/bin/generate-schema-statements.py", line 612, in eval_section 
   assert p.returncode == 0 
AssertionError

 

Error message: 

 

put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]). 
put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]). 
put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]). 
put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]). 
put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]). 
put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]). 
put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]). 
put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]). 
put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]). 
put: Specified block size (524288) is less than the cell size (1048576) of the erasure coding policy (ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2]).

 

IMPALA-12327 added a specific block size setting for copying files from iceberg_lineitem_multiblock. Without the blocksize setting, the put operation succeeds.

Issue Links

• is caused by: IMPALA-11619 Improve Iceberg V2 reads with a custom Iceberg Position Delete operator