[HUDI-3660] config hoodie.logfile.max.size not work

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Not A Bug
• Fix Version/s: 0.12.0
• Component/s: configs
• Assignee: Unassigned
• Reporter: YuAngZhang
• Created: 18/Mar/22 08:32
• Updated: 26/Jul/22 15:54
• Resolved: 26/Jul/22 15:54

Description

log file not rollover when file size more than 10M 

it seems the method HoodieLogFormatWriter.rolloverIfNeeded not work, the 

file system wrapped to HoodieWrapperFileSystem，the pos of 

FSDataInputStream  always set to 0

SET 'execution.checkpointing.interval' = '30min';
CREATE TABLE sink(
   role_id VARCHAR(20),
   log_id VARCHAR(10),
   origin_json string,
   ts TIMESTAMP(3),
   ds date,
   `ds` date
 )
 PARTITIONED BY (`ds`)
 WITH (
   'connector' = 'hudi',
   'path' = 'hdfs:///user/dl/hudi_nsh/',
   'table.type' = 'MERGE_ON_READ',
   'compaction.trigger.strategy'='num_commits',
   'compaction.delta_commits'='5',
   'hoodie.cleaner.commits.retained'='1',
   'hoodie.datasource.write.recordkey.field'='role_id,log_id,ts',
   'write.batch.size'='10',
   'hoodie.logfile.max.size'='10',
   'hive_sync.enable'='true',
   'hive_sync.mode' = 'hms',
   'hive_sync.metastore.uris' = 'thrift://fuxi-luoge-148:9083',
   'hive_sync.jdbc_url'='jdbc:hive2://',
   'hive_sync.table'='sink5',
   'hive_sync.db'='test',
   'hive_sync.username'='',
   'hive_sync.password'=''
 );