[HUDI-4734] Add table config change validation in deltastreamer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.1
• Component/s: deltastreamer
• Labels: hudi-on-call pull-request-available
• Story Points: 2
• Assignee: Jonathan Vexler
• Reporter: sivabalan narayanan
• Created: 29/Aug/22 13:57
• Updated: 29/Sep/22 02:54
• Resolved: 29/Sep/22 02:54

Description

looks like we are missing proper table config validation in deltastreamer. 

   if (fs.exists(new Path(cfg.targetBasePath))) {
        HoodieTableMetaClient meta =
            HoodieTableMetaClient.builder().setConf(new Configuration(fs.getConf())).setBasePath(cfg.targetBasePath).setLoadActiveTimelineOnLoad(false).build();
        tableType = meta.getTableType();
        // This will guarantee there is no surprise with table type
        ValidationUtils.checkArgument(tableType.equals(HoodieTableType.valueOf(cfg.tableType)),
            "Hoodie table is of type " + tableType + " but passed in CLI argument is " + cfg.tableType);

        // Load base file format
        // This will guarantee there is no surprise with base file type
        String baseFileFormat = meta.getTableConfig().getBaseFileFormat().toString();
        ValidationUtils.checkArgument(baseFileFormat.equals(cfg.baseFileFormat) || cfg.baseFileFormat == null,
            "Hoodie table's base file format is of type " + baseFileFormat + " but passed in CLI argument is "
                + cfg.baseFileFormat);
        cfg.baseFileFormat = baseFileFormat;
        this.cfg.baseFileFormat = baseFileFormat;
      } else {
        tableType = HoodieTableType.valueOf(cfg.tableType);
        if (cfg.baseFileFormat == null) {
          cfg.baseFileFormat = "PARQUET"; // default for backward compatibility
        }
      } 

Issue Links

• links to: GitHub Pull Request #6753
• links to: issue-6487