[HUDI-723] SqlTransformer's schema sometimes is not registered.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: deltastreamer
• Labels: bug-bash-0.6.0 pull-request-available
• Assignee: hong dongdong
• Reporter: Alexander Filipchik
• Created: 19/Mar/20 02:41
• Updated: 16/May/20 09:39
• Resolved: 16/May/20 09:37

Description

If schema is inferred from RowBasedSchemaProvider when SQL transformer is used it also needs to be registered. 

 

Current way only works if SchemaProvider has a valid target schema. Is one wants to use schema from SQL transformation, the result of RowBasedSchemaProvider.getTargetSchema needs to be passed into something like:

private void setupWriteClient(SchemaProvider schemaProvider) {
  LOG.info("Setting up Hoodie Write Client");
  registerAvroSchemas(schemaProvider);
  HoodieWriteConfig hoodieCfg = getHoodieClientConfig(schemaProvider);
  writeClient = new HoodieWriteClient<>(jssc, hoodieCfg, true);
  onInitializingHoodieWriteClient.apply(writeClient);
}

Existent method will not work as it is checking for:

if ((null != schemaProvider) && (null == writeClient)) {

and writeClient is already configured. 

 

Issue Links

• is depended upon by: HUDI-901 Bug Bash 0.6.0 Tracking Ticket
• links to: GitHub Pull Request #1518