[GOBBLIN-1399] provide a way to specify writer path and file name format via config

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.15.0
• Fix Version/s: 0.17.0
• Component/s: gobblin-api
• Assignee: Hung Tran
• Reporter: Jay Sen
• Created: 27/Feb/21 22:27
• Updated: 14/Jun/23 04:13

Description

currently gobblin has hard coded specification for writer's path and file name

primarily it has namespace and tablename and default - 3 way to have writer's path and file name.

// code placeholder

switch (getWriterFilePathType(state)) {
  case NAMESPACE_TABLE:
    // writer.file.path.format = <extract.namespace>/<extract.table.name>/
    return getNamespaceTableWriterFilePath(state);
  case TABLENAME:
    // <extract.table.name>
    return WriterUtils.getTableNameWriterFilePath(state);
  default:
    return WriterUtils.getDefaultWriterFilePath(state, numBranches, branchId);
}

 

 Filename:

namespace.replaceAll("\\.", "/") + "/" + table + "/" + extractId + "_"
    + (isFull ? "full" : "append");

 

There is no way user can add any other parameters like version, batchId.

Also it would be awesome to have any configuration value to be part of the writer path, which can be defined by the format like this

 

Unable to find source-code formatter for language: java. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yamlextract.type = increments
writer.file.path.format="<extract.table.name>/<extract.extract.id>/<extract.type>"
writer.file.name.format="part.<writer_id>_batch_<dataset.batch_id>.<branch_id>.<format_extension>"

Notice the values (like "dataset.batch_id" comes from the runtime config( state.getProp() ), so it allows you to have any kind of flexible path and file name based on your use-case.

This will be enabled by the feature flag, so existing functionality can remains the same.