[GOBBLIN-130] FsRenameCommitStep.execute doesn't perform same steps as ParallelRunner.movePath

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: Bug:Generic Core:Publisher
• External issue URL: https://github.com/linkedin/gobblin/issues/781
• Assignee: Unassigned
• Reporter: Joel Baranick
• Created: 19/Jul/17 07:40
• Updated: 07/Aug/17 08:35

Description

When exactly-once job publishing is enabled `FsRenameCommitStep` is used to ensure that the files make it to the destination. However, it doesn't result in the same state as the method used when exactly-once job publishing is disabled: `ParallelRunner.movePath`. The `ParallelRunner.movePath` method both moves the path and sets the group, but `FsRenameCommitStep.execute` only moves the file.

1. 1. 1. ParallelRunner.movePath

``` java
if (fs.exists(src)) {
HadoopUtils.movePath(fs, src, dstFs, dst, overwrite, dstFs.getConf());
if (group.isPresent())

{ HadoopUtils.setGroup(dstFs, dst, group.get()); }

}
```

1. 1. 1. FsRenameCommitStep.execute

``` java
log.info(String.format(Moving %s to %s, this.srcPath, this.dstPath));
HadoopUtils.movePath(this.srcFs, this.srcPath, this.dstFs, this.dstPath, this.dstFs.getConf());
```

Github Url : https://github.com/linkedin/gobblin/issues/781
Github Reporter : jbaranick
Github Created At : 2016-03-01T16:44:23Z
Github Updated At : 2017-01-12T04:44:04Z

Comments

---

jbaranick wrote on 2016-03-02T13:52:27Z : @zliu41 you might want to look at this one

Github Url : https://github.com/linkedin/gobblin/issues/781#issuecomment-191246073

---

zliu41 wrote on 2016-03-02T16:20:10Z : Good catch. It's an easy fix but we'll probably do some surgery to exactly-once in the next few weeks so this will be fixed as part of that.

Github Url : https://github.com/linkedin/gobblin/issues/781#issuecomment-191310800