[HADOOP-12469] distcp should not ignore the ignoreFailures option

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.1
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: tools/distcp
• Hadoop Flags: Reviewed
• Assignee: Mingliang Liu
• Reporter: Gera Shegalov
• Created: 09/Oct/15 01:30
• Updated: 30/Aug/16 01:22
• Resolved: 04/May/16 17:25

Description

RetriableFileCopyCommand.CopyReadException is double-wrapped via

1. via RetriableCommand::execute
2. via CopyMapper#copyFileWithRetry

before CopyMapper::handleFailure tests

if (ignoreFailures && exception.getCause() instanceof
            RetriableFileCopyCommand.CopyReadException

which is always false.

Orthogonally, ignoring failures should be mutually exclusive with the atomic option otherwise an incomplete dir is eligible for commit defeating the purpose.

Issue Links

• is cloned by: HADOOP-12473 distcp's ignoring failures option should be mutually exclusive with the atomic option