[HADOOP-16107] FilterFileSystem doesn't wrap all create() or new builder calls; may skip CRC logic

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.3, 3.3.0
• Fix Version/s: 3.3.0
• Component/s: fs
• Target Version/s: 3.3.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 12/Feb/19 17:06
• Updated: 27/Feb/19 22:32
• Resolved: 27/Feb/19 20:58

Description

LocalFS is a subclass of filterFS, but overrides create and open so that checksums are created and read.

MAPREDUCE-7184 has thrown up that the new builder openFile() call is being forwarded to the innerFS without CRC checking. Reviewing/fixing that has shown that some of the create methods aren't being correctly wrapped, so not generating CRCs

• createFile() builder

The following create calls

  public FSDataOutputStream createNonRecursive(final Path f,
      final FsPermission permission,
      final EnumSet<CreateFlag> flags,
      final int bufferSize,
      final short replication,
      final long blockSize,
      final Progressable progress) throws IOException;

  public FSDataOutputStream create(final Path f,
      final FsPermission permission,
      final EnumSet<CreateFlag> flags,
      final int bufferSize,
      final short replication,
      final long blockSize,
      final Progressable progress,
      final Options.ChecksumOpt checksumOpt) throws IOException {
    return super.create(f, permission, flags, bufferSize, replication,
        blockSize, progress, checksumOpt);
  }

This means that applications using these methods, directly or indirectly to create files aren't actually generating checksums.

Fix: implement these methods & relay to local create calls, not to the inner FS.

Issue Links

• causes: MAPREDUCE-7184 TestJobCounters byte counters omitting crc file bytes read
• is related to: HADOOP-16150 checksumFS doesn't wrap concat(): concatenated files don't have checksums
• is related to: HADOOP-16102 FilterFileSystem does not implement getScheme