Description
LocalFS is a subclass of filterFS, but overrides create and open so that checksums are created and read.
MAPREDUCE-7184 has thrown up that the new builder openFile() call is being forwarded to the innerFS without CRC checking. Reviewing/fixing that has shown that some of the create methods aren't being correctly wrapped, so not generating CRCs
- createFile() builder
The following create calls
public FSDataOutputStream createNonRecursive(final Path f, final FsPermission permission, final EnumSet<CreateFlag> flags, final int bufferSize, final short replication, final long blockSize, final Progressable progress) throws IOException; public FSDataOutputStream create(final Path f, final FsPermission permission, final EnumSet<CreateFlag> flags, final int bufferSize, final short replication, final long blockSize, final Progressable progress, final Options.ChecksumOpt checksumOpt) throws IOException { return super.create(f, permission, flags, bufferSize, replication, blockSize, progress, checksumOpt); }
This means that applications using these methods, directly or indirectly to create files aren't actually generating checksums.
Fix: implement these methods & relay to local create calls, not to the inner FS.
Attachments
Attachments
Issue Links
- causes
-
MAPREDUCE-7184 TestJobCounters byte counters omitting crc file bytes read
- Resolved
- is related to
-
HADOOP-16150 checksumFS doesn't wrap concat(): concatenated files don't have checksums
- Resolved
-
HADOOP-16102 FilterFileSystem does not implement getScheme
- Resolved