[HDFS-6984] Serialize FileStatus via protobuf

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-beta1
• Labels: BB2015-05-TBR
• Target Version/s: 3.0.0-beta1
• Hadoop Flags: Incompatible change, Reviewed

Release Note

FileStatus and FsPermission Writable serialization is deprecated and its implementation (incompatibly) replaced with protocol buffers. The FsPermissionProto record moved from hdfs.proto to acl.proto. HdfsFileStatus is now a subtype of FileStatus. FsPermissionExtension with its associated flags for ACLs, encryption, and erasure coding has been deprecated; users should query these attributes on the FileStatus object directly. The FsPermission instance in AclStatus no longer retains or reports these extended attributes (likely unused).

• Assignee: Christopher Douglas
• Reporter: Colin McCabe
• Created: 02/Sep/14 21:23
• Updated: 06/Oct/23 02:59
• Resolved: 02/Aug/17 19:31

Description

FileStatus was a Writable in Hadoop 2 and earlier. Originally, we used this to serialize it and send it over the wire. But in Hadoop 2 and later, we have the protobuf HdfsFileStatusProto which serves to serialize this information. The protobuf form is preferable, since it allows us to add new fields in a backwards-compatible way. Another issue is that already a lot of subclasses of FileStatus don't override the Writable methods of the superclass, breaking the interface contract that read(status.write) should be equal to the original status.

In Hadoop 3, we should just make FileStatus serialize itself via protobuf so that we don't have to deal with these issues. It's probably too late to do this in Hadoop 2, since user code may be relying on the existing FileStatus serialization there.

Issue Links

• blocks: HDFS-7878 API - expose a unique file identifier
• breaks: HADOOP-14730 Support protobuf FileStatus in AdlFileSystem
• breaks: SPARK-21817 Pass FSPermissions to LocatedFileStatus from InMemoryFileIndex
• causes: HADOOP-15289 FileStatus.readFields() assertion incorrect
• depends upon: HADOOP-13626 Remove distcp dependency on FileStatus serialization
• is related to: SPARK-45432 Remove deprecated Hadoop-2 `LocatedFileStatus` constructor
• is related to: HADOOP-13895 Make FileStatus Serializable
• is related to: HDFS-12250 Reduce usage of FsPermissionExtension in unit tests