[HADOOP-19176] S3A Xattr headers need hdfs-compatible prefix

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0, 3.3.6
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 15/May/24 13:02
• Updated: 15/May/24 13:02

Description

x3a xattr list needs a prefix compatible with hdfs or existing code which tries to copy attributes between stores can break

we need a prefix of

{user/trusted/security/system/raw}

.

now, problem: currently xattrs are used by the magic committer to propagate file size progress; renaming the prefix will break existing code. But as it's read only we could modify spark to look for both old and new values.

org.apache.hadoop.HadoopIllegalArgumentException: An XAttr name must be prefixed with user/trusted/security/system/raw, followed by a '.'
	at org.apache.hadoop.hdfs.XAttrHelper.buildXAttr(XAttrHelper.java:77) 
	at org.apache.hadoop.hdfs.DFSClient.setXAttr(DFSClient.java:2835) 
	at org.apache.hadoop.hdfs.DistributedFileSystem$59.doCall(DistributedFileSystem.java:3106) 
	at org.apache.hadoop.hdfs.DistributedFileSystem$59.doCall(DistributedFileSystem.java:3102) 
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.setXAttr(DistributedFileSystem.java:3115) 
	at org.apache.hadoop.fs.FileSystem.setXAttr(FileSystem.java:3097)

Issue Links

• relates to: HIVE-25609 Preserve XAttrs in normal file copy case.