[SPARK-23003] Deterministic name to s3 partition while writing to s3 by Spark

• Type: Brainstorming
• Status: Resolved
• Priority: Minor
• Resolution: Invalid
• Affects Version/s: 2.1.1
• Component/s: Input/Output, Spark Core

Environment

aws emr-5.7.0

• Assignee: Unassigned
• Reporter: Dhruv sharma
• Created: 09/Jan/18 10:30
• Updated: 09/Jan/18 13:30
• Resolved: 09/Jan/18 13:30

Description

While writing to s3 how can one control the name of the partition being written to s3 i.e.
lets say df.write().json() writes 5 partitions with some hashed name in an S3 bucket/keyPath.
Is it possible that somehow that name is deterministic?