[HUDI-2387] Too many HEAD requests from Hudi to S3

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: 0.8.0
• Component/s: Usability
• Labels: AWS
• Environment: AWS Glue with PySpark
• Assignee: Unassigned
• Reporter: Sourav T
• Created: 01/Sep/21 16:20
• Updated: 20/Sep/21 03:55
• Resolved: 20/Sep/21 03:55

Description

We are using Apache Hudi from AWS Glue (with PySpark runtime) to store data on S3 bucket. We are observing a very high number of S3 HEAD requests originating from what we believe from Hudi. 

Many a time due to this high number of requests, S3 throws "Status Code: 503; Error Code: SlowDown" causing data losses. 

Is there any any out-of-box feature to debug this further to confirm which Hudi feature causing this?