[SPARK-24273] Failure while using .checkpoint method to private S3 store via S3A connector

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: Spark Shell
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Jami Malikzade
• Created: 14/May/18 17:46
• Updated: 08/Oct/19 05:41
• Resolved: 08/Oct/19 05:41

Description

We are getting following error:

com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 416, AWS Service: Amazon S3, AWS Request ID: tx000000000000000014126-005ae9bfd9-9ed9ac2-default, AWS Error Code: InvalidRange, AWS Error Message: null, S3 Extended Request ID: 9ed9ac2-default-default"

when we use checkpoint method as below.

val streamBucketDF = streamPacketDeltaDF
 .filter('timeDelta > maxGap && 'timeDelta <= 30000)
 .withColumn("bucket", when('timeDelta <= mediumGap, "medium")
 .otherwise("large")
 )
 .checkpoint()

Do you have idea how to prevent invalid range in header to be sent, or how it can be workarounded or fixed?

Thanks.

Issue Links

• is duplicated by: ZEPPELIN-3477 Spark interpreter returns AWS Error Code: InvalidRange, AWS Error Message: null
• relates to: HADOOP-13293 add a special 0 byte input stream for empty blobs