[LIVY-740] Trying to access jar on s3 using livy

• Type: Question
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 0.9.0
• Component/s: API
• Labels: beginner
• Environment: Spark - 2.3.0.2.6.5.1175-1 ; hdp - 2.6.5.1175-1 ; ; AWS jar versions in spark: ; ; /aws-java-sdk-core-1.10.6.jar ; /aws-java-sdk-kms-1.10.6.jar ; /aws-java-sdk-s3-1.10.6.jar ; /hadoop-aws-2.7.3.2.6.5.1175-1.jar
• Assignee: Unassigned
• Reporter: poonam chaudhari
• Created: 10/Jan/20 04:30
• Updated: 12/Nov/22 22:10

Description

curl -X POST --data '{"file": "s3://livy-test/spark-examples_2.11-2.4.4.jar", "className": "org.apache.spark.examples.SparkPi","conf":{"spark.hadoop.fs.s3.awsAccessKeyId":"**","spark.hadoop.fs.s3.awsSecretAccessKey":"**","spark.executor.extraJavaOptions":"-Dcom.amazonaws.services.s3.enableV4=true","spark.driver.extraJavaOptions":"Dcom.amazonaws.services.s3.enableV4=true"}}' -H "Content-Type: application/json" -H "X-Requested-By: admin" http://hostname:8999/batches

 

I am trying to submit spark job to like this but it fails with below error:

AM I missing something here? I have verified aws secret many times.The same secret is working with aws cli. Can you please guide me here?