[SPARK-36761] spark-examples_2.12-3.0.2.jar DFSReadWriteTest S3A Implementation

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.2
• Component/s: Examples
• Assignee: Unassigned
• Reporter: Raj
• Created: 15/Sep/21 04:33
• Updated: 12/Dec/22 18:10

Description

Dear Team,

I am using Spark3 to test the s3a storage writing. Part of test i am invoking DFSReadWriteTest from spark-examples_2.12-3.0.2.jar file. I am passing the arguments as below

spark-submit --verbose --driver-java-options "-Dlog4j.configuration=file:/home/myid/log4j.properties" --conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=file:/home/myid/log4j.properties'" --driver-class-path "/usr/hdp/3.1.5.0-152/hadoop/ceph-rgw-sts-auth-6.jar,/opt/spark3/jars/hadoop-aws-3.1.1.3.1.5.0-152.jar" --class org.apache.spark.examples.DFSReadWriteTest --deploy-mode client --executor-memory 1G --num-executors 3 --conf "spark.hadoop.fs.s3a.refreshTokenFile='/home/myid/keycloaktoken/tokenfile'" /opt/spark3/examples/jars/spark-examples_2.12-3.0.2.jar "/home/myid/sparkreadtest.txt" "s3a://d14/import/rajtestsp33"

The Program fails with the message Wrong FS ( It seems the file system comparison fails in checkpath)

Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: s3a://d14/import/rajtestsp33/dfs_read_write_test, expected: hdfs://pphdpException in thread "main" java.lang.IllegalArgumentException: Wrong FS: s3a://d14/import/rajtestsp33/dfs_read_write_test, expected: hdfs://pphdp at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:730) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:234) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1577) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1574) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1589) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1683) at org.apache.spark.examples.DFSReadWriteTest$.main(DFSReadWriteTest.scala:115) at org.apache.spark.examples.DFSReadWriteTest.main(DFSReadWriteTest.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

And the same program successfully ran with the Spark2 with HDP 3.1.5.0-152 and spark-examples_2.11-2.3.2.3.1.5.0-152.jar file.

Any inputs appreciated. 

 

Thanks

Raj