[SPARK-29047] use spark-submit Not a file: hdfs://

• Type: Question
• Status: Resolved
• Priority: Critical
• Resolution: Not A Problem
• Affects Version/s: 2.1.3
• Component/s: Deploy, Spark Submit

Environment

spark1.3 ；hadoop2.6

• Assignee: Unassigned
• Reporter: bailin
• Created: 11/Sep/19 06:45
• Updated: 12/Dec/22 18:10
• Resolved: 16/Sep/19 01:32

Description

when i submit a spark application : 

/spark-submit --class com.yto.log.SparkStatCleanJobYARN --name TopNStatJobYARN --master yarn --executor-memory 1G  --num-executors 1 /home/hadoop/lib/sql-1.0-jar-with-dependencies.jar hdfs://hadoop001:8020/clean 20161110

the hadfs file is already existed，but not recognition 

 

19/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process : 219/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process : 219/09/11 14:18:50 ERROR datasources.FileFormatWriter: Aborting job null.java.io.IOException: Not a file: hdfs://hadoop001:8020/clean/day= at org.apache.hadoop.mapred.FileInputFormat.getSplits