Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29047

use spark-submit Not a file: hdfs://

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Resolved
    • Priority: Critical
    • Resolution: Not A Problem
    • Affects Version/s: 2.1.3
    • Fix Version/s: None
    • Component/s: Deploy, Spark Submit
    • Labels:
      None
    • Environment:

      spark1.3 ;hadoop2.6

      Description

      when i submit a spark application : 

      /spark-submit --class com.yto.log.SparkStatCleanJobYARN --name TopNStatJobYARN --master yarn --executor-memory 1G  --num-executors 1 /home/hadoop/lib/sql-1.0-jar-with-dependencies.jar hdfs://hadoop001:8020/clean 20161110

      the hadfs file is already existed,but not recognition 

       

      19/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process : 219/09/11 14:18:50 INFO mapred.FileInputFormat: Total input paths to process : 219/09/11 14:18:50 ERROR datasources.FileFormatWriter: Aborting job null.java.io.IOException: Not a file: hdfs://hadoop001:8020/clean/day= at org.apache.hadoop.mapred.FileInputFormat.getSplits
      

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              我心凉凉的 bailin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: