[SPARK-3284] saveAsParquetFile not working on windows

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.0.2
• Component/s: Windows

Environment

Windows

• Assignee: Unassigned
• Reporter: Pravesh Jain
• Created: 28/Aug/14 10:20
• Updated: 11/Jun/15 12:33
• Resolved: 11/Jun/15 12:33

Description

object parquet {

  case class Person(name: String, age: Int)

  def main(args: Array[String]) {

    val sparkConf = new SparkConf().setMaster("local").setAppName("HdfsWordCount")
    val sc = new SparkContext(sparkConf)
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    // createSchemaRDD is used to implicitly convert an RDD to a SchemaRDD.
    import sqlContext.createSchemaRDD

    val people = sc.textFile("C:/Users/pravesh.jain/Desktop/people/people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt))
    people.saveAsParquetFile("C:/Users/pravesh.jain/Desktop/people/people.parquet")

    val parquetFile = sqlContext.parquetFile("C:/Users/pravesh.jain/Desktop/people/people.parquet")
  }
}

gives the error

Exception in thread "main" java.lang.NullPointerException at org.apache.spark.parquet$.main(parquet.scala:16)

which is the line saveAsParquetFile.

This works fine in linux but using in eclipse in windows gives the error.

Issue Links

• duplicates: SPARK-6961 Cannot save data to parquet files when executing from Windows from a Maven Project