Major Description
Currently, if we build Spark, it will generate a 'spark-version-info.properties' and merged into spark-core_2.11-*.jar. However, the script 'build/spark-build-info' which generates this file can only be executed with bash environment.
Without this file, errors like below will happen when submitting Spark application, which break the whole submitting phrase at beginning.
ERROR ApplicationMaster: Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:194) at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:394) at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:247) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:759) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:757) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) Caused by: java.util.concurrent.ExecutionException: Boxed Error at scala.concurrent.impl.Promise$.resolver(Promise.scala:55) at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244) at scala.concurrent.Promise$class.tryFailure(Promise.scala:112) at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:648) Caused by: java.lang.ExceptionInInitializerError at org.apache.spark.package$.<init>(package.scala:91) at org.apache.spark.package$.<clinit>(package.scala) at org.apache.spark.SparkContext$$anonfun$3.apply(SparkContext.scala:187) at org.apache.spark.SparkContext$$anonfun$3.apply(SparkContext.scala:187) at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) at org.apache.spark.SparkContext.logInfo(SparkContext.scala:76) at org.apache.spark.SparkContext.<init>(SparkContext.scala:187) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2287) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:822) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:814) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:814) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:630) Caused by: org.apache.spark.SparkException: Error while locating file spark-version-info.properties at org.apache.spark.package$SparkBuildInfo$.liftedTree1$1(package.scala:75) at org.apache.spark.package$SparkBuildInfo$.<init>(package.scala:61) at org.apache.spark.package$SparkBuildInfo$.<clinit>(package.scala) ... 19 more Caused by: java.lang.NullPointerException at java.util.Properties$LineReader.readLine(Properties.java:434) at java.util.Properties.load0(Properties.java:353) at java.util.Properties.load(Properties.java:341) at org.apache.spark.package$SparkBuildInfo$.liftedTree1$1(package.scala:64) ... 21 more
We need to provide method to generate 'spark-version-info.properties' in Windows environment.
