Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.3.0, 1.3.1
-
None
-
Spark 1.3.1, aws, persistent hdfs version 2 with ebs storage, pyspark, 8 x c3.8xlarge nodes.
spark-conf
spark.executor.memory 50g
spark.driver.cores 32
spark.driver.memory 50gspark.default.parallelism 512
spark.sql.shuffle.partitions 512spark.task.maxFailures 30
spark.executor.logs.rolling.maxRetainedFiles 2
spark.executor.logs.rolling.size.maxBytes 102400
spark.executor.logs.rolling.strategy sizespark.shuffle.spill false
spark.sql.parquet.cacheMetadata true
spark.sql.parquet.filterPushdown true
spark.sql.codegen truespark.akka.threads = 64
Spark 1.3.1, aws, persistent hdfs version 2 with ebs storage, pyspark, 8 x c3.8xlarge nodes. spark-conf spark.executor.memory 50g spark.driver.cores 32 spark.driver.memory 50g spark.default.parallelism 512 spark.sql.shuffle.partitions 512 spark.task.maxFailures 30 spark.executor.logs.rolling.maxRetainedFiles 2 spark.executor.logs.rolling.size.maxBytes 102400 spark.executor.logs.rolling.strategy size spark.shuffle.spill false spark.sql.parquet.cacheMetadata true spark.sql.parquet.filterPushdown true spark.sql.codegen true spark.akka.threads = 64
Description
I have 2.6 billion rows in parquet format and I'm trying to use the new schema merging feature (I was enforcing a consistent schema manually before in 0.8-1.2 which was annoying).
I have approximate 200 parquet files with key=<date>. When I load the dataframe with the sqlcontext that process is understandably slow because I assume it's reading all the meta data from the parquet files and doing the initial schema merging. So that's ok.
However the problem I have is that once I have the dataframe. Doing any operation on the dataframe seems to have a 10-30 second lag before it actually starts processing the Job and shows up as an Active Job in the Spark Manager. This was an instant operation in all previous versions of Spark. Once the job actually is running the performance is fantastic, however this job submission lag is horrible.
I'm wondering if there is a bug with recomputing the schema merging. Running top on the master node shows some thread maxed out on 1 cpu during the lagging time which makes me think it's not net i/o but something pre-processing before job submission.
Attachments
Issue Links
- links to
