Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.3.0, 2.3.1
-
None
Description
The "INSERT OVERWRITE LOCAL DIRECTORY" features use the local staging directory to load data into the specified output directory. If the executor is executed on a non-driver node, the data files allocated on that node will not be written to the specified output directory.
The "spark-sql" application has two executors, one on the driver node and one on the non-driver node.
spark-sql> select * from aa;
1 10000
2 30000
2 10000
3 30000
As follows, view the data file of the aa table on HDFS,
/spark/aa/part-00000-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
/spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
/spark/aa/part-00002-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
/spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
spark-sql> insert overwrite local directory "/test" (select * from aa);
As follows, view the contents of the output directory "/test", there are only two data files,
part-00000-16312194-288a-4085-9cb8-de0a685f7af0-c000
part-00002-16312194-288a-4085-9cb8-de0a685f7af0-c000
_SUCCESS
As follows, view the executor log on non-driver node, the other two data files is Is assigned to this node. And the execution results of the tasks is saved to the local staging directory.
INFO org.apache.spark.rdd.HadoopRDD: Input split: hdfs://nameservice/spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_20180920151705_0002_m_000001_0' to file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000001
INFO org.apache.spark.rdd.HadoopRDD: Input split: hdfs://nameservice/spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_20180920151705_0002_m_000003_0' to file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000003