[SPARK-21128] Running R tests multiple times failed due to pre-exiting "spark-warehouse" / "metastore_db"

• Type: Test
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: SparkR
• Target Version/s: 2.2.1, 2.3.0
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 17/Jun/17 10:18
• Updated: 12/Dec/22 17:51
• Resolved: 18/Jun/17 18:27

Description

Currently, running R tests multiple times fails due to pre-exiting "spark-warehouse" / "metastore_db" as below:

SparkSQL functions: Spark package found in SPARK_HOME: .../spark
...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................1234.......................

Failed -------------------------------------------------------------------------
1. Failure: No extra files are created in SPARK_HOME by starting session and making calls (@test_sparkSQL.R#3384)
length(list1) not equal to length(list2).
1/1 mismatches
[1] 25 - 23 == 2


2. Failure: No extra files are created in SPARK_HOME by starting session and making calls (@test_sparkSQL.R#3384)
sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
10/25 mismatches
x[16]: "metastore_db"
y[16]: "pkg"

x[17]: "pkg"
y[17]: "R"

x[18]: "R"
y[18]: "README.md"

x[19]: "README.md"
y[19]: "run-tests.sh"

x[20]: "run-tests.sh"
y[20]: "SparkR_2.2.0.tar.gz"

x[21]: "metastore_db"
y[21]: "pkg"

x[22]: "pkg"
y[22]: "R"

x[23]: "R"
y[23]: "README.md"

x[24]: "README.md"
y[24]: "run-tests.sh"

x[25]: "run-tests.sh"
y[25]: "SparkR_2.2.0.tar.gz"


3. Failure: No extra files are created in SPARK_HOME by starting session and making calls (@test_sparkSQL.R#3388)
length(list1) not equal to length(list2).
1/1 mismatches
[1] 25 - 23 == 2


4. Failure: No extra files are created in SPARK_HOME by starting session and making calls (@test_sparkSQL.R#3388)
sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
10/25 mismatches
x[16]: "metastore_db"
y[16]: "pkg"

x[17]: "pkg"
y[17]: "R"

x[18]: "R"
y[18]: "README.md"

x[19]: "README.md"
y[19]: "run-tests.sh"

x[20]: "run-tests.sh"
y[20]: "SparkR_2.2.0.tar.gz"

x[21]: "metastore_db"
y[21]: "pkg"

x[22]: "pkg"
y[22]: "R"

x[23]: "R"
y[23]: "README.md"

x[24]: "README.md"
y[24]: "run-tests.sh"

x[25]: "run-tests.sh"
y[25]: "SparkR_2.2.0.tar.gz"


DONE ===========================================================================

It looks we should remove both "spark-warehouse" and "metastore_db" before listing files into sparkRFilesBefore.

Issue Links

• links to: [Github] Pull Request #18335 (HyukjinKwon)
• links to: [Github] Pull Request #19166 (felixcheung)