Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21128

Running R tests multiple times failed due to pre-exiting "spark-warehouse" / "metastore_db"

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.1, 2.3.0
    • Component/s: SparkR
    • Labels:
      None

      Description

      Currently, running R tests multiple times fails due to pre-exiting "spark-warehouse" / "metastore_db" as below:

      SparkSQL functions: Spark package found in SPARK_HOME: .../spark
      ...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................1234.......................
      
      Failed -------------------------------------------------------------------------
      1. Failure: No extra files are created in SPARK_HOME by starting session and making calls (@test_sparkSQL.R#3384)
      length(list1) not equal to length(list2).
      1/1 mismatches
      [1] 25 - 23 == 2
      
      
      2. Failure: No extra files are created in SPARK_HOME by starting session and making calls (@test_sparkSQL.R#3384)
      sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
      10/25 mismatches
      x[16]: "metastore_db"
      y[16]: "pkg"
      
      x[17]: "pkg"
      y[17]: "R"
      
      x[18]: "R"
      y[18]: "README.md"
      
      x[19]: "README.md"
      y[19]: "run-tests.sh"
      
      x[20]: "run-tests.sh"
      y[20]: "SparkR_2.2.0.tar.gz"
      
      x[21]: "metastore_db"
      y[21]: "pkg"
      
      x[22]: "pkg"
      y[22]: "R"
      
      x[23]: "R"
      y[23]: "README.md"
      
      x[24]: "README.md"
      y[24]: "run-tests.sh"
      
      x[25]: "run-tests.sh"
      y[25]: "SparkR_2.2.0.tar.gz"
      
      
      3. Failure: No extra files are created in SPARK_HOME by starting session and making calls (@test_sparkSQL.R#3388)
      length(list1) not equal to length(list2).
      1/1 mismatches
      [1] 25 - 23 == 2
      
      
      4. Failure: No extra files are created in SPARK_HOME by starting session and making calls (@test_sparkSQL.R#3388)
      sort(list1, na.last = TRUE) not equal to sort(list2, na.last = TRUE).
      10/25 mismatches
      x[16]: "metastore_db"
      y[16]: "pkg"
      
      x[17]: "pkg"
      y[17]: "R"
      
      x[18]: "R"
      y[18]: "README.md"
      
      x[19]: "README.md"
      y[19]: "run-tests.sh"
      
      x[20]: "run-tests.sh"
      y[20]: "SparkR_2.2.0.tar.gz"
      
      x[21]: "metastore_db"
      y[21]: "pkg"
      
      x[22]: "pkg"
      y[22]: "R"
      
      x[23]: "R"
      y[23]: "README.md"
      
      x[24]: "README.md"
      y[24]: "run-tests.sh"
      
      x[25]: "run-tests.sh"
      y[25]: "SparkR_2.2.0.tar.gz"
      
      
      DONE ===========================================================================
      

      It looks we should remove both "spark-warehouse" and "metastore_db" before listing files into sparkRFilesBefore.

        Attachments

          Activity

            People

            • Assignee:
              hyukjin.kwon Hyukjin Kwon
              Reporter:
              hyukjin.kwon Hyukjin Kwon
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: