Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12582

IndexShuffleBlockResolverSuite fails in windows

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.1, 2.0.0
    • Component/s: Tests, Windows
    • Labels:
      None

      Description

      IndexShuffleBlockResolverSuite fails in my windows develop machine.

      [info] IndexShuffleBlockResolverSuite:
      [info] - commit shuffle files multiple times *** FAILED *** (388 milliseconds)
      [info]   Array(10, 0, 20) equaled Array(10, 0, 20) (IndexShuffleBlockResolverSuite.scala:108)
      [info]   org.scalatest.exceptions.TestFailedException:
      .....
      .....
      [info] Exception encountered when attempting to run a suite with class name: org.apache.spark.shuffle.sort.IndexShuffleB
      lockResolverSuite *** ABORTED *** (2 seconds, 234 milliseconds)
      [info]   java.io.IOException: Failed to delete: C:\Users\yyu29\Documents\codes.next\spark\target\tmp\spark-0e81a15a-e712
      -4b1c-a089-f421db149e65
      [info]   at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:940)
      [info]   at org.apache.spark.shuffle.sort.IndexShuffleBlockResolverSuite.afterEach(IndexShuffleBlockResolverSuite.scala:
      60)
      [info]   at org.scalatest.BeforeAndAfterEach$class.afterEach(BeforeAndAfterEach.scala:205)
      [info]   at org.apache.spark.shuffle.sort.IndexShuffleBlockResolverSuite.afterEach(IndexShuffleBlockResolverSuite.scala:
      36)
      [info]   at org.scalatest.BeforeAndAfterEach$class.afterEach(BeforeAndAfterEach.scala:220)
      [info]   at org.apache.spark.shuffle.sort.IndexShuffleBlockResolverSuite.afterEach(IndexShuffleBlockResolverSuite.scala:
      36)
      

      Root cause is when "afterEach" wants to clean up data, some files are still open. For example:

          // The dataFile should be the previous one
          val in = new FileInputStream(dataFile)
          val firstByte = new Array[Byte](1)
          in.read(firstByte)
          assert(firstByte(0) === 0)
      

      Lack of "in.close()".
      In Linux, it is not a problem, you can still delete a file even it is open, but this does not work in windows, which will report "resource is busy".

      Another issue is this IndexShuffleBlockResolverSuite.scala is a scala file but it is placed in "test/java".

        Attachments

          Activity

            People

            • Assignee:
              yucai yucai
              Reporter:
              yucai yucai
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: