Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17611

YarnShuffleServiceSuite swallows exceptions, doesn't really test a few things

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: Tests, YARN
    • Labels:
      None

      Description

      If you look at the logs, you'll see exceptions like this:

      16/09/20 10:03:45.081 pool-1-thread-1-ScalaTest-running-YarnShuffleServiceSuite ERROR YarnShuffleService: Exception when initializing application application_0_0001
      java.lang.NullPointerException
              at io.netty.buffer.Unpooled.wrappedBuffer(Unpooled.java:183)
              at org.apache.spark.network.util.JavaUtils.bytesToString(JavaUtils.java:80)
              at org.apache.spark.network.sasl.ShuffleSecretManager.registerApp(ShuffleSecretManager.java:62)
              at org.apache.spark.network.yarn.YarnShuffleService.initializeApplication(YarnShuffleService.java:254)
              at org.apache.spark.network.yarn.YarnShuffleServiceSuite$$anonfun$1.apply$mcV$sp(YarnShuffleServiceSuite.scala:89)
              at org.apache.spark.network.yarn.YarnShuffleServiceSuite$$anonfun$1.apply(YarnShuffleServiceSuite.scala:81)
              at org.apache.spark.network.yarn.YarnShuffleServiceSuite$$anonfun$1.apply(YarnShuffleServiceSuite.scala:81)
      

      The underlying reason is that the suite is testing with auth on, but providing a null auth secret. But because of catch blocks in the shuffle service code, and failure to properly check things in the suite, the tests pass even though the underlying code is failing.

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Masiero Vanzin
              Reporter:
              vanzin Marcelo Masiero Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: