Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5425

ConcurrentModificationException during SparkConf creation

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.1, 1.2.0
    • Fix Version/s: 1.1.2, 1.2.2, 1.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      This fragment of code:

        if (loadDefaults) {
          // Load any spark.* system properties
          for ((k, v) <- System.getProperties.asScala if k.startsWith("spark.")) {
            settings(k) = v
          }
        }
      

      causes

      ERROR 09:43:15  SparkMaster service caused error in state STARTINGjava.util.ConcurrentModificationException: null
      	at java.util.Hashtable$Enumerator.next(Hashtable.java:1167) ~[na:1.7.0_60]
      	at scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:458) ~[scala-library-2.10.4.jar:na]
      	at scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:454) ~[scala-library-2.10.4.jar:na]
      	at scala.collection.Iterator$class.foreach(Iterator.scala:727) ~[scala-library-2.10.4.jar:na]
      	at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) ~[scala-library-2.10.4.jar:na]
      	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[scala-library-2.10.4.jar:na]
      	at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[scala-library-2.10.4.jar:na]
      	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) ~[scala-library-2.10.4.jar:na]
      	at org.apache.spark.SparkConf.<init>(SparkConf.scala:53) ~[spark-core_2.10-1.2.1_dse-20150121.075638-2.jar:1.2.1_dse-SNAPSHOT]
      	at org.apache.spark.SparkConf.<init>(SparkConf.scala:47) ~[spark-core_2.10-1.2.1_dse-20150121.075638-2.jar:1.2.1_dse-SNAPSHOT]
      

      when there is another thread which modifies system properties at the same time.

      This bug https://issues.scala-lang.org/browse/SI-7775 is somehow related to the issue and shows that the problem has been also found elsewhere.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jlewandowski Jacek Lewandowski
                Reporter:
                jlewandowski Jacek Lewandowski
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: