[SPARK-4952] Handle ConcurrentModificationExceptions in SparkEnv.environmentDetails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.2, 1.2.1, 1.3.0
• Component/s: Spark Core, YARN
• Target Version/s: 1.1.2, 1.2.1, 1.3.0
• Assignee: Guoqiang Li
• Reporter: Guoqiang Li
• Created: 24/Dec/14 06:42
• Updated: 08/Mar/15 14:17
• Resolved: 27/Dec/14 07:32

Description

the log:

Exception in thread "main" 14/12/24 12:00:25 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> tuan200, PROXY_URI_BASES -> http://host:9082/proxy/application_1414231702825_488625), /proxy/application_1414231702825_488625
java.util.ConcurrentModificationException
        at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
        at scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:458)
        at scala.collection.convert.Wrappers$JPropertiesWrapper$$anon$3.next(Wrappers.scala:454)
        at scala.collection.Iterator$class.toStream(Iterator.scala:1143)
        at scala.collection.AbstractIterator.toStream(Iterator.scala:1157)
        at scala.collection.Iterator$$anonfun$toStream$1.apply(Iterator.scala:1143)
        at scala.collection.Iterator$$anonfun$toStream$1.apply(Iterator.scala:1143)
        at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
        at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
        at scala.collection.immutable.Stream$$anonfun$filteredTail$1.apply(Stream.scala:1149)
        at scala.collection.immutable.Stream$$anonfun$filteredTail$1.apply(Stream.scala:1149)
        at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
        at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
        at scala.collection.immutable.Stream.length(Stream.scala:284)
        at scala.collection.SeqLike$class.sorted(SeqLike.scala:608)
        at scala.collection.AbstractSeq.sorted(Seq.scala:40)
        at org.apache.spark.SparkEnv$.environmentDetails(SparkEnv.scala:324)
        at org.apache.spark.SparkContext.postEnvironmentUpdate(SparkContext.scala:1319)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:344)
        at com.zhe800.toona.als.computation.DealCF$.main(DealCF.scala:497)
        at com.zhe800.toona.als.computation.DealCF.main(DealCF.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
14/12/24 12:00:25 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter

Issue Links

• is duplicated by: SPARK-3457 ConcurrentModificationException starting up pyspark
• links to: [Github] Pull Request #3788 (witgo)