[SPARK-10067] Long delay (16 seconds) when running local session on offline machine

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.4.1
• Component/s: Spark Core

Environment

Mac 10.10.5, java 1.8.0_51 from IntelliJ 14.1

• Assignee: Unassigned
• Reporter: Daniel Pinyol
• Created: 17/Aug/15 22:15
• Updated: 20/Aug/15 07:14
• Resolved: 19/Aug/15 10:15

Description

If I run this

        SparkContext sc = new SparkContext("local", "test");

on a machine with no network, it hangs during 15 or 16 seconds during this point, and then it successfully resumes. Looks like the problem is when checking the kerberos realm (see callstack below).
Is there anyway to avoid this annoying delay? I reviewed https://spark.apache.org/docs/latest/configuration.html, but couldn't find any solution.

thanks

"main@1" prio=5 tid=0x1 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at java.net.PlainDatagramSocketImpl.peekData(PlainDatagramSocketImpl.java:-1)
	  - locked <0x758> (a java.net.PlainDatagramSocketImpl)
	  at java.net.DatagramSocket.receive(DatagramSocket.java:787)
	  - locked <0x732> (a java.net.DatagramSocket)
	  - locked <0x759> (a java.net.DatagramPacket)
	  at com.sun.jndi.dns.DnsClient.doUdpQuery(DnsClient.java:413)
	  at com.sun.jndi.dns.DnsClient.query(DnsClient.java:207)
	  at com.sun.jndi.dns.Resolver.query(Resolver.java:81)
	  at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:434)
	  at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:235)
	  at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:141)
	  at com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:103)
	  at sun.security.krb5.KrbServiceLocator.getKerberosService(KrbServiceLocator.java:85)
	  at sun.security.krb5.Config.checkRealm(Config.java:1120)
	  at sun.security.krb5.Config.getRealmFromDNS(Config.java:1093)
	  at sun.security.krb5.Config.getDefaultRealm(Config.java:987)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:483)
	  at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:75)
	  at org.apache.hadoop.security.authentication.util.KerberosName.<clinit>(KerberosName.java:85)
	  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:225)
	  - locked <0x57d> (a java.lang.Class)
	  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214)
	  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
	  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
	  at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2162)
	  at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2162)
	  at scala.Option.getOrElse(Option.scala:121)
	  at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2162)
	  at org.apache.spark.SparkContext.<init>(SparkContext.scala:301)
	  at org.apache.spark.SparkContext.<init>(SparkContext.scala:155)
	  at org.apache.spark.SparkContext.<init>(SparkContext.scala:170)
	  at DataFrameSandbox.<init>(DataFrameSandbox.java:31)
	  at DataFrameSandbox.main(DataFrameSandbox.java:45)