[TOREE-516] Kerberos error while working with Toree Kernel

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Workaround
• Affects Version/s: 0.3.0
• Fix Version/s: Not Applicable
• Component/s: Kernel
• Labels: security
• Environment: RHEL 7.7, HDP 3 , Spark 2.3.2
• Assignee: Luciano Resende
• Reporter: Bharath
• Created: 30/Mar/20 19:57
• Updated: 24/Aug/20 16:16
• Resolved: 16/Jul/20 19:36

Description

We have a Kerberied HDP 3.1.0 cluster, configured Jupyter/Toree - It throws Keberos error, we have a valid kerberos ticket in place. Here is the kernel.json

{ "argv": [ "/opt/jupyterhub/share/jupyter/kernels/apache_toree_scala/bin/run.sh", "--profile", "\{connection_file}

"
],
"env":

{ "DEFAULT_INTERPRETER": "Scala", "TOREE_SPARK_OPTS": "--master yarn --deploy-mode cluster", "TOREE_OPTS": "", "HADOOP_CONF_DIR": "/etc/hadoop/conf", "SPARK_HOME": "/usr/hdp/3.1.0.0-78/spark2", "PYTHONPATH": "/opt/jupyterhub/bin/python:/opt/anaconda3/bin/python:/usr/hdp/3.1.0.0-78/spark2/python/lib/py4j-0.10.7-src.zip:/usr/hdp/3.1.0.0-78/spark2/python:/usr/hdp/3.1.0.0-78/spark2/python/lib", "PYTHON_EXEC": "python", "HBASE_CONF_DIR": "/etc/hbase/conf", "HBASE_HOME": "/usr/hdp/current/hbase-client", "JAVA_HOME": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64" }

,
"display_name": "Toree - Scala-kernel",
"language": "scala",
"interrupt_mode": "signal",
"metadata": {}
}

Error Message:
20/03/26 18:32:28 INFO RpcRetryingCallerImpl: Call exception, tries=8, retries=36, started=18874 ms ago, cancelled=false, msg=Call to compute-4.datalake.ntt/xxx.xxx.xx.xxx:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)], details=row 'name_data:exchg_yearly,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=compute-4.datalake.ntt

With above configuration, kernel dies sometimes.

Please suggest if the way we are passing configs in Kernel.Json is the right way ?