Major Description
STR:
- Install Ambari 2.4
- HDP 2.5 with Hive and Atlas
- Kerberize the cluster
The hive hook fails because 2 configs are missing from hive-atlas-application.properties,
atlas.jaas.KafkaClient.option.keyTab=/etc/security/keytabs/hive.service.keytab atlas.jaas.KafkaClient.option.principal=hive/_HOST@EXAMPLE.COM
Impact: HiveHook related tests are failing.
2016-07-29 10:25:50,087 INFO [Atlas Logger 1]: producer.ProducerConfig (AbstractConfig.java:logAll(178)) - ProducerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
bootstrap.servers = [atlas-r6-bug-62789-1023re-2.openstacklocal:6667, atlas-r6-bug-62789-1023re-1.openstacklocal:6667]
ssl.keystore.type = JKS
sasl.mechanism = GSSAPI
max.block.ms = 60000
interceptor.classes = null
ssl.truststore.password = null
client.id =
ssl.endpoint.identification.algorithm = null
request.timeout.ms = 30000
acks = 1
receive.buffer.bytes = 32768
ssl.truststore.type = JKS
retries = 0
ssl.truststore.location = null
ssl.keystore.password = null
send.buffer.bytes = 131072
compression.type = none
metadata.fetch.timeout.ms = 60000
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
buffer.memory = 33554432
timeout.ms = 30000
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
sasl.kerberos.service.name = kafka
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
block.on.buffer.full = false
ssl.key.password = null
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
batch.size = 16384
ssl.keystore.location = null
ssl.cipher.suites = null
.protocol = PLAINTEXTSASL
max.request.size = 1048576
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
linger.ms = 0
2016-07-29 10:25:50,091 INFO [Atlas Logger 1]: producer.KafkaProducer (KafkaProducer.java:close(658)) - Closing the Kafka producer with timeoutMillis = 0 ms.
2016-07-29 10:25:50,091 INFO [Atlas Logger 1]: hook.AtlasHook (AtlasHook.java:notifyEntitiesInternal(131)) - Failed to notify atlas for entity [[{Id='(type: hive_db, id: <unassigned>)', traits=[], values={owner=public, ownerType=2, qualifiedName=default@cl1, clusterName=cl1, name=default, description=Default Hive database, location=hdfs://atlas-r6-bug-62789-1023re-1.openstacklocal:8020/apps/hive/warehouse, parameters={}}}, {Id='(type: hive_table, id: <unassigned>)', traits=[], values={owner=hrt_qa, temporary=false, lastAccessTime=Fri Jul 29 10:25:49 UTC 2016, qualifiedName=default.t2@cl1, columns=[{Id='(type: hive_column, id: <unassigned>)', traits=[], values={owner=hrt_qa, qualifiedName=default.t2.abc@cl1, name=abc, comment=null, type=string, table=(type: hive_table, id: <unassigned>)}}], sd={Id='(type: hive_storagedesc, id: <unassigned>)', traits=[], values={qualifiedName=default.t2@cl1_storage, storedAsSubDirectories=false, location=hdfs://atlas-r6-bug-62789-1023re-1.openstacklocal:8020/apps/hive/warehouse/t2, compressed=false, inputFormat=org.apache.hadoop.mapred.TextInputFormat, outputFormat=org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, parameters={}, serdeInfo=org.apache.atlas.typesystem.Struct@7648946d, table=(type: hive_table, id: <unassigned>), numBuckets=-1}}, tableType=MANAGED_TABLE, createTime=Fri Jul 29 10:25:49 UTC 2016, name=t2, comment=null, partitionKeys=[], parameters={totalSize=0, numRows=0, rawDataSize=0, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}, numFiles=0, transient_lastDdlTime=1469787949}, retention=0, db={Id='(type: hive_db, id: <unassigned>)', traits=[], values={owner=public, ownerType=2, qualifiedName=default@cl1, clusterName=cl1, name=default, description=Default Hive database, location=hdfs://atlas-r6-bug-62789-1023re-1.openstacklocal:8020/apps/hive/warehouse, parameters={}}}}}]]. Retrying
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:335)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:188)
at org.apache.atlas.kafka.KafkaNotification.createProducer(KafkaNotification.java:312)
at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:220)
at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:84)
at org.apache.atlas.hook.AtlasHook.notifyEntitiesInternal(AtlasHook.java:126)
at org.apache.atlas.hook.AtlasHook.notifyEntities(AtlasHook.java:111)
at org.apache.atlas.hook.AtlasHook.notifyEntities(AtlasHook.java:157)
at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:274)
at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:82)
at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.KafkaException: javax..auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:71)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:277)
... 15 more
Caused by: javax..auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
at com.sun..auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:940)
at com.sun..auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun..auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax..auth.login.LoginContext.invoke(LoginContext.java:755)
at javax..auth.login.LoginContext.access$000(LoginContext.java:195)
at javax..auth.login.LoginContext$4.run(LoginContext.java:682)
at javax..auth.login.LoginContext$4.run(LoginContext.java:680)
at java..AccessController.doPrivileged(Native Method)
at javax..auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax..auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.kafka.common..authenticator.AbstractLogin.login(AbstractLogin.java:69)
at org.apache.kafka.common..kerberos.KerberosLogin.login(KerberosLogin.java:110)
at org.apache.kafka.common..authenticator.LoginManager.<init>(LoginManager.java:46)
at org.apache.kafka.common..authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
... 18 more
This occurs because the Kerberos Descriptors does not allow using the same identity name more than once.
AMBARI-17993 just made it possible to have unique names still reference the same identity via a "reference" tag.
Attachments
Attachments
Issue Links
- links to
