Details
Critical Description
I configured Ranger and Atlas in Hive , the Hive service started successfully, but when I excute DDL SQL such as "Create Table", it seems that the Atlas kakfa can not receive any messages.
Then I removed Ranger configurtion in Hive , the Atlas kakfa can receive messages.
Although I granted all database permissions to Hive or Hadoop user in Ranger,but nothing help.
I found some logs in Hive ,it seems that Atlas Hook was created successfully.
This is "including ranger configurtion" logs , When the statement finishes, Kafka's log is not printed :
2021-03-08T08:44:10,510 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: ql.Driver (Driver.java:compile(554)) - Compiling command(queryId=hive_20210308084410_0089cdc7-45b6-46b3-9921-05c71e950cf4): USE `default`
2021-03-08T08:44:10,572 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:get(119)) - Looking for atlas-application.properties in classpath
2021-03-08T08:44:10,572 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:get(132)) - Loading atlas-application.propertiesfrom file:/etc/hive/conf.dist/atlas-application.properties
2021-03-08T08:44:10,591 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(284)) - Using graphdb backend 'janus'
2021-03-08T08:44:10,591 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(295)) - Using storage backend 'hbase2'
2021-03-08T08:44:10,591 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(306)) - Using index backend 'solr'
2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(310)) - Atlas is running in MODE: PROD.
2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(316)) - Setting solr-wait-searcher property 'true'
2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(320)) - Setting index.search.map-name property 'false'
2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.db-cache = true
2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20
2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.db-cache-size = 0.5
2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.tx-cache-size = 15000
2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.tx-dirty-size = 120
2021-03-08T08:44:10,613 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: kafka.KafkaNotification (KafkaNotification.java:<init>(105)) - ==> KafkaNotification()
2021-03-08T08:44:10,616 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: security.SecurityUtil (SecurityUtil.java:getPassword(48)) - Attempting to retrieve password for key truststore.password from configured credential provider path jceks://file/etc/hive/conf/atlas.jceks
2021-03-08T08:44:10,720 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: kafka.KafkaNotification (KafkaNotification.java:<init>(137)) - <== KafkaNotification()
2021-03-08T08:44:10,734 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: hook.AtlasHook (AtlasHook.java:<clinit>(147)) - Created Atlas Hook
And this is "removed ranger config" logs ,you can see that at the end of the logs, Kafka successfully created and sent the message .
2021-03-08T09:33:01,560 INFO [aaff2ff0-f790-4ec3-8add-b4b84d235ba6 HiveServer2-Handler-Pool: Thread-86([])]: conf.HiveConf (HiveConf.java:getLogIdVar(5070)) - Using the default value passed in for log id: aaff2ff0-f790-4ec3-8add-b4b84d235ba6
2021-03-08T09:33:01,560 INFO [aaff2ff0-f790-4ec3-8add-b4b84d235ba6 HiveServer2-Handler-Pool: Thread-86([])]: session.SessionState (SessionState.java:resetThreadName(452)) - Resetting thread name to HiveServer2-Handler-Pool: Thread-86
2021-03-08T09:33:01,747 INFO [HiveServer2-Background-Pool: Thread-84([])]: metadata.Hive (Hive.java:logDumpPhase(4669)) - Dumping metastore api call timing information for : execution phase
2021-03-08T09:33:01,747 INFO [HiveServer2-Background-Pool: Thread-84([])]: metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(4657)) - Total time spent in this metastore function was greater than 1000ms : dropTable_(String, String, boolean, boolean, boolean, )=3864
2021-03-08T09:33:01,748 INFO [HiveServer2-Background-Pool: Thread-84([])]: ql.Driver (Driver.java:execute(2531)) - Completed executing command(queryId=hive_20210308093257_3cb85365-3a9f-4984-aa29-1bc2c6ac7926); Timetaken: 3.913 seconds
2021-03-08T09:33:01,748 INFO [HiveServer2-Background-Pool: Thread-84([])]: ql.Driver (SessionState.java:printInfo(1227)) - OK
2021-03-08T09:33:01,748 INFO [HiveServer2-Background-Pool: Thread-84([])]: ql.Driver (Driver.java:checkConcurrency(285)) - Concurrency mode is disabled, not creating a lock manager
2021-03-08T09:33:01,827 INFO [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logAll(279)) - ProducerConfig values:
acks = 1
batch.size = 16384
bootstrap.servers = [172.30.198.23:9092]
buffer.memory = 33554432
client.id =
compression.type = none
connections.max.idle.ms = 540000
enable.idempotence = false
interceptor.classes = []
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer
2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'key.deserializer' was supplied but isn't a known config.
2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'value.deserializer' was supplied but isn't a known config.
2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'hook.group.id' was supplied but isn't a known config.
2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'max.poll.records' was supplied but isn't a known config.
2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'data' was supplied but isn't a known config.
2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'auto.commit.interval.ms' was supplied but isn't a known config.
2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'offsets.topic.replication.factor' was supplied but isn't a known config.
2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'zookeeper.connection.timeout.ms' was supplied but isn't a known config.
2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'zookeeper.session.timeout.ms' was supplied but isn't a known config.
2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'enable.auto.commit' was supplied but isn't a known config.
2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'zookeeper.connect' was supplied but isn't a known config.
2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'poll.timeout.ms' was supplied but isn't a known config.
2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'zookeeper.sync.time.ms' was supplied but isn't a known config.
2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'session.timeout.ms' was supplied but isn't a known config.
2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'auto.offset.reset' was supplied but isn't a known config.
2021-03-08T09:33:01,874 INFO [Atlas Notifier 0([])]: utils.AppInfoParser (AppInfoParser.java:<init>(109)) - Kafka version : 2.0.0
2021-03-08T09:33:01,874 INFO [Atlas Notifier 0([])]: utils.AppInfoParser (AppInfoParser.java:<init>(110)) - Kafka commitId : 3402a8361b734732
2021-03-08T09:33:01,946 INFO [kafka-producer-network-thread | producer-1([])]: clients.Metadata (Metadata.java:update(273)) - Cluster ID: ytZ5Kgb9RXOOp0KEZL6iXw
2021-03-08T09:33:02,731 INFO [HiveServer2-Handler-Pool: Thread-77([])]: conf.HiveConf (HiveConf.java:getLogIdVar(5070)) - Using the default value passed in for log id: aaff2ff0-f790-4ec3-8add-b4b84d235ba6
2021-03-08T09:33:02,731 INFO [HiveServer2-Handler-Pool: Thread-77([])]: session.SessionState (SessionState.java:updateThreadName(441)) - Updating thread name to aaff2ff0-f790-4ec3-8add-b4b84d235ba6 HiveServer2-Handler-Pool: Thread-77
Can someone give me some advice? Thanks very much!