Uploaded image for project: 'Ranger'
  1. Ranger
  2. RANGER-3198

Ranger and Atlas are incompatible on Hive 3.1

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: admin, audit, plugins, Ranger
    • Labels:
      None
    • Environment:
      Ranger Version: 2.1
      Atlas Version: 2.1
      Hive Version: 3.1

      Description

      I configured Ranger and Atlas in Hive , the Hive service started successfully, but when I excute DDL SQL such as "Create Table", it seems that the Atlas kakfa can not receive any messages.

      Then I removed Ranger configurtion in Hive , the Atlas kakfa can receive messages.

       

      Although I granted all database permissions to Hive or Hadoop user in Ranger,but nothing help.

      I found some logs in Hive ,it seems that Atlas Hook was created successfully.

      This is "including ranger configurtion" logs , When the statement finishes, Kafka's log is not printed :

      2021-03-08T08:44:10,510 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: ql.Driver (Driver.java:compile(554)) - Compiling command(queryId=hive_20210308084410_0089cdc7-45b6-46b3-9921-05c71e950cf4): USE `default`
      2021-03-08T08:44:10,572 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:get(119)) - Looking for atlas-application.properties in classpath
      2021-03-08T08:44:10,572 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:get(132)) - Loading atlas-application.propertiesfrom file:/etc/hive/conf.dist/atlas-application.properties
      2021-03-08T08:44:10,591 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(284)) - Using graphdb backend 'janus'
      2021-03-08T08:44:10,591 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(295)) - Using storage backend 'hbase2'
      2021-03-08T08:44:10,591 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(306)) - Using index backend 'solr'
      2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(310)) - Atlas is running in MODE: PROD.
      2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(316)) - Setting solr-wait-searcher property 'true'
      2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefaults(320)) - Setting index.search.map-name property 'false'
      2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.db-cache = true
      2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20
      2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.db-cache-size = 0.5
      2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.tx-cache-size = 15000
      2021-03-08T08:44:10,592 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: atlas.ApplicationProperties (ApplicationProperties.java:setDefault(334)) - Property (set to default) atlas.graph.cache.tx-dirty-size = 120
      2021-03-08T08:44:10,613 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: kafka.KafkaNotification (KafkaNotification.java:<init>(105)) - ==> KafkaNotification()
      2021-03-08T08:44:10,616 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: security.SecurityUtil (SecurityUtil.java:getPassword(48)) - Attempting to retrieve password for key truststore.password from configured credential provider path jceks://file/etc/hive/conf/atlas.jceks
      2021-03-08T08:44:10,720 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: kafka.KafkaNotification (KafkaNotification.java:<init>(137)) - <== KafkaNotification()
      2021-03-08T08:44:10,734 INFO [1d58ddb3-463a-4b28-aadc-e62ed5bbc9fc HiveServer2-Handler-Pool: Thread-100([])]: hook.AtlasHook (AtlasHook.java:<clinit>(147)) - Created Atlas Hook

       

      And this is "removed ranger config" logs ,you can see that at the end of the logs, Kafka successfully created and sent the message .

      2021-03-08T09:33:01,560 INFO [aaff2ff0-f790-4ec3-8add-b4b84d235ba6 HiveServer2-Handler-Pool: Thread-86([])]: conf.HiveConf (HiveConf.java:getLogIdVar(5070)) - Using the default value passed in for log id: aaff2ff0-f790-4ec3-8add-b4b84d235ba6
      2021-03-08T09:33:01,560 INFO [aaff2ff0-f790-4ec3-8add-b4b84d235ba6 HiveServer2-Handler-Pool: Thread-86([])]: session.SessionState (SessionState.java:resetThreadName(452)) - Resetting thread name to HiveServer2-Handler-Pool: Thread-86
      2021-03-08T09:33:01,747 INFO [HiveServer2-Background-Pool: Thread-84([])]: metadata.Hive (Hive.java:logDumpPhase(4669)) - Dumping metastore api call timing information for : execution phase
      2021-03-08T09:33:01,747 INFO [HiveServer2-Background-Pool: Thread-84([])]: metadata.Hive (Hive.java:dumpAndClearMetaCallTiming(4657)) - Total time spent in this metastore function was greater than 1000ms : dropTable_(String, String, boolean, boolean, boolean, )=3864
      2021-03-08T09:33:01,748 INFO [HiveServer2-Background-Pool: Thread-84([])]: ql.Driver (Driver.java:execute(2531)) - Completed executing command(queryId=hive_20210308093257_3cb85365-3a9f-4984-aa29-1bc2c6ac7926); Timetaken: 3.913 seconds
      2021-03-08T09:33:01,748 INFO [HiveServer2-Background-Pool: Thread-84([])]: ql.Driver (SessionState.java:printInfo(1227)) - OK
      2021-03-08T09:33:01,748 INFO [HiveServer2-Background-Pool: Thread-84([])]: ql.Driver (Driver.java:checkConcurrency(285)) - Concurrency mode is disabled, not creating a lock manager
      2021-03-08T09:33:01,827 INFO [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logAll(279)) - ProducerConfig values:
      acks = 1
      batch.size = 16384
      bootstrap.servers = [172.30.198.23:9092]
      buffer.memory = 33554432
      client.id =
      compression.type = none
      connections.max.idle.ms = 540000
      enable.idempotence = false
      interceptor.classes = []
      key.serializer = class org.apache.kafka.common.serialization.StringSerializer
      linger.ms = 0
      max.block.ms = 60000
      max.in.flight.requests.per.connection = 5
      max.request.size = 1048576
      metadata.max.age.ms = 300000
      metric.reporters = []
      metrics.num.samples = 2
      metrics.recording.level = INFO
      metrics.sample.window.ms = 30000
      partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
      receive.buffer.bytes = 32768
      reconnect.backoff.max.ms = 1000
      reconnect.backoff.ms = 50
      request.timeout.ms = 30000
      retries = 0
      retry.backoff.ms = 100
      sasl.client.callback.handler.class = null
      sasl.jaas.config = null
      sasl.kerberos.kinit.cmd = /usr/bin/kinit
      sasl.kerberos.min.time.before.relogin = 60000
      sasl.kerberos.service.name = null
      sasl.kerberos.ticket.renew.jitter = 0.05
      sasl.kerberos.ticket.renew.window.factor = 0.8
      sasl.login.callback.handler.class = null
      sasl.login.class = null
      sasl.login.refresh.buffer.seconds = 300
      sasl.login.refresh.min.period.seconds = 60
      sasl.login.refresh.window.factor = 0.8
      sasl.login.refresh.window.jitter = 0.05
      sasl.mechanism = GSSAPI
      security.protocol = PLAINTEXT
      send.buffer.bytes = 131072
      ssl.cipher.suites = null
      ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
      ssl.endpoint.identification.algorithm = https
      ssl.key.password = null
      ssl.keymanager.algorithm = SunX509
      ssl.keystore.location = null
      ssl.keystore.password = null
      ssl.keystore.type = JKS
      ssl.protocol = TLS
      ssl.provider = null
      ssl.secure.random.implementation = null
      ssl.trustmanager.algorithm = PKIX
      ssl.truststore.location = null
      ssl.truststore.password = [hidden]
      ssl.truststore.type = JKS
      transaction.timeout.ms = 60000
      transactional.id = null
      value.serializer = class org.apache.kafka.common.serialization.StringSerializer

      2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'key.deserializer' was supplied but isn't a known config.
      2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'value.deserializer' was supplied but isn't a known config.
      2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'hook.group.id' was supplied but isn't a known config.
      2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'max.poll.records' was supplied but isn't a known config.
      2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'data' was supplied but isn't a known config.
      2021-03-08T09:33:01,871 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'auto.commit.interval.ms' was supplied but isn't a known config.
      2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'offsets.topic.replication.factor' was supplied but isn't a known config.
      2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'zookeeper.connection.timeout.ms' was supplied but isn't a known config.
      2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'zookeeper.session.timeout.ms' was supplied but isn't a known config.
      2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'enable.auto.commit' was supplied but isn't a known config.
      2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'zookeeper.connect' was supplied but isn't a known config.
      2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'poll.timeout.ms' was supplied but isn't a known config.
      2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'zookeeper.sync.time.ms' was supplied but isn't a known config.
      2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'session.timeout.ms' was supplied but isn't a known config.
      2021-03-08T09:33:01,872 WARN [Atlas Notifier 0([])]: producer.ProducerConfig (AbstractConfig.java:logUnused(287)) - The configuration 'auto.offset.reset' was supplied but isn't a known config.
      2021-03-08T09:33:01,874 INFO [Atlas Notifier 0([])]: utils.AppInfoParser (AppInfoParser.java:<init>(109)) - Kafka version : 2.0.0
      2021-03-08T09:33:01,874 INFO [Atlas Notifier 0([])]: utils.AppInfoParser (AppInfoParser.java:<init>(110)) - Kafka commitId : 3402a8361b734732
      2021-03-08T09:33:01,946 INFO [kafka-producer-network-thread | producer-1([])]: clients.Metadata (Metadata.java:update(273)) - Cluster ID: ytZ5Kgb9RXOOp0KEZL6iXw
      2021-03-08T09:33:02,731 INFO [HiveServer2-Handler-Pool: Thread-77([])]: conf.HiveConf (HiveConf.java:getLogIdVar(5070)) - Using the default value passed in for log id: aaff2ff0-f790-4ec3-8add-b4b84d235ba6
      2021-03-08T09:33:02,731 INFO [HiveServer2-Handler-Pool: Thread-77([])]: session.SessionState (SessionState.java:updateThreadName(441)) - Updating thread name to aaff2ff0-f790-4ec3-8add-b4b84d235ba6 HiveServer2-Handler-Pool: Thread-77

       

       

      Can someone give me some advice? Thanks very much!

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Sekiro Sekiro
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: