[HUDI-3113] Kafka Connect create Multiple Embedded Timeline Services

• Type: Improvement
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Fix Version/s: 1.1.0
• Component/s: kafka-connect
• Epic Link: Hudi Kafka Connect
• Assignee: Ethan Guo (this is the old account; please use "yihua")
• Reporter: cdmikechen
• Created: 28/Dec/21 07:58
• Updated: 21/Dec/23 15:27

Description

After Kafka Connect started, I've found that hudi will create not one Embedded Timeline Service.

[2021-12-28 07:52:57,154] INFO Starting Timeline service !! (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,155] WARN Unable to find driver bind address from spark config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,169] INFO Creating View Manager with storage type :MEMORY (org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,170] INFO Creating in-memory based Table View (org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,184] INFO Logging initialized @27658ms to org.apache.hudi.org.eclipse.jetty.util.log.Slf4jLog (org.apache.hudi.org.eclipse.jetty.util.log)
[2021-12-28 07:52:57,502] INFO
           __                      __ _
          / /____ _ _   __ ____ _ / /(_)____
     __  / // __ `/| | / // __ `// // // __ \
    / /_/ // /_/ / | |/ // /_/ // // // / / /
    \____/ \__,_/  |___/ \__,_//_//_//_/ /_/

        https://javalin.io/documentation
 (io.javalin.Javalin)
[2021-12-28 07:52:57,504] INFO Starting Javalin ... (io.javalin.Javalin)
[2021-12-28 07:52:57,650] INFO Listening on http://localhost:43691/ (io.javalin.Javalin)
[2021-12-28 07:52:57,650] INFO Javalin started in 151ms \o/ (io.javalin.Javalin)
[2021-12-28 07:52:57,650] INFO Starting Timeline server on port :43691 (org.apache.hudi.timeline.service.TimelineService)
[2021-12-28 07:52:57,650] INFO Started embedded timeline server at 172.17.0.7:43691 (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,661] INFO Start Transaction Coordinator for topic hudi-test-topic partition 0 (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
[2021-12-28 07:52:57,690] INFO Loaded instants upto : Option\{val=[==>20211228075022280__commit__INFLIGHT]} (org.apache.hudi.common.table.timeline.HoodieActiveTimeline)
[2021-12-28 07:52:57,822] INFO Retrieved Raw Kafka offsets from Hudi Commit File 0=100 (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
[2021-12-28 07:52:57,823] INFO Initialized the kafka offset commits \{0=100} (org.apache.hudi.connect.transaction.ConnectTransactionCoordinator)
[2021-12-28 07:52:57,858] INFO The value of hoodie.datasource.write.keygenerator.type is empty, using SIMPLE (org.apache.hudi.keygen.factory.HoodieAvroKeyGeneratorFactory)
[2021-12-28 07:52:57,928] INFO AdminClientConfig values:
    bootstrap.servers = [10.3.101.60:9092]
    client.dns.lookup = use_all_dns_ips
    client.id =
    connections.max.idle.ms = 300000
    default.api.timeout.ms = 60000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retries = 2147483647
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = PLAINTEXT
    security.providers = null
    send.buffer.bytes = 131072
    socket.connection.setup.timeout.max.ms = 127000
    socket.connection.setup.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.certificate.chain = null
    ssl.keystore.key = null
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.certificates = null
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
 (org.apache.kafka.clients.admin.AdminClientConfig)
[2021-12-28 07:52:57,930] INFO Kafka version: 6.1.1-ccs (org.apache.kafka.common.utils.AppInfoParser)
[2021-12-28 07:52:57,930] INFO Kafka commitId: c209f70c6c2e52ae (org.apache.kafka.common.utils.AppInfoParser)
[2021-12-28 07:52:57,930] INFO Kafka startTimeMs: 1640677977929 (org.apache.kafka.common.utils.AppInfoParser)
[2021-12-28 07:52:57,937] INFO Starting Timeline service !! (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,938] WARN Unable to find driver bind address from spark config (org.apache.hudi.client.embedded.EmbeddedTimelineService)
[2021-12-28 07:52:57,938] INFO Creating View Manager with storage type :MEMORY (org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,938] INFO Creating in-memory based Table View (org.apache.hudi.common.table.view.FileSystemViewManager)
[2021-12-28 07:52:57,943] INFO Latest number of partitions for topic hudi-test-topic is 1 (org.apache.hudi.connect.utils.KafkaConnectUtils)
[2021-12-28 07:52:57,947] INFO
           __                      __ _
          / /____ _ _   __ ____ _ / /(_)____
     __  / // __ `/| | / // __ `// // // __ \
    / /_/ // /_/ / | |/ // /_/ // // // / / /
    \____/ \__,_/  |___/ \__,_//_//_//_/ /_/

        https://javalin.io/documentation
 (io.javalin.Javalin)
[2021-12-28 07:52:57,947] INFO Starting Javalin ... (io.javalin.Javalin)
[2021-12-28 07:52:57,959] INFO Loading HoodieTableMetaClient from hdfs://hdp-syzh-cluster/hive/warehouse/default.db/hudi_test_topic (org.apache.hudi.common.table.HoodieTableMetaClient)
[2021-12-28 07:52:57,960] INFO Listening on http://localhost:41798/ (io.javalin.Javalin)
[2021-12-28 07:52:57,960] INFO Javalin started in 14ms \o/ (io.javalin.Javalin)
[2021-12-28 07:52:57,960] INFO Starting Timeline server on port :41798 (org.apache.hudi.timeline.service.TimelineService)
[2021-12-28 07:52:57,962] INFO Started embedded timeline server at 172.17.0.7:41798 (org.apache.hudi.client.embedded.EmbeddedTimelineService)

 

I think a Kafka Connect should only start one Embedded Timeline Service in the current environment.