Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10710

MirrorMaker 2 creates all combinations of herders

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.5.1
    • 2.8.0
    • mirrormaker
    • None

    Description

      We are using MM2 distributed to synchronize topics from a "Central" broker down to multiple "Local" brokers. 

      replica_CENTRAL->replica_OLS.enabled = true
      replica_CENTRAL->replica_OLS.topics = _schemas
      replica_CENTRAL->replica_OLS.replication.factor = 3

      replica_CENTRAL->replica_HBG.enabled = true
      replica_CENTRAL->replica_HBG.topics = _schemas
      replica_CENTRAL->replica_HBG.replication.factor = 3

      ...

      many more

      ...

      replica_CENTRAL->replica_VIT.enabled = true
      replica_CENTRAL->replica_VIT.topics = _schemas
      replica_CENTRAL->replica_VIT.replication.factor = 3

      replica_CENTRAL->replica_UGO.enabled = true
      replica_CENTRAL->replica_UGO.topics = _schemas
      replica_CENTRAL->replica_UGO.replication.factor = 3

       

      When looking into the Mirror Maker logs, we discover that a herder is created for each combination even if we specifically don't describe a link between 2 clusters

       

      Exemples:

      [2020-11-12 08:43:30,351] INFO creating herder for replica_VIT->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:33,697] INFO creating herder for replica_CNO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:36,020] INFO creating herder for replica_CENTRAL->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:38,508] INFO creating herder for replica_UMO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:40,438] INFO creating herder for replica_CNO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:42,898] INFO creating herder for replica_ARA->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:45,457] INFO creating herder for replica_TST->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:47,860] INFO creating herder for replica_UMO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:50,248] INFO creating herder for replica_OLS->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:52,817] INFO creating herder for replica_UGO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:54,903] INFO creating herder for replica_HBG->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:57,081] INFO creating herder for replica_OLS->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:43:59,215] INFO creating herder for replica_TST->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:01,481] INFO creating herder for replica_OLS->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:03,320] INFO creating herder for replica_CNO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:05,337] INFO creating herder for replica_VIT->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:07,494] INFO creating herder for replica_CENTRAL->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:09,595] INFO creating herder for replica_ARA->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:12,201] INFO creating herder for replica_VLD->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:14,360] INFO creating herder for replica_HBG->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:16,490] INFO creating herder for replica_OLS->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:18,680] INFO creating herder for replica_ARA->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:20,779] INFO creating herder for replica_VLD->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:22,596] INFO creating herder for replica_CENTRAL->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:24,579] INFO creating herder for replica_VIT->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:26,076] INFO creating herder for replica_TST->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:27,868] INFO creating herder for replica_OLS->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:30,047] INFO creating herder for replica_TST->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:32,295] INFO creating herder for replica_OLS->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:33,777] INFO creating herder for replica_CNO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:36,083] INFO creating herder for replica_CENTRAL->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:38,857] INFO creating herder for replica_ARA->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:40,629] INFO creating herder for replica_CNO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:43,170] INFO creating herder for replica_UGO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:45,154] INFO creating herder for replica_HBG->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:46,981] INFO creating herder for replica_UMO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:49,042] INFO creating herder for replica_UGO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:51,881] INFO creating herder for replica_HBG->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:54,406] INFO creating herder for replica_HBG->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:56,378] INFO creating herder for replica_TST->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:44:58,825] INFO creating herder for replica_ARA->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:00,394] INFO creating herder for replica_CNO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:02,060] INFO creating herder for replica_UGO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:04,513] INFO creating herder for replica_TST->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:06,071] INFO creating herder for replica_VLD->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:08,700] INFO creating herder for replica_CNO->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:10,533] INFO creating herder for replica_VIT->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:12,539] INFO creating herder for replica_CNO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:14,749] INFO creating herder for replica_HBG->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:16,222] INFO creating herder for replica_UMO->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:18,308] INFO creating herder for replica_VIT->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:20,212] INFO creating herder for replica_HBG->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:22,618] INFO creating herder for replica_VLD->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:24,663] INFO creating herder for replica_UGO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:26,333] INFO creating herder for replica_ARA->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:28,408] INFO creating herder for replica_CNO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:30,480] INFO creating herder for replica_HBG->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:32,810] INFO creating herder for replica_VIT->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:34,904] INFO creating herder for replica_OLS->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:36,893] INFO creating herder for replica_UMO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:38,364] INFO creating herder for replica_UMO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:40,781] INFO creating herder for replica_OLS->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:43,168] INFO creating herder for replica_VIT->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:45,215] INFO creating herder for replica_CENTRAL->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:47,305] INFO creating herder for replica_UMO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 08:45:49,331] INFO creating herder for replica_VLD->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:16,537] INFO creating herder for replica_VIT->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:20,193] INFO creating herder for replica_CNO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:22,410] INFO creating herder for replica_CENTRAL->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:25,009] INFO creating herder for replica_UMO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:26,949] INFO creating herder for replica_CNO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:29,604] INFO creating herder for replica_ARA->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:32,121] INFO creating herder for replica_TST->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:34,883] INFO creating herder for replica_UMO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:37,323] INFO creating herder for replica_OLS->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:39,755] INFO creating herder for replica_UGO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:42,106] INFO creating herder for replica_HBG->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:44,425] INFO creating herder for replica_OLS->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:46,484] INFO creating herder for replica_TST->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:49,388] INFO creating herder for replica_OLS->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:51,300] INFO creating herder for replica_CNO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:54,560] INFO creating herder for replica_VIT->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:00:57,861] INFO creating herder for replica_CENTRAL->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:00,673] INFO creating herder for replica_ARA->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:03,222] INFO creating herder for replica_VLD->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:05,627] INFO creating herder for replica_HBG->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:07,752] INFO creating herder for replica_OLS->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:09,947] INFO creating herder for replica_ARA->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:12,492] INFO creating herder for replica_VLD->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:14,265] INFO creating herder for replica_CENTRAL->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:16,102] INFO creating herder for replica_VIT->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:17,661] INFO creating herder for replica_TST->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:19,440] INFO creating herder for replica_OLS->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:22,479] INFO creating herder for replica_TST->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:24,830] INFO creating herder for replica_OLS->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:26,478] INFO creating herder for replica_CNO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:29,179] INFO creating herder for replica_CENTRAL->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:31,616] INFO creating herder for replica_ARA->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:33,467] INFO creating herder for replica_CNO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:35,816] INFO creating herder for replica_UGO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:37,937] INFO creating herder for replica_HBG->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:40,195] INFO creating herder for replica_UMO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:42,318] INFO creating herder for replica_UGO->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:44,688] INFO creating herder for replica_HBG->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:47,083] INFO creating herder for replica_HBG->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:49,199] INFO creating herder for replica_TST->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:51,516] INFO creating herder for replica_ARA->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:53,042] INFO creating herder for replica_CNO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:54,575] INFO creating herder for replica_UGO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:57,081] INFO creating herder for replica_TST->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:01:58,540] INFO creating herder for replica_VLD->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:01,584] INFO creating herder for replica_CNO->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:03,401] INFO creating herder for replica_VIT->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:05,714] INFO creating herder for replica_CNO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:07,664] INFO creating herder for replica_HBG->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:09,246] INFO creating herder for replica_UMO->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:11,565] INFO creating herder for replica_VIT->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:13,451] INFO creating herder for replica_HBG->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:15,872] INFO creating herder for replica_VLD->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:18,094] INFO creating herder for replica_UGO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:19,597] INFO creating herder for replica_ARA->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:21,576] INFO creating herder for replica_CNO->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:23,834] INFO creating herder for replica_HBG->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:26,410] INFO creating herder for replica_VIT->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:28,516] INFO creating herder for replica_OLS->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:30,344] INFO creating herder for replica_UMO->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:31,850] INFO creating herder for replica_UMO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:34,094] INFO creating herder for replica_OLS->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:36,743] INFO creating herder for replica_VIT->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:38,778] INFO creating herder for replica_CENTRAL->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:40,785] INFO creating herder for replica_UMO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:42,816] INFO creating herder for replica_VLD->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:45,225] INFO creating herder for replica_VLD->replica_CENTRAL (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:46,659] INFO creating herder for replica_ARA->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:48,729] INFO creating herder for replica_TST->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:50,722] INFO creating herder for replica_TST->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:52,387] INFO creating herder for replica_UGO->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:54,326] INFO creating herder for replica_CENTRAL->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:56,085] INFO creating herder for replica_ARA->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:57,914] INFO creating herder for replica_VLD->replica_UMO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:02:59,665] INFO creating herder for replica_ARA->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:01,717] INFO creating herder for replica_OLS->replica_VLD (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:03,832] INFO creating herder for replica_UGO->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:06,278] INFO creating herder for replica_VLD->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:08,417] INFO creating herder for replica_CENTRAL->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:10,731] INFO creating herder for replica_UGO->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:12,712] INFO creating herder for replica_TST->replica_VIT (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:15,082] INFO creating herder for replica_VIT->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:17,439] INFO creating herder for replica_VLD->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:19,800] INFO creating herder for replica_HBG->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:21,554] INFO creating herder for replica_UMO->replica_HBG (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:23,536] INFO creating herder for replica_VIT->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:26,041] INFO creating herder for replica_CENTRAL->replica_TST (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:28,090] INFO creating herder for replica_UGO->replica_CNO (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:30,353] INFO creating herder for replica_CENTRAL->replica_ARA (org.apache.kafka.connect.mirror.MirrorMaker)
      [2020-11-12 09:03:32,826] INFO creating herder for replica_UMO->replica_OLS (org.apache.kafka.connect.mirror.MirrorMaker)

      So much that we reached the limit of our user property LimitNOFILE recently when trying to add a new "Local" cluster.

      I believe this behavior leads to unecessary connections and resource usage that could be easily avoided by just limiting the herder creation only to elements specifically described in the mirrormaker.properties file

      https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java#L130-L136

       

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #9589

          Activity

            People

              Unassigned Unassigned
              Twobeeb Julien Chanaud
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: