Flume
  1. Flume
  2. FLUME-114

Multi Masters not syncing updated configs properly

    Details

    • Type: Bug Bug
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v0.9.4
    • Fix Version/s: None
    • Component/s: Master
    • Labels:
      None

      Description

      If you have multiple masters and apply a config change to one of them. The config doesn't properly get past along to the other masters to apply to there listening nodes.

        Issue Links

          Activity

          Hide
          Disabled imported user added a comment -

          Hi Nick -

          Can you share your master configuration setup? (How many masters, how you have zookeeper configured etc)

          Henry

          Show
          Disabled imported user added a comment - Hi Nick - Can you share your master configuration setup? (How many masters, how you have zookeeper configured etc) Henry
          Hide
          Nicholas Verbeck added a comment -

          Setup looks like http://dpaste.com/225082/ minus that each master the flume.master.serverid has been change to 0,1,2 respectively.

          Show
          Nicholas Verbeck added a comment - Setup looks like http://dpaste.com/225082/ minus that each master the flume.master.serverid has been change to 0,1,2 respectively.
          Hide
          Disabled imported user added a comment - - edited

          I've seen this too in my 3 node setup. I usually have to go to the stuck node and issue a 'refreshAll' to fix. Sometimes I get a failed (something about not being able to increment a version number). Also, this is using an external zookeeper cluster (but on the same 3 machines) – if that matters. Restarting flume-master fixes that problem.

          Show
          Disabled imported user added a comment - - edited I've seen this too in my 3 node setup. I usually have to go to the stuck node and issue a 'refreshAll' to fix. Sometimes I get a failed (something about not being able to increment a version number). Also, this is using an external zookeeper cluster (but on the same 3 machines) – if that matters. Restarting flume-master fixes that problem.
          Hide
          Disabled imported user added a comment -

          I have a similar setup of 3 masters and external zookeeper cluster. Now for example I added 10 nodes to this setup in which some goes to master A, some to master B and remaining to master C. If I configure 1 node on master A as a collector (using source as autocollectorsource and sink to file) it works good for only the number of nodes configured on master A. for master B it is not recognizing the first collector. This gets worse after flume master restart as restart moves collector node to another master server and than it does not recognize the initial setup.
          Any suggestions?

          Ashok

          Show
          Disabled imported user added a comment - I have a similar setup of 3 masters and external zookeeper cluster. Now for example I added 10 nodes to this setup in which some goes to master A, some to master B and remaining to master C. If I configure 1 node on master A as a collector (using source as autocollectorsource and sink to file) it works good for only the number of nodes configured on master A. for master B it is not recognizing the first collector. This gets worse after flume master restart as restart moves collector node to another master server and than it does not recognize the initial setup. Any suggestions? Ashok
          Hide
          Jonathan Hsieh added a comment -

          These are related. FLUME-382 should allow multiple masters for configurations that do not require configuration translations.

          Show
          Jonathan Hsieh added a comment - These are related. FLUME-382 should allow multiple masters for configurations that do not require configuration translations.
          Hide
          Nicholas Verbeck added a comment -

          With the current trunk this seems to be working when using external ZK machines. I however have yet to try with built in ZK.

          Show
          Nicholas Verbeck added a comment - With the current trunk this seems to be working when using external ZK machines. I however have yet to try with built in ZK.
          Hide
          Alexander Alten-Lorenz added a comment -

          deprecated, flumeNG (1.0x)

          Show
          Alexander Alten-Lorenz added a comment - deprecated, flumeNG (1.0x)

            People

            • Assignee:
              Unassigned
              Reporter:
              Nicholas Verbeck
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development