Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.5.0
    • Component/s: Data Collection
    • Labels:
      None
    • Environment:

      Java 6, Mac OS X 10.6

      Description

      1) Start the agent with some "X" collectors.
      2) Update the conf/collectors file with "Y" collector entry and removing "X" collector entry.
      3) Doing telnet to port 9093
      4) Firing command "reloadcollectors".

      After reloading the collectors, agent responses as "OK reloadCollectors done". But agents logs shows trying to connect to old collector "X" and does not connect to new collector.

        Activity

        Hide
        eyang Eric Yang added a comment -

        Committed by Ari. Thanks.

        Show
        eyang Eric Yang added a comment - Committed by Ari. Thanks.
        Hide
        eyang Eric Yang added a comment -

        Yes, please commit it for me. Thanks (Taking time off to bond with my new born.)

        Show
        eyang Eric Yang added a comment - Yes, please commit it for me. Thanks (Taking time off to bond with my new born.)
        Hide
        asrabkin Ari Rabkin added a comment -

        Eric, is this committable?

        Show
        asrabkin Ari Rabkin added a comment - Eric, is this committable?
        Hide
        stuti Stuti Awasthi added a comment -

        Hi Eric.

        Thanks for providing the patch. I have tested it and it works fine

        Show
        stuti Stuti Awasthi added a comment - Hi Eric. Thanks for providing the patch. I have tested it and it works fine
        Hide
        eyang Eric Yang added a comment -

        Simple fix, but takes a long time to debug.

        Show
        eyang Eric Yang added a comment - Simple fix, but takes a long time to debug.
        Hide
        eyang Eric Yang added a comment -

        Rearrange reset collector list logic so ChukwaHttpSender gets updated correctly.

        Show
        eyang Eric Yang added a comment - Rearrange reset collector list logic so ChukwaHttpSender gets updated correctly.
        Hide
        eyang Eric Yang added a comment -

        It looks like the reload command has been send through AgentControlSocketListener -> HttpConnector, but ChukwaHttpSender is stuck on the original list. The maximum retries is very large in reliablySend by default (144000 Retries). When ChukwaHttpSender reaches maximum retry count with the original list, it will throw IOException("no collectors") to HttpConnector. HttpConnector will cause agent to shutdown due to no collector to try. The updated collector list is never really deployed to ChukwaHttpSender.

        Collector list change should be notifying ChukwaHttpSender as soon as reloadConfiguration is called.

        Show
        eyang Eric Yang added a comment - It looks like the reload command has been send through AgentControlSocketListener -> HttpConnector, but ChukwaHttpSender is stuck on the original list. The maximum retries is very large in reliablySend by default (144000 Retries). When ChukwaHttpSender reaches maximum retry count with the original list, it will throw IOException("no collectors") to HttpConnector. HttpConnector will cause agent to shutdown due to no collector to try. The updated collector list is never really deployed to ChukwaHttpSender. Collector list change should be notifying ChukwaHttpSender as soon as reloadConfiguration is called.

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development