Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-737

Add heartbeat adaptor to push chukwa agent status to remote http server

    Details

    • Type: Improvement
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7.0
    • Fix Version/s: None
    • Component/s: Data Collection
    • Labels:
      None

      Description

      Provide a generic HeartbeatAdaptor which can periodically send status reports to a remote http server. The adaptor can be configured with arbitrary number of component status checkers and it will send out a single status report for all components.

      Example - to send out the status of chukwa agent as well as a rest interface (in this case chukwa agent rest), we add the following adaptor:
      add HeartbeatAdaptor DefaultProcessor (ChukwaStatusChecker, HttpStatusChecker Chukwa.rest.server http://localhost:9090/rest/v1/adaptor) 3 0

      This will send out status reports such as -
      {"time":1424213867980,"components":[

      {"host":"host.example.com","component":"Chukwa.Agent","adaptor.count":1}

      ,

      {"status":"running","component":"Chukwa.rest.server","uri":"http:\/\/localhost:9090\/rest\/v1"}

      ]}

      Additional configuration needed in chukwa-agent-conf.xml
      chukwa.http.writer.host
      chukwa.http.writer.port
      if PipelineConnector is used, add org.apache.hadoop.chukwa.datacollection.writer.HttpWriter to head of pipeline

      1. CHUKWA-737.patch
        23 kB
        shreyas subramanya
      2. CHUKWA-737.patch2
        23 kB
        shreyas subramanya
      3. CHUKWA-737.patch3
        24 kB
        shreyas subramanya

        Activity

        Hide
        eyang Eric Yang added a comment -

        I think the rest api has moved to v2 in Chukwa 0.6. Could we update the url to v2 instead?

        Show
        eyang Eric Yang added a comment - I think the rest api has moved to v2 in Chukwa 0.6. Could we update the url to v2 instead?
        Hide
        shreyass123 shreyas subramanya added a comment -

        Changed rest url to v2. Thanks!

        Show
        shreyass123 shreyas subramanya added a comment - Changed rest url to v2. Thanks!
        Hide
        eyang Eric Yang added a comment -

        TestHeartbeatAdaptor is registering adaptor for test, but it does not clean up registered adaptor after the test. This causes TestAdaptorController to fail because the number of expected adaptor registered is 1 instead of 2.

        Show
        eyang Eric Yang added a comment - TestHeartbeatAdaptor is registering adaptor for test, but it does not clean up registered adaptor after the test. This causes TestAdaptorController to fail because the number of expected adaptor registered is 1 instead of 2.
        Hide
        shreyass123 shreyas subramanya added a comment -

        Fixed the test cases to shutdown the adaptor correctly

        Show
        shreyass123 shreyas subramanya added a comment - Fixed the test cases to shutdown the adaptor correctly
        Hide
        eyang Eric Yang added a comment -

        +1 Thanks Shreyas.

        Show
        eyang Eric Yang added a comment - +1 Thanks Shreyas.
        Hide
        shreyass123 shreyas subramanya added a comment -

        Thanks Eric, committed this.

        Show
        shreyass123 shreyas subramanya added a comment - Thanks Eric, committed this.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Chukwa-trunk #523 (See https://builds.apache.org/job/Chukwa-trunk/523/)
        CHUKWA-737. Heartbeat adaptor to push status to remote http server (Shreyas Subramanya) (shreyas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1662365)

        • /chukwa/trunk/CHANGES.txt
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/HeartbeatAdaptor.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/ChukwaStatusChecker.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/HttpStatusChecker.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/StatusChecker.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/StatusCheckerException.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/writer/HttpWriter.java
        • /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/TestHeartbeatAdaptor.java
        • /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/agent/rest/TestAdaptorController.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Chukwa-trunk #523 (See https://builds.apache.org/job/Chukwa-trunk/523/ ) CHUKWA-737 . Heartbeat adaptor to push status to remote http server (Shreyas Subramanya) (shreyas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1662365 ) /chukwa/trunk/CHANGES.txt /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/HeartbeatAdaptor.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/ChukwaStatusChecker.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/HttpStatusChecker.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/StatusChecker.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/StatusCheckerException.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/writer/HttpWriter.java /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/TestHeartbeatAdaptor.java /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/agent/rest/TestAdaptorController.java
        Hide
        eyang Eric Yang added a comment -

        This is causing build to fail. It looks like the same issue where Adaptor has not been removed cleanly before the next test is executed. Therefore the data feeding the test case is producing incorrect input to the next test case. I think you want to remove heartbeat adaptor before TestHeartbeatAdaptor test case exits.

        Show
        eyang Eric Yang added a comment - This is causing build to fail. It looks like the same issue where Adaptor has not been removed cleanly before the next test is executed. Therefore the data feeding the test case is producing incorrect input to the next test case. I think you want to remove heartbeat adaptor before TestHeartbeatAdaptor test case exits.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Chukwa-trunk #527 (See https://builds.apache.org/job/Chukwa-trunk/527/)
        CHUKWA-737. Heartbeat adaptor to push status to remote http server (Shreyas Subramanya) (shreyas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1662365)

        • /chukwa/trunk/CHANGES.txt
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/HeartbeatAdaptor.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/ChukwaStatusChecker.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/HttpStatusChecker.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/StatusChecker.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/StatusCheckerException.java
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/writer/HttpWriter.java
        • /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/TestHeartbeatAdaptor.java
        • /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/agent/rest/TestAdaptorController.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Chukwa-trunk #527 (See https://builds.apache.org/job/Chukwa-trunk/527/ ) CHUKWA-737 . Heartbeat adaptor to push status to remote http server (Shreyas Subramanya) (shreyas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1662365 ) /chukwa/trunk/CHANGES.txt /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/HeartbeatAdaptor.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/ChukwaStatusChecker.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/HttpStatusChecker.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/StatusChecker.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/heartbeat/StatusCheckerException.java /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/writer/HttpWriter.java /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/TestHeartbeatAdaptor.java /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/agent/rest/TestAdaptorController.java

          People

          • Assignee:
            shreyass123 shreyas subramanya
            Reporter:
            shreyass123 shreyas subramanya
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development