Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Data Collection
    • Labels:
      None

      Description

      To some extent, we rely on backpressure from collectors to throttle agent sends. This should be cleaned up and tested.

      1. CHUKWA-373.patch
        7 kB
        Ari Rabkin
      2. CHUKWA-373.patch
        13 kB
        Ari Rabkin

        Activity

        Hide
        asrabkin Ari Rabkin added a comment -

        Mostly just test code, so I will take silence for consent by Tuesday or so.

        Show
        asrabkin Ari Rabkin added a comment - Mostly just test code, so I will take silence for consent by Tuesday or so.
        Hide
        asrabkin Ari Rabkin added a comment -

        I just committed this.

        Show
        asrabkin Ari Rabkin added a comment - I just committed this.
        Hide
        eyang Eric Yang added a comment -

        Test fails:

        [junit] Running org.apache.hadoop.chukwa.datacollection.collector.TestBackpressure
        [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 21.722 sec

        The test result is inconsistent, it fails sometimes.

        Test Log output:

        Testsuite: org.apache.hadoop.chukwa.datacollection.collector.TestBackpressure
        Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 21.83 sec

        ------------- Standard Output ---------------
        data rate was 193 kb /second
        ------------- ---------------- ---------------
        ------------- Standard Error -----------------
        Error in RetryListOfCollectors() opening file: collectors, double check that you have set the CHUKWA_CONF_DIR environment variable. Also, ensure file exists and is in classpath
        2009-08-18 11:17:25.907::INFO: Logging to STDERR via org.mortbay.log.StdErrLog
        2009-08-18 11:17:25.012::INFO: jetty-6.1.11
        2009-08-18 11:17:25.054::INFO: Started SocketConnector@0.0.0.0:9991
        ------------- ---------------- ---------------
        Testcase: testBackpressure(org.apache.hadoop.chukwa.datacollection.collector.TestBackpressure): FAILED
        null
        junit.framework.AssertionFailedError
        at org.apache.hadoop.chukwa.datacollection.collector.TestBackpressure.testBackpressure(TestBackpressure.java:50)

        Show
        eyang Eric Yang added a comment - Test fails: [junit] Running org.apache.hadoop.chukwa.datacollection.collector.TestBackpressure [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 21.722 sec The test result is inconsistent, it fails sometimes. Test Log output: Testsuite: org.apache.hadoop.chukwa.datacollection.collector.TestBackpressure Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 21.83 sec ------------- Standard Output --------------- data rate was 193 kb /second ------------- ---------------- --------------- ------------- Standard Error ----------------- Error in RetryListOfCollectors() opening file: collectors, double check that you have set the CHUKWA_CONF_DIR environment variable. Also, ensure file exists and is in classpath 2009-08-18 11:17:25.907::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2009-08-18 11:17:25.012::INFO: jetty-6.1.11 2009-08-18 11:17:25.054::INFO: Started SocketConnector@0.0.0.0:9991 ------------- ---------------- --------------- Testcase: testBackpressure(org.apache.hadoop.chukwa.datacollection.collector.TestBackpressure): FAILED null junit.framework.AssertionFailedError at org.apache.hadoop.chukwa.datacollection.collector.TestBackpressure.testBackpressure(TestBackpressure.java:50)
        Hide
        asrabkin Ari Rabkin added a comment -

        The test was intended to also verify performance of collectors; I may have been substantially too optimistic. I'll fix this.

        Show
        asrabkin Ari Rabkin added a comment - The test was intended to also verify performance of collectors; I may have been substantially too optimistic. I'll fix this.
        Hide
        eyang Eric Yang added a comment -

        Fresh checkout, the build didn't compile after I applied the new patch.

        compile-src:
        [javac] Compiling 2 source files to /Users/eyang/sandbox/chukwa/trunk/build/contrib/chukwa-xtrace/classes
        [javac] /Users/eyang/sandbox/chukwa/trunk/contrib/xtrace/src/java/edu/berkeley/chukwa_xtrace/XtrAdaptor.java:116: warning: [deprecation] hardStop() in org.apache.hadoop.chukwa.datacollection.adaptor.Adaptor has been deprecated
        [javac] public void hardStop() throws AdaptorException {
        [javac] ^
        [javac] /Users/eyang/sandbox/chukwa/trunk/contrib/xtrace/src/java/edu/berkeley/chukwa_xtrace/XtrAdaptor.java:121: warning: [deprecation] shutdown() in org.apache.hadoop.chukwa.datacollection.adaptor.Adaptor has been deprecated
        [javac] public long shutdown() throws AdaptorException {
        [javac] ^
        [javac] /Users/eyang/sandbox/chukwa/trunk/contrib/xtrace/src/java/edu/berkeley/chukwa_xtrace/XtrAdaptor.java:105: getCurrentStatus() in edu.berkeley.chukwa_xtrace.XtrAdaptor cannot implement getCurrentStatus() in org.apache.hadoop.chukwa.datacollection.adaptor.Adaptor; overridden method does not throw org.apache.hadoop.chukwa.datacollection.adaptor.AdaptorException
        [javac] public String getCurrentStatus() throws AdaptorException {
        [javac] ^
        [javac] 1 error
        [javac] 2 warnings

        BUILD FAILED
        /Users/eyang/sandbox/chukwa/trunk/build.xml:460: The following error occurred while executing this line:
        /Users/eyang/sandbox/chukwa/trunk/contrib/build.xml:30: The following error occurred while executing this line:
        /Users/eyang/sandbox/chukwa/trunk/contrib/xtrace/build.xml:211: Compile failed; see the compiler error output for details.

        Show
        eyang Eric Yang added a comment - Fresh checkout, the build didn't compile after I applied the new patch. compile-src: [javac] Compiling 2 source files to /Users/eyang/sandbox/chukwa/trunk/build/contrib/chukwa-xtrace/classes [javac] /Users/eyang/sandbox/chukwa/trunk/contrib/xtrace/src/java/edu/berkeley/chukwa_xtrace/XtrAdaptor.java:116: warning: [deprecation] hardStop() in org.apache.hadoop.chukwa.datacollection.adaptor.Adaptor has been deprecated [javac] public void hardStop() throws AdaptorException { [javac] ^ [javac] /Users/eyang/sandbox/chukwa/trunk/contrib/xtrace/src/java/edu/berkeley/chukwa_xtrace/XtrAdaptor.java:121: warning: [deprecation] shutdown() in org.apache.hadoop.chukwa.datacollection.adaptor.Adaptor has been deprecated [javac] public long shutdown() throws AdaptorException { [javac] ^ [javac] /Users/eyang/sandbox/chukwa/trunk/contrib/xtrace/src/java/edu/berkeley/chukwa_xtrace/XtrAdaptor.java:105: getCurrentStatus() in edu.berkeley.chukwa_xtrace.XtrAdaptor cannot implement getCurrentStatus() in org.apache.hadoop.chukwa.datacollection.adaptor.Adaptor; overridden method does not throw org.apache.hadoop.chukwa.datacollection.adaptor.AdaptorException [javac] public String getCurrentStatus() throws AdaptorException { [javac] ^ [javac] 1 error [javac] 2 warnings BUILD FAILED /Users/eyang/sandbox/chukwa/trunk/build.xml:460: The following error occurred while executing this line: /Users/eyang/sandbox/chukwa/trunk/contrib/build.xml:30: The following error occurred while executing this line: /Users/eyang/sandbox/chukwa/trunk/contrib/xtrace/build.xml:211: Compile failed; see the compiler error output for details.
        Hide
        asrabkin Ari Rabkin added a comment -

        Yes. That's what CHUKWA-377 fixes. I was waiting for a +1 before committing the fix. But I'll go do it now.

        Show
        asrabkin Ari Rabkin added a comment - Yes. That's what CHUKWA-377 fixes. I was waiting for a +1 before committing the fix. But I'll go do it now.
        Hide
        asrabkin Ari Rabkin added a comment -

        I notice that some of the test-case fixes here are inadvertently redundant with CHUKWA-380. I'll do the merge on my end, if we wind up committing both.

        Show
        asrabkin Ari Rabkin added a comment - I notice that some of the test-case fixes here are inadvertently redundant with CHUKWA-380 . I'll do the merge on my end, if we wind up committing both.
        Hide
        asrabkin Ari Rabkin added a comment -

        Also, I will take silence for consent if I don't hear back by thursday afternoon or so – this is a bugfix, needed to make trunk unit tests pass more reliably, and not a refactoring

        Show
        asrabkin Ari Rabkin added a comment - Also, I will take silence for consent if I don't hear back by thursday afternoon or so – this is a bugfix, needed to make trunk unit tests pass more reliably, and not a refactoring
        Hide
        asrabkin Ari Rabkin added a comment -

        Unit tests pass, and I just committed this.

        Show
        asrabkin Ari Rabkin added a comment - Unit tests pass, and I just committed this.
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #112 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/112/)
        . Fix Backpressure test case

        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #112 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/112/ ) . Fix Backpressure test case

          People

          • Assignee:
            asrabkin Ari Rabkin
            Reporter:
            asrabkin Ari Rabkin
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development