Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.3.0
    • Component/s: None
    • Labels:
      None

      Description

      Looks like this test:
      org.apache.flume.sink.elasticsearch.TestElasticSearchSink.shouldIndexFiveEventsOverThreeBatches

      fails intermittently:
      https://builds.apache.org/job/flume-trunk/353/
      https://builds.apache.org/job/flume-trunk/341/

      1. FLUME-1862.patch
        2 kB
        Cameron Gandevia

        Activity

        Hari Shreedharan created issue -
        Hide
        Hari Shreedharan added a comment -

        Cameron Gandevia - Do you know whats happening here?

        Show
        Hari Shreedharan added a comment - Cameron Gandevia - Do you know whats happening here?
        Hide
        Cameron Gandevia added a comment -

        I think the tests were timing out because they were trying to recover the index during startup. I was able to reproduce the problem by running the unit test 10+ times. I have submitted a patch that uses the none gateway type which seems to have fixed the issue.

        Show
        Cameron Gandevia added a comment - I think the tests were timing out because they were trying to recover the index during startup. I was able to reproduce the problem by running the unit test 10+ times. I have submitted a patch that uses the none gateway type which seems to have fixed the issue.
        Cameron Gandevia made changes -
        Field Original Value New Value
        Attachment FLUME-1862.patch [ 12565907 ]
        Cameron Gandevia made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hari Shreedharan added a comment -

        Is the reason for the failure that the index is being recovered from disk? Looks like the failure is in the transaction commit?

        org.apache.flume.EventDeliveryException: Failed to commit transaction. Transaction rolled back.
        	at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:229)
        	at org.apache.flume.sink.elasticsearch.TestElasticSearchSink.shouldIndexFiveEventsOverThreeBatches(TestElasticSearchSink.java:139)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
        	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
        	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
        	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
        	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
        	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
        	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
        	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
        	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
        	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
        	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
        	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
        	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
        	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
        	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
        	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
        	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
        	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
        Caused by: org.apache.flume.EventDeliveryException: failure in bulk execution:
        [0]: index [flume-2013-01-18], type [log], id [JmKw9Ua5RTOoWtyzhM6QaA], message [UnavailableShardsException[[flume-2013-01-18][3] [2] shardIt, [0] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@2cd5a3]]
        [1]: index [flume-2013-01-18], type [log], id [ia_lCgKNT4K4aGJbbTzHyw], message [UnavailableShardsException[[flume-2013-01-18][4] [2] shardIt, [0] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@4a6ccc]]
        	at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:206)
        
        Show
        Hari Shreedharan added a comment - Is the reason for the failure that the index is being recovered from disk? Looks like the failure is in the transaction commit? org.apache.flume.EventDeliveryException: Failed to commit transaction. Transaction rolled back. at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:229) at org.apache.flume.sink.elasticsearch.TestElasticSearchSink.shouldIndexFiveEventsOverThreeBatches(TestElasticSearchSink.java:139) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) Caused by: org.apache.flume.EventDeliveryException: failure in bulk execution: [0]: index [flume-2013-01-18], type [log], id [JmKw9Ua5RTOoWtyzhM6QaA], message [UnavailableShardsException[[flume-2013-01-18][3] [2] shardIt, [0] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@2cd5a3]] [1]: index [flume-2013-01-18], type [log], id [ia_lCgKNT4K4aGJbbTzHyw], message [UnavailableShardsException[[flume-2013-01-18][4] [2] shardIt, [0] active : Timeout waiting for [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@4a6ccc]] at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:206)
        Hide
        Hari Shreedharan added a comment -

        FYI, I have never been able to reproduce this on my mac.

        Show
        Hari Shreedharan added a comment - FYI, I have never been able to reproduce this on my mac.
        Hide
        Hari Shreedharan added a comment -

        I am committing this. The test passes with this patch - if the test fails again, we can revisit this.

        Show
        Hari Shreedharan added a comment - I am committing this. The test passes with this patch - if the test fails again, we can revisit this.
        Hide
        Hari Shreedharan added a comment -

        Patch committed, rev: 4515117d56cff69accdf54ecbca0825ca6d71ab3. Thanks Cameron!

        Show
        Hari Shreedharan added a comment - Patch committed, rev: 4515117d56cff69accdf54ecbca0825ca6d71ab3. Thanks Cameron!
        Hari Shreedharan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Assignee Cameron Gandevia [ gnoremac ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #354 (See https://builds.apache.org/job/flume-trunk/354/)
        FLUME-1862. ElasticSearchSink test is flakey. (Revision 4515117d56cff69accdf54ecbca0825ca6d71ab3)

        Result = SUCCESS
        harishreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4515117d56cff69accdf54ecbca0825ca6d71ab3
        Files :

        • flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
        Show
        Hudson added a comment - Integrated in flume-trunk #354 (See https://builds.apache.org/job/flume-trunk/354/ ) FLUME-1862 . ElasticSearchSink test is flakey. (Revision 4515117d56cff69accdf54ecbca0825ca6d71ab3) Result = SUCCESS harishreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4515117d56cff69accdf54ecbca0825ca6d71ab3 Files : flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
        Mike Percy made changes -
        Fix Version/s v1.3.0 [ 12322140 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        7h 3m 1 Cameron Gandevia 22/Jan/13 05:46
        Patch Available Patch Available Resolved Resolved
        1h 42m 1 Hari Shreedharan 22/Jan/13 07:29

          People

          • Assignee:
            Cameron Gandevia
            Reporter:
            Hari Shreedharan
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development