[STORM-3167] Flaky test in metrics_test.clj

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: storm-core
• Labels: pull-request-available
• Assignee: Stig Rohde Døssing
• Reporter: Stig Rohde Døssing
• Created: 01/Aug/18 14:31
• Updated: 03/Aug/18 15:42
• Resolved: 03/Aug/18 15:42

Description

classname: org.apache.storm.metrics-test / testname: test-builtin-metrics-2
Uncaught exception, not in assertion.
expected: nil
 actual: java.util.ConcurrentModificationException: null
 at java.util.ArrayList$Itr.checkForComodification (ArrayList.java:907)
 java.util.ArrayList$Itr.next (ArrayList.java:857)
 com.google.common.collect.AbstractMapBasedMultimap$WrappedCollection$WrappedIterator.next (AbstractMapBasedMultimap.java:486)
 clojure.lang.PersistentVector.create (PersistentVector.java:105)
 clojure.lang.LazilyPersistentVector.create (LazilyPersistentVector.java:32)
 clojure.core$vec.invoke (core.clj:361)
 org.apache.storm.util$clojurify_structure$fn__206.invoke (util.clj:85)
 clojure.walk$prewalk.invoke (walk.clj:64)
 clojure.core$partial$fn__4527.invoke (core.clj:2493)
 clojure.core$map$fn__4553.invoke (core.clj:2622)
 clojure.lang.LazySeq.sval (LazySeq.java:40)
 clojure.lang.LazySeq.seq (LazySeq.java:49)
 clojure.lang.RT.seq (RT.java:507)
 clojure.core/seq (core.clj:137)
 clojure.core.protocols$seq_reduce.invoke (protocols.clj:30)
 clojure.core.protocols/fn (protocols.clj:101)
 clojure.core.protocols$fn__6452$G__6447__6465.invoke (protocols.clj:13)
 clojure.core$reduce.invoke (core.clj:6519)
 clojure.core$into.invoke (core.clj:6600)
 clojure.walk$walk.invoke (walk.clj:49)
 clojure.walk$prewalk.invoke (walk.clj:64)
 clojure.core$partial$fn__4527.invoke (core.clj:2493)
 clojure.core$map$fn__4553.invoke (core.clj:2624)
 clojure.lang.LazySeq.sval (LazySeq.java:40)
 clojure.lang.LazySeq.seq (LazySeq.java:49)
 clojure.lang.RT.seq (RT.java:507)
 clojure.core/seq (core.clj:137)
 clojure.core.protocols$seq_reduce.invoke (protocols.clj:30)
 clojure.core.protocols/fn (protocols.clj:101)
 clojure.core.protocols$fn__6452$G__6447__6465.invoke (protocols.clj:13)
 clojure.core$reduce.invoke (core.clj:6519)
 clojure.core$into.invoke (core.clj:6600)
 clojure.walk$walk.invoke (walk.clj:49)
 clojure.walk$prewalk.invoke (walk.clj:64)
 org.apache.storm.util$clojurify_structure.invoke (util.clj:83)
 org.apache.storm.metrics_test$wait_for_atleast_N_buckets_BANG_$reify__1258.exec (metrics_test.clj:79)
 org.apache.storm.Testing.whileTimeout (Testing.java:103)
 org.apache.storm.metrics_test$wait_for_atleast_N_buckets_BANG_.invoke (metrics_test.clj:77)
 org.apache.storm.metrics_test$assert_metric_running_sum_BANG_.invoke (metrics_test.clj:98)
 org.apache.storm.metrics_test/fn (metrics_test.clj:326)
 clojure.test$test_var$fn__7670.invoke (test.clj:704)
 clojure.test$test_var.invoke (test.clj:704)
 clojure.test$test_vars$fn__7692$fn__7697.invoke (test.clj:722)
 clojure.test$default_fixture.invoke (test.clj:674)
 clojure.test$test_vars$fn__7692.invoke (test.clj:722)
 clojure.test$default_fixture.invoke (test.clj:674)
 clojure.test$test_vars.invoke (test.clj:718)
 clojure.test$test_all_vars.invoke (test.clj:728)
 clojure.test$test_ns.invoke (test.clj:747)
 clojure.core$map$fn__4553.invoke (core.clj:2624)
 clojure.lang.LazySeq.sval (LazySeq.java:40)
 clojure.lang.LazySeq.seq (LazySeq.java:49)
 clojure.lang.Cons.next (Cons.java:39)
 clojure.lang.RT.boundedLength (RT.java:1735)
 clojure.lang.RestFn.applyTo (RestFn.java:130)
 clojure.core$apply.invoke (core.clj:632)
 clojure.test$run_tests.doInvoke (test.clj:762)
 clojure.lang.RestFn.invoke (RestFn.java:408)
 org.apache.storm.testrunner$eval5125$iter__5126__5130$fn__5131$fn__5132$fn__5133.invoke (test_runner.clj:107)
 org.apache.storm.testrunner$eval5125$iter__5126__5130$fn__5131$fn__5132.invoke (test_runner.clj:53)
 org.apache.storm.testrunner$eval5125$iter__5126__5130$fn__5131.invoke (test_runner.clj:52)
 clojure.lang.LazySeq.sval (LazySeq.java:40)
 clojure.lang.LazySeq.seq (LazySeq.java:49)
 clojure.lang.RT.seq (RT.java:507)
 clojure.core/seq (core.clj:137)
 clojure.core$dorun.invoke (core.clj:3009)
 org.apache.storm.testrunner$eval5125.invoke (test_runner.clj:52)
 clojure.lang.Compiler.eval (Compiler.java:6782)
 clojure.lang.Compiler.load (Compiler.java:7227)
 clojure.lang.Compiler.loadFile (Compiler.java:7165)
 clojure.main$load_script.invoke (main.clj:275)
 clojure.main$script_opt.invoke (main.clj:337)
 clojure.main$main.doInvoke (main.clj:421)
 clojure.lang.RestFn.invoke (RestFn.java:421)
 clojure.lang.Var.invoke (Var.java:383)
 clojure.lang.AFn.applyToHelper (AFn.java:156)
 clojure.lang.Var.applyTo (Var.java:700)
 clojure.main.main (main.java:37)

 

It looks to me like the issue is that the FakeMetricsConsumer.getTaskIdToBuckets returns a view of a map that may be modified at any time (the getTaskIdToBuckets is synchronized on the map). When the method returns, the lock is released, but the return value is a view of the map, rather than a copy. This makes iteration over the return value unsafe. The method should instead copy the map before returning it.

Issue Links

• links to: GitHub Pull Request #2785