[SOLR-13068] many cloud/autoscaling tests are using System.currentTimeMillis() for timing comparisons (under the covers)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 7.7, 8.0
• Assignee: Chris M. Hostetter
• Reporter: Chris M. Hostetter
• Created: 12/Dec/18 22:33
• Updated: 11/Feb/19 09:20
• Resolved: 17/Dec/18 18:23

Description

After rewriting TriggerSetPropertiesIntegrationTest in SOLR-13054 to use better concurrency handling/signalling and log the timestamps the triggers were firing at, i noticed we stil got a failure from Uwe's "Windows" jenkins machine (on the http2 branch, but after my fix was merged to that branch. The nature of the failure seemed to suggest that the JVM's ScheduledExecutorService.scheduleWithFixedDelay wasn't living up to it's contract – and was running successive iterations before the full delay had lapsed.

But then i realized that in spite of using timeSource.getTimeNs() in the test, the TimeSource (being used in the test) was a lie – and under the covers System.currentTimeInMillis is being used (via TimeSource.CURRENT_TIME ) ... which IIUC is susceptible to clock drift, particularly in VMs like those used on Uwe's jenkins machines...

Any code in the following tests that relies on the TimeSource for doing comparisons or delta calculations should be suspect...

hossman@tray:~/lucene/dev [master] $ find solr/core/src/test -name \*.java | xargs grep CURRENT_TIME
solr/core/src/test/org/apache/solr/cloud/autoscaling/NodeAddedTriggerTest.java:  private static final TimeSource timeSource = TimeSource.CURRENT_TIME;
solr/core/src/test/org/apache/solr/cloud/autoscaling/ExecutePlanActionTest.java:          "mock_trigger_name", Collections.singletonList(TimeSource.CURRENT_TIME.getTimeNs()),
solr/core/src/test/org/apache/solr/cloud/autoscaling/TriggerIntegrationTest.java:  static final TimeSource timeSource = TimeSource.CURRENT_TIME;
solr/core/src/test/org/apache/solr/cloud/autoscaling/NodeLostTriggerTest.java:  private final TimeSource timeSource = TimeSource.CURRENT_TIME;
solr/core/src/test/org/apache/solr/cloud/autoscaling/sim/TestSimExecutePlanAction.java:          "mock_trigger_name", Collections.singletonList(TimeSource.CURRENT_TIME.getTimeNs()),
solr/core/src/test/org/apache/solr/cloud/HttpPartitionTest.java:    TimeOut timeOut = new TimeOut(ms, TimeUnit.MILLISECONDS, TimeSource.CURRENT_TIME);
solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java:    TimeOut timeOut = new TimeOut(10, TimeUnit.SECONDS, TimeSource.CURRENT_TIME);
solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java:    TimeOut timeOut = new TimeOut(10, TimeUnit.SECONDS, TimeSource.CURRENT_TIME);
hossman@tray:~/lucene/dev [master] $ find -name \*.java | xargs grep TriggerIntegrationTest.timeSource
./solr/core/src/test/org/apache/solr/cloud/autoscaling/SearchRateTriggerIntegrationTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;
./solr/core/src/test/org/apache/solr/cloud/autoscaling/TriggerSetPropertiesIntegrationTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;
./solr/core/src/test/org/apache/solr/cloud/autoscaling/RestoreTriggerStateTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;
./solr/core/src/test/org/apache/solr/cloud/autoscaling/NodeLostTriggerIntegrationTest.java:          long currentTimeNanos = TriggerIntegrationTest.timeSource.getTimeNs();
./solr/core/src/test/org/apache/solr/cloud/autoscaling/MetricTriggerIntegrationTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;
./solr/core/src/test/org/apache/solr/cloud/autoscaling/NodeAddedTriggerIntegrationTest.java:          long currentTimeNanos = TriggerIntegrationTest.timeSource.getTimeNs();
./solr/core/src/test/org/apache/solr/cloud/autoscaling/TriggerCooldownIntegrationTest.java:import static org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.timeSource;

Issue Links

• is related to: SOLR-13054 TriggerSetPropertiesIntegrationTest cleanup