Start from commit 9538af0 brought by jira HADOOP-12348, this test is failing consistently in trunk. Please double check this.

Test failure message is like:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 146.403 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
testSynchronousEviction(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter)  Time elapsed: 12.973 sec  <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<0>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.junit.Assert.assertEquals(Assert.java:542)
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.LazyPersistTestCase.verifyRamDiskJMXMetric(LazyPersistTestCase.java:483)
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter.testSynchronousEviction(TestLazyWriter.java:75)

Attached patch..
Please review...

Surendra Singh Lilhore thanks for taking this issue...Following testcases are failing from HADOOP-12348, It's better to address all as part of this issue and change the defect summary..you can check https://builds.apache.org/job/PreCommit-HDFS-Build/12407/testReport/ ( same is mentioned in that issue description)

All Failed Tests

Test Name
Duration
Age
 org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory.testReleaseOnEviction	10 sec	2
 org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testSynchronousEviction	13 sec	2
 org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testFallbackToDiskFull	0.77 sec	2
 org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter.testSynchronousEviction

Thanks Brahma Reddy Battula for comment..
HDFS-9074, HDFS-9073 already created for other test cases..

I think, the correct fix would be to fix at the metric system level.

Say, allow metric system to support cache TTL in millis, only for tests.
May be we can have a config in "hadoop-metrics2" with name, "periodMillis" and this will have priority over "period" which is in seconds.
So, any future tests which tests based on JMX metrics need not wait till the metric is updated.
Default value of 'periodMillis' can be decided based on requirement, may be 100msec or even 10msec.

Any thoughts?

Thanks Vinayakumar B for suggestion...
Attached patch , Please review ...

-1 overall

This message was automatically generated.

-1 overall

This message was automatically generated.

Failures are not completely gone. Only TestJMXGet is passed.

I think may be even 10 msec is more in Jenkins. As all tests passes locally with this, but not in jenkins.
We can make it to 1 msec, but changing that, will affect all tests, which are not monitoring metrics.
So I recommend, we can keep configuring periodMillis for tests, along with that we can wait for the metric to be updated to expected value.
Now tests need not wait for max 10 sec for the update. We can keep 100msec for periodMillis.

So IMO, its safe to combine v1 and v2 patches.

Thanks Vinayakumar B for review and suggestion.
Attached updated patch..

002:
1. Combine v1 and v2 patches
2. moved LazyPersistTestCase#waitForMetric() in DFSTestUtil so other test can can use it.

Please review..

-1 overall

This message was automatically generated.

hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement

For these two test separate jira is there HDFS-9074, HDFS-9073.

other test cases are passed locally.

 
-  protected void waitForMetric(final String metricName, final int expectedValue)
-      throws TimeoutException, InterruptedException {
-    GenericTestUtils.waitFor(new Supplier<Boolean>() {
-      @Override
-      public Boolean get() {
-        try {
-          final int currentValue = Integer.parseInt(jmx.getValue(metricName));
-          LOG.info("Waiting for " + metricName +
-                       " to reach value " + expectedValue +
-                       ", current value = " + currentValue);
-          return currentValue == expectedValue;
-        } catch (Exception e) {
-          throw new UnhandledException("Test failed due to unexpected exception", e);
-        }
-      }
-    }, 1000, Integer.MAX_VALUE);
-  }
-

I think instead of above complete removal of method, you can call DFSTestUtil#waitforMetric(..) from here, so that changes in other files need not be required.

Thanks Vinayakumar B for review ..
Attached updated patch, Please review ..

+1 for the latest patch.

Committed to trunk and branch-2.

Thanks Surendra Singh Lilhore for the contribution.
Thanks Haohui Mai for reporting.

Thanks Mingliang Liu and Brahma Reddy Battula for helping in analysis.

ABORTED: Integrated in Hadoop-Mapreduce-trunk-Java8 #388 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/388/)
HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter is failing in trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev a4405674919d14be89bc4da22db2f417b5ae6ac3)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyWriter.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-metrics2.properties
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java

-1 overall

This message was automatically generated.

FAILURE: Integrated in Hadoop-trunk-Commit #8454 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8454/)
HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter is failing in trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev a4405674919d14be89bc4da22db2f417b5ae6ac3)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyWriter.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-metrics2.properties
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2336 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2336/)
HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter is failing in trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev a4405674919d14be89bc4da22db2f417b5ae6ac3)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyWriter.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-metrics2.properties
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #394 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/394/)
HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter is failing in trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev a4405674919d14be89bc4da22db2f417b5ae6ac3)

• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyWriter.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-metrics2.properties

FAILURE: Integrated in Hadoop-Yarn-trunk #1128 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1128/)
HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter is failing in trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev a4405674919d14be89bc4da22db2f417b5ae6ac3)

• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-metrics2.properties
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyWriter.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #373 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/373/)
HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter is failing in trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev a4405674919d14be89bc4da22db2f417b5ae6ac3)

• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-metrics2.properties
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyWriter.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java

FAILURE: Integrated in Hadoop-Hdfs-trunk #2313 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2313/)
HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter is failing in trunk (Contributed by Surendra Singh Lilhore) (vinayakumarb: rev a4405674919d14be89bc4da22db2f417b5ae6ac3)

• hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
• hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyWriter.java
• hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-metrics2.properties
• hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java