Description
Check the access time of block file to avoid scanning recently changed blocks, reducing disk IO.
Attachments
Attachments
- HDFS-15207.002.patch
- 25 kB
- Yang Yun
- HDFS-15207.003.patch
- 8 kB
- Yang Yun
- HDFS-15207.004.patch
- 9 kB
- Yang Yun
- HDFS-15207.005.patch
- 9 kB
- Yang Yun
- HDFS-15207.006.patch
- 9 kB
- Yang Yun
- HDFS-15207.007.patch
- 9 kB
- Yang Yun
- HDFS-15207.008.patch
- 9 kB
- Yang Yun
- HDFS-15207.patch
- 7 kB
- Yang Yun
- HDFS-15207.patch
- 4 kB
- Yang Yun
Activity
I think this is good, let's improve the readability a little:
- At the start of the else added in VolumeScanner, let's comment what we have in the description of this JIRA: Check the access time of block file to avoid scanning recently changed blocks, reducing disk IO.
- No need to check replica != null in line 549, if Im not wrong, instanceof already handles that.
- When catching the IOException, let's log it as debug. It doesn't hurt and potentially can help debuggability.
- In the test, when doing the waitFor, let's do it with a lambda.
- Let's avoid adding the 200ms sleep.
- Let's make readFiles private and add it next to the function where we use.
Thanks elgoiri for the review.
Modified according to your comments.
I'm sorry I didn't catch one comoent "In the test, when doing the waitFor, let's do it with a lambda.". Should we run the waitFor in a thread?
In adition, I add a option to enbale this feature for some test cases of TestBlockScanner is affected.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 50s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 21m 48s | trunk passed |
+1 | compile | 1m 13s | trunk passed |
+1 | checkstyle | 0m 59s | trunk passed |
+1 | mvnsite | 1m 33s | trunk passed |
+1 | shadedclient | 18m 48s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 32s | trunk passed |
+1 | javadoc | 0m 50s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 20s | the patch passed |
+1 | compile | 1m 10s | the patch passed |
-1 | javac | 1m 10s | hadoop-hdfs-project_hadoop-hdfs generated 1 new + 579 unchanged - 0 fixed = 580 total (was 579) |
-0 | checkstyle | 0m 53s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 501 unchanged - 0 fixed = 504 total (was 501) |
+1 | mvnsite | 1m 21s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 15m 38s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 36s | the patch passed |
+1 | javadoc | 0m 51s | the patch passed |
Other Tests | |||
-1 | unit | 117m 58s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
192m 42s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.TestEncryptionZonesWithKMS |
hadoop.tools.TestHdfsConfigFields | |
hadoop.hdfs.server.balancer.TestBalancer | |
hadoop.hdfs.TestEncryptionZones | |
hadoop.hdfs.server.datanode.TestBlockScanner | |
hadoop.hdfs.TestRollingUpgrade |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.7 Server=19.03.7 Image:yetus/hadoop:c44943d1fc3 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12995813/HDFS-15207.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux f860d8d9e4d7 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 004e955 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_242 |
findbugs | v3.1.0-RC1 |
javac | https://builds.apache.org/job/PreCommit-HDFS-Build/28902/artifact/out/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/28902/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28902/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/28902/testReport/ |
Max. process+thread count | 3073 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/28902/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Thanks hadoop_yangyun for the update.
Can you number the patches with HDFS-15207.001.patch and so on?
For the lambda, instead of doing:
GenericTestUtils.waitFor(new Supplier<Boolean>() { ... });
Do:
GenericTestUtils.waitFor(() -> { ... });
Thanks elgoiri for the review.
Got the lambda, thanks!
For the public function readFiles, is a method of private class TestContext.
new patch HDFS-15207.001.patch is added.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 45s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 19m 44s | trunk passed |
+1 | compile | 1m 1s | trunk passed |
+1 | checkstyle | 0m 49s | trunk passed |
+1 | mvnsite | 1m 6s | trunk passed |
+1 | shadedclient | 16m 2s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 47s | trunk passed |
+1 | javadoc | 0m 38s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 2s | the patch passed |
+1 | compile | 0m 56s | the patch passed |
-1 | javac | 0m 56s | hadoop-hdfs-project_hadoop-hdfs generated 1 new + 579 unchanged - 0 fixed = 580 total (was 579) |
-0 | checkstyle | 0m 44s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 501 unchanged - 0 fixed = 502 total (was 501) |
+1 | mvnsite | 1m 1s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | shadedclient | 14m 23s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 54s | the patch passed |
+1 | javadoc | 0m 37s | the patch passed |
Other Tests | |||
-1 | unit | 106m 54s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 33s | The patch does not generate ASF License warnings. |
172m 4s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
hadoop.hdfs.tools.TestDFSZKFailoverController |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.7 Server=19.03.7 Image:yetus/hadoop:c44943d1fc3 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12995991/HDFS-15207.001.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
uname | Linux 57e636464541 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 3859fa7 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_242 |
findbugs | v3.1.0-RC1 |
javac | https://builds.apache.org/job/PreCommit-HDFS-Build/28913/artifact/out/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/28913/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28913/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/28913/testReport/ |
Max. process+thread count | 3144 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/28913/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
One warning of checkstyle is for the Variable 'skipRecentAccessed', which is member of inside class Conf, keeping the same style with other variables. I think we can ignore it.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockScanner.java:117: final boolean skipRecentAccessed;:19: Variable 'skipRecentAccessed' must be private and have accessor methods. [VisibilityModifier]
A minor style comment, let's make the Thread a lambda too:
new Thread(() -> { ... }).start();
For the waitFor, make it 10 millis or so, 1 milli seems a little too much.
Thanks elgoiri for the review.
Simplified the test case. Please help review again.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 50s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 20m 20s | trunk passed |
+1 | compile | 1m 2s | trunk passed |
+1 | checkstyle | 0m 51s | trunk passed |
+1 | mvnsite | 1m 7s | trunk passed |
+1 | shadedclient | 16m 8s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 50s | trunk passed |
+1 | javadoc | 0m 38s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 2s | the patch passed |
+1 | compile | 0m 57s | the patch passed |
-1 | javac | 0m 57s | hadoop-hdfs-project_hadoop-hdfs generated 1 new + 579 unchanged - 0 fixed = 580 total (was 579) |
-0 | checkstyle | 0m 44s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 501 unchanged - 0 fixed = 504 total (was 501) |
+1 | mvnsite | 1m 1s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | shadedclient | 14m 15s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 54s | the patch passed |
+1 | javadoc | 0m 37s | the patch passed |
Other Tests | |||
-1 | unit | 105m 21s | hadoop-hdfs in the patch passed. |
-1 | asflicense | 0m 33s | The patch generated 4 ASF License warnings. |
171m 16s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.TestMaintenanceState |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.7 Server=19.03.7 Image:yetus/hadoop:c44943d1fc3 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12996230/HDFS-15207.002.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
uname | Linux e66d98081e9b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 44afe11 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_242 |
findbugs | v3.1.0-RC1 |
javac | https://builds.apache.org/job/PreCommit-HDFS-Build/28917/artifact/out/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/28917/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28917/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/28917/testReport/ |
asflicense | https://builds.apache.org/job/PreCommit-HDFS-Build/28917/artifact/out/patch-asflicense-problems.txt |
Max. process+thread count | 2961 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/28917/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Thanks hadoop_yangyun for your works. This improvement seems to depend on access time of local file attribution. Any case that access this file from external process but DataNode? And there are also some random reads to get local file attribution. Is it possible to add lastScanTime for ReplicaInfo to determine if scan or skip in the next time? Of cause, it will occupy extra heap memory. Thanks again.
Thanks hexiaoqiao for the suggestion.
Yes, the access time changes if it is accessed from external process. The block file is readable even if it is accessed externally. and access time not changed if only get the attribution.
Add lastScanTime for ReplicaInfo may be a more accurate solution, change the lastScanTime when Datanode read it.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 46s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 19m 31s | trunk passed |
+1 | compile | 1m 0s | trunk passed |
+1 | checkstyle | 0m 48s | trunk passed |
+1 | mvnsite | 1m 6s | trunk passed |
+1 | shadedclient | 16m 22s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 45s | trunk passed |
+1 | javadoc | 0m 39s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 2s | the patch passed |
+1 | compile | 0m 57s | the patch passed |
-1 | javac | 0m 57s | hadoop-hdfs-project_hadoop-hdfs generated 1 new + 579 unchanged - 0 fixed = 580 total (was 579) |
-0 | checkstyle | 0m 46s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 501 unchanged - 0 fixed = 502 total (was 501) |
+1 | mvnsite | 1m 2s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | shadedclient | 14m 14s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 55s | the patch passed |
+1 | javadoc | 0m 39s | the patch passed |
Other Tests | |||
-1 | unit | 108m 11s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
173m 20s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.7 Server=19.03.7 Image:yetus/hadoop:c44943d1fc3 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12996239/HDFS-15207.003.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
uname | Linux 65399ab7a704 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 44afe11 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_242 |
findbugs | v3.1.0-RC1 |
javac | https://builds.apache.org/job/PreCommit-HDFS-Build/28919/artifact/out/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/28919/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28919/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/28919/testReport/ |
Max. process+thread count | 3015 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/28919/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Patch makes sense tom me. Thanks for working on this hadoop_yangyun
In the test code:
assertTrue("Should not run to here", false);
+1 after the test code change.
Can you use LambdaTestUtils#intercept() instead, or use the more traditional fail()
VolumeScanner#runLoop() is quite long now. Time to refactor it. Can be a separate jira to refactor that method.
Another thing, let's merge the else and the if as it makes it less verbose and easier to reason about:
} else if (conf.skipRecentAccessed) { // Check the access time of block file to avoid scanning recently // changed blocks, reducing disk IO. ... }
Thanks weichiu and elgoiri for the review.
Updated to new patch HDFS-15207.004.patch with following changes,
- Use fail() than assertTrue in the test code.
- merge the else and the if in the VolumeScanner.
LamdaTestUtils with a wait inside is kind of weird, so I guess using fail() in this case is fine.
+1 on HDFS-15207.004.patch.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 37s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 18m 3s | trunk passed |
+1 | compile | 1m 3s | trunk passed |
+1 | checkstyle | 0m 52s | trunk passed |
+1 | mvnsite | 1m 8s | trunk passed |
+1 | shadedclient | 14m 52s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 37s | trunk passed |
+1 | javadoc | 0m 43s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 59s | the patch passed |
+1 | compile | 0m 54s | the patch passed |
-1 | javac | 0m 54s | hadoop-hdfs-project_hadoop-hdfs generated 1 new + 585 unchanged - 0 fixed = 586 total (was 585) |
-0 | checkstyle | 0m 46s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 501 unchanged - 0 fixed = 502 total (was 501) |
+1 | mvnsite | 1m 1s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | shadedclient | 13m 3s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 41s | the patch passed |
+1 | javadoc | 0m 40s | the patch passed |
Other Tests | |||
-1 | unit | 93m 42s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
154m 30s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks | |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:e6455cc864d |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12998838/HDFS-15207.004.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
uname | Linux 53b428e7119a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / e6455cc |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_242 |
findbugs | v3.1.0-RC1 |
javac | https://builds.apache.org/job/PreCommit-HDFS-Build/29077/artifact/out/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/29077/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/29077/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/29077/testReport/ |
Max. process+thread count | 4180 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/29077/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Is there something we can do for:
VolumeScanner.java:[549,47] [deprecation] getReplica(String,long) in FsDatasetSpi has been deprecated
Thanks elgoiri for the review.
Updated to HDFS-15207.005.patch with new way that uses getBlockLocalPathInfo instead of deprecated getReplica.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 47s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 24m 38s | trunk passed |
+1 | compile | 1m 21s | trunk passed |
+1 | checkstyle | 1m 3s | trunk passed |
+1 | mvnsite | 1m 31s | trunk passed |
+1 | shadedclient | 19m 43s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 45s | trunk passed |
+1 | javadoc | 0m 46s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 19s | the patch passed |
+1 | compile | 1m 8s | the patch passed |
+1 | javac | 1m 8s | the patch passed |
-0 | checkstyle | 0m 55s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 501 unchanged - 0 fixed = 502 total (was 501) |
+1 | mvnsite | 1m 12s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | shadedclient | 16m 30s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 49s | the patch passed |
+1 | javadoc | 0m 46s | the patch passed |
Other Tests | |||
-1 | unit | 118m 41s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 39s | The patch does not generate ASF License warnings. |
198m 24s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
hadoop.hdfs.tools.TestDFSZKFailoverController | |
hadoop.hdfs.server.datanode.TestBPOfferService | |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | |
hadoop.hdfs.server.balancer.TestBalancer | |
hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:e6455cc864d |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12999035/HDFS-15207.005.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
uname | Linux 1216d2fdb046 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / ab7495d |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_242 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/29086/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/29086/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/29086/testReport/ |
Max. process+thread count | 2873 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/29086/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
I was committing and I realized that we should not contract the import Assert.
Thanks elgoiri for the review, update to HDFS-15207.006.patch and remove the Assert.*
It just occurred to me that certain vendors (e.g. Cloudera) recommends noatime for DataNodes: https://docs.cloudera.com/documentation/enterprise/5-6-x/topics/install_cdh_file_system.html
What does this change mean if atime is disabled on the file system?
Regardless, please go ahead with the commit. This is optional so it won't cause regression even if this is not compatible with noatime setup.
if the atime is disabled, the atime will never change and keep the initial value. this feature will not work even it's enabled.
Thanks
Would you mind to add to the hdfs-default.xml that this feature will not work if the DataNode volume has noatime mount option?
+1 after that.
Thanks weichiu for the review.
Added the info to the hdfs-default.xml in new patch HDFS-15207.007.patch.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 2m 34s | Docker mode activated. |
Prechecks | |||
+1 | dupname | 0m 0s | No case conflicting files found. |
+1 | @author | 0m 1s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
-1 | mvninstall | 24m 53s | root in trunk failed. |
+1 | compile | 1m 6s | trunk passed |
+1 | checkstyle | 0m 52s | trunk passed |
+1 | mvnsite | 1m 12s | trunk passed |
-1 | shadedclient | 18m 12s | branch has errors when building and testing our client artifacts. |
+1 | javadoc | 0m 40s | trunk passed |
0 | spotbugs | 2m 59s | Used deprecated FindBugs config; considering switching to SpotBugs. |
+1 | findbugs | 2m 57s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 7s | the patch passed |
+1 | compile | 1m 4s | the patch passed |
+1 | javac | 1m 4s | the patch passed |
-0 | checkstyle | 0m 48s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 501 unchanged - 0 fixed = 502 total (was 501) |
+1 | mvnsite | 1m 7s | the patch passed |
+1 | whitespace | 0m 1s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
-1 | shadedclient | 16m 10s | patch has errors when building and testing our client artifacts. |
+1 | javadoc | 0m 37s | the patch passed |
+1 | findbugs | 3m 4s | the patch passed |
Other Tests | |||
-1 | unit | 109m 49s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
186m 56s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | |
hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | |
hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/PreCommit-HDFS-Build/29312/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13003164/HDFS-15207.008.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
uname | Linux 5d9abcd9943c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / 6e416a83d1e |
Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
mvninstall | https://builds.apache.org/job/PreCommit-HDFS-Build/29312/artifact/out/branch-mvninstall-root.txt |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/29312/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/29312/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/29312/testReport/ |
Max. process+thread count | 2684 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/29312/console |
versions | git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1 |
Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
Pushed to trunk and branch-3.3
There are conflicts in branch-3.2 and lower branches. If those branches are desired, please provide patch files against them. Thanks!
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #18269 (See https://builds.apache.org/job/Hadoop-trunk-Commit/18269/)
HDFS-15207. VolumeScanner skip to scan blocks accessed during recent (weichiu: rev 50caba1a92cb36ce78307d47ed7624ce216562fc)
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockScanner.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/VolumeScanner.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockScanner.java
HDFS-15207This message was automatically generated.