Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.7.3
-
Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
-
Component/s: None
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
If a block is removed, it will be removed from the block map. When the clearCorruptLazyPersistFiles() tries to delete the block, it may already be deleted and generate a null pointer exception.
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.clearCorruptLazyPersistFiles(FSNamesystem.java:3820)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem$LazyPersistFileScrubber.run(FSNamesystem.java:3851)
at java.lang.Thread.run(Thread.java:745)
Issue Links
- relates to
-
HDFS-11869
Backport HDFS-11078 to branch 2.7
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
I don't find this fix landed in branch-2.8 or branch-2.8.2. Konstantin Sharenkov, which branches do you commit the patch to? We should explicitly mention it in committing the patch.
Looks jira id is missed while committing..Bytheway I updated the CHANGES.txt in branch-2.7.
I just committed this. Thank you Íñigo Goiri.
+1 for the patch. Looks like the failed tests are quite flaky.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 53s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 15m 51s | trunk passed |
| +1 | compile | 1m 1s | trunk passed |
| +1 | checkstyle | 0m 47s | trunk passed |
| +1 | mvnsite | 1m 28s | trunk passed |
| +1 | mvneclipse | 0m 19s | trunk passed |
| +1 | findbugs | 2m 6s | trunk passed |
| +1 | javadoc | 0m 47s | trunk passed |
| +1 | mvninstall | 1m 8s | the patch passed |
| +1 | compile | 1m 8s | the patch passed |
| +1 | javac | 1m 8s | the patch passed |
| +1 | checkstyle | 0m 45s | the patch passed |
| +1 | mvnsite | 1m 11s | the patch passed |
| +1 | mvneclipse | 0m 15s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 2m 6s | the patch passed |
| +1 | javadoc | 0m 52s | the patch passed |
| -1 | unit | 102m 31s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
| 135m 19s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancer |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.hdfs.server.namenode.TestDecommissioningStatus | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12836383/HDFS-11078.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 186f06bf25a1 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8e0f83e |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/19548/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/19548/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/19548/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 22s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 7m 9s | trunk passed |
| +1 | compile | 0m 50s | trunk passed |
| +1 | checkstyle | 0m 31s | trunk passed |
| +1 | mvnsite | 0m 59s | trunk passed |
| +1 | mvneclipse | 0m 16s | trunk passed |
| +1 | findbugs | 1m 43s | trunk passed |
| +1 | javadoc | 0m 40s | trunk passed |
| +1 | mvninstall | 0m 48s | the patch passed |
| +1 | compile | 0m 44s | the patch passed |
| +1 | javac | 0m 44s | the patch passed |
| +1 | checkstyle | 0m 27s | the patch passed |
| +1 | mvnsite | 1m 3s | the patch passed |
| +1 | mvneclipse | 0m 11s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 52s | the patch passed |
| +1 | javadoc | 0m 37s | the patch passed |
| -1 | unit | 76m 43s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
| 96m 33s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12836383/HDFS-11078.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 68a791d51034 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 808f5d8 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17368/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17368/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17368/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Moved message to info.
Since this can happen legitimately and does not require any further attention from users, logging at debug or info at most seems preferable.
There are no other tests related to the scrubber from what I can tell.
I would ignore the addition of tests for this particular case.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 22s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 7m 49s | trunk passed |
| +1 | compile | 0m 46s | trunk passed |
| +1 | checkstyle | 0m 29s | trunk passed |
| +1 | mvnsite | 0m 53s | trunk passed |
| +1 | mvneclipse | 0m 13s | trunk passed |
| +1 | findbugs | 1m 57s | trunk passed |
| +1 | javadoc | 0m 48s | trunk passed |
| +1 | mvninstall | 1m 3s | the patch passed |
| +1 | compile | 0m 57s | the patch passed |
| +1 | javac | 0m 57s | the patch passed |
| +1 | checkstyle | 0m 31s | the patch passed |
| +1 | mvnsite | 1m 7s | the patch passed |
| +1 | mvneclipse | 0m 13s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 2m 0s | the patch passed |
| +1 | javadoc | 0m 45s | the patch passed |
| +1 | unit | 78m 14s | hadoop-hdfs in the patch passed. |
| +1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
| 99m 50s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12836250/HDFS-11078.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 226e3d5501b8 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7ba74be |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17360/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17360/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Checking if the block is not in the system anymore in the LazyPresistFileScrubber. I'm open to a better log message.
After applying HDFS-9555, this is not that critical but still it would be nice to add a check for this.
Ok. As Brahma Reddy Battula mentioned above, we are missing jira ID in commit message here, so the patch get actually landed.