[HDFS-2430] The number of failed or low-resource volumes the NN can tolerate should be configurable - ASF JIRA

Hide

Permalink

Milind Bhandarkar added a comment - 11/Oct/11 00:25

+1.

Show

Milind Bhandarkar added a comment - 11/Oct/11 00:25 +1.

Hide

Permalink

Suresh Srinivas added a comment - 11/Oct/11 19:05

Aaron, with the patch from ~~HDFS-2422~~, the javadoc for NameNodeResourceChecker is not valid. It says <code>hasAvailableDiskSpace</code> - which will return true if and only if the NameNode has disk space available on all volumes which are configured to be checked. Can you please address that in ~~HDFS-2430~~.

Show

Suresh Srinivas added a comment - 11/Oct/11 19:05 Aaron, with the patch from HDFS-2422 , the javadoc for NameNodeResourceChecker is not valid. It says <code>hasAvailableDiskSpace</code> - which will return true if and only if the NameNode has disk space available on all volumes which are configured to be checked . Can you please address that in HDFS-2430 .

Hide

Permalink

Aaron T. Myers added a comment - 14/Oct/11 07:42

Thanks for catching that, Suresh. I updated the method javadoc but didn't notice that the class javadoc mentions this behavior as well. I will be sure to fix this comment when I address this JIRA.

Show

Aaron T. Myers added a comment - 14/Oct/11 07:42 Thanks for catching that, Suresh. I updated the method javadoc but didn't notice that the class javadoc mentions this behavior as well. I will be sure to fix this comment when I address this JIRA.

Hide

Permalink

Aaron T. Myers added a comment - 14/Oct/11 08:10

While just being able to configure the number of tolerated dir failures from the single pool of dirs specified by the dfs.namenode.name.dir config would be an improvement, I actually think the best solution would to be able to specify multiple distinct pools of name dirs, where the number of failed volumes tolerated could be configured per-pool. This will be useful since not all name dirs are created equal. For example, if an operator has 4 name dirs configured, 3 of which are local, with the other being on a remote machine mounted via NFS, the operator might want to configure the NN to tolerate up to 2 failures of the 3 local dirs, but stop immediately if the NFS dir goes away. I think such a scheme will be necessary for an NFS-based HA solution, as described as one of the options in ~~HDFS-1623~~, but this configuration can be useful generally as well, so might as well be developed on trunk.

I can imagine two high-level designs:

The operator can configure exactly two distinct pools of name dirs, configured via something like dfs.namenode.name.dirs.required and dfs.namenode.name.dirs.redundant. If any single dir specified in the .required config goes offline, the NN will not continue to operate. The number of acceptable failed dirs in the .redundant pool would be configurable by a third option, dfs.name.node.name.dirs.failures.tolerated.
The operator can specify N distinct pools of name dirs, configured via something like dfs.namenode.name.dirs.pool.0, dfs.namenode.name.dirs.pool.1, etc. For each of these configured pools, the number of failed volumes tolerated could be configured individually, e.g. dfs.namenode.name.dirs.failures.tolerated.pool.0, dfs.namenode.name.dirs.failures.tolerated.pool.1, etc.

Under either of these schemes, it would be an error to specify the same dir in multiple pools.

Any thoughts? Option 2 is obviously more flexible, but I don't want to thrust our operators deeper into configuration hell than they already are.

Show

Aaron T. Myers added a comment - 14/Oct/11 08:10 While just being able to configure the number of tolerated dir failures from the single pool of dirs specified by the dfs.namenode.name.dir config would be an improvement, I actually think the best solution would to be able to specify multiple distinct pools of name dirs, where the number of failed volumes tolerated could be configured per-pool. This will be useful since not all name dirs are created equal. For example, if an operator has 4 name dirs configured, 3 of which are local, with the other being on a remote machine mounted via NFS, the operator might want to configure the NN to tolerate up to 2 failures of the 3 local dirs, but stop immediately if the NFS dir goes away. I think such a scheme will be necessary for an NFS-based HA solution, as described as one of the options in HDFS-1623 , but this configuration can be useful generally as well, so might as well be developed on trunk. I can imagine two high-level designs: The operator can configure exactly two distinct pools of name dirs, configured via something like dfs.namenode.name.dirs.required and dfs.namenode.name.dirs.redundant . If any single dir specified in the .required config goes offline, the NN will not continue to operate. The number of acceptable failed dirs in the .redundant pool would be configurable by a third option, dfs.name.node.name.dirs.failures.tolerated . The operator can specify N distinct pools of name dirs, configured via something like dfs.namenode.name.dirs.pool.0 , dfs.namenode.name.dirs.pool.1 , etc. For each of these configured pools, the number of failed volumes tolerated could be configured individually, e.g. dfs.namenode.name.dirs.failures.tolerated.pool.0 , dfs.namenode.name.dirs.failures.tolerated.pool.1 , etc. Under either of these schemes, it would be an error to specify the same dir in multiple pools. Any thoughts? Option 2 is obviously more flexible, but I don't want to thrust our operators deeper into configuration hell than they already are.

Hide

Permalink

Todd Lipcon added a comment - 19/Oct/11 22:29

I think option #1 is better - agree #2 is more flexible but I can't think of any real use cases.

The other thing that needs to be considered in this design, though, is that we need to think separately about edits storage vs name directories. Edits may be stored in other implementations in the not-so-distant future (eg bookkeeper). So it's not just the name dirs, but also the edits managers which need to have such a configuration.

Show

Todd Lipcon added a comment - 19/Oct/11 22:29 I think option #1 is better - agree #2 is more flexible but I can't think of any real use cases. The other thing that needs to be considered in this design, though, is that we need to think separately about edits storage vs name directories. Edits may be stored in other implementations in the not-so-distant future (eg bookkeeper). So it's not just the name dirs, but also the edits managers which need to have such a configuration.

Hide

Permalink

Aaron T. Myers added a comment - 27/Oct/11 06:46

Thanks a lot for the comments, Todd. Here's a patch which addresses the issue. This implements option 1 as described above.

I deviated slightly from what I initially described regarding the configuration parameters. Instead of two new configs, there's just a new "required" config. The previous edit dir config options are treated as they were before - as redundant directories. Some of these may be marked "required" using the new config introduced by this patch.

Show

Aaron T. Myers added a comment - 27/Oct/11 06:46 Thanks a lot for the comments, Todd. Here's a patch which addresses the issue. This implements option 1 as described above. I deviated slightly from what I initially described regarding the configuration parameters. Instead of two new configs, there's just a new "required" config. The previous edit dir config options are treated as they were before - as redundant directories. Some of these may be marked "required" using the new config introduced by this patch.

Hide

Permalink

Hadoop QA added a comment - 27/Oct/11 08:44

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501035/HDFS-2430.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestAbandonBlock

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1472//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1472//console

This message is automatically generated.

Show

Hadoop QA added a comment - 27/Oct/11 08:44 -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501035/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestAbandonBlock +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1472//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1472//console This message is automatically generated.

Hide

Permalink

Aaron T. Myers added a comment - 27/Oct/11 17:59

The test failure seems unrelated. I just tried running that test locally with the patch applied and it passed without issue.

Show

Aaron T. Myers added a comment - 27/Oct/11 17:59 The test failure seems unrelated. I just tried running that test locally with the patch applied and it passed without issue.

Hide

Permalink

Eli Collins added a comment - 28/Oct/11 00:25

Agree that option #1 is better. The notion of a NN "required" dir here parallels the notion of a "critical" dir in the DN (proposed in HDFS-1848). I'll check out the patch.

Show

Eli Collins added a comment - 28/Oct/11 00:25 Agree that option #1 is better. The notion of a NN "required" dir here parallels the notion of a "critical" dir in the DN (proposed in HDFS-1848 ). I'll check out the patch.

Hide

Permalink

Aaron T. Myers added a comment - 28/Oct/11 00:52

Hold off a minute, Eli. I just realized that I completely forgot to make the number of failures in the redundant pool configurable. Let me fix that before you spend time reviewing it.

Show

Aaron T. Myers added a comment - 28/Oct/11 00:52 Hold off a minute, Eli. I just realized that I completely forgot to make the number of failures in the redundant pool configurable. Let me fix that before you spend time reviewing it.

Hide

Permalink

Aaron T. Myers added a comment - 28/Oct/11 23:35

Here's a patch which does everything the previous patch did, as well as implements the configurable number of redundant dirs, as well as adds a proper unit test for NameNodeResourcePolicy.

Please review away.

Show

Aaron T. Myers added a comment - 28/Oct/11 23:35 Here's a patch which does everything the previous patch did, as well as implements the configurable number of redundant dirs, as well as adds a proper unit test for NameNodeResourcePolicy . Please review away.

Hide

Permalink

Hadoop QA added a comment - 29/Oct/11 02:26

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501394/HDFS-2430.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 8 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestAbandonBlock
org.apache.hadoop.hdfs.server.namenode.TestClusterId
org.apache.hadoop.cli.TestHDFSCLI

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1491//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1491//console

This message is automatically generated.

Show

Hadoop QA added a comment - 29/Oct/11 02:26 -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501394/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.hdfs.server.namenode.TestClusterId org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1491//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1491//console This message is automatically generated.

Hide

Permalink

Aaron T. Myers added a comment - 29/Oct/11 06:30

Here's a patch which fixes the TestClusterId failure. That test was getting the configured cluster ID using a backdoor that wasn't bothering to set any edits dirs. The fix was:

--- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
+++ hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
@@ -49,7 +49,7 @@ public class TestClusterId {
   private String getClusterId(Configuration config) throws IOException {
     // see if cluster id not empty.
     Collection<URI> dirsToFormat = FSNamesystem.getNamespaceDirs(config);
-    Collection<URI> editsToFormat = new ArrayList<URI>(0);
+    Collection<URI> editsToFormat = FSNamesystem.getNamespaceEditsDirs(config);
     FSImage fsImage = new FSImage(config, dirsToFormat, editsToFormat);
     
     Iterator<StorageDirectory> sdit =

The TestAbandonBlock and TestHDFSCLI test failures are unrelated.

Show

Aaron T. Myers added a comment - 29/Oct/11 06:30 Here's a patch which fixes the TestClusterId failure. That test was getting the configured cluster ID using a backdoor that wasn't bothering to set any edits dirs. The fix was: --- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java +++ hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java @@ -49,7 +49,7 @@ public class TestClusterId { private String getClusterId(Configuration config) throws IOException { // see if cluster id not empty. Collection<URI> dirsToFormat = FSNamesystem.getNamespaceDirs(config); - Collection<URI> editsToFormat = new ArrayList<URI>(0); + Collection<URI> editsToFormat = FSNamesystem.getNamespaceEditsDirs(config); FSImage fsImage = new FSImage(config, dirsToFormat, editsToFormat); Iterator<StorageDirectory> sdit = The TestAbandonBlock and TestHDFSCLI test failures are unrelated.

Hide

Permalink

Hadoop QA added a comment - 29/Oct/11 08:43

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501431/HDFS-2430.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestAbandonBlock
org.apache.hadoop.cli.TestHDFSCLI

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1497//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1497//console

This message is automatically generated.

Show

Hadoop QA added a comment - 29/Oct/11 08:43 -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501431/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1497//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1497//console This message is automatically generated.

Hide

Permalink

Eli Collins added a comment - 19/Nov/11 03:32

The api for NNResourcePolicy#areResourcesAvailable is a little strange, if a required resource is not available we return false, but if there are not enough redundant resources we throw a RTE. Do callers need to distinguish between these two cases? If not, we should return false in both cases.
If so, there should be two separate apis. I'd make the method name better reflect the purpose in the comment (whether we can log edits), the method is generic but the particular policy is edits-specific.

Does the dir checking use canonical paths so we're sure the checks pass in case of symlinks?

Nits:

NNResourcePolicy needs param javadocs
NNResourcePolicy, NNResourceChecker, and CheckableNNResource need @InterfaceAudience.Private annotations

Otherwise looks great.

Show

Eli Collins added a comment - 19/Nov/11 03:32 The api for NNResourcePolicy#areResourcesAvailable is a little strange, if a required resource is not available we return false, but if there are not enough redundant resources we throw a RTE. Do callers need to distinguish between these two cases? If not, we should return false in both cases. If so, there should be two separate apis. I'd make the method name better reflect the purpose in the comment (whether we can log edits), the method is generic but the particular policy is edits-specific. Does the dir checking use canonical paths so we're sure the checks pass in case of symlinks? Nits: NNResourcePolicy needs param javadocs NNResourcePolicy, NNResourceChecker, and CheckableNNResource need @InterfaceAudience.Private annotations Otherwise looks great.

Hide

Permalink

Eli Collins added a comment - 20/Nov/11 03:52

Per ~~HDFS-208~~ might be worth considering other warnings to log eg if there's only a single redundant resource left. Could also standardize the location/format of the log check and output to make life easier for monitoring tools. Ditto for metrics (feel free to address in another jira).

Show

Eli Collins added a comment - 20/Nov/11 03:52 Per HDFS-208 might be worth considering other warnings to log eg if there's only a single redundant resource left. Could also standardize the location/format of the log check and output to make life easier for monitoring tools. Ditto for metrics (feel free to address in another jira).

Hide

Permalink

Aaron T. Myers added a comment - 30/Nov/11 22:36

Thanks a lot for the review, Eli. Comments below.

The api for NNResourcePolicy#areResourcesAvailable is a little strange, if a required resource is not available we return false, but if there are not enough redundant resources we throw a RTE. Do callers need to distinguish between these two cases? If not, we should return false in both cases.
If so, there should be two separate apis. I'd make the method name better reflect the purpose in the comment (whether we can log edits), the method is generic but the particular policy is edits-specific.

It only throws an RTE if the configured number of redundant resources is less than the configured minimum number of redundant resources. i.e. it throws only in the case of misconfiguration. If indeed the number of configured redundant resources is greater than the minimum, but the number of available redundant resources is below the minimum, the method will return false. Does that seem reasonable?

Does the dir checking use canonical paths so we're sure the checks pass in case of symlinks?

Yes indeed. FSNS#getStorageDirs ends up calling o.a.h.hdfs.server.common.Util#fileAsURI which calls File#getCanonicalFile.

I'll upload a patch in a moment which addresses the nits.

Show

Aaron T. Myers added a comment - 30/Nov/11 22:36 Thanks a lot for the review, Eli. Comments below. The api for NNResourcePolicy#areResourcesAvailable is a little strange, if a required resource is not available we return false, but if there are not enough redundant resources we throw a RTE. Do callers need to distinguish between these two cases? If not, we should return false in both cases. If so, there should be two separate apis. I'd make the method name better reflect the purpose in the comment (whether we can log edits), the method is generic but the particular policy is edits-specific. It only throws an RTE if the configured number of redundant resources is less than the configured minimum number of redundant resources. i.e. it throws only in the case of misconfiguration. If indeed the number of configured redundant resources is greater than the minimum, but the number of available redundant resources is below the minimum, the method will return false. Does that seem reasonable? Does the dir checking use canonical paths so we're sure the checks pass in case of symlinks? Yes indeed. FSNS#getStorageDirs ends up calling o.a.h.hdfs.server.common.Util#fileAsURI which calls File#getCanonicalFile . I'll upload a patch in a moment which addresses the nits.

Hide

Permalink

Aaron T. Myers added a comment - 30/Nov/11 23:50

Updated patch addressing Eli's comments.

Show

Aaron T. Myers added a comment - 30/Nov/11 23:50 Updated patch addressing Eli's comments.

Hide

Permalink

Aaron T. Myers added a comment - 30/Nov/11 23:53

Per ~~HDFS-208~~ might be worth considering other warnings to log eg if there's only a single redundant resource left. Could also standardize the location/format of the log check and output to make life easier for monitoring tools. Ditto for metrics (feel free to address in another jira).

I've filed HDFS-2613 to address this.

Show

Aaron T. Myers added a comment - 30/Nov/11 23:53 Per HDFS-208 might be worth considering other warnings to log eg if there's only a single redundant resource left. Could also standardize the location/format of the log check and output to make life easier for monitoring tools. Ditto for metrics (feel free to address in another jira). I've filed HDFS-2613 to address this.

Hide

Permalink

Hadoop QA added a comment - 01/Dec/11 00:53

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12505685/HDFS-2430.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1617//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1617//console

This message is automatically generated.

Show

Hadoop QA added a comment - 01/Dec/11 00:53 +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505685/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1617//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1617//console This message is automatically generated.

Hide

Permalink

Eli Collins added a comment - 01/Dec/11 00:59

It only throws an RTE if the configured number of redundant resources is less than the configured minimum number of redundant resources. i.e. it throws only in the case of misconfiguration. If indeed the number of configured redundant resources is greater than the minimum, but the number of available redundant resources is below the minimum, the method will return false. Does that seem reasonable?

Ah, I think it would be more clear if the static (configuration) one-time check was separated from the dynamic check (eg so you don't have to check all callers to reason about the method) but I don't feel strongly. If you keep the current behavior please add a throws clause to the javadoc and add a comment per your above explanation. If you want to address it feel free to do so in a follow on jira. +1 to the latest patch.

Show

Eli Collins added a comment - 01/Dec/11 00:59 It only throws an RTE if the configured number of redundant resources is less than the configured minimum number of redundant resources. i.e. it throws only in the case of misconfiguration. If indeed the number of configured redundant resources is greater than the minimum, but the number of available redundant resources is below the minimum, the method will return false. Does that seem reasonable? Ah, I think it would be more clear if the static (configuration) one-time check was separated from the dynamic check (eg so you don't have to check all callers to reason about the method) but I don't feel strongly. If you keep the current behavior please add a throws clause to the javadoc and add a comment per your above explanation. If you want to address it feel free to do so in a follow on jira. +1 to the latest patch.

Hide

Permalink

Aaron T. Myers added a comment - 05/Dec/11 22:38

Thanks a lot for the review, Eli. I've added an @throws comment per your suggestion.

The attached patch is also rebased on trunk, which had a few conflicts since ~~HDFS-1580~~ was committed.

Show

Aaron T. Myers added a comment - 05/Dec/11 22:38 Thanks a lot for the review, Eli. I've added an @throws comment per your suggestion. The attached patch is also rebased on trunk, which had a few conflicts since HDFS-1580 was committed.

Hide

Permalink

Eli Collins added a comment - 05/Dec/11 23:07

+1 rebase on 1580 looks correct.

Show

Eli Collins added a comment - 05/Dec/11 23:07 +1 rebase on 1580 looks correct.

Hide

Permalink

Hadoop QA added a comment - 05/Dec/11 23:08

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506178/HDFS-2430.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestDistributedFileSystem
org.apache.hadoop.hdfs.TestConnCache
org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
org.apache.hadoop.hdfs.TestSafeMode
org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit
org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat
org.apache.hadoop.hdfs.TestFileAppend
org.apache.hadoop.hdfs.TestDatanodeDeath
org.apache.hadoop.hdfs.server.namenode.TestFileLimit
org.apache.hadoop.hdfs.TestLease
org.apache.hadoop.hdfs.TestShortCircuitLocalRead
org.apache.hadoop.hdfs.TestDFSRename
org.apache.hadoop.hdfs.TestPipelines
org.apache.hadoop.hdfs.TestSetrepDecreasing
org.apache.hadoop.hdfs.TestFileAppend2
org.apache.hadoop.security.TestRefreshUserMappings
org.apache.hadoop.tools.TestDelegationTokenFetcher
org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes
org.apache.hadoop.hdfs.TestHFlush
org.apache.hadoop.hdfs.TestLeaseRecovery
org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount
org.apache.hadoop.fs.TestGlobPaths
org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction
org.apache.hadoop.fs.TestResolveHdfsSymlink
org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart
org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
org.apache.hadoop.hdfs.TestDfsOverAvroRpc
org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
org.apache.hadoop.hdfs.TestAbandonBlock
org.apache.hadoop.fs.TestUrlStreamHandler
org.apache.hadoop.hdfs.server.namenode.TestTransferFsImage
org.apache.hadoop.hdfs.TestBlockMissingException
org.apache.hadoop.fs.viewfs.TestViewFsFileStatusHdfs
org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
org.apache.hadoop.hdfs.TestFileCreation
org.apache.hadoop.fs.TestFcHdfsPermission
org.apache.hadoop.hdfs.TestCrcCorruption
org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
org.apache.hadoop.hdfs.TestListFilesInFileContext
org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
org.apache.hadoop.hdfs.TestListPathServlet
org.apache.hadoop.hdfs.TestMultiThreadedHflush
org.apache.hadoop.hdfs.TestDFSFinalize
org.apache.hadoop.hdfs.TestLocalDFS
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.TestDFSStartupVersions
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.tools.TestGetGroups
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
org.apache.hadoop.hdfs.TestDataTransferProtocol
org.apache.hadoop.hdfs.TestDFSClientRetries
org.apache.hadoop.hdfs.TestAppendDifferentChecksum
org.apache.hadoop.hdfs.TestFileCreationDelete
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade
org.apache.hadoop.hdfs.TestMiniDFSCluster
org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
org.apache.hadoop.hdfs.TestFileCreationClient
org.apache.hadoop.hdfs.server.datanode.TestWriteToReplica
org.apache.hadoop.hdfs.TestListFilesInDFS
org.apache.hadoop.hdfs.TestDatanodeConfig
org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean
org.apache.hadoop.hdfs.server.namenode.TestMetaSave
org.apache.hadoop.hdfs.server.namenode.TestClusterId
org.apache.hadoop.hdfs.server.namenode.TestFsck
org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
org.apache.hadoop.hdfs.TestDFSMkdirs
org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace
org.apache.hadoop.fs.TestFcHdfsCreateMkdir
org.apache.hadoop.hdfs.server.namenode.TestAllowFormat
org.apache.hadoop.hdfs.TestModTime
org.apache.hadoop.hdfs.TestDFSRollback
org.apache.hadoop.security.TestPermission
org.apache.hadoop.hdfs.server.namenode.TestEditLogFileOutputStream
org.apache.hadoop.hdfs.server.namenode.TestEditLog
org.apache.hadoop.hdfs.server.namenode.TestStreamFile
org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs
org.apache.hadoop.hdfs.TestDecommission
org.apache.hadoop.fs.TestHDFSFileContextMainOperations
org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
org.apache.hadoop.hdfs.server.datanode.TestDiskError
org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
org.apache.hadoop.hdfs.server.namenode.TestSecondaryWebUi
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
org.apache.hadoop.hdfs.TestFileCorruption
org.apache.hadoop.hdfs.TestClientBlockVerification
org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
org.apache.hadoop.hdfs.TestFileAppend3
org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
org.apache.hadoop.hdfs.TestFileAppend4
org.apache.hadoop.hdfs.TestBlocksScheduledCounter
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
org.apache.hadoop.hdfs.server.namenode.TestStartup
org.apache.hadoop.hdfs.server.datanode.TestDatanodeJsp
org.apache.hadoop.hdfs.TestWriteRead
org.apache.hadoop.hdfs.TestRenameWhileOpen
org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
org.apache.hadoop.hdfs.TestDFSRemove
org.apache.hadoop.fs.viewfs.TestViewFsHdfs
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
org.apache.hadoop.hdfs.TestDatanodeReport
org.apache.hadoop.hdfs.TestHDFSFileSystemContract
org.apache.hadoop.hdfs.TestDFSClientExcludedNodes
org.apache.hadoop.hdfs.TestFSInputChecker
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
org.apache.hadoop.hdfs.server.datanode.TestBlockReport
org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
org.apache.hadoop.hdfs.TestSetTimes
org.apache.hadoop.hdfs.TestPread
org.apache.hadoop.hdfs.security.TestClientProtocolWithDelegationToken
org.apache.hadoop.hdfs.server.namenode.TestEditLogRace
org.apache.hadoop.hdfs.security.TestDelegationToken
org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional
org.apache.hadoop.hdfs.TestDFSShellGenericOptions
org.apache.hadoop.hdfs.TestDFSPermission
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.TestHDFSTrash
org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
org.apache.hadoop.fs.TestFcHdfsSymlink
org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
org.apache.hadoop.hdfs.TestFileStatus
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
org.apache.hadoop.hdfs.server.namenode.TestNameNodeJspHelper
org.apache.hadoop.hdfs.TestReadWhileWriting
org.apache.hadoop.hdfs.TestHftpFileSystem
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
org.apache.hadoop.hdfs.server.namenode.TestSafeMode
org.apache.hadoop.hdfs.TestLeaseRenewer
org.apache.hadoop.hdfs.TestDatanodeRegistration
org.apache.hadoop.hdfs.server.namenode.TestCorruptFilesJsp
org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures
org.apache.hadoop.hdfs.TestLargeBlock
org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
org.apache.hadoop.hdfs.TestParallelRead
org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
org.apache.hadoop.hdfs.server.datanode.TestInterDatanodeProtocol
org.apache.hadoop.hdfs.server.datanode.TestDatanodeRestart
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.TestFSOutputSummer
org.apache.hadoop.hdfs.TestFileCreationEmpty
org.apache.hadoop.hdfs.TestBalancerBandwidth
org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
org.apache.hadoop.fs.permission.TestStickyBit
org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader
org.apache.hadoop.hdfs.security.token.block.TestBlockToken
org.apache.hadoop.hdfs.TestHftpDelegationToken
org.apache.hadoop.hdfs.TestRestartDFS
org.apache.hadoop.hdfs.TestGetBlocks
org.apache.hadoop.hdfs.TestSmallBlock
org.apache.hadoop.hdfs.TestSetrepIncreasing
org.apache.hadoop.hdfs.TestHDFSServerPorts
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings
org.apache.hadoop.hdfs.TestReplication
org.apache.hadoop.hdfs.TestDFSUpgrade
org.apache.hadoop.tools.TestJMXGet
org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
org.apache.hadoop.hdfs.TestQuota
org.apache.hadoop.hdfs.TestFileConcurrentReader

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1641//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1641//console

This message is automatically generated.

Show

Hadoop QA added a comment - 05/Dec/11 23:08 -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506178/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.TestConnCache org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite org.apache.hadoop.hdfs.TestSafeMode org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.server.namenode.TestFileLimit org.apache.hadoop.hdfs.TestLease org.apache.hadoop.hdfs.TestShortCircuitLocalRead org.apache.hadoop.hdfs.TestDFSRename org.apache.hadoop.hdfs.TestPipelines org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.security.TestRefreshUserMappings org.apache.hadoop.tools.TestDelegationTokenFetcher org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes org.apache.hadoop.hdfs.TestHFlush org.apache.hadoop.hdfs.TestLeaseRecovery org.apache.hadoop.fs.loadGenerator.TestLoadGenerator org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount org.apache.hadoop.fs.TestGlobPaths org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction org.apache.hadoop.fs.TestResolveHdfsSymlink org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.fs.TestUrlStreamHandler org.apache.hadoop.hdfs.server.namenode.TestTransferFsImage org.apache.hadoop.hdfs.TestBlockMissingException org.apache.hadoop.fs.viewfs.TestViewFsFileStatusHdfs org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.fs.TestFcHdfsPermission org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.TestListPathServlet org.apache.hadoop.hdfs.TestMultiThreadedHflush org.apache.hadoop.hdfs.TestDFSFinalize org.apache.hadoop.hdfs.TestLocalDFS org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestDFSStartupVersions org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.tools.TestGetGroups org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps org.apache.hadoop.hdfs.TestDataTransferProtocol org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.hdfs.TestAppendDifferentChecksum org.apache.hadoop.hdfs.TestFileCreationDelete org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade org.apache.hadoop.hdfs.TestMiniDFSCluster org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.hdfs.server.datanode.TestWriteToReplica org.apache.hadoop.hdfs.TestListFilesInDFS org.apache.hadoop.hdfs.TestDatanodeConfig org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean org.apache.hadoop.hdfs.server.namenode.TestMetaSave org.apache.hadoop.hdfs.server.namenode.TestClusterId org.apache.hadoop.hdfs.server.namenode.TestFsck org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks org.apache.hadoop.hdfs.TestDFSMkdirs org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace org.apache.hadoop.fs.TestFcHdfsCreateMkdir org.apache.hadoop.hdfs.server.namenode.TestAllowFormat org.apache.hadoop.hdfs.TestModTime org.apache.hadoop.hdfs.TestDFSRollback org.apache.hadoop.security.TestPermission org.apache.hadoop.hdfs.server.namenode.TestEditLogFileOutputStream org.apache.hadoop.hdfs.server.namenode.TestEditLog org.apache.hadoop.hdfs.server.namenode.TestStreamFile org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.fs.TestHDFSFileContextMainOperations org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode org.apache.hadoop.hdfs.server.datanode.TestDiskError org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.server.namenode.TestSecondaryWebUi org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestFileCorruption org.apache.hadoop.hdfs.TestClientBlockVerification org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery org.apache.hadoop.hdfs.TestFileAppend3 org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestBlocksScheduledCounter org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus org.apache.hadoop.hdfs.server.namenode.TestStartup org.apache.hadoop.hdfs.server.datanode.TestDatanodeJsp org.apache.hadoop.hdfs.TestWriteRead org.apache.hadoop.hdfs.TestRenameWhileOpen org.apache.hadoop.hdfs.TestDFSUpgradeFromImage org.apache.hadoop.hdfs.TestDFSRemove org.apache.hadoop.fs.viewfs.TestViewFsHdfs org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.TestDatanodeReport org.apache.hadoop.hdfs.TestHDFSFileSystemContract org.apache.hadoop.hdfs.TestDFSClientExcludedNodes org.apache.hadoop.hdfs.TestFSInputChecker org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner org.apache.hadoop.hdfs.TestSetTimes org.apache.hadoop.hdfs.TestPread org.apache.hadoop.hdfs.security.TestClientProtocolWithDelegationToken org.apache.hadoop.hdfs.server.namenode.TestEditLogRace org.apache.hadoop.hdfs.security.TestDelegationToken org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional org.apache.hadoop.hdfs.TestDFSShellGenericOptions org.apache.hadoop.hdfs.TestDFSPermission org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser org.apache.hadoop.fs.TestFcHdfsSymlink org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark org.apache.hadoop.hdfs.TestFileStatus org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.hdfs.server.namenode.TestNameNodeJspHelper org.apache.hadoop.hdfs.TestReadWhileWriting org.apache.hadoop.hdfs.TestHftpFileSystem org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.server.namenode.TestSafeMode org.apache.hadoop.hdfs.TestLeaseRenewer org.apache.hadoop.hdfs.TestDatanodeRegistration org.apache.hadoop.hdfs.server.namenode.TestCorruptFilesJsp org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork org.apache.hadoop.hdfs.TestParallelRead org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage org.apache.hadoop.hdfs.server.datanode.TestInterDatanodeProtocol org.apache.hadoop.hdfs.server.datanode.TestDatanodeRestart org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes org.apache.hadoop.hdfs.TestFSOutputSummer org.apache.hadoop.hdfs.TestFileCreationEmpty org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.TestWriteConfigurationToDFS org.apache.hadoop.fs.permission.TestStickyBit org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader org.apache.hadoop.hdfs.security.token.block.TestBlockToken org.apache.hadoop.hdfs.TestHftpDelegationToken org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.hdfs.TestSmallBlock org.apache.hadoop.hdfs.TestSetrepIncreasing org.apache.hadoop.hdfs.TestHDFSServerPorts org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.TestDFSUpgrade org.apache.hadoop.tools.TestJMXGet org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration org.apache.hadoop.hdfs.TestQuota org.apache.hadoop.hdfs.TestFileConcurrentReader +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1641//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1641//console This message is automatically generated.

Hide

Permalink

Aaron T. Myers added a comment - 05/Dec/11 23:30

Something went majorly wrong with that test run - OOMs and "Unable to create native thread." I've also run a selection of the failed tests on my local box, and they all passed.

I've just kicked Jenkins to re-run test-patch. We'll see how that goes.

Show

Aaron T. Myers added a comment - 05/Dec/11 23:30 Something went majorly wrong with that test run - OOMs and "Unable to create native thread." I've also run a selection of the failed tests on my local box, and they all passed. I've just kicked Jenkins to re-run test-patch. We'll see how that goes.

Hide

Permalink

Hadoop QA added a comment - 05/Dec/11 23:57

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506178/HDFS-2430.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestDistributedFileSystem
org.apache.hadoop.hdfs.TestConnCache
org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
org.apache.hadoop.hdfs.TestSafeMode
org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit
org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat
org.apache.hadoop.hdfs.TestFileAppend
org.apache.hadoop.hdfs.TestDatanodeDeath
org.apache.hadoop.hdfs.server.namenode.TestFileLimit
org.apache.hadoop.hdfs.TestLease
org.apache.hadoop.hdfs.TestShortCircuitLocalRead
org.apache.hadoop.hdfs.TestPipelines
org.apache.hadoop.hdfs.TestSetrepDecreasing
org.apache.hadoop.hdfs.TestLeaseRecovery2
org.apache.hadoop.hdfs.TestFileAppend2
org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes
org.apache.hadoop.hdfs.TestHFlush
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner
org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount
org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction
org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart
org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
org.apache.hadoop.hdfs.server.namenode.TestTransferFsImage
org.apache.hadoop.hdfs.TestBlockMissingException
org.apache.hadoop.hdfs.TestClientReportBadBlock
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
org.apache.hadoop.hdfs.TestMultiThreadedHflush
org.apache.hadoop.hdfs.TestDFSFinalize
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.TestDFSStartupVersions
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.tools.TestGetGroups
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps
org.apache.hadoop.hdfs.TestDataTransferProtocol
org.apache.hadoop.hdfs.TestDFSClientRetries
org.apache.hadoop.hdfs.TestAppendDifferentChecksum
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade
org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
org.apache.hadoop.hdfs.server.datanode.TestWriteToReplica
org.apache.hadoop.hdfs.TestListFilesInDFS
org.apache.hadoop.hdfs.TestDatanodeConfig
org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean
org.apache.hadoop.hdfs.server.namenode.TestMetaSave
org.apache.hadoop.hdfs.server.namenode.TestFsck
org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace
org.apache.hadoop.hdfs.server.namenode.TestAllowFormat
org.apache.hadoop.hdfs.TestModTime
org.apache.hadoop.hdfs.TestDFSRollback
org.apache.hadoop.hdfs.server.namenode.TestEditLogFileOutputStream
org.apache.hadoop.hdfs.server.namenode.TestEditLog
org.apache.hadoop.hdfs.server.namenode.TestStreamFile
org.apache.hadoop.hdfs.TestDecommission
org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
org.apache.hadoop.hdfs.server.datanode.TestDiskError
org.apache.hadoop.hdfs.server.datanode.TestTransferRbw
org.apache.hadoop.hdfs.server.namenode.TestSecondaryWebUi
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
org.apache.hadoop.hdfs.TestFileCorruption
org.apache.hadoop.hdfs.TestClientBlockVerification
org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
org.apache.hadoop.hdfs.TestFileAppend3
org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
org.apache.hadoop.hdfs.TestBlocksScheduledCounter
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
org.apache.hadoop.hdfs.server.namenode.TestStartup
org.apache.hadoop.hdfs.server.datanode.TestDatanodeJsp
org.apache.hadoop.hdfs.TestWriteRead
org.apache.hadoop.hdfs.TestRenameWhileOpen
org.apache.hadoop.hdfs.TestDFSRemove
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
org.apache.hadoop.hdfs.TestDFSClientExcludedNodes
org.apache.hadoop.hdfs.TestFSInputChecker
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
org.apache.hadoop.hdfs.server.datanode.TestBlockReport
org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
org.apache.hadoop.hdfs.TestSetTimes
org.apache.hadoop.hdfs.TestPread
org.apache.hadoop.hdfs.security.TestClientProtocolWithDelegationToken
org.apache.hadoop.hdfs.server.namenode.TestEditLogRace
org.apache.hadoop.hdfs.security.TestDelegationToken
org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional
org.apache.hadoop.hdfs.TestDFSShellGenericOptions
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.TestHDFSTrash
org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
org.apache.hadoop.hdfs.TestFileStatus
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
org.apache.hadoop.hdfs.server.namenode.TestNameNodeJspHelper
org.apache.hadoop.hdfs.TestHftpFileSystem
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
org.apache.hadoop.hdfs.server.namenode.TestSafeMode
org.apache.hadoop.hdfs.TestDatanodeRegistration
org.apache.hadoop.hdfs.server.namenode.TestCorruptFilesJsp
org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures
org.apache.hadoop.hdfs.TestLargeBlock
org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
org.apache.hadoop.hdfs.TestParallelRead
org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
org.apache.hadoop.hdfs.server.datanode.TestInterDatanodeProtocol
org.apache.hadoop.hdfs.server.datanode.TestDatanodeRestart
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.TestFSOutputSummer
org.apache.hadoop.hdfs.TestFileCreationEmpty
org.apache.hadoop.hdfs.TestBalancerBandwidth
org.apache.hadoop.hdfs.TestWriteConfigurationToDFS
org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader
org.apache.hadoop.hdfs.security.token.block.TestBlockToken
org.apache.hadoop.hdfs.TestGetBlocks
org.apache.hadoop.hdfs.TestSmallBlock
org.apache.hadoop.hdfs.TestSetrepIncreasing
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings
org.apache.hadoop.hdfs.TestReplication
org.apache.hadoop.hdfs.TestDFSUpgrade
org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
org.apache.hadoop.hdfs.TestQuota
org.apache.hadoop.hdfs.TestFileConcurrentReader

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1642//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1642//console

This message is automatically generated.

Show

Hadoop QA added a comment - 05/Dec/11 23:57 -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506178/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.TestConnCache org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite org.apache.hadoop.hdfs.TestSafeMode org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat org.apache.hadoop.hdfs.TestFileAppend org.apache.hadoop.hdfs.TestDatanodeDeath org.apache.hadoop.hdfs.server.namenode.TestFileLimit org.apache.hadoop.hdfs.TestLease org.apache.hadoop.hdfs.TestShortCircuitLocalRead org.apache.hadoop.hdfs.TestPipelines org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes org.apache.hadoop.hdfs.TestHFlush org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount org.apache.hadoop.hdfs.server.namenode.TestBlockUnderConstruction org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker org.apache.hadoop.hdfs.server.namenode.TestTransferFsImage org.apache.hadoop.hdfs.TestBlockMissingException org.apache.hadoop.hdfs.TestClientReportBadBlock org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.TestMultiThreadedHflush org.apache.hadoop.hdfs.TestDFSFinalize org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestDFSStartupVersions org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.tools.TestGetGroups org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps org.apache.hadoop.hdfs.TestDataTransferProtocol org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.hdfs.TestAppendDifferentChecksum org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete org.apache.hadoop.hdfs.server.datanode.TestWriteToReplica org.apache.hadoop.hdfs.TestListFilesInDFS org.apache.hadoop.hdfs.TestDatanodeConfig org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean org.apache.hadoop.hdfs.server.namenode.TestMetaSave org.apache.hadoop.hdfs.server.namenode.TestFsck org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace org.apache.hadoop.hdfs.server.namenode.TestAllowFormat org.apache.hadoop.hdfs.TestModTime org.apache.hadoop.hdfs.TestDFSRollback org.apache.hadoop.hdfs.server.namenode.TestEditLogFileOutputStream org.apache.hadoop.hdfs.server.namenode.TestEditLog org.apache.hadoop.hdfs.server.namenode.TestStreamFile org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode org.apache.hadoop.hdfs.server.datanode.TestDiskError org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.server.namenode.TestSecondaryWebUi org.apache.hadoop.hdfs.server.datanode.TestDataNodeMXBean org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestFileCorruption org.apache.hadoop.hdfs.TestClientBlockVerification org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog org.apache.hadoop.hdfs.TestFileAppend3 org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.TestBlocksScheduledCounter org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus org.apache.hadoop.hdfs.server.namenode.TestStartup org.apache.hadoop.hdfs.server.datanode.TestDatanodeJsp org.apache.hadoop.hdfs.TestWriteRead org.apache.hadoop.hdfs.TestRenameWhileOpen org.apache.hadoop.hdfs.TestDFSRemove org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.TestDFSClientExcludedNodes org.apache.hadoop.hdfs.TestFSInputChecker org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner org.apache.hadoop.hdfs.TestSetTimes org.apache.hadoop.hdfs.TestPread org.apache.hadoop.hdfs.security.TestClientProtocolWithDelegationToken org.apache.hadoop.hdfs.server.namenode.TestEditLogRace org.apache.hadoop.hdfs.security.TestDelegationToken org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional org.apache.hadoop.hdfs.TestDFSShellGenericOptions org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark org.apache.hadoop.hdfs.TestFileStatus org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.hdfs.server.namenode.TestNameNodeJspHelper org.apache.hadoop.hdfs.TestHftpFileSystem org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.server.namenode.TestSafeMode org.apache.hadoop.hdfs.TestDatanodeRegistration org.apache.hadoop.hdfs.server.namenode.TestCorruptFilesJsp org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork org.apache.hadoop.hdfs.TestParallelRead org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage org.apache.hadoop.hdfs.server.datanode.TestInterDatanodeProtocol org.apache.hadoop.hdfs.server.datanode.TestDatanodeRestart org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes org.apache.hadoop.hdfs.TestFSOutputSummer org.apache.hadoop.hdfs.TestFileCreationEmpty org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.TestWriteConfigurationToDFS org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader org.apache.hadoop.hdfs.security.token.block.TestBlockToken org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.hdfs.TestSmallBlock org.apache.hadoop.hdfs.TestSetrepIncreasing org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.TestDFSUpgrade org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration org.apache.hadoop.hdfs.TestQuota org.apache.hadoop.hdfs.TestFileConcurrentReader +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1642//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1642//console This message is automatically generated.

Hide

Permalink

Aaron T. Myers added a comment - 07/Dec/11 01:42

Same sort of odd failure. I've just kicked Jenkins again. Third time's a charm?

Show

Aaron T. Myers added a comment - 07/Dec/11 01:42 Same sort of odd failure. I've just kicked Jenkins again. Third time's a charm?

Hide

Permalink

Hadoop QA added a comment - 07/Dec/11 03:03

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506178/HDFS-2430.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 11 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
org.apache.hadoop.hdfs.TestParallelRead
org.apache.hadoop.hdfs.TestCrcCorruption
org.apache.hadoop.hdfs.server.namenode.TestGenericJournalConf
org.apache.hadoop.hdfs.TestQuota
org.apache.hadoop.hdfs.security.token.block.TestBlockToken
org.apache.hadoop.hdfs.TestFileAppend3
org.apache.hadoop.hdfs.security.TestDelegationToken
org.apache.hadoop.hdfs.tools.TestGetGroups

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1652//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1652//console

This message is automatically generated.

Show

Hadoop QA added a comment - 07/Dec/11 03:03 -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506178/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.TestParallelRead org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.server.namenode.TestGenericJournalConf org.apache.hadoop.hdfs.TestQuota org.apache.hadoop.hdfs.security.token.block.TestBlockToken org.apache.hadoop.hdfs.TestFileAppend3 org.apache.hadoop.hdfs.security.TestDelegationToken org.apache.hadoop.hdfs.tools.TestGetGroups +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1652//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1652//console This message is automatically generated.

Hide

Permalink

Aaron T. Myers added a comment - 07/Dec/11 19:35

Most of those test failures seem to be spurious, but the recently-introduced TestGenericJournalConf failure does seem legitimate. I'm looking into that.

Show

Aaron T. Myers added a comment - 07/Dec/11 19:35 Most of those test failures seem to be spurious, but the recently-introduced TestGenericJournalConf failure does seem legitimate. I'm looking into that.

Hide

Permalink

Aaron T. Myers added a comment - 07/Dec/11 19:58

Updated patch to address the TestGenericJournalConf failure. The only difference between this patch and the last is:

diff --git hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apa
index 5679900..00fe43f 100644
— hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java
+++ hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java
@@ -118,6 +118,7 @@ public class TestGenericJournalConf {
DummyJournalManager.class.getName());
conf.set(DFSConfigKeys.DFS_NAMENODE_EDITS_DIR_KEY,
"dummy://test");
+ conf.setInt(DFSConfigKeys.DFS_NAMENODE_CHECKED_VOLUMES_MINIMUM_KEY, 0);
try {
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(0).build();
cluster.waitActive();

Show

Aaron T. Myers added a comment - 07/Dec/11 19:58 Updated patch to address the TestGenericJournalConf failure. The only difference between this patch and the last is: diff --git hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apa index 5679900..00fe43f 100644 — hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java +++ hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java @@ -118,6 +118,7 @@ public class TestGenericJournalConf { DummyJournalManager.class.getName()); conf.set(DFSConfigKeys.DFS_NAMENODE_EDITS_DIR_KEY, "dummy://test"); + conf.setInt(DFSConfigKeys.DFS_NAMENODE_CHECKED_VOLUMES_MINIMUM_KEY, 0); try { cluster = new MiniDFSCluster.Builder(conf).numDataNodes(0).build(); cluster.waitActive();

Hide

Permalink

Aaron T. Myers added a comment - 07/Dec/11 19:59

I ran all the tests which failed in the last run on my local box, and they all passed:

-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.hdfs.TestQuota
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.869 sec
Running org.apache.hadoop.hdfs.tools.TestGetGroups
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.481 sec
Running org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.276 sec
Running org.apache.hadoop.hdfs.security.TestDelegationToken
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.952 sec
Running org.apache.hadoop.hdfs.security.token.block.TestBlockToken
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.599 sec
Running org.apache.hadoop.hdfs.TestFileAppend3
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.35 sec
Running org.apache.hadoop.hdfs.TestParallelRead
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.063 sec
Running org.apache.hadoop.hdfs.server.namenode.TestGenericJournalConf
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.19 sec

Results :

Tests run: 36, Failures: 0, Errors: 0, Skipped: 0

Show

Aaron T. Myers added a comment - 07/Dec/11 19:59 I ran all the tests which failed in the last run on my local box, and they all passed: ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.TestQuota Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.869 sec Running org.apache.hadoop.hdfs.tools.TestGetGroups Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.481 sec Running org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.276 sec Running org.apache.hadoop.hdfs.security.TestDelegationToken Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.952 sec Running org.apache.hadoop.hdfs.security.token.block.TestBlockToken Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.599 sec Running org.apache.hadoop.hdfs.TestFileAppend3 Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.35 sec Running org.apache.hadoop.hdfs.TestParallelRead Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.063 sec Running org.apache.hadoop.hdfs.server.namenode.TestGenericJournalConf Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.19 sec Results : Tests run: 36, Failures: 0, Errors: 0, Skipped: 0

Hide

Permalink

Eli Collins added a comment - 07/Dec/11 20:10

+1 to the delta, lgtm

Show

Eli Collins added a comment - 07/Dec/11 20:10 +1 to the delta, lgtm

Hide

Permalink

Hadoop QA added a comment - 07/Dec/11 21:38

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12506503/HDFS-2430.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 15 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1657//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1657//console

This message is automatically generated.

Show

Hadoop QA added a comment - 07/Dec/11 21:38 +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506503/HDFS-2430.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1657//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1657//console This message is automatically generated.

Hide

Permalink

Aaron T. Myers added a comment - 07/Dec/11 21:47

Thanks a lot for the reviews, Eli. I've just committed this to trunk.

Show

Aaron T. Myers added a comment - 07/Dec/11 21:47 Thanks a lot for the reviews, Eli. I've just committed this to trunk.

Hide

Permalink

Hudson added a comment - 07/Dec/11 22:17

Integrated in Hadoop-Hdfs-trunk-Commit #1448 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1448/)
~~HDFS-2430~~. The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650
Files :

/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

Show

Hudson added a comment - 07/Dec/11 22:17 Integrated in Hadoop-Hdfs-trunk-Commit #1448 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1448/ ) HDFS-2430 . The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

Hide

Permalink

Hudson added a comment - 07/Dec/11 22:18

Integrated in Hadoop-Common-trunk-Commit #1376 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1376/)
~~HDFS-2430~~. The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650
Files :

/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

Show

Hudson added a comment - 07/Dec/11 22:18 Integrated in Hadoop-Common-trunk-Commit #1376 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1376/ ) HDFS-2430 . The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

Hide

Permalink

Hudson added a comment - 07/Dec/11 22:28

Integrated in Hadoop-Mapreduce-trunk-Commit #1400 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1400/)
~~HDFS-2430~~. The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650
Files :

/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

Show

Hudson added a comment - 07/Dec/11 22:28 Integrated in Hadoop-Mapreduce-trunk-Commit #1400 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1400/ ) HDFS-2430 . The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

Hide

Permalink

Hudson added a comment - 08/Dec/11 15:33

Integrated in Hadoop-Mapreduce-trunk #921 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/921/)
~~HDFS-2430~~. The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650
Files :

/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

Show

Hudson added a comment - 08/Dec/11 15:33 Integrated in Hadoop-Mapreduce-trunk #921 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/921/ ) HDFS-2430 . The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

Hide

Permalink

Hudson added a comment - 08/Dec/11 15:42

Integrated in Hadoop-Hdfs-trunk #888 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/888/)
~~HDFS-2430~~. The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650
Files :

/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

Show

Hudson added a comment - 08/Dec/11 15:42 Integrated in Hadoop-Hdfs-trunk #888 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/888/ ) HDFS-2430 . The number of failed or low-resource volumes the NN can tolerate should be configurable. Contributed by Aaron T. Myers atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1211650 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckableNameNodeResource.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourceChecker.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java

The number of failed or low-resource volumes the NN can tolerate should be configurable

Details

Description

Attachments

Activity

People

Dates

Development

Agile