Details
-
Type:
Sub-task
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: balancer & mover, namenode
-
Labels:None
-
Hadoop Flags:Reviewed
Description
The result of getStoragePolicy command is always 'unspecified' even we has set a StoragePolicy on a directory.But the real placement of blocks is correct.
The result of fsck is not correct either.
$ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD Set storage policy COLD on /tmp/cold $ hdfs storagepolicies -getStoragePolicy -path /tmp/cold The storage policy of /tmp/cold is unspecified $ hdfs fsck -storagepolicies /tmp/cold Blocks NOT satisfying the specified storage policy: Storage Policy Specified Storage Policy # of blocks % of blocks ARCHIVE:4(COLD) HOT 5 55.5556% ARCHIVE:3(COLD) HOT 4 44.4444%
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Hi Jing Zhao
I tried it by super user then found it works well.
So this issue should become "getStoragePolicy should not need superuser privilege".
@jing zhao and Hui Zheng thanks a lot for your inputs..just we need remove the superuser check in FSDirStatAndListingOp#getFileInfo, Attached the patch for same.. I verfied manually,following is the result for same.Did not seen better way to write the testcase.
$ ./hdfs storagepolicies -setStoragePolicy -path /BBP -policy COLD Set storage policy COLD on /BBP
without Patch
host1 bin# ./hdfs storagepolicies -getStoragePolicy -path /BBP The storage policy of /BBP is unspecified
with Patch
host1 bin# ./hdfs storagepolicies -getStoragePolicy -path /BBP
The storage policy of /BBP:
BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 16m 0s | Findbugs (version ) appears to be broken on trunk. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | javac | 7m 52s | There were no new javac warning messages. |
| +1 | javadoc | 10m 12s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 0m 45s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 35s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 35s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 2m 40s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | native | 3m 10s | Pre-build of native portion |
| -1 | hdfs tests | 161m 44s | Tests failed in hadoop-hdfs. |
| 205m 0s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.hdfs.TestAppendSnapshotTruncate |
| hadoop.hdfs.server.datanode.TestDatanodeProtocolRetryPolicy | |
| hadoop.hdfs.TestDistributedFileSystem |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12747027/HDFS-8805.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 206d493 |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11829/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11829/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11829/console |
This message was automatically generated.
Thanks for working on this, Brahma Reddy Battula. The patch looks good to me overall. Some minor comments:
- I think we also need to update getFileInfo(FSDirectory, String, boolean) where we do not need to check isSuperUser anymore.
- Also the parameter includeStoragePolicy can be removed from getFileInfo(FSDirectory, String, boolean, boolean, boolean).
- nitty pick: it may be more natural to have "i.isSymlink() ? HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED : i.getStoragePolicyID()" in the following code.
+ byte policyId = + !i.isSymlink() ? i.getStoragePolicyID() + : HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED;
Jing Zhao thanks a lot for your review..Updated the patch based on your comments kindly review..
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 17m 8s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| -1 | javac | 1m 36s | The patch appears to cause the build to fail. |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12747074/HDFS-8805-002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f8f6091 |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11831/console |
This message was automatically generated.
Resubmitted the patch to run jenkin's again..
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 17m 5s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| -1 | javac | 1m 35s | The patch appears to cause the build to fail. |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12747087/HDFS-8805-002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d19d187 |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11832/console |
This message was automatically generated.
Hi Brahma Reddy Battula, thanks for updating the patch! Looks like the 002 patch cannot compile. Do you want to fix it?
Also the parameter includeStoragePolicy can be removed from getFileInfo(FSDirectory, String, boolean, boolean, boolean).
FSDirWriteFileOp,FSDirAppendOp and FSDirectory are using this API.Now I removed from this methods also which is not required.
Kindly review the latest patch.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| -1 | pre-patch | 21m 20s | Findbugs (version ) appears to be broken on trunk. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | javac | 9m 39s | There were no new javac warning messages. |
| +1 | javadoc | 11m 17s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 28s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 0m 42s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 57s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 2m 34s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | native | 3m 5s | Pre-build of native portion |
| -1 | hdfs tests | 152m 25s | Tests failed in hadoop-hdfs. |
| 204m 5s |
| Reason | Tests |
|---|---|
| Timed out tests | org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12749180/HDFS-8805-003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6d4eee7 |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11928/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11928/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11928/console |
This message was automatically generated.
Thanks for updating the patch, Brahma Reddy Battula! The new patch removes the includeStoragePolicy parameter from both getFileInfo methods. Actually we still need the parameter for the first one: getFileInfo(FSDirectory, String, INodesInPath, boolean, boolean), considering it is called by getAuditFileInfo which does not require the storage policy information. We only need to remove the parameter from the other getFileInfo: getFileInfo(FSDirectory, String, boolean, boolean, boolean).
Jing Zhao If I remove from getFileInfo(FSDirectory, String, boolean, boolean, boolean), I need to remove from getFileInfo(FSDirectory, String, INodesInPath, boolean, boolean) right, same variable (includeStoragePolicy) is passed from first one second one right..? do I miss something..?
Jing Zhao thanks for clarification.. uploaded the patch..Kindly Review..
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 17m 57s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | javac | 7m 49s | There were no new javac warning messages. |
| +1 | javadoc | 9m 53s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 25s | The applied patch does not increase the total number of release audit warnings. |
| +1 | checkstyle | 1m 25s | There were no new checkstyle issues. |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 25s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
| +1 | findbugs | 2m 32s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| +1 | native | 3m 6s | Pre-build of native portion |
| -1 | hdfs tests | 175m 30s | Tests failed in hadoop-hdfs. |
| 220m 40s |
| Reason | Tests |
|---|---|
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12749759/HDFS-8805-004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fa1d84a |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/11959/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/11959/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/11959/console |
This message was automatically generated.
Test case failures are unrelated to this issue..
Thanks for updating the patch, Brahma Reddy Battula. +1 for the 004 patch. I will commit it shortly.
I've committed this to trunk and branch-2. Thanks for the contribution, Brahma Reddy Battula! Thanks for reporting the issue, Hui Zheng!
FAILURE: Integrated in Hadoop-trunk-Commit #8284 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8284/)
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. Contributed by Brahma Reddy Battula. (jing9: rev 1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
FAILURE: Integrated in Hadoop-Yarn-trunk #1015 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1015/)
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. Contributed by Brahma Reddy Battula. (jing9: rev 1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #285 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/285/)
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. Contributed by Brahma Reddy Battula. (jing9: rev 1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2212 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2212/)
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. Contributed by Brahma Reddy Battula. (jing9: rev 1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #282 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/282/)
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. Contributed by Brahma Reddy Battula. (jing9: rev 1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2231 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2231/)
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. Contributed by Brahma Reddy Battula. (jing9: rev 1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #274 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/274/)
HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. Contributed by Brahma Reddy Battula. (jing9: rev 1fc3c779a422bafdb86ad1a5b2349802dda1cb62)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java
Thanks for reporting the issue, Hui! The reason of the issue is that currently we only expose the storage policy information to super user (FSDirStatAndListing#getFileInfo). I think it's ok to remove this restriction.