Reloading FSImage and FSEditLog may erase user and group information

• Key: HADOOP-4395
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Tsz Wo (Nicholas), SZE
• Reporter: Tsz Wo (Nicholas), SZE
• Fix Version/s: 0.18.2

Description

The bug can be reproduced as following:

bash-3.2$ ./bin/hadoop fs -lsr                 
-rw-r--r--   3 tsz supergroup       1366 2008-10-10 17:52 /user/tsz/a.txt
-rw-r--r--   3 tsz supergroup       1366 2008-10-10 17:52 /user/tsz/b.txt
bash-3.2$ ./bin/hadoop fs -chown sze a.txt
bash-3.2$ ./bin/hadoop fs -lsr
-rw-r--r--   3 sze supergroup       1366 2008-10-10 17:52 /user/tsz/a.txt
-rw-r--r--   3 tsz supergroup       1366 2008-10-10 17:52 /user/tsz/b.txt

Then, restart cluster ...

bash-3.2$ ./bin/hadoop fs -lsr
-rw-r--r--   3 sze                  1366 2008-10-10 17:52 /user/tsz/a.txt
-rw-r--r--   3 tsz supergroup       1366 2008-10-10 17:52 /user/tsz/b.txt

The group information for a.txt is missing.

Comments

[Tsz Wo (Nicholas), SZE]

HADOOP-3364 changes FSImage and FSEditLog loading. The OP_SET_OWNER command is misinterpreted.

[Tsz Wo (Nicholas), SZE]

4395_20081010.patch: fix the problem. see whether I could add a unit test.

[Tsz Wo (Nicholas), SZE]

Tested 4395_20081010.patch manually. It fixed the problem. Submitting ...

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391924/4395_20081010.patch
against trunk revision 703609.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/console

This message is automatically generated.

[Tsz Wo (Nicholas), SZE]

4395_20081011.patch: updated a test.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391936/4395_20081011.patch
against trunk revision 703609.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/console

This message is automatically generated.

[Tsz Wo (Nicholas), SZE]

The failed core test, TestJobQueueInformation, is not related to this issue. See HADOOP-4378.

[Chris Douglas]

+1 Patch looks good

[Tsz Wo (Nicholas), SZE]

4395_20081011_0.18.patch: for 0.18

[Tsz Wo (Nicholas), SZE]

I just committed this.

[Raghu Angadi]

I think a considerable part of the reason why this bug got in is lack of javadoc contract for setOwner() methods. Though these are supposed to be hdfs internal, javadoc should have been there. But this comment is too late for this jira I guess.

[Hudson]

Integrated in Hadoop-trunk #634 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/634/)