Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4395

Reloading FSImage and FSEditLog may erase user and group information

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Blocker
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 0.18.2
        • Component/s: None
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          The bug can be reproduced as following:

          bash-3.2$ ./bin/hadoop fs -lsr                 
-rw-r--r--   3 tsz supergroup       1366 2008-10-10 17:52 /user/tsz/a.txt
-rw-r--r--   3 tsz supergroup       1366 2008-10-10 17:52 /user/tsz/b.txt
bash-3.2$ ./bin/hadoop fs -chown sze a.txt
bash-3.2$ ./bin/hadoop fs -lsr
-rw-r--r--   3 sze supergroup       1366 2008-10-10 17:52 /user/tsz/a.txt
-rw-r--r--   3 tsz supergroup       1366 2008-10-10 17:52 /user/tsz/b.txt

          Then, restart cluster ...

          bash-3.2$ ./bin/hadoop fs -lsr
-rw-r--r--   3 sze                  1366 2008-10-10 17:52 /user/tsz/a.txt
-rw-r--r--   3 tsz supergroup       1366 2008-10-10 17:52 /user/tsz/b.txt

          The group information for a.txt is missing.

          1. Text File
            4395_20081010.patch
            1 kB
            Tsz Wo Nicholas Sze
          2. Text File
            4395_20081011_0.18.patch
            5 kB
            Tsz Wo Nicholas Sze
          3. Text File
            4395_20081011.patch
            5 kB
            Tsz Wo Nicholas Sze

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            szetszwo Tsz Wo Nicholas Sze added a comment -

            HADOOP-3364 changes FSImage and FSEditLog loading. The OP_SET_OWNER command is misinterpreted.

            Show
            szetszwo Tsz Wo Nicholas Sze added a comment - HADOOP-3364 changes FSImage and FSEditLog loading. The OP_SET_OWNER command is misinterpreted.
            Hide
            Permalink
            szetszwo Tsz Wo Nicholas Sze added a comment -

            4395_20081010.patch: fix the problem. see whether I could add a unit test.

            Show
            szetszwo Tsz Wo Nicholas Sze added a comment - 4395_20081010.patch: fix the problem. see whether I could add a unit test.
            Hide
            Permalink
            szetszwo Tsz Wo Nicholas Sze added a comment -

            Tested 4395_20081010.patch manually. It fixed the problem. Submitting ...

            Show
            szetszwo Tsz Wo Nicholas Sze added a comment - Tested 4395_20081010.patch manually. It fixed the problem. Submitting ...
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12391924/4395_20081010.patch
            against trunk revision 703609.

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no tests are needed for this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

            -1 core tests. The patch failed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/console

            This message is automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391924/4395_20081010.patch against trunk revision 703609. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3445/console This message is automatically generated.
            Hide
            Permalink
            szetszwo Tsz Wo Nicholas Sze added a comment - - edited

            4395_20081011.patch: updated a test.

            Show
            szetszwo Tsz Wo Nicholas Sze added a comment - - edited 4395_20081011.patch: updated a test.
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12391936/4395_20081011.patch
            against trunk revision 703609.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

            -1 core tests. The patch failed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/console

            This message is automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12391936/4395_20081011.patch against trunk revision 703609. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3447/console This message is automatically generated.
            Hide
            Permalink
            szetszwo Tsz Wo Nicholas Sze added a comment -

            The failed core test, TestJobQueueInformation, is not related to this issue. See HADOOP-4378.

            Show
            szetszwo Tsz Wo Nicholas Sze added a comment - The failed core test, TestJobQueueInformation, is not related to this issue. See HADOOP-4378 .
            Hide
            Permalink
            chris.douglas Chris Douglas added a comment -

            +1 Patch looks good

            Show
            chris.douglas Chris Douglas added a comment - +1 Patch looks good
            Hide
            Permalink
            szetszwo Tsz Wo Nicholas Sze added a comment -

            4395_20081011_0.18.patch: for 0.18

            Show
            szetszwo Tsz Wo Nicholas Sze added a comment - 4395_20081011_0.18.patch: for 0.18
            Hide
            Permalink
            szetszwo Tsz Wo Nicholas Sze added a comment -

            I just committed this.

            Show
            szetszwo Tsz Wo Nicholas Sze added a comment - I just committed this.
            Hide
            Permalink
            rangadi Raghu Angadi added a comment -

            I think a considerable part of the reason why this bug got in is lack of javadoc contract for setOwner() methods. Though these are supposed to be hdfs internal, javadoc should have been there. But this comment is too late for this jira I guess.

            Show
            rangadi Raghu Angadi added a comment - I think a considerable part of the reason why this bug got in is lack of javadoc contract for setOwner() methods. Though these are supposed to be hdfs internal, javadoc should have been there. But this comment is too late for this jira I guess.
            Hide
            Permalink
            hudson Hudson added a comment -

            Integrated in Hadoop-trunk #634 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/634/)

            Show
            hudson Hudson added a comment - Integrated in Hadoop-trunk #634 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/634/ )

              People

              • Assignee:
                szetszwo Tsz Wo Nicholas Sze
                Reporter:
                szetszwo Tsz Wo Nicholas Sze
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development