Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha
    • Component/s: tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      • SetOwner operations can change both the user and group which a file or directory belongs to, or just one of those. Currently, in the XML serialization/deserialization code, we don't handle the case where just the group is set, not the user. We should handle this case.
      • consistently serialize generation stamp as GENSTAMP.
      1. HDFS-3206.001.patch
        10 kB
        Colin Patrick McCabe
      2. HDFS-3206.002.patch
        10 kB
        Colin Patrick McCabe
      3. HDFS-3206.003.patch
        10 kB
        Colin Patrick McCabe
      4. HDFS-3206.004.patch
        10 kB
        Colin Patrick McCabe

        Activity

        Hide
        Colin Patrick McCabe added a comment -
        • fix
        Show
        Colin Patrick McCabe added a comment - fix
        Hide
        Colin Patrick McCabe added a comment -

        also, formerly we had some operations serializing generation stamp as GENSTAMP, and some as GENERATION_STAMP. The patch I posted makes it consistent.

        Show
        Colin Patrick McCabe added a comment - also, formerly we had some operations serializing generation stamp as GENSTAMP, and some as GENERATION_STAMP. The patch I posted makes it consistent.
        Hide
        Bikas Saha added a comment -

        At a higher level, is it ok for no user owner to exist and only have a group owner?
        The current XML code seems to imply that a user will always be there. If that is true, and this bug showed up then it implies some namenode code is broken because it allows ownership to be group only.

        Aside from that, the XML serde code should not reflect namenode behavior. So the fix looks good.

        Show
        Bikas Saha added a comment - At a higher level, is it ok for no user owner to exist and only have a group owner? The current XML code seems to imply that a user will always be there. If that is true, and this bug showed up then it implies some namenode code is broken because it allows ownership to be group only. Aside from that, the XML serde code should not reflect namenode behavior. So the fix looks good.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521538/HDFS-3206.001.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
        org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2197//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2197//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521538/HDFS-3206.001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2197//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2197//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        Hi Bikras,

        The edit log reflects changes to the image, not the contents of the image themselves. It's a series of deltas. So it is quite possible to have an OP_SET_OWNER that contains no user name, despite the fact that all files have an owning user associated with them. It just means that at some point, someone changed the group in which the file resides, without changing the owner at the same time.

        Show
        Colin Patrick McCabe added a comment - Hi Bikras, The edit log reflects changes to the image, not the contents of the image themselves. It's a series of deltas. So it is quite possible to have an OP_SET_OWNER that contains no user name, despite the fact that all files have an owning user associated with them. It just means that at some point, someone changed the group in which the file resides, without changing the owner at the same time.
        Hide
        Colin Patrick McCabe added a comment -
        • put OP_END_LOG_SEGMENT at the end of the edit log, which it would be in a real edit log.
        Show
        Colin Patrick McCabe added a comment - put OP_END_LOG_SEGMENT at the end of the edit log, which it would be in a real edit log.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12521618/HDFS-3206.002.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2207//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2207//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521618/HDFS-3206.002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2207//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2207//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12523033/HDFS-3206.003.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2292//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2292//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523033/HDFS-3206.003.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2292//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2292//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -
        • here's a patch with the binary part included. Hopefully jenkins won't choke on it...
        Show
        Colin Patrick McCabe added a comment - here's a patch with the binary part included. Hopefully jenkins won't choke on it...
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12523258/HDFS-3206.004.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2302//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2302//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523258/HDFS-3206.004.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2302//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2302//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        I just want to comment here that the test failure is the result of Jenkins not applying my binary diff. I filed https://issues.apache.org/jira/browse/HADOOP-8291 in hopes the Jenkins will someday support binary diffs.

        Show
        Colin Patrick McCabe added a comment - I just want to comment here that the test failure is the result of Jenkins not applying my binary diff. I filed https://issues.apache.org/jira/browse/HADOOP-8291 in hopes the Jenkins will someday support binary diffs.
        Hide
        Colin Patrick McCabe added a comment -

        I just want to comment here that the test failure is the result of Jenkins not applying my binary diff. I filed https://issues.apache.org/jira/browse/HADOOP-8291 in hopes the Jenkins will someday support binary diffs.

        Show
        Colin Patrick McCabe added a comment - I just want to comment here that the test failure is the result of Jenkins not applying my binary diff. I filed https://issues.apache.org/jira/browse/HADOOP-8291 in hopes the Jenkins will someday support binary diffs.
        Hide
        Eli Collins added a comment -

        +1 looks good. The binary patch didn't apply for me so I generated the new editsStored via:

        ./bin/hdfs oev -i ~/src/hadoop-trunk1/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml -o editsStored -p binary
        

        and ran TestOfflineEditsViewer for sanity.

        Show
        Eli Collins added a comment - +1 looks good. The binary patch didn't apply for me so I generated the new editsStored via: ./bin/hdfs oev -i ~/src/hadoop-trunk1/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml -o editsStored -p binary and ran TestOfflineEditsViewer for sanity.
        Hide
        Eli Collins added a comment -

        I've committed this and merged to branch-2. Thanks Colin.

        Show
        Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Colin.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2174/)
        HDFS-3206. Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2174/ ) HDFS-3206 . Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2101 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2101/)
        HDFS-3206. Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2101 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2101/ ) HDFS-3206 . Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2118 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2118/)
        HDFS-3206. Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768)

        Result = ABORTED
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2118 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2118/ ) HDFS-3206 . Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1019 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1019/)
        HDFS-3206. Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1019 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1019/ ) HDFS-3206 . Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1054 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1054/)
        HDFS-3206. Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1054 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1054/ ) HDFS-3206 . Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml

          People

          • Assignee:
            Colin Patrick McCabe
            Reporter:
            Colin Patrick McCabe
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development