Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1138

Modification times are being overwritten when FSImage loads

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      A very easy way to spot the bug is to do a second restart in TestRestartDFS and check that the modification time on root is the same as it was before the second restart.

      The problem is modifying time of the parent if the modification time of the child is greater than parent's in addToParent.

      So if you have /DIR/File then on creation of a file modification time of the DIR will be set, but on cluster restart, or when secondary is checkpointing and reading the image it will add DIR to "/" and write the new modification time for "/" which is the modification time of DIR.

      This is clearly a bug. I will attach a patch with one more parameter being passed from the loadFSImage that says to not propagate the time.

      1. HDFS-1138
        6 kB
        Dmytro Molkov
      2. HDFS-1138.2.patch
        7 kB
        Dmytro Molkov

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12444237/HDFS-1138.2.patch
        against trunk revision 943214.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444237/HDFS-1138.2.patch against trunk revision 943214. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        I just committed this. Thanks Dmytro!

        Show
        dhruba borthakur added a comment - I just committed this. Thanks Dmytro!
        Hide
        Eli Collins added a comment -

        +1 Patch looks good.

        Show
        Eli Collins added a comment - +1 Patch looks good.
        Hide
        Dmytro Molkov added a comment -

        This one addresses Eli's comments.

        Show
        Dmytro Molkov added a comment - This one addresses Eli's comments.
        Hide
        Eli Collins added a comment -

        Nice catch Dmytro! Patch looks good to me.

        • I'd remove the new addChild method and make both callers explicitly pass whether they propagate the modification time.
        • Would update the comment from "update modification time of the parent directory" to something like "Optionally update the parent's modification time, we shouldn't if we're just loading an image."
        Show
        Eli Collins added a comment - Nice catch Dmytro! Patch looks good to me. I'd remove the new addChild method and make both callers explicitly pass whether they propagate the modification time. Would update the comment from "update modification time of the parent directory" to something like "Optionally update the parent's modification time, we shouldn't if we're just loading an image."
        Hide
        dhruba borthakur added a comment -

        The behaviour you describe is clearly a bug. can somebody please review this one?

        Show
        dhruba borthakur added a comment - The behaviour you describe is clearly a bug. can somebody please review this one?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12444137/HDFS-1138
        against trunk revision 942863.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/172/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/172/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/172/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/172/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444137/HDFS-1138 against trunk revision 942863. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/172/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/172/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/172/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/172/console This message is automatically generated.
        Hide
        Dmytro Molkov added a comment -

        Submitting for testing on Hudson.
        But please take a look and leave your comments

        Show
        Dmytro Molkov added a comment - Submitting for testing on Hudson. But please take a look and leave your comments

          People

          • Assignee:
            Dmytro Molkov
            Reporter:
            Dmytro Molkov
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development