Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-711

hdfsUtime does not handle atime = 0 or mtime = 0 correctly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 2.0.2-alpha
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      in HADOOP/src/c++/libhdfs/hdfs.h

      The following function document is incorrect:
      /* @param mtime new modification time or 0 for only set access time in seconds
      @param atime new access time or 0 for only set modification time in seconds
      */
      int hdfsUtime(hdfsFS fs, const char* path, tTime mtime, tTime atime);

      Currently, setting mtime or atime to 0 has no special meaning. That is, file last modified time will change to 0 if the mtime argument is 0.

      libhdfs should translate mtime = 0 or atime = 0 to the special value -1, which in HDFS means "don't change this time."

      1. HDFS-711.003.patch
        1 kB
        Colin Patrick McCabe
      2. HDFS-711.002.patch
        2 kB
        Colin Patrick McCabe
      3. HDFS-711.001.patch
        0.6 kB
        Colin Patrick McCabe

        Activity

        Hide
        Colin Patrick McCabe added a comment -
        • fix incorrect hdfsUtime behavior
        Show
        Colin Patrick McCabe added a comment - fix incorrect hdfsUtime behavior
        Hide
        Eli Collins added a comment -

        Since time_t is signed why not have the same interfaces as ClientProtocol#seTimes, ie the user has to pass -1 to indicate times should not be updated?

        Show
        Eli Collins added a comment - Since time_t is signed why not have the same interfaces as ClientProtocol#seTimes, ie the user has to pass -1 to indicate times should not be updated?
        Hide
        Colin Patrick McCabe added a comment -

        Unfortunately the decision was made to use time_t as the parameter for hdfsUtime. That type is sometimes signed, and sometimes unsigned, so it doesn't really make sense to require users to pass 1, in my opinion. (It would work on modern versions of Linux, though.) (The use of time_t causes other problems- it is only 32 bit on 32-bit Linux systems, for example.)

        It would be nice to have a better API that exposed the true resolution of the HDFS utime (currently we're limited to seconds only, whereas HDFS tracks milliseconds.) However, that's a separate issue, I think...

        Show
        Colin Patrick McCabe added a comment - Unfortunately the decision was made to use time_t as the parameter for hdfsUtime. That type is sometimes signed, and sometimes unsigned, so it doesn't really make sense to require users to pass 1, in my opinion. (It would work on modern versions of Linux, though.) (The use of time_t causes other problems - it is only 32 bit on 32-bit Linux systems, for example.) It would be nice to have a better API that exposed the true resolution of the HDFS utime (currently we're limited to seconds only, whereas HDFS tracks milliseconds.) However, that's a separate issue, I think...
        Hide
        Colin Patrick McCabe added a comment -

        er, the wiki markup went nuts there. Ignore the strikeout.

        Show
        Colin Patrick McCabe added a comment - er, the wiki markup went nuts there. Ignore the strikeout.
        Hide
        Eli Collins added a comment -

        What systems have an unsigned time_t? Thought Unix std was signed.

        IMO using -1 makes sense because it (1) matches the HDFS API and (2) allows you to set zero values which this patch doesn't allow because it overloads zero right?

        Show
        Eli Collins added a comment - What systems have an unsigned time_t? Thought Unix std was signed. IMO using -1 makes sense because it (1) matches the HDFS API and (2) allows you to set zero values which this patch doesn't allow because it overloads zero right?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12535400/HDFS-711.001.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestDatanodeBlockScanner

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2752//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2752//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2752//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535400/HDFS-711.001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2752//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2752//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2752//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        Wikipedia says that QNX has an unsigned time_t. However, we probably don't need to care about that.

        I guess it's probably ok to use the -1 => don't set convention for now. Hopefully the next version of the API will not use time_t, which will clear this problem up.

        Show
        Colin Patrick McCabe added a comment - Wikipedia says that QNX has an unsigned time_t. However, we probably don't need to care about that. I guess it's probably ok to use the -1 => don't set convention for now. Hopefully the next version of the API will not use time_t, which will clear this problem up.
        Hide
        Colin Patrick McCabe added a comment -
        • use -1 rather than 0 as the "don't set" value.
        Show
        Colin Patrick McCabe added a comment - use -1 rather than 0 as the "don't set" value.
        Hide
        Eli Collins added a comment -

        * @param mtime new modification time or -1 for only set access time in seconds

        Shouldn't this be something like "-1 for no change otherwise set the acccess time in seconds"?

        Show
        Eli Collins added a comment - * @param mtime new modification time or -1 for only set access time in seconds Shouldn't this be something like "-1 for no change otherwise set the acccess time in seconds"?
        Hide
        Colin Patrick McCabe added a comment -
        • better wording for hdfsUtime
        Show
        Colin Patrick McCabe added a comment - better wording for hdfsUtime
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12535488/HDFS-711.003.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestHDFSTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2757//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2757//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2757//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535488/HDFS-711.003.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestHDFSTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2757//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2757//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2757//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        +1 looks good. Findbugs warnings are HDFS-3615. Test failure is HDFS-3603. Can add a test in your other unit test patch.

        Show
        Eli Collins added a comment - +1 looks good. Findbugs warnings are HDFS-3615 . Test failure is HDFS-3603 . Can add a test in your other unit test patch.
        Hide
        Eli Collins added a comment -

        I've committed this and merged to branch-2. Thanks Colin!

        Show
        Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Colin!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1098 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1098/)
        HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1098 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1098/ ) HDFS-711 . hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2434 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2434/)
        HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2434 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2434/ ) HDFS-711 . hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2502 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2502/)
        HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2502 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2502/ ) HDFS-711 . hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2452 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2452/)
        HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2452 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2452/ ) HDFS-711 . hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1131 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1131/)
        HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1131 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1131/ ) HDFS-711 . hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/)
        HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/ ) HDFS-711 . hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/)
        HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2441 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/ ) HDFS-711 . hdfsUtime does not handle atime = 0 or mtime = 0 correctly. Contributed by Colin Patrick McCabe (Revision 1358810) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1358810 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.h

          People

          • Assignee:
            Colin Patrick McCabe
            Reporter:
            freestyler
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development