Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2436

FSNamesystem.setTimes(..) expects the path is a file.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.23.0, 0.24.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      FSNamesystem.setTimes(..) does not work if the path is a directory.

      Arpit found this bug when testing webhdfs:

      settimes api is working when called on a file, but when called on a dir it returns a 404. I should be able to set time on both a file and a directory.

      1. HDFS-2436.patch
        4 kB
        Uma Maheswara Rao G
      2. HDFS-2436.patch
        5 kB
        Uma Maheswara Rao G
      3. HDFS-2436.patch
        5 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Nicholas, I just filed the JIRA HDFS-2712.
          Goal for that JIRA is to move the atime and mtime fields down to iNodeFile as this will save the memory consumption with dirs and also support setTimes access only for directories. We can continue the further discussion in that JIRA.

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Thanks Nicholas, I just filed the JIRA HDFS-2712 . Goal for that JIRA is to move the atime and mtime fields down to iNodeFile as this will save the memory consumption with dirs and also support setTimes access only for directories. We can continue the further discussion in that JIRA. Thanks Uma
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > You mean to move the fields(atime and mtime) to INodeFile if it supports only for files?

          If mtime and atime are supported only for files, it makes sense to move the fields to INodeFile since it reduces memory usage.

          Show
          Tsz Wo Nicholas Sze added a comment - > You mean to move the fields(atime and mtime) to INodeFile if it supports only for files? If mtime and atime are supported only for files, it makes sense to move the fields to INodeFile since it reduces memory usage.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Konstnatin,
          I was not known about previos discussion and thought, that was a missing behaviour.
          Also i did not find any test for not supporting on directories. .
          And other point already mentioned by Nicholas.

          It is quite confusing whether atime and mtime are supported for directories. Let file a JIRA to fix it.

          You mean to move the fields(atime and mtime) to INodeFile if it supports only for files?

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Thanks Konstnatin, I was not known about previos discussion and thought, that was a missing behaviour. Also i did not find any test for not supporting on directories. . And other point already mentioned by Nicholas. It is quite confusing whether atime and mtime are supported for directories. Let file a JIRA to fix it. You mean to move the fields(atime and mtime) to INodeFile if it supports only for files? Thanks Uma
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... Directories do not have access times.

          In our code, directories do have access times (and mtime) since the fields are defined in INode but not INodeFile. Also atime and mtime are returned in getFileInfo(..) for directories. It is quite confusing whether atime and mtime are supported for directories. Let file a JIRA to fix it.

          Show
          Tsz Wo Nicholas Sze added a comment - > ... Directories do not have access times. In our code, directories do have access times (and mtime) since the fields are defined in INode but not INodeFile. Also atime and mtime are returned in getFileInfo(..) for directories. It is quite confusing whether atime and mtime are supported for directories. Let file a JIRA to fix it.
          Hide
          Konstantin Shvachko added a comment -

          And sorry for coming late.

          Show
          Konstantin Shvachko added a comment - And sorry for coming late.
          Hide
          Konstantin Shvachko added a comment -

          Guys, accesTime and modificationTime are not supported for directories. It would have been too expensive as any update of a file should trigger update of all directories on the path.
          So making times settable for directories is controversial. It was intentionally implemented on files only. See discussions in HADOOP-1869 e.g. Dhruba's comment saying:
          Implements access time for files. Directories do not have access times.

          Aren't we creating a confusion here by letting people setTimes() which generally are unsupported?
          I agree though that the error message was bad. It should clearly say setTimes() is not supported for directories rather than throwing FileNotFoundException.
          I saw this behavior in DistCp, which tries to setTimes() on directories, similar to WebHdfs.

          Show
          Konstantin Shvachko added a comment - Guys, accesTime and modificationTime are not supported for directories. It would have been too expensive as any update of a file should trigger update of all directories on the path. So making times settable for directories is controversial. It was intentionally implemented on files only. See discussions in HADOOP-1869 e.g. Dhruba's comment saying: Implements access time for files. Directories do not have access times. Aren't we creating a confusion here by letting people setTimes() which generally are unsupported? I agree though that the error message was bad. It should clearly say setTimes() is not supported for directories rather than throwing FileNotFoundException. I saw this behavior in DistCp, which tries to setTimes() on directories, similar to WebHdfs.
          Hide
          Uma Maheswara Rao G added a comment -

          Ok, since we are not sure that, applications may depend on current behaviour, we will not fix this on 20.205 branch.

          Show
          Uma Maheswara Rao G added a comment - Ok, since we are not sure that, applications may depend on current behaviour, we will not fix this on 20.205 branch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          This issue applicable for 20 security as well. Arpit raised this issue against to 20.205 branch!
          I will post the patch for 205 as well.
          Please let me know, if you have any concerns with 20.205 for some reason.

          For 205, the patch will change the behavior. I am not sure if it may break some application.

          Show
          Tsz Wo Nicholas Sze added a comment - This issue applicable for 20 security as well. Arpit raised this issue against to 20.205 branch! I will post the patch for 205 as well. Please let me know, if you have any concerns with 20.205 for some reason. For 205, the patch will change the behavior. I am not sure if it may break some application.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #72 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/72/)
          svn merge -c 1190708 from trunk for HDFS-2436.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #72 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/72/ ) svn merge -c 1190708 from trunk for HDFS-2436 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #878 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/878/)
          HDFS-2436. Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #878 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/878/ ) HDFS-2436 . Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #847 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/847/)
          HDFS-2436. Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #847 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/847/ ) HDFS-2436 . Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #54 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/54/)
          svn merge -c 1190708 from trunk for HDFS-2436.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #54 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/54/ ) svn merge -c 1190708 from trunk for HDFS-2436 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Nicholas,
          This issue applicable for 20 security as well. Arpit raised this issue against to 20.205 branch!
          I will post the patch for 205 as well.
          Please let me know, if you have any concerns with 20.205 for some reason.

          Thanks,
          Uma

          Show
          Uma Maheswara Rao G added a comment - Hi Nicholas, This issue applicable for 20 security as well. Arpit raised this issue against to 20.205 branch! I will post the patch for 205 as well. Please let me know, if you have any concerns with 20.205 for some reason. Thanks, Uma
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1270 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1270/)
          HDFS-2436. Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1270 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1270/ ) HDFS-2436 . Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1194 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1194/)
          HDFS-2436. Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1194 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1194/ ) HDFS-2436 . Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #101 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/101/)
          svn merge -c 1190708 from trunk for HDFS-2436.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #101 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/101/ ) svn merge -c 1190708 from trunk for HDFS-2436 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #101 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/101/)
          svn merge -c 1190708 from trunk for HDFS-2436.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #101 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/101/ ) svn merge -c 1190708 from trunk for HDFS-2436 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #108 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/108/)
          svn merge -c 1190708 from trunk for HDFS-2436.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #108 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/108/ ) svn merge -c 1190708 from trunk for HDFS-2436 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190711 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1219 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1219/)
          HDFS-2436. Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1219 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1219/ ) HDFS-2436 . Change FSNamesystem.setTimes(..) for allowing setting times on directories. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190708 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetTimes.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Uma!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Uma!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501353/HDFS-2436.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestBalancerBandwidth
          org.apache.hadoop.cli.TestHDFSCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1489//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1489//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501353/HDFS-2436.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1489//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1489//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Hi Nicholas, Thanks a lot for the review!

          Yes, no use with that null check. I might missed while just removing the inode.isDirectory() check. Thanks for noticing.

          Updated the patch with the comment fix.

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Hi Nicholas, Thanks a lot for the review! Yes, no use with that null check. I might missed while just removing the inode.isDirectory() check. Thanks for noticing. Updated the patch with the comment fix. Thanks Uma
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Uma, the patch looks good.

          The new getINode(..) does not have to check null. The existing getFileINode(..) should use getINode(..) and do not have to acquire the read lock, i.e.

            INodeFile getFileINode(String src) throws UnresolvedLinkException {
              INode inode = getINode(src);
              if (inode == null || inode.isDirectory())
                return null;
              assert !inode.isLink();      
              return (INodeFile)inode;
            }
          
          Show
          Tsz Wo Nicholas Sze added a comment - Hi Uma, the patch looks good. The new getINode(..) does not have to check null. The existing getFileINode(..) should use getINode(..) and do not have to acquire the read lock, i.e. INodeFile getFileINode( String src) throws UnresolvedLinkException { INode inode = getINode(src); if (inode == null || inode.isDirectory()) return null ; assert !inode.isLink(); return (INodeFile)inode; }
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501318/HDFS-2436.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestBalancerBandwidth
          org.apache.hadoop.cli.TestHDFSCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1485//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1485//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501318/HDFS-2436.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1485//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1485//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          TestSetTimes passing in my environment!
          Looks source assertions has been triggered in hudson. Fixed the test.

          Show
          Uma Maheswara Rao G added a comment - TestSetTimes passing in my environment! Looks source assertions has been triggered in hudson. Fixed the test.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12501288/HDFS-2436.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestSetTimes
          org.apache.hadoop.hdfs.TestBalancerBandwidth
          org.apache.hadoop.cli.TestHDFSCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1484//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1484//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501288/HDFS-2436.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestSetTimes org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1484//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1484//console This message is automatically generated.
          Hide
          Uma Maheswara Rao G added a comment -

          Arpit also found that there is a NPE if calling getFileChecksum(..) with a directory.

          it will throw the NullPointerException on non existing files/directories.

          Show
          Uma Maheswara Rao G added a comment - Arpit also found that there is a NPE if calling getFileChecksum(..) with a directory. it will throw the NullPointerException on non existing files/directories.
          Hide
          Uma Maheswara Rao G added a comment -

          20Security is throwing the NullPointerException.

          I am palnning to sync the behaviors with trunk! And will file separate JIRA for that.

          Filed the separate JIRA HDFS-2509

          Show
          Uma Maheswara Rao G added a comment - 20Security is throwing the NullPointerException. I am palnning to sync the behaviors with trunk! And will file separate JIRA for that. Filed the separate JIRA HDFS-2509
          Hide
          Uma Maheswara Rao G added a comment -

          Arpit also found that there is a NPE if calling getFileChecksum(..) with a directory.

          This is because, getBlockLoactions will return null on dirs

           private synchronized LocatedBlocks getBlockLocationsInternal(String src,
                                                                 long offset, 
                                                                 long length,
                                                                 int nrBlocksToReturn,
                                                                 boolean doAccessTime, 
                                                                 boolean needBlockToken)
                                                                 throws IOException {
              INodeFile inode = dir.getFileINode(src);
              if(inode == null) {
                return null;
              }
          

          Above code snippet is from 20Security branch.

          Looks this is already handled in trunk code

           INodeFile inode = dir.getFileINode(src);
                  if (inode == null) {
                    throw new FileNotFoundException("File does not exist: " + src);
                  }
          

          I just verified in trunk, i did not find the null pointer exception. It throws FileNotFoundException.

          20Security is throwing the NullPointerException.

          I am palnning to sync the behaviors with trunk! And will file separate JIRA for that.

          Thanks,
          Uma

          Show
          Uma Maheswara Rao G added a comment - Arpit also found that there is a NPE if calling getFileChecksum(..) with a directory. This is because, getBlockLoactions will return null on dirs private synchronized LocatedBlocks getBlockLocationsInternal( String src, long offset, long length, int nrBlocksToReturn, boolean doAccessTime, boolean needBlockToken) throws IOException { INodeFile inode = dir.getFileINode(src); if (inode == null ) { return null ; } Above code snippet is from 20Security branch. Looks this is already handled in trunk code INodeFile inode = dir.getFileINode(src); if (inode == null ) { throw new FileNotFoundException( "File does not exist: " + src); } I just verified in trunk, i did not find the null pointer exception. It throws FileNotFoundException. 20Security is throwing the NullPointerException. I am palnning to sync the behaviors with trunk! And will file separate JIRA for that. Thanks, Uma
          Hide
          Uma Maheswara Rao G added a comment -

          Here is the patch for settimes on directories!

          Show
          Uma Maheswara Rao G added a comment - Here is the patch for settimes on directories!
          Hide
          Arpit Gupta added a comment -

          Also the delegation token rest api's throw a 500 stating a npe in the code. We should also see if we can better handle the case where the user tries to get the delegation token when security is off. 500 is not the ideal response as the user has no idea what they did wrong.

          Show
          Arpit Gupta added a comment - Also the delegation token rest api's throw a 500 stating a npe in the code. We should also see if we can better handle the case where the user tries to get the delegation token when security is off. 500 is not the ideal response as the user has no idea what they did wrong.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Arpit also found that there is a NPE if calling getFileChecksum(..) with a directory.

          2011-10-24 15:23:16,569 WARN org.apache.hadoop.hdfs.web.resources.ExceptionHandler: INTERNAL_SERVER_ERROR
          java.lang.NullPointerException
          	at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:783)
          	at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:771)
          	at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods$3.run(DatanodeWebHdfsMethods.java:259)
          	...
          
          Show
          Tsz Wo Nicholas Sze added a comment - Arpit also found that there is a NPE if calling getFileChecksum(..) with a directory. 2011-10-24 15:23:16,569 WARN org.apache.hadoop.hdfs.web.resources.ExceptionHandler: INTERNAL_SERVER_ERROR java.lang.NullPointerException at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:783) at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:771) at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods$3.run(DatanodeWebHdfsMethods.java:259) ...
          Hide
          Tsz Wo Nicholas Sze added a comment -

          This is not webhdfs specific. DistributedFileSystem.setTime(..) also won't work for directories. FSNamesystem.setTimes(..) somehow expects the path is a file.

          Show
          Tsz Wo Nicholas Sze added a comment - This is not webhdfs specific. DistributedFileSystem.setTime(..) also won't work for directories. FSNamesystem.setTimes(..) somehow expects the path is a file.

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Arpit Gupta
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development