Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2565

DFSPath cache of FileStatus can become stale

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Paths returned from DFS internally cache their FileStatus, so that getStatus(Path) does not require another RPC. This cache is never refreshed and become stale, resulting in program error.

      This should not be fixed until FileSystem#listStatus() is removed by HADOOP-2563, and user code is thus no longer dependent on this cache for good performance.

      1. 2565_20080424.patch
        5 kB
        Tsz Wo Nicholas Sze
      2. 2565_20080604.patch
        4 kB
        Tsz Wo Nicholas Sze
      3. 2565_20080604b.patch
        4 kB
        Tsz Wo Nicholas Sze
      4. 2565_20080605.patch
        4 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          hairong Hairong Kuang added a comment -

          I just committed this. Thanks, Nicholas!

          Show
          hairong Hairong Kuang added a comment - I just committed this. Thanks, Nicholas!
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          2565_20080605.patch: updated with trunk

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - 2565_20080605.patch: updated with trunk
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          See HADOOP-3493 for the failed TestStreamingFailure.

          No test added since there are already a lot of tests using it.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - See HADOOP-3493 for the failed TestStreamingFailure. No test added since there are already a lot of tests using it.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12383419/2565_20080604b.patch
          against trunk revision 663370.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2581/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2581/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2581/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2581/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12383419/2565_20080604b.patch against trunk revision 663370. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2581/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2581/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2581/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2581/console This message is automatically generated.
          Hide
          cutting Doug Cutting added a comment -

          +1 The patch looks good to me. Thanks!

          Show
          cutting Doug Cutting added a comment - +1 The patch looks good to me. Thanks!
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          2565_20080604b.patch: did the rename

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - 2565_20080604b.patch: did the rename
          Hide
          cutting Doug Cutting added a comment -

          This looks good.

          Can you please rename 'makeQualifiedPath4FileStatus' to just be 'makeQualified' or perhaps 'qualifyPath'?

          Show
          cutting Doug Cutting added a comment - This looks good. Can you please rename 'makeQualifiedPath4FileStatus' to just be 'makeQualified' or perhaps 'qualifyPath'?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          2565_20080604.patch: HADOOP-3095 just has been committed. This patch removes DfsPath and keeps creating fully-qualified Path.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - 2565_20080604.patch: HADOOP-3095 just has been committed. This patch removes DfsPath and keeps creating fully-qualified Path.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -
          • Regrading to FileStatus,
            • I am not sure whether Path is required to be fully-qualified but I am afraid there are user codes depending on it.
            • Strictly speaking, FileStatus is currently mutable and it is impossible to make it immutable since it extends Writable. We cannot prevent the readFields method changing the content of the object. I will update the patch to keep the original code if it is desire.
          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Regrading to FileStatus, I am not sure whether Path is required to be fully-qualified but I am afraid there are user codes depending on it. Strictly speaking, FileStatus is currently mutable and it is impossible to make it immutable since it extends Writable. We cannot prevent the readFields method changing the content of the object. I will update the patch to keep the original code if it is desire. Linked it to HADOOP-3095
          Hide
          cutting Doug Cutting added a comment -

          A few comments:

          • FileStatus should remain immutable, I think. If DFS wants to ensure that the FileStatus instances that it generates always have a fully-qualified Path, then it should construct them with a fully-qualified Path, no?
          • Until HADOOP-3095 is fixed, this will make mapreduce slow. Should we make this depend on HADOOP-3095?
          Show
          cutting Doug Cutting added a comment - A few comments: FileStatus should remain immutable, I think. If DFS wants to ensure that the FileStatus instances that it generates always have a fully-qualified Path, then it should construct them with a fully-qualified Path, no? Until HADOOP-3095 is fixed, this will make mapreduce slow . Should we make this depend on HADOOP-3095 ?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          2565_20080424.patch: remove DfsPath

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - 2565_20080424.patch: remove DfsPath

            People

            • Assignee:
              szetszwo Tsz Wo Nicholas Sze
              Reporter:
              cutting Doug Cutting
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development