Hadoop Common
  1. Hadoop Common
  2. HADOOP-7015

RawLocalFileSystem#listStatus does not deal with a directory whose entries are changing ( e.g. in a multi-thread or multi-process environment)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. hadoop-7015-2.patch
      1 kB
      Eli Collins
    2. hadoop-7015.patch
      1 kB
      Sanjay Radia

      Issue Links

        Activity

        Hide
        Sanjay Radia added a comment -

        RawLocalFileSystem#listStatus does a list followed by a getFileStatus for each entry.
        It should ignore entries that do not exist since they may have been deleted after the list operation.

        Show
        Sanjay Radia added a comment - RawLocalFileSystem#listStatus does a list followed by a getFileStatus for each entry. It should ignore entries that do not exist since they may have been deleted after the list operation.
        Hide
        Sanjay Radia added a comment -

        Patch attached.

        Show
        Sanjay Radia added a comment - Patch attached.
        Hide
        Doug Cutting added a comment -

        I think you can save a few lines by using Arrays#copyOf to truncate.

        Also, can you please add a test case?

        Show
        Doug Cutting added a comment - I think you can save a few lines by using Arrays#copyOf to truncate. Also, can you please add a test case?
        Hide
        Sanjay Radia added a comment -

        We don't have good MT tests for our file systems and they are not easy to write.
        However, the loadGenerator (an MT filesystem load generator) is a good MT tests for the basic file system operations; I discovered this bug when running it and it now passes with my patch. I think we should incorporate the loadGenerator as part of our commit tests (since it runs for a while); I was planning to file a jira to do that.
        Would incorporating the loadGenerator as part of our commit tests be good enough?
        (Will fix the Arrays#copyOf ....).

        Show
        Sanjay Radia added a comment - We don't have good MT tests for our file systems and they are not easy to write. However, the loadGenerator (an MT filesystem load generator) is a good MT tests for the basic file system operations; I discovered this bug when running it and it now passes with my patch. I think we should incorporate the loadGenerator as part of our commit tests (since it runs for a while); I was planning to file a jira to do that. Would incorporating the loadGenerator as part of our commit tests be good enough? (Will fix the Arrays#copyOf ....).
        Hide
        Sanjay Radia added a comment -

        Forgot to add: when I ran the load generator with 5 threads the bug was not uncovered. With 20 the bug was uncovered.

        Show
        Sanjay Radia added a comment - Forgot to add: when I ran the load generator with 5 threads the bug was not uncovered. With 20 the bug was uncovered.
        Hide
        Doug Cutting added a comment -

        > Would incorporating the loadGenerator as part of our commit tests be good enough?

        Sounds good to me. Can you please file a Jira for that and have this one depend on it?

        Show
        Doug Cutting added a comment - > Would incorporating the loadGenerator as part of our commit tests be good enough? Sounds good to me. Can you please file a Jira for that and have this one depend on it?
        Hide
        Eli Collins added a comment -

        +1 Looks good to me.

        Updated the patch to use Arrays.copyOf.

        Show
        Eli Collins added a comment - +1 Looks good to me. Updated the patch to use Arrays.copyOf.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12459762/hadoop-7015-2.patch
        against trunk revision 1035876.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/106//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/106//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/106//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459762/hadoop-7015-2.patch against trunk revision 1035876. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/106//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/106//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/106//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        I don't see any additional javadoc warnings with this patch (same # warnings are reported before and after applying this patch). I've committed this. Thanks Sanjay!

        Show
        Eli Collins added a comment - I don't see any additional javadoc warnings with this patch (same # warnings are reported before and after applying this patch). I've committed this. Thanks Sanjay!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #432 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/432/)
        HADOOP-7015. RawLocalFileSystem#listStatus does not deal with a directory whose entries are changing (e.g. in a multi-thread or multi-process environment). Contributed by Sanjay Radia

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #432 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/432/ ) HADOOP-7015 . RawLocalFileSystem#listStatus does not deal with a directory whose entries are changing (e.g. in a multi-thread or multi-process environment). Contributed by Sanjay Radia
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #520 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/520/)

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #520 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/520/ )

          People

          • Assignee:
            Sanjay Radia
            Reporter:
            Sanjay Radia
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development