Hadoop Common
  1. Hadoop Common
  2. HADOOP-8328

Duplicate FileSystem Statistics object for 'file' scheme

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: fs
    • Labels:
      None

      Description

      Because of a change in HADOOP-8013, there are duplicate Statistics objects in FileSystem's statistics table: one for LocalFileSystem and one for RawLocalFileSystem. This causes MapReduce local file system counters to be incorrect some of the time.

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1073 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1073/)
        HADOOP-8328. Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1073 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1073/ ) HADOOP-8328 . Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1038 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1038/)
        HADOOP-8328. Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085)

        Result = FAILURE
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1038 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1038/ ) HADOOP-8328 . Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #251 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/251/)
        HADOOP-8328. Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335127)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335127
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #251 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/251/ ) HADOOP-8328 . Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335127) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335127 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2213 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2213/)
        HADOOP-8328. Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085)

        Result = ABORTED
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2213 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2213/ ) HADOOP-8328 . Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085) Result = ABORTED tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Hide
        Robert Joseph Evans added a comment -

        I pulled this into branch-0.23 too.

        Show
        Robert Joseph Evans added a comment - I pulled this into branch-0.23 too.
        Hide
        Tom White added a comment -

        I just committed this.

        Show
        Tom White added a comment - I just committed this.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2195 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2195/)
        HADOOP-8328. Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2195 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2195/ ) HADOOP-8328 . Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2270 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2270/)
        HADOOP-8328. Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2270 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2270/ ) HADOOP-8328 . Duplicate FileSystem Statistics object for 'file' scheme. (Revision 1335085) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335085 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocalFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
        Hide
        Alejandro Abdelnur added a comment -

        +1 lgtm

        Show
        Alejandro Abdelnur added a comment - +1 lgtm
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525502/HADOOP-8328.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/931//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/931//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525502/HADOOP-8328.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/931//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/931//console This message is automatically generated.
        Hide
        Tom White added a comment -

        Here's a patch that restores the former behaviour (i.e. having a single Statistics object for the "file" schema).

        Show
        Tom White added a comment - Here's a patch that restores the former behaviour (i.e. having a single Statistics object for the "file" schema).
        Hide
        Tom White added a comment -

        Task's updateCounters() method uses FileSystem.getAllStatistics() to get all the statistics for all filesystems. However, when there are two Statistics objects for the 'file' scheme, it will use the first one, whichever that happens to be (it is unstable). MapReduce uses the RawLocalFileSystem to read and write local files (e.g. when merging), so if the Statistics object for LocalFileSystem happens to come first out of the Map, then the counters will be too low.

        The quick fix would be to change LocalFileSystem's initialize method to create a single Statistics object - for RawLocalFileSystem - to bring the behaviour in line with branch-1 (i.e. pre-HADOOP-8013). There might be a better longer-term fix though.

        Show
        Tom White added a comment - Task's updateCounters() method uses FileSystem.getAllStatistics() to get all the statistics for all filesystems. However, when there are two Statistics objects for the 'file' scheme, it will use the first one, whichever that happens to be (it is unstable). MapReduce uses the RawLocalFileSystem to read and write local files (e.g. when merging), so if the Statistics object for LocalFileSystem happens to come first out of the Map, then the counters will be too low. The quick fix would be to change LocalFileSystem's initialize method to create a single Statistics object - for RawLocalFileSystem - to bring the behaviour in line with branch-1 (i.e. pre- HADOOP-8013 ). There might be a better longer-term fix though.

          People

          • Assignee:
            Tom White
            Reporter:
            Tom White
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development