Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5896

InputSplits should indicate which locations have the block cached in memory

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. MAPREDUCE-5896.patch
      14 kB
      Sandy Ryza
    2. MAPREDUCE-5896-1.patch
      24 kB
      Sandy Ryza
    3. MAPREDUCE-5896-2.patch
      24 kB
      Sandy Ryza

      Issue Links

        Activity

        Sandy Ryza created issue -
        Sandy Ryza made changes -
        Field Original Value New Value
        Link This issue is depended upon by SPARK-1767 [ SPARK-1767 ]
        Hide
        Sandy Ryza added a comment -

        Given HDFS's plans for hierarchical storage management, I think it would be good to make this extensible to handle storage mediums beyond memory. I talked this over with Andrew Wang and Aaron T. Myers and we think the right interface would be something like a SplitLocationInfo class, with isInMemory() and isOnDisk() methods. We can later add isInFlash() and possibly even getDisk() to return which disk the data is on.

        InputSplits would have a "SplitLocationInfo[] getLocationInfo()" method that returns info about how the data is stored on each host returned by getLocations().

        Show
        Sandy Ryza added a comment - Given HDFS's plans for hierarchical storage management, I think it would be good to make this extensible to handle storage mediums beyond memory. I talked this over with Andrew Wang and Aaron T. Myers and we think the right interface would be something like a SplitLocationInfo class, with isInMemory() and isOnDisk() methods. We can later add isInFlash() and possibly even getDisk() to return which disk the data is on. InputSplits would have a "SplitLocationInfo[] getLocationInfo()" method that returns info about how the data is stored on each host returned by getLocations().
        Sandy Ryza made changes -
        Attachment MAPREDUCE-5896.patch [ 12646096 ]
        Sandy Ryza made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Sandy Ryza added a comment -

        Uploaded a POC patch. I'll add some tests if others think the APIs make sense.

        Show
        Sandy Ryza added a comment - Uploaded a POC patch. I'll add some tests if others think the APIs make sense.
        Hide
        Tom White added a comment -

        From a compatibility point of view I think the changes are OK, although SplitLocationInfo should not (yet) be marked as @Stable. Also, mapred.FileSplit has a new getSplitLocationInfo() method, but it should be getLocationInfo(). Add an @Override annotation to catch this.

        How would the split calculation in FileInputFormat change to use hosts with a cached copy of a particular block? It would be worth creating a trial implementation to test that the changes are sufficient.

        Show
        Tom White added a comment - From a compatibility point of view I think the changes are OK, although SplitLocationInfo should not (yet) be marked as @Stable. Also, mapred.FileSplit has a new getSplitLocationInfo() method, but it should be getLocationInfo(). Add an @Override annotation to catch this. How would the split calculation in FileInputFormat change to use hosts with a cached copy of a particular block? It would be worth creating a trial implementation to test that the changes are sufficient.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12646096/MAPREDUCE-5896.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4620//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4620//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12646096/MAPREDUCE-5896.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4620//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4620//console This message is automatically generated.
        Hide
        Sandy Ryza added a comment -

        What should the criteria be for marking this stable? I'd like to start using this in downstream projects (Spark, and I believe Tez could benefit as well) as soon as possible, and an Evolving annotation would prevent this.

        Show
        Sandy Ryza added a comment - What should the criteria be for marking this stable? I'd like to start using this in downstream projects (Spark, and I believe Tez could benefit as well) as soon as possible, and an Evolving annotation would prevent this.
        Hide
        Tom White added a comment -

        @Evolving would mean that it could change incompatibly between 2.4.0 and 2.5.0 say, but not between 2.4.0 and 2.4.1 - and that might be the right trade off here. Hopefully SplitLocationInfo would not need to be changed incompatibly, but it's hard to know for sure until it's being used.

        Show
        Tom White added a comment - @Evolving would mean that it could change incompatibly between 2.4.0 and 2.5.0 say, but not between 2.4.0 and 2.4.1 - and that might be the right trade off here. Hopefully SplitLocationInfo would not need to be changed incompatibly, but it's hard to know for sure until it's being used.
        Hide
        Sandy Ryza added a comment -

        Updated patch sets all the new APIs to Evolving, fixes the typo that Tom noticed, includes cached hosts in mapred.FileInputFormat split generation, and adds tests.

        Show
        Sandy Ryza added a comment - Updated patch sets all the new APIs to Evolving, fixes the typo that Tom noticed, includes cached hosts in mapred.FileInputFormat split generation, and adds tests.
        Sandy Ryza made changes -
        Attachment MAPREDUCE-5896-1.patch [ 12647178 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12647178/MAPREDUCE-5896-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4630//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4630//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12647178/MAPREDUCE-5896-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4630//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4630//console This message is automatically generated.
        Hide
        Tom White added a comment -

        +1 for the API changes.

        Show
        Tom White added a comment - +1 for the API changes.
        Hide
        Karthik Kambatla (Inactive) added a comment -

        Review comments:

        1. Can we make InputSplitLocationInfo extend InputSplit? It doesn't make sense for any class to implement only InputSplitLocationInfo without implementing InputSplit.
        2. I am uncomfortable having to depend on index matching between InputSplit#getLocations and InputSplit#getLocationInfo. Do you think it would make sense to include the string corresponding to the location in SplitLocationInfo itself? We could deprecate InputSplit#getLocations(). Users are to be expected to use getLocationInfos instead.
        3. Nothing to do with this patch. It is unfortunate that mapreduce.InputSplit doesn't implement mapred.InputSplit. Would it be easy to fix it?
        4. Nit: The following two constants should probably be in SplitLocationInfo?
            private static final SplitLocationInfo ON_DISK = new SplitLocationInfo(false);
            private static final SplitLocationInfo IN_MEMORY = new SplitLocationInfo(true);
          
        5. Nit: Instead of assigning ON_DISK by default, would it make sense to set it post null-check after the loop for checking if it is in memory.
             for (int i = 0; i < hosts.length; i++) {
               hostInfos[i] = ON_DISK;
               // because N will be tiny, scanning is probably faster than a HashSet
               for (String inMemoryHost : inMemoryHosts) {
          
        Show
        Karthik Kambatla (Inactive) added a comment - Review comments: Can we make InputSplitLocationInfo extend InputSplit? It doesn't make sense for any class to implement only InputSplitLocationInfo without implementing InputSplit. I am uncomfortable having to depend on index matching between InputSplit#getLocations and InputSplit#getLocationInfo. Do you think it would make sense to include the string corresponding to the location in SplitLocationInfo itself? We could deprecate InputSplit#getLocations(). Users are to be expected to use getLocationInfos instead. Nothing to do with this patch. It is unfortunate that mapreduce.InputSplit doesn't implement mapred.InputSplit. Would it be easy to fix it? Nit: The following two constants should probably be in SplitLocationInfo? private static final SplitLocationInfo ON_DISK = new SplitLocationInfo( false ); private static final SplitLocationInfo IN_MEMORY = new SplitLocationInfo( true ); Nit: Instead of assigning ON_DISK by default, would it make sense to set it post null-check after the loop for checking if it is in memory. for ( int i = 0; i < hosts.length; i++) { hostInfos[i] = ON_DISK; // because N will be tiny, scanning is probably faster than a HashSet for ( String inMemoryHost : inMemoryHosts) {
        Hide
        Sandy Ryza added a comment -

        Can we make InputSplitLocationInfo extend InputSplit? It doesn't make sense for any class to implement only InputSplitLocationInfo without implementing InputSplit.

        Will do.

        Nothing to do with this patch. It is unfortunate that mapreduce.InputSplit doesn't implement mapred.InputSplit. Would it be easy to fix it?

        Not entirely sure the reasoning there, but as this stuff can have binary compatibility implications in mysterious ways, I'd rather not touch it if we don't need to.

        The following two constants should probably be in SplitLocationInfo?

        They're only used in FileSplit and not in SplitLocationInfo - is there utility in moving them away from where they're used? I'd like to avoid adding these constants to the API because, when we include additional storage types, each SplitLocationInfo could end up as a union of storage types - needing to add a ON_DISK_AND_IN_FLASH_AND_IN_MEMORY would be ugly.

        Instead of assigning ON_DISK by default, would it make sense to set it post null-check after the loop for checking if it is in memory.

        Any advantage to this? It would add extra code, an extra branch, and I don't think be particularly more readable.

        Do you think it would make sense to include the string corresponding to the location in SplitLocationInfo itself?

        Will do.

        Show
        Sandy Ryza added a comment - Can we make InputSplitLocationInfo extend InputSplit? It doesn't make sense for any class to implement only InputSplitLocationInfo without implementing InputSplit. Will do. Nothing to do with this patch. It is unfortunate that mapreduce.InputSplit doesn't implement mapred.InputSplit. Would it be easy to fix it? Not entirely sure the reasoning there, but as this stuff can have binary compatibility implications in mysterious ways, I'd rather not touch it if we don't need to. The following two constants should probably be in SplitLocationInfo? They're only used in FileSplit and not in SplitLocationInfo - is there utility in moving them away from where they're used? I'd like to avoid adding these constants to the API because, when we include additional storage types, each SplitLocationInfo could end up as a union of storage types - needing to add a ON_DISK_AND_IN_FLASH_AND_IN_MEMORY would be ugly. Instead of assigning ON_DISK by default, would it make sense to set it post null-check after the loop for checking if it is in memory. Any advantage to this? It would add extra code, an extra branch, and I don't think be particularly more readable. Do you think it would make sense to include the string corresponding to the location in SplitLocationInfo itself? Will do.
        Sandy Ryza made changes -
        Attachment MAPREDUCE-5896-2.patch [ 12649975 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12649975/MAPREDUCE-5896-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4650//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4650//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12649975/MAPREDUCE-5896-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4650//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4650//console This message is automatically generated.
        Hide
        Karthik Kambatla (Inactive) added a comment -

        Looks good to me. +1. Will commit this later today.

        Show
        Karthik Kambatla (Inactive) added a comment - Looks good to me. +1. Will commit this later today.
        Karthik Kambatla (Inactive) made changes -
        Summary Allow InputSplits to indicate which locations have the block cached in memory InputSplits should indicate which locations have the block cached in memory
        Hide
        Karthik Kambatla (Inactive) added a comment -

        Thanks for the contribution, Sandy. Just committed this to trunk and branch-2.

        Show
        Karthik Kambatla (Inactive) added a comment - Thanks for the contribution, Sandy. Just committed this to trunk and branch-2.
        Karthik Kambatla (Inactive) made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.5.0 [ 12326265 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5731 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5731/)
        MAPREDUCE-5896. InputSplits should indicate which locations have the block cached in memory. (Sandy Ryza via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603670)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputSplitWithLocationInfo.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SplitLocationInfo.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/InputSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5731 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5731/ ) MAPREDUCE-5896 . InputSplits should indicate which locations have the block cached in memory. (Sandy Ryza via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603670 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputSplitWithLocationInfo.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SplitLocationInfo.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/InputSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #588 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/588/)
        MAPREDUCE-5896. InputSplits should indicate which locations have the block cached in memory. (Sandy Ryza via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603670)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputSplitWithLocationInfo.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SplitLocationInfo.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/InputSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #588 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/588/ ) MAPREDUCE-5896 . InputSplits should indicate which locations have the block cached in memory. (Sandy Ryza via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603670 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputSplitWithLocationInfo.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SplitLocationInfo.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/InputSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1779 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1779/)
        MAPREDUCE-5896. InputSplits should indicate which locations have the block cached in memory. (Sandy Ryza via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603670)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputSplitWithLocationInfo.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SplitLocationInfo.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/InputSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1779 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1779/ ) MAPREDUCE-5896 . InputSplits should indicate which locations have the block cached in memory. (Sandy Ryza via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603670 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputSplitWithLocationInfo.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SplitLocationInfo.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/InputSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1806 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1806/)
        MAPREDUCE-5896. InputSplits should indicate which locations have the block cached in memory. (Sandy Ryza via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603670)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputSplitWithLocationInfo.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SplitLocationInfo.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/InputSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileSplit.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1806 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1806/ ) MAPREDUCE-5896 . InputSplits should indicate which locations have the block cached in memory. (Sandy Ryza via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603670 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputSplitWithLocationInfo.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SplitLocationInfo.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/InputSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileSplit.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java
        Karthik Kambatla (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Sandy Ryza
            Reporter:
            Sandy Ryza
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development