Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mrv2, performance
    • Labels:
      None

      Description

      With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance.

      1. MAPREDUCE-3813-20120205.txt
        4 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          This is critical for the scan benchmark given the benchmarks low runtimes.

          Show
          Vinod Kumar Vavilapalli added a comment - This is critical for the scan benchmark given the benchmarks low runtimes.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          This should fix it. Also, added a simple test.

          Show
          Vinod Kumar Vavilapalli added a comment - This should fix it. Also, added a simple test.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12513387/MAPREDUCE-3813-20120205.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1785//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1785//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513387/MAPREDUCE-3813-20120205.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1785//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1785//console This message is automatically generated.
          Hide
          Allen Wittenauer added a comment -

          If this is to cache hostname resolution, this shouldn't be necessary as Java already does this for you for free. See the discussion under HDFS-34.

          Show
          Allen Wittenauer added a comment - If this is to cache hostname resolution, this shouldn't be necessary as Java already does this for you for free. See the discussion under HDFS-34 .
          Hide
          Allen Wittenauer added a comment -

          Woops, also meant to point out that properly configured hosts have an OS-level cache for hostnames. So unless there are some benchmarks to back up the claims, I'm not convinced that hostname resolution should be an issue (with the caveat that the machines should be configured correctly).

          Show
          Allen Wittenauer added a comment - Woops, also meant to point out that properly configured hosts have an OS-level cache for hostnames. So unless there are some benchmarks to back up the claims, I'm not convinced that hostname resolution should be an issue (with the caveat that the machines should be configured correctly).
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Allen, this is not for hostname resolution. This is the rack resolver plugin which finds out the racks for a given set of hosts. Not every rack resolver plugin is fast enough, so we need a cache to keep track of hosts whose racks are already resolved.

          Show
          Vinod Kumar Vavilapalli added a comment - Allen, this is not for hostname resolution. This is the rack resolver plugin which finds out the racks for a given set of hosts. Not every rack resolver plugin is fast enough, so we need a cache to keep track of hosts whose racks are already resolved.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Vinod!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Vinod!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1744 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1744/)
          MAPREDUCE-3813. Added a cache for resolved racks. Contributed by Vinod K V.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1744 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1744/ ) MAPREDUCE-3813 . Added a cache for resolved racks. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #498 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/498/)
          Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813. Added a cache for resolved racks.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #498 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/498/ ) Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813 . Added a cache for resolved racks. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1672 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1672/)
          MAPREDUCE-3813. Added a cache for resolved racks. Contributed by Vinod K V.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1672 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1672/ ) MAPREDUCE-3813 . Added a cache for resolved racks. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #488 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/488/)
          Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813. Added a cache for resolved racks.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #488 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/488/ ) Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813 . Added a cache for resolved racks. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #509 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/509/)
          Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813. Added a cache for resolved racks.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #509 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/509/ ) Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813 . Added a cache for resolved racks. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1687 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1687/)
          MAPREDUCE-3813. Added a cache for resolved racks. Contributed by Vinod K V.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1687 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1687/ ) MAPREDUCE-3813 . Added a cache for resolved racks. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #949 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/)
          MAPREDUCE-3813. Added a cache for resolved racks. Contributed by Vinod K V.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #949 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/ ) MAPREDUCE-3813 . Added a cache for resolved racks. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #162 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/)
          Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813. Added a cache for resolved racks.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #162 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/ ) Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813 . Added a cache for resolved racks. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #184 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/)
          Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813. Added a cache for resolved racks.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #184 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/ ) Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813 . Added a cache for resolved racks. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241210 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #982 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/)
          MAPREDUCE-3813. Added a cache for resolved racks. Contributed by Vinod K V.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #982 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/ ) MAPREDUCE-3813 . Added a cache for resolved racks. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241209 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java

            People

            • Assignee:
              Vinod Kumar Vavilapalli
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development