Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4288

ClusterStatus.getMapTasks() and ClusterStatus.getReduceTasks() is giving one when no job is running

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      When no job is running in the cluster invoke the ClusterStatus.getMapTasks() and ClusterStatus.getReduceTasks() API's
      Observed that these API's are returning one instead of zero(as no job is running)

        Issue Links

          Activity

          Hide
          Devaraj K added a comment -

          It is coming due to the hard-coded values in the below code,

          ResourceMgrDelegate.java
            public ClusterMetrics getClusterMetrics() throws IOException,
                InterruptedException {
              GetClusterMetricsRequest request = recordFactory.newRecordInstance(GetClusterMetricsRequest.class);
              GetClusterMetricsResponse response = applicationsManager.getClusterMetrics(request);
              YarnClusterMetrics metrics = response.getClusterMetrics();
              ClusterMetrics oldMetrics = new ClusterMetrics(1, 1, 1, 1, 1, 1, 
                  metrics.getNumNodeManagers() * 10, metrics.getNumNodeManagers() * 2, 1,
                  metrics.getNumNodeManagers(), 0, 0);
              return oldMetrics;
            }
          

          Here we cannot get runningMaps, runningReduces, occupiedMapSlots...etc from RM because the yarn cluster is completely based on the resources and resource usages.

          It doesn't look good to show these hard-coded values always to the user when they try to get cluster status using the JobClient.getClusterStatus() API.

          Any thoughts on this?

          Show
          Devaraj K added a comment - It is coming due to the hard-coded values in the below code, ResourceMgrDelegate.java public ClusterMetrics getClusterMetrics() throws IOException, InterruptedException { GetClusterMetricsRequest request = recordFactory.newRecordInstance(GetClusterMetricsRequest.class); GetClusterMetricsResponse response = applicationsManager.getClusterMetrics(request); YarnClusterMetrics metrics = response.getClusterMetrics(); ClusterMetrics oldMetrics = new ClusterMetrics(1, 1, 1, 1, 1, 1, metrics.getNumNodeManagers() * 10, metrics.getNumNodeManagers() * 2, 1, metrics.getNumNodeManagers(), 0, 0); return oldMetrics; } Here we cannot get runningMaps, runningReduces, occupiedMapSlots...etc from RM because the yarn cluster is completely based on the resources and resource usages. It doesn't look good to show these hard-coded values always to the user when they try to get cluster status using the JobClient.getClusterStatus() API. Any thoughts on this?
          Hide
          Karthik Kambatla (Inactive) added a comment -

          In YARN, the ClusterMetrics should only correspond to numNodeManagers, numActiveJobs(), numActiveContainers(), availableResources(). Other job/app-specific metrics should move to the corresponding AMs. JobStatus would be a good place to have these metrics.

          Subsequently, JobClient.getClusterStatus() can correspond to the job-specific metrics (would be a misnomer).

          Comments?

          Show
          Karthik Kambatla (Inactive) added a comment - In YARN, the ClusterMetrics should only correspond to numNodeManagers, numActiveJobs(), numActiveContainers(), availableResources(). Other job/app-specific metrics should move to the corresponding AMs. JobStatus would be a good place to have these metrics. Subsequently, JobClient.getClusterStatus() can correspond to the job-specific metrics (would be a misnomer). Comments?
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Couldn't get around to this, marking it as unassigned should anyone be interested.

          Will pick it up again if still available later.

          Show
          Karthik Kambatla (Inactive) added a comment - Couldn't get around to this, marking it as unassigned should anyone be interested. Will pick it up again if still available later.
          Hide
          Rohith Sharma K S added a comment -

          The issue is still exist in the latest code base. This issue require more discussion on whether to change the hard coded values which may break comatibility for the MR client OR bring out a new design to handle this dielema. If not planning to fix , I think this can be closed as wont fix.

          Show
          Rohith Sharma K S added a comment - The issue is still exist in the latest code base. This issue require more discussion on whether to change the hard coded values which may break comatibility for the MR client OR bring out a new design to handle this dielema. If not planning to fix , I think this can be closed as wont fix.
          Hide
          Rohith Sharma K S added a comment -

          Linking to similar issue in MR

          Show
          Rohith Sharma K S added a comment - Linking to similar issue in MR

            People

            • Assignee:
              Unassigned
              Reporter:
              Nishan Shetty
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development