Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4288

ClusterStatus.getMapTasks() and ClusterStatus.getReduceTasks() is giving one when no job is running

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      When no job is running in the cluster invoke the ClusterStatus.getMapTasks() and ClusterStatus.getReduceTasks() API's
      Observed that these API's are returning one instead of zero(as no job is running)

        Activity

        Hide
        Devaraj K added a comment -

        It is coming due to the hard-coded values in the below code,

        ResourceMgrDelegate.java
          public ClusterMetrics getClusterMetrics() throws IOException,
              InterruptedException {
            GetClusterMetricsRequest request = recordFactory.newRecordInstance(GetClusterMetricsRequest.class);
            GetClusterMetricsResponse response = applicationsManager.getClusterMetrics(request);
            YarnClusterMetrics metrics = response.getClusterMetrics();
            ClusterMetrics oldMetrics = new ClusterMetrics(1, 1, 1, 1, 1, 1, 
                metrics.getNumNodeManagers() * 10, metrics.getNumNodeManagers() * 2, 1,
                metrics.getNumNodeManagers(), 0, 0);
            return oldMetrics;
          }
        

        Here we cannot get runningMaps, runningReduces, occupiedMapSlots...etc from RM because the yarn cluster is completely based on the resources and resource usages.

        It doesn't look good to show these hard-coded values always to the user when they try to get cluster status using the JobClient.getClusterStatus() API.

        Any thoughts on this?

        Show
        Devaraj K added a comment - It is coming due to the hard-coded values in the below code, ResourceMgrDelegate.java public ClusterMetrics getClusterMetrics() throws IOException, InterruptedException { GetClusterMetricsRequest request = recordFactory.newRecordInstance(GetClusterMetricsRequest.class); GetClusterMetricsResponse response = applicationsManager.getClusterMetrics(request); YarnClusterMetrics metrics = response.getClusterMetrics(); ClusterMetrics oldMetrics = new ClusterMetrics(1, 1, 1, 1, 1, 1, metrics.getNumNodeManagers() * 10, metrics.getNumNodeManagers() * 2, 1, metrics.getNumNodeManagers(), 0, 0); return oldMetrics; } Here we cannot get runningMaps, runningReduces, occupiedMapSlots...etc from RM because the yarn cluster is completely based on the resources and resource usages. It doesn't look good to show these hard-coded values always to the user when they try to get cluster status using the JobClient.getClusterStatus() API. Any thoughts on this?
        Hide
        Karthik Kambatla added a comment -

        In YARN, the ClusterMetrics should only correspond to numNodeManagers, numActiveJobs(), numActiveContainers(), availableResources(). Other job/app-specific metrics should move to the corresponding AMs. JobStatus would be a good place to have these metrics.

        Subsequently, JobClient.getClusterStatus() can correspond to the job-specific metrics (would be a misnomer).

        Comments?

        Show
        Karthik Kambatla added a comment - In YARN, the ClusterMetrics should only correspond to numNodeManagers, numActiveJobs(), numActiveContainers(), availableResources(). Other job/app-specific metrics should move to the corresponding AMs. JobStatus would be a good place to have these metrics. Subsequently, JobClient.getClusterStatus() can correspond to the job-specific metrics (would be a misnomer). Comments?
        Hide
        Karthik Kambatla added a comment -

        Couldn't get around to this, marking it as unassigned should anyone be interested.

        Will pick it up again if still available later.

        Show
        Karthik Kambatla added a comment - Couldn't get around to this, marking it as unassigned should anyone be interested. Will pick it up again if still available later.

          People

          • Assignee:
            Unassigned
            Reporter:
            Nishan Shetty, Huawei
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development