[EAGLE-565] Because of the "type" field when has subqueue ,response of RM REST API doesn‘t match the SchedulerWrapper object

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: v0.4.0
• Environment: hadoop2.6.0-cdh5.4.3
• Assignee: Lingang Deng
• Reporter: Lingang Deng
• Created: 23/Sep/16 01:30
• Updated: 22/Jan/17 07:04
• Resolved: 22/Jan/17 07:04

Description

I got the exception

Caused by: org.codehaus.jackson.map.JsonMappingException: Can not deserialize instance of java.lang.String out of START_ARRAY token
 at [Source: java.io.ByteArrayInputStream@1578809b; line: 1, column: 2350] (through reference chain: org.apache.eagle.hadoop.queue.model.scheduler.SchedulerWrapper["scheduler"]->org.apache.eagle.hadoop.queue.model.scheduler.Scheduler["schedulerInfo"]->org.apache.eagle.hadoop.queue.model.scheduler.SchedulerInfo["queues"]->org.apache.eagle.hadoop.queue.model.scheduler.Queues["queue"]->org.apache.eagle.hadoop.queue.model.scheduler.Queue["queues"]->org.apache.eagle.hadoop.queue.model.scheduler.Queues["queue"]->org.apache.eagle.hadoop.queue.model.scheduler.Queue["type"])

The response of https://some.server.address:50030/ws/v1/cluster/scheduler?anonymous=true, the "type" field maybe return Array, but the right is String, and the field is declared as String in class Queue. Then, I got the root cause, there is a YARN JIRA, YARN-4785.Let me give a case:

{
    "scheduler": {
        "schedulerInfo": {
            "type": "capacityScheduler",
            "capacity": 100,
            "usedCapacity": 0,
            "maxCapacity": 100,
            "queueName": "root",
            "queues": {
                "queue": [
                    {
                        "type": "capacitySchedulerLeafQueueInfo",
                        "capacity": 35,
                        "usedCapacity": 0,
                        "maxCapacity": 100,
                        "absoluteCapacity": 35,
                        "absoluteMaxCapacity": 100,
                        "absoluteUsedCapacity": 0,
                        "numApplications": 0,
                        "queueName": "A",
                        "state": "RUNNING",
                        "resourcesUsed": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "hideReservationQueues": false,
                        "nodeLabels": [
                            "*"
                        ],
                        "numActiveApplications": 0,
                        "numPendingApplications": 0,
                        "numContainers": 0,
                        "maxApplications": 20000,
                        "maxApplicationsPerUser": 40000,
                        "maxActiveApplications": 20,
                        "maxActiveApplicationsPerUser": 14,
                        "userLimit": 100,
                        "users": null,
                        "userLimitFactor": 2
                    },
                    {
                        "capacity": 35,
                        "usedCapacity": 0,
                        "maxCapacity": 35,
                        "absoluteCapacity": 35,
                        "absoluteMaxCapacity": 35,
                        "absoluteUsedCapacity": 0,
                        "numApplications": 0,
                        "queueName": "B",
                        "state": "RUNNING",
                        "queues": {
                            "queue": [
                                {
                                    "capacity": 37,
                                    "usedCapacity": 0,
                                    "maxCapacity": 100,
                                    "absoluteCapacity": 12.95,
                                    "absoluteMaxCapacity": 35,
                                    "absoluteUsedCapacity": 0,
                                    "numApplications": 0,
                                    "queueName": "B1",
                                    "state": "RUNNING",
                                    "queues": {
                                        "queue": [
                                            {
                                                "type": "capacitySchedulerLeafQueueInfo",
                                                "capacity": 45,
                                                "usedCapacity": 0,
                                                "maxCapacity": 100,
                                                "absoluteCapacity": 5.8275,
                                                "absoluteMaxCapacity": 35,
                                                "absoluteUsedCapacity": 0,
                                                "numApplications": 0,
                                                "queueName": "B11",
                                                "state": "RUNNING",
                                                "resourcesUsed": {
                                                    "memory": 0,
                                                    "vCores": 0
                                                },
                                                "hideReservationQueues": false,
                                                "nodeLabels": [
                                                    "*"
                                                ],
                                                "numActiveApplications": 0,
                                                "numPendingApplications": 0,
                                                "numContainers": 0,
                                                "maxApplications": 20000,
                                                "maxApplicationsPerUser": 200000,
                                                "maxActiveApplications": 7,
                                                "maxActiveApplicationsPerUser": 20,
                                                "userLimit": 100,
                                                "users": null,
                                                "userLimitFactor": 10
                                            },
                                            {
                                                "type": "capacitySchedulerLeafQueueInfo",
                                                "capacity": 55,
                                                "usedCapacity": 0,
                                                "maxCapacity": 100,
                                                "absoluteCapacity": 7.1225004,
                                                "absoluteMaxCapacity": 35,
                                                "absoluteUsedCapacity": 0,
                                                "numApplications": 0,
                                                "queueName": "B12",
                                                "state": "RUNNING",
                                                "resourcesUsed": {
                                                    "memory": 0,
                                                    "vCores": 0
                                                },
                                                "hideReservationQueues": false,
                                                "nodeLabels": [
                                                    "*"
                                                ],
                                                "numActiveApplications": 0,
                                                "numPendingApplications": 0,
                                                "numContainers": 0,
                                                "maxApplications": 712,
                                                "maxApplicationsPerUser": 712,
                                                "maxActiveApplications": 2,
                                                "maxActiveApplicationsPerUser": 1,
                                                "userLimit": 100,
                                                "users": null,
                                                "userLimitFactor": 1
                                            }
                                        ]
                                    },
                                    "resourcesUsed": {
                                        "memory": 0,
                                        "vCores": 0
                                    },
                                    "hideReservationQueues": false,
                                    "nodeLabels": [
                                        "*"
                                    ]
                                },
                                {
                                    "type": [
                                        "capacitySchedulerLeafQueueInfo"
                                    ],
                                    "capacity": 63,
                                    "usedCapacity": 0,
                                    "maxCapacity": 100,
                                    "absoluteCapacity": 22.05,
                                    "absoluteMaxCapacity": 35,
                                    "absoluteUsedCapacity": 0,
                                    "numApplications": 0,
                                    "queueName": "B2",
                                    "state": "RUNNING",
                                    "resourcesUsed": {
                                        "memory": 0,
                                        "vCores": 0
                                    },
                                    "hideReservationQueues": false,
                                    "nodeLabels": [
                                        "*"
                                    ],
                                    "numActiveApplications": 0,
                                    "numPendingApplications": 0,
                                    "numContainers": 0,
                                    "maxApplications": 20000,
                                    "maxApplicationsPerUser": 30000,
                                    "maxActiveApplications": 7,
                                    "maxActiveApplicationsPerUser": 8,
                                    "userLimit": 100,
                                    "users": null,
                                    "userLimitFactor": 1.5
                                }
                            ]
                        },
                        "resourcesUsed": {
                            "memory": 0,
                            "vCores": 0
                        },
                        "hideReservationQueues": false,
                        "nodeLabels": [
                            "*"
                        ]
                    }
                ]
            }
        }
    }
}

Issue Links

• is part of: EAGLE-561 JsonMappingException in Hadoop queue feeder
• links to: GitHub Pull Request #450
• links to: GitHub Pull Request #451
• links to: GitHub Pull Request #455