Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4709

NMWebServices produces incorrect JSON for containers

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Following exception is thrown when we run below command.

      root@varun-Inspiron-5558:/opt1/hadoop3/bin# ./yarn logs -applicationId application_1455999168135_0002 -am ALL -logFiles ALL

      Container: container_e31_1455999168135_0002_01_000001
      =======================================================
      LogType:syslogstderrstdout
      Log Upload Time:Sun Feb 21 01:44:55 +0530 2016
      Log Contents:
      java.lang.Exception: Cannot find this log on the local disk.
      End of LogType:syslogstderrstdout

      LogType:syslog
      Log Upload Time:Sun Feb 21 01:44:55 +0530 2016
      Log Contents:
      2016-02-21 01:44:49,565 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1455999168135_0002_000001
      2016-02-21 01:44:49,914 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
      /************************************************************

      This is because we annotate containerLogFiles list with XmlElementWrapper which generates XML output as under. And when we read this XML at client side, reading the value associated with containerLogFiles also leads to one value being syslogstderrstdout because both parent and child tags are same. This leads to the exception.

      <containerLogFiles>
        <containerLogFiles>syslog</containerLogFiles>
        <containerLogFiles>stderr</containerLogFiles>
        <containerLogFiles>stdout</containerLogFiles>
      </containerLogFiles>
      

      Moreover, as we use XMLElementWrapper, the JSON generated is as under. This JSON cannot be properly parsed by JSON parser(as a list). This is because child containerLogsFiles entries are treated as a key-value pair(map) and hence only last entry i.e. stdout is picked up. This was found while working on YARN-4517. This makes output unusable.
      This will be an issue for 2 REST endpoints i.e. /ws/v1/node/containers and /ws/v1/node/containers/{{containerId}}

        "containerLogFiles":[
          {
            "containerLogFiles":"syslog",
            "containerLogFiles":"stderr",
            "containerLogFiles":"stdout"
          }
        ]
      

      Ideally the JSON output should be as under.

      "containerLogFiles":["syslog","stderr","stdout"]
      

      We can indicate in the JAXB context to ignore the outer wrapper while marshalling to JSON. But this can only be done at class level. If we do so for ContainerInfo, it would break backward compatibility.
      Hence, to fix it we can remove XmlElementWrapper annotation for containerLogFiles list.
      Another solution would be to wrap the list inside another class.

      But going with former as of now as we do not specify XmlElementWrapper for lists at most of the places in our code.

        Attachments

        1. YARN-4709.02.patch
          7 kB
          Varun Saxena
        2. YARN-4709.01.patch
          7 kB
          Varun Saxena

          Issue Links

            Activity

              People

              • Assignee:
                varun_saxena Varun Saxena
                Reporter:
                varun_saxena Varun Saxena
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: