Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10781

The Thread of the NM aggregate log is exhausted and no other Application can aggregate the log

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.9.2, 3.3.0
    • Fix Version/s: None
    • Component/s: yarn
    • Labels:
      None

      Description

      We observed more than 100 applications running on one NM.Most of these applications are SparkStreaming applications, but these applications do not have running Containers.When the offline application running on it finishes, the log cannot be reported to HDFS. When we killed a large number of SparkStreaming applications, we found that a large number of log files were being created on the NN side, causing the read and write performance on the NN side to degrade significantly.Causes the business application to time out。

        Attachments

        1. applications.png
          82 kB
          Xiping Zhang
        2. containers.png
          82 kB
          Xiping Zhang
        3. containers.png
          83 kB
          Xiping Zhang

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zhangxiping Xiping Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: