Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5523

Yarn running container log fetching causes OutOfMemoryError

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: log-aggregation
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Fetching a 256MB log from container caused OOM on client

      $ yarn logs -applicationId application_1470931023753_0001 -log_files log_2016-08-11-21_3.done > logs
      16/08/11 21:58:11 INFO impl.TimelineClientImpl: Timeline service address: http://xxxx:8188/ws/v1/timeline/
      16/08/11 21:58:11 INFO client.RMProxy: Connecting to ResourceManager at xxxx:8050
      16/08/11 21:58:12 INFO client.AHSProxy: Connecting to Application History server at xxxx:10200
      Can not find any log file matching the pattern: [log_2016-08-11-21_3.done] for the container: container_e04_1470931023753_0001_01_000001 within the application: application_1470931023753_0001
      Can not find any log file matching the pattern: [log_2016-08-11-21_3.done] for the container: container_e04_1470931023753_0001_01_000002 within the application: application_1470931023753_0001
      Can not find any log file matching the pattern: [log_2016-08-11-21_3.done] for the container: container_e04_1470931023753_0001_01_000003 within the application: application_1470931023753_0001
      Can not find any log file matching the pattern: [log_2016-08-11-21_3.done] for the container: container_e04_1470931023753_0001_01_000004 within the application: application_1470931023753_0001
      Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
        at java.lang.StringBuilder.append(StringBuilder.java:190)
        at com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:172)
        at com.sun.jersey.core.util.ReaderWriter.readFromAsString(ReaderWriter.java:157)
        at com.sun.jersey.core.provider.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:114)
        at com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:73)
        at com.sun.jersey.core.impl.provider.entity.StringProvider.readFrom(StringProvider.java:58)
        at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:553)
        at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506)
        at org.apache.hadoop.yarn.client.cli.LogsCLI.printContainerLogsFromRunningApplication(LogsCLI.java:477)
        at org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:950)
        at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:280)
        at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:102)
        at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:307)
      

        Attachments

        1. YARN-5523.trunk.3.patch
          12 kB
          Xuan Gong
        2. YARN-5523.trunk.3.patch
          12 kB
          Varun Saxena
        3. YARN-5523.trunk.2.patch
          11 kB
          Xuan Gong
        4. YARN-5523.trunk.1.patch
          3 kB
          Xuan Gong
        5. YARN-5523.branch-2.2.patch
          11 kB
          Xuan Gong
        6. YARN-5523.branch-2.2.patch
          12 kB
          Varun Saxena
        7. YARN-5523.1.patch
          3 kB
          Xuan Gong

          Activity

            People

            • Assignee:
              xgong Xuan Gong
              Reporter:
              prasanth_j Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: