Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15891

Make YARN logs less noisy

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.1.0
    • Component/s: YARN
    • Labels:
      None

      Description

      Spark can generate a lot of logs when running in YARN mode. The problem is already a little bad with normal configuration, but is even worse with dynamic allocation on.

      The first problem is that for every executor Spark launches, it will print the whole command and all the env variables it's setting, even though those are exactly the same for every executor. That's not too bad with a handful of executors, but get annoying pretty soon after that. Dynamic allocation makes that problem worse since there executors constantly being started and then going away.

      Also, there's a lot of logging generated by the dynamic allocation backend code in the YARN module. We should audit those and make sure they all make sense, and whether / how to reduce the amount of logging.

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Masiero Vanzin
              Reporter:
              vanzin Marcelo Masiero Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: