Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6302

Fail the node if Linux Container Executor is not configured properly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We have a cluster that has one node with misconfigured Linux Container Executor. Every time an AM or regular container is launched on the cluster, it will fail. The node will still have resources available, so it keeps failing apps until the administrator notices the issue and decommissions the node. AM Blacklisting only helps, if the application is already running.

      As a possible improvement, when the LCE is used on the cluster and a NM gets certain errors back from the LCE, like error 24 configuration not found, we should not try to allocate anything on the node anymore or shut down the node entirely. That kind of problem normally does not fix itself and it means that nothing can really run on that node.

      Application application_1488920587909_0010 failed 2 times due to AM Container for appattempt_1488920587909_0010_000002 exited with exitCode: -1000
      Failing this attempt.Diagnostics: Application application_1488920587909_0010 initialization failed (exitCode=24) with output:
      For more detailed output, check the application tracking page: http://node-1.domain.com:8088/cluster/app/application_1488920587909_0010 Then click on links to logs of each attempt.
      . Failing the application.
      

        Attachments

        1. YARN-6302.000.patch
          57 kB
          Miklos Szegedi
        2. YARN-6302.001.patch
          59 kB
          Miklos Szegedi
        3. YARN-6302.002.patch
          59 kB
          Miklos Szegedi
        4. YARN-6302.003.patch
          59 kB
          Miklos Szegedi
        5. YARN-6302.005.branch-2.patch
          59 kB
          Miklos Szegedi

          Issue Links

            Activity

              People

              • Assignee:
                miklos.szegedi@cloudera.com Miklos Szegedi
                Reporter:
                miklos.szegedi@cloudera.com Miklos Szegedi
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: