Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha, 0.23.3
    • Fix Version/s: 2.0.2-alpha, 0.23.3
    • Component/s: nodemanager
    • Labels:
      None

      Description

      The NM's token secret manager will reject all container tokens after the secret key is activated which means the NM will not launch any containers including AMs. The whole yarn cluster becomes inoperable in 1d.

      1. YARN-60-20120831-branch-23.txt
        41 kB
        Vinod Kumar Vavilapalli
      2. YARN-60-20120829.txt
        38 kB
        Vinod Kumar Vavilapalli
      3. YARN-60-20120829.2.txt
        42 kB
        Vinod Kumar Vavilapalli
      4. YARN-60-20120829.1.txt
        41 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Daryn Sharp created issue -
          Daryn Sharp made changes -
          Field Original Value New Value
          Link This issue is broken by YARN-39 [ YARN-39 ]
          Daryn Sharp made changes -
          Summary NMs rejects all container tokens after 2nd secret key roll NMs rejects all container tokens after secret key rolls
          Description The NM's token secret manager will reject all container tokens after the third secret key is activated which means the NM will not launch _any_ containers including AMs. The whole yarn cluster becomes inoperable in 2d 15m (default 1d key roll + 15m activation delay).

          The NM tracks the previous 2 secrets, so either the RM is generating tokens with an old secret, or more likely the NMs are going out of sync when discarding the oldest secret.
          The NM's token secret manager will reject all container tokens after the secret key is activated which means the NM will not launch _any_ containers including AMs. The whole yarn cluster becomes inoperable in 1d.
          Vinod Kumar Vavilapalli made changes -
          Attachment YARN-60-20120829.txt [ 12543010 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment YARN-60-20120829.1.txt [ 12543021 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment YARN-60-20120829.2.txt [ 12543022 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment YARN-60-20120831-branch-23.txt [ 12543330 ]
          Vinod Kumar Vavilapalli made changes -
          Parent YARN-47 [ 12605130 ]
          Issue Type Bug [ 1 ] Sub-task [ 7 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.1.0-alpha [ 12322547 ]
          Fix Version/s 0.23.3 [ 12322841 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322548 ]
          Fix Version/s 2.1.0-alpha [ 12322547 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Vinod Kumar Vavilapalli
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development