Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3292

[Umbrella] Tests/documentation and/or tools for YARN rolling upgrades backwards/forward compatibility verification

    Details

      Description

      YARN-666 added the support to YARN rolling upgrade. In order to support this feature, we made changes from many perspectives. There were many assumptions made together with these existing changes. Future code changes may break these assumptions by accident, and hence break the YARN rolling upgrades feature.

      To simplify YARN RU regression tests, maybe we would like to create a set of tools/tests that can verify YARN RU backward compatibility.

      On the very first step, we may want to have a compatibility checker for important protocols and APIs. We may also want to incorporate these tools into our test Jenkins runs, if necessary.

        Issue Links

          Activity

          Hide
          gtCarrera9 Li Lu added a comment -

          So far as we can see, YARN requires the following components to be compatible in a rolling upgrade (please feel free to add more in the discussion):

          • Protocols: both public protocols and private wiring protocols
          • RM/NM/ATS state stores: RM/NM/ATS data version numbers and the data store/read schema for each state store.
          • APIs
          • Security tokens
          • Configurations

          We may want to provide a suite of tools and/or unit tests that can verify if an incoming YARN patch will break the compatibility to the previous version. In the very first stage, we may want to finish the following tasks:

          1. Implement a protobuf compatibility checker to check if a patch breaks the compatibility with existing client and internal protocols
          2. Extend the protobuf compatibility checker in step 1 to check RM state store
          3. Look into the possibility to further extend the protobuf checker to NM/ATS(v1) state store (I’m not very sure now, we can merge this with step 2 if a simple extension is possible).
          4. Implement a diff-based java API compatibility checker
          5. Wire up the implemented tools to Jenkins test runs
          6. Finish formal write ups for the YARN rolling upgrade standard

          Please feel free to discuss more about our first step goal. Thanks!

          Show
          gtCarrera9 Li Lu added a comment - So far as we can see, YARN requires the following components to be compatible in a rolling upgrade (please feel free to add more in the discussion): Protocols: both public protocols and private wiring protocols RM/NM/ATS state stores: RM/NM/ATS data version numbers and the data store/read schema for each state store. APIs Security tokens Configurations We may want to provide a suite of tools and/or unit tests that can verify if an incoming YARN patch will break the compatibility to the previous version. In the very first stage, we may want to finish the following tasks: Implement a protobuf compatibility checker to check if a patch breaks the compatibility with existing client and internal protocols Extend the protobuf compatibility checker in step 1 to check RM state store Look into the possibility to further extend the protobuf checker to NM/ATS(v1) state store (I’m not very sure now, we can merge this with step 2 if a simple extension is possible). Implement a diff-based java API compatibility checker Wire up the implemented tools to Jenkins test runs Finish formal write ups for the YARN rolling upgrade standard Please feel free to discuss more about our first step goal. Thanks!
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Edited title to reflect that this is about rolling upgrades and to focus also on forward compatibility.

          I'd also like to see if w can whip up a doc or two for listing general guidelines for developers in addition to our existing compatibility guide. Edited the title to also include that.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Edited title to reflect that this is about rolling upgrades and to focus also on forward compatibility. I'd also like to see if w can whip up a doc or two for listing general guidelines for developers in addition to our existing compatibility guide. Edited the title to also include that.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          +1, this will be super useful in successfully supporting YARN rolling upgrades!

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - +1, this will be super useful in successfully supporting YARN rolling upgrades!

            People

            • Assignee:
              gtCarrera9 Li Lu
              Reporter:
              gtCarrera9 Li Lu
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development