Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7290

ShuffleHeader should be compatible between client&server when rolling update

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 2.6.0
    • 2.9.0, 3.3.0
    • mrv2

    Description

      Recently, we need to add a property in ShuffleHeader class. And there were a problem that client&server may not compatible because of serialization/deserialization when we were rolling update NodeManager instance. 

      To solve the problem, we made an improvement to help shuffle header version upgrade smoothly. 

      In HeaderVersion class, shuffle client uses targetVersion(eg 1.2.0) to negotiate with shuffle server. In that case, shuffle server has three cases:

      1. shuffle server does not have the newest code which is able to support version chosen, and we have to use defaultVersion(eg 1.0.0) instead*.*
      2. shuffle server can support older version(eg 1.1.0), then it can reply to shuffle client with compatibleVersion(eg 1.1.0), and finally, client and server can use the same version.
      3. shuffle server can support older version(eg 1.3.0), then it can reply to shuffle client with compatibleVersion(eg 1.2.0), and finally, client and server can use the same version.

      Then, client and server can serialize&deserialize property using the available version.

       

      /**
       * unified hard code header version for new properties.
       * @throws IOException
       */
      private void writeByVersion(DataOutput out) throws IOException {
        if (headerVersion == null || headerVersion.compareTo(DEFAULT_HEADER_VERSION_INSTANCE) == 0) {
          return;
        }
        // if current version larger then target version,
        // we should serialize the properties owned by the version in order.
        if (headerVersion.compareTo(HEADER_VERSION_INSTANCE_V1_1) >= 0) {
          // here write version properties
        }
      }
      
      

       

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              lichaojacobs chaoli
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m