Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4150 Versioning and rolling upgrades for MR2
  3. MAPREDUCE-5963

ShuffleHandler DB schema should be versioned with compatible/incompatible changes

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.1
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      ShuffleHandler persist job shuffle info into DB schema, which should be versioned with compatible/incompatible changes to support rolling upgrade.

      1. MAPREDUCE-5963.patch
        12 kB
        Junping Du
      2. MAPREDUCE-5963-v2.1.patch
        11 kB
        Junping Du
      3. MAPREDUCE-5963-v2.patch
        12 kB
        Junping Du

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          12d 23h 56m 1 Junping Du 22/Jul/14 03:54
          Patch Available Patch Available Resolved Resolved
          16h 43m 1 Jason Lowe 22/Jul/14 20:38
          Resolved Resolved Closed Closed
          131d 7h 32m 1 Arun C Murthy 01/Dec/14 03:10
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320355 ]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1840 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1840/)
          MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with compatible/incompatible changes. Contributed by Junping Du (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612652)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1840 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1840/ ) MAPREDUCE-5963 . ShuffleHandler DB schema should be versioned with compatible/incompatible changes. Contributed by Junping Du (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612652 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1813 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1813/)
          MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with compatible/incompatible changes. Contributed by Junping Du (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612652)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1813 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1813/ ) MAPREDUCE-5963 . ShuffleHandler DB schema should be versioned with compatible/incompatible changes. Contributed by Junping Du (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612652 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #621 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/621/)
          MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with compatible/incompatible changes. Contributed by Junping Du (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612652)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #621 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/621/ ) MAPREDUCE-5963 . ShuffleHandler DB schema should be versioned with compatible/incompatible changes. Contributed by Junping Du (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612652 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Jason Lowe made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Fix Version/s 2.6.0 [ 12327180 ]
          Resolution Fixed [ 1 ]
          Hide
          Jason Lowe added a comment -

          Thanks, Junping! I committed this to trunk and branch-2.

          Show
          Jason Lowe added a comment - Thanks, Junping! I committed this to trunk and branch-2.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #5941 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5941/)
          MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with compatible/incompatible changes. Contributed by Junping Du (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612652)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #5941 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5941/ ) MAPREDUCE-5963 . ShuffleHandler DB schema should be versioned with compatible/incompatible changes. Contributed by Junping Du (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1612652 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          Hide
          Jason Lowe added a comment -

          +1 lgtm. Committing this.

          Show
          Jason Lowe added a comment - +1 lgtm. Committing this.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12657123/MAPREDUCE-5963-v2.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4761//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4761//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12657123/MAPREDUCE-5963-v2.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4761//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4761//console This message is automatically generated.
          Junping Du made changes -
          Attachment MAPREDUCE-5963-v2.1.patch [ 12657123 ]
          Hide
          Junping Du added a comment -

          In latest patch, fix the findbug warning.

          Show
          Junping Du added a comment - In latest patch, fix the findbug warning.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12657039/MAPREDUCE-5963-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4759//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4759//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12657039/MAPREDUCE-5963-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4759//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4759//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4759//console This message is automatically generated.
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Junping Du made changes -
          Attachment MAPREDUCE-5963-v2.patch [ 12657039 ]
          Hide
          Junping Du added a comment -

          Update patch to latest trunk and do some fix on visibility of interfaces, checking on old/new schema version, etc.

          Show
          Junping Du added a comment - Update patch to latest trunk and do some fix on visibility of interfaces, checking on old/new schema version, etc.
          Junping Du made changes -
          Link This issue depends upon YARN-2045 [ YARN-2045 ]
          Junping Du made changes -
          Attachment MAPREDUCE-5963.patch [ 12654820 ]
          Hide
          Junping Du added a comment -

          Upload a draft patch. It should need some code in YARN-2045.

          Show
          Junping Du added a comment - Upload a draft patch. It should need some code in YARN-2045 .
          Junping Du made changes -
          Parent MAPREDUCE-4150 [ 12550798 ]
          Issue Type New Feature [ 2 ] Sub-task [ 7 ]
          Junping Du made changes -
          Project Hadoop YARN [ 12313722 ] Hadoop Map/Reduce [ 12310941 ]
          Key YARN-2265 MAPREDUCE-5963
          Affects Version/s 2.4.1 [ 12326698 ]
          Affects Version/s 2.4.1 [ 12326697 ]
          Target Version/s 2.6.0 [ 12327197 ] 2.6.0 [ 12327180 ]
          Component/s nodemanager [ 12319323 ]
          Junping Du made changes -
          Field Original Value New Value
          Parent YARN-666 [ 12647127 ]
          Issue Type Sub-task [ 7 ] New Feature [ 2 ]
          Hide
          Junping Du added a comment -

          Actually, this need to be moved to MAPREDUCE project...

          Show
          Junping Du added a comment - Actually, this need to be moved to MAPREDUCE project...
          Junping Du created issue -

            People

            • Assignee:
              Junping Du
              Reporter:
              Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development