Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.2
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      2.7 RM app page is broken by the cherry pick of YARN-3248 on 23/Sep. It broke the work around 2.7 patch of YARN-3544 to let it still use container report. Currently, our cluster's 2.7 RM app page is completely broken due to 500 error, which is caused by when user UGI is null, completed app can not retrieve its container report, and in that code path, it doesn't catch ContainerNotFoundException, but throw the exception, therefore cause the 500 error.
      Running app is also broken because of the way it construct containerID by

       "ContainerId.newContainerId(
                    appAttemptReport.getApplicationAttemptId(), 1)" 
      

      ,
      which will not include epoch number in ID, so it will also get ContainerNotFoundException and throw 500 error.
      Propose to use the branch-2 version of YARN-3544, instead of the work around 2.7 patch because branch 2 patch on 2.7 is no longer blocked.

      1. YARN-4281-branch-2.7-3.patch
        11 kB
        Chang Li
      2. YARN-4281-branch-2.7.2.patch
        11 kB
        Chang Li
      3. YARN-4281-branch-2.7.patch
        11 kB
        Chang Li
      4. YARN-4281.2.7.modify.patch
        11 kB
        Chang Li

        Issue Links

          Activity

          Hide
          djp Junping Du added a comment -

          Ok. Thanks for the confirmation, Jason!

          Show
          djp Junping Du added a comment - Ok. Thanks for the confirmation, Jason!
          Hide
          jlowe Jason Lowe added a comment -

          Using 2.6.4, I tried visiting the RM app page of both a live app and a completed application and both worked just fine. So it doesn't appear that this is needed for branch-2.6.

          Show
          jlowe Jason Lowe added a comment - Using 2.6.4, I tried visiting the RM app page of both a live app and a completed application and both worked just fine. So it doesn't appear that this is needed for branch-2.6.
          Hide
          djp Junping Du added a comment -

          Hi Chang Li and Jason Lowe, does branch-2.6 has the same issue? If so, we may consider to backport the fix to branch-2.6 also?

          Show
          djp Junping Du added a comment - Hi Chang Li and Jason Lowe , does branch-2.6 has the same issue? If so, we may consider to backport the fix to branch-2.6 also?
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Chang! I committed this to branch-2.7.

          Show
          jlowe Jason Lowe added a comment - Thanks, Chang! I committed this to branch-2.7.
          Hide
          jlowe Jason Lowe added a comment -

          +1, committing this.

          Show
          jlowe Jason Lowe added a comment - +1, committing this.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 14m 52s Findbugs (version ) appears to be broken on branch-2.7.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          -1 patch 0m 7s The patch command could not apply the patch.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767684/YARN-4281-branch-2.7-3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision branch-2.7 / 3f3829e
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9503/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 14m 52s Findbugs (version ) appears to be broken on branch-2.7. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch 0m 7s The patch command could not apply the patch. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767684/YARN-4281-branch-2.7-3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision branch-2.7 / 3f3829e Console output https://builds.apache.org/job/PreCommit-YARN-Build/9503/console This message was automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          Thanks Jason Lowe for point out the problem! Fixed that in 2.7-3 patch

          Show
          lichangleo Chang Li added a comment - Thanks Jason Lowe for point out the problem! Fixed that in 2.7-3 patch
          Hide
          jlowe Jason Lowe added a comment -

          There's still extra whitespace added by this deletion (note it doesn't delete one of the lines around the block deleted, resulting in the extra whitespace):

          --- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java
          +++ hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java
          @@ -20,21 +20,14 @@
           
           import static org.apache.hadoop.yarn.webapp.view.JQueryUI._INFO_WRAP;
           
          -import java.security.PrivilegedExceptionAction;
          -import java.util.Collection;
          -import java.util.Set;
           
           import org.apache.commons.lang.StringEscapeUtils;
           import org.apache.commons.logging.Log;
          

          Otherwise looks good.

          Show
          jlowe Jason Lowe added a comment - There's still extra whitespace added by this deletion (note it doesn't delete one of the lines around the block deleted, resulting in the extra whitespace): --- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java +++ hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java @@ -20,21 +20,14 @@ import static org.apache.hadoop.yarn.webapp.view.JQueryUI._INFO_WRAP; -import java.security.PrivilegedExceptionAction; -import java.util.Collection; -import java.util.Set; import org.apache.commons.lang.StringEscapeUtils; import org.apache.commons.logging.Log; Otherwise looks good.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767669/YARN-4281-branch-2.7.2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6c8b6f3
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9501/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767669/YARN-4281-branch-2.7.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6c8b6f3 Console output https://builds.apache.org/job/PreCommit-YARN-Build/9501/console This message was automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          Thanks Jason Lowe for review! updated my patch and addressed your concerns in 2.7.2.patch.

          Show
          lichangleo Chang Li added a comment - Thanks Jason Lowe for review! updated my patch and addressed your concerns in 2.7.2.patch.
          Hide
          jlowe Jason Lowe added a comment -

          The intent is to bring this code better inline with what's happening on branch-2, now that YARN-3248 was committed to branch-2.7 (and broke the RM UI as a result). As such I have a few whitespace nits on the patch which when fixed should help remove some potentials for merge conflicts when cherry-picking other changes from branch-2. For reference I was diffing the patched version of AppBlock.java and RMAppBlock.java against commit c9ee316045b83b18cb068aa4de739a1f4b50f02a which is where YARN-3544 went into branch-2. Any diffs that don't belong to another commit delta between branch-2 and branch-2.7 are what's flagged below.

          For this patch hunk, the original YARN-3248 did not have the appReport = null line, just the appReport declaration. Also there's an additional whitespace line that was added by this patch which should not be there:

          AppBlock.java
          @@ -87,8 +86,9 @@ protected void render(Block html) {
                 return;
               }
           
          -    callerUGI = getCallerUGI();
          -    ApplicationReport appReport;
          +
          +    UserGroupInformation callerUGI = getCallerUGI();
          +    ApplicationReport appReport = null;
               try {
                 final GetApplicationReportRequest request =
                     GetApplicationReportRequest.newInstance(appID);
          

          Similarly, the patch leaves an additional whitespace line where the import code was deleted and adds a new whitespace line where it was added back in:

          RMAppBlock.java
          @@ -20,21 +20,14 @@
           
           import static org.apache.hadoop.yarn.webapp.view.JQueryUI._INFO_WRAP;
           
          -import java.security.PrivilegedExceptionAction;
          -import java.util.Collection;
          -import java.util.Set;
           
           import org.apache.commons.lang.StringEscapeUtils;
           import org.apache.commons.logging.Log;
          
          [...]
          
           import com.google.inject.Inject;
           
          +
          +import java.util.Collection;
          +import java.util.Set;
          +
           public class RMAppBlock extends AppBlock{
           
             private static final Log LOG = LogFactory.getLog(RMAppBlock.class);
          
          
          Show
          jlowe Jason Lowe added a comment - The intent is to bring this code better inline with what's happening on branch-2, now that YARN-3248 was committed to branch-2.7 (and broke the RM UI as a result). As such I have a few whitespace nits on the patch which when fixed should help remove some potentials for merge conflicts when cherry-picking other changes from branch-2. For reference I was diffing the patched version of AppBlock.java and RMAppBlock.java against commit c9ee316045b83b18cb068aa4de739a1f4b50f02a which is where YARN-3544 went into branch-2. Any diffs that don't belong to another commit delta between branch-2 and branch-2.7 are what's flagged below. For this patch hunk, the original YARN-3248 did not have the appReport = null line, just the appReport declaration. Also there's an additional whitespace line that was added by this patch which should not be there: AppBlock.java @@ -87,8 +86,9 @@ protected void render(Block html) { return; } - callerUGI = getCallerUGI(); - ApplicationReport appReport; + + UserGroupInformation callerUGI = getCallerUGI(); + ApplicationReport appReport = null; try { final GetApplicationReportRequest request = GetApplicationReportRequest.newInstance(appID); Similarly, the patch leaves an additional whitespace line where the import code was deleted and adds a new whitespace line where it was added back in: RMAppBlock.java @@ -20,21 +20,14 @@ import static org.apache.hadoop.yarn.webapp.view.JQueryUI._INFO_WRAP; -import java.security.PrivilegedExceptionAction; -import java.util.Collection; -import java.util.Set; import org.apache.commons.lang.StringEscapeUtils; import org.apache.commons.logging.Log; [...] import com.google.inject.Inject; + +import java.util.Collection; +import java.util.Set; + public class RMAppBlock extends AppBlock{ private static final Log LOG = LogFactory.getLog(RMAppBlock.class);
          Hide
          jlowe Jason Lowe added a comment -

          Marking this a blocker for 2.7.2, just like YARN-3544 was a blocker originally.

          Show
          jlowe Jason Lowe added a comment - Marking this a blocker for 2.7.2, just like YARN-3544 was a blocker originally.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767639/YARN-4281-branch-2.7.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision branch-2 / 1a4bd5b
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9494/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767639/YARN-4281-branch-2.7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision branch-2 / 1a4bd5b Console output https://builds.apache.org/job/PreCommit-YARN-Build/9494/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767633/YARN-4281.2.7.modify.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6381ddc
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9493/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767633/YARN-4281.2.7.modify.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6381ddc Console output https://builds.apache.org/job/PreCommit-YARN-Build/9493/console This message was automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          uploaded 2.7.modify patch, which revert the YARN-3544 2.7 patch change, and apply the YARN-3544 trunk patch and solve minor merge conflicts

          Show
          lichangleo Chang Li added a comment - uploaded 2.7.modify patch, which revert the YARN-3544 2.7 patch change, and apply the YARN-3544 trunk patch and solve minor merge conflicts

            People

            • Assignee:
              lichangleo Chang Li
              Reporter:
              lichangleo Chang Li
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development