Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4163

CreateFlatHiveTableStep has not yarn app url when hive job running

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: v3.0.0-alpha
    • Fix Version/s: v3.0.0
    • Component/s: Job Engine, Web
    • Labels:
      None

      Description

      CreateFlatHiveTableStep has yarn app url on the monitor web page only when job finished, but SparkExecutable has yarn app url when job running.
      this is because of SparkExecutable`s logger has logger listener:

      final PatternedLogger patternedLogger = new PatternedLogger(logger, new PatternedLogger.ILogListener() {
       @Override
       public void onLogEvent(String infoKey, Map<String, String> info) {
       // only care three properties here
       if (ExecutableConstants.SPARK_JOB_ID.equals(infoKey)
       || ExecutableConstants.YARN_APP_ID.equals(infoKey)
       || ExecutableConstants.YARN_APP_URL.equals(infoKey)) {
       getManager().addJobInfo(getId(), info);
       }
       }
       });

      sometimes creating flat hive table hangs, so user wants to have yarn app url when hive job running  like attachment.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              xiaoge chuxiao

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment