Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4163

CreateFlatHiveTableStep has not yarn app url when hive job running

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • v3.0.0-alpha
    • v3.0.0
    • Job Engine, Web
    • None

    Description

      CreateFlatHiveTableStep has yarn app url on the monitor web page only when job finished, but SparkExecutable has yarn app url when job running.
      this is because of SparkExecutable`s logger has logger listener:

      final PatternedLogger patternedLogger = new PatternedLogger(logger, new PatternedLogger.ILogListener() {
       @Override
       public void onLogEvent(String infoKey, Map<String, String> info) {
       // only care three properties here
       if (ExecutableConstants.SPARK_JOB_ID.equals(infoKey)
       || ExecutableConstants.YARN_APP_ID.equals(infoKey)
       || ExecutableConstants.YARN_APP_URL.equals(infoKey)) {
       getManager().addJobInfo(getId(), info);
       }
       }
       });

      sometimes creating flat hive table hangs, so user wants to have yarn app url when hive job running  like attachment.

      Attachments

        1. flathivetablerunning图.jpg
          48 kB
          chuxiao
        2. KYLIN-4163.master.001.patch
          4 kB
          chuxiao

        Issue Links

          Activity

            People

              Unassigned Unassigned
              xiaoge chuxiao
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: