Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3653

After kylin configured with hive data source with beeline, build failed if two jobs for creating flat table submitted at same time

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v2.4.1
    • Fix Version/s: v2.6.0
    • Component/s: Job Engine
    • Labels:
      None

      Description

      The root cause is in org.apache.kylin.common.util.HiveCmdBuilder#build

              case BEELINE:
                  String tmpHqlPath = null;
                  StringBuilder hql = new StringBuilder();
                  try {
                      tmpHqlPath = "/tmp/" + System.currentTimeMillis() + ".hql";
                      for (String statement : statements) {
                          hql.append(statement);
                          hql.append("\n");
                      }
                      String createFileCmd = String.format(CREATE_HQL_TMP_FILE_TEMPLATE, tmpHqlPath, hql);
                      buf.append(createFileCmd);
                      buf.append("\n");
                      buf.append(beelineShell);
                      buf.append(" ");
                      buf.append(beelineParams);
                      buf.append(parseProps());
                      buf.append(" -f ");
                      buf.append(tmpHqlPath);
                      buf.append(";ret_code=$?;rm -f ");
                      buf.append(tmpHqlPath);
                      buf.append(";exit $ret_code");
                  } finally {
                      if (tmpHqlPath != null && logger.isDebugEnabled()) {
                          logger.debug("The SQL to execute in beeline: \n" + hql);
                      }
                  }
                  break;
      

      A temp hql file will be created when creating flat table, the file name is the current time stamp. But if two jobs triggered with same time, there will be only one file, and jobs may be failed

        Attachments

          Activity

            People

            • Assignee:
              nichunen nichunen
              Reporter:
              nichunen nichunen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: