Uploaded image for project: 'Tajo (Retired)'
  1. Tajo (Retired)
  2. TAJO-1144

INSERT INTO with output rotation may overwrite previous written output file.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Invalid
    • None
    • None
    • Physical Operator
    • None

    Description

      Tajo can write multiple output files according to a given maximum file size. We can this feature 'output file rotation'.

      BTW, INSERT with output file rotation may overwrite the previous written output file. See the source code in StoreTableExec.

      public void openNewFile(int suffixId) throws IOException {
          String prevFile = null;
      
          lastFileName = context.getOutputPath();
          if (suffixId > 0) {
            prevFile = lastFileName.toString();
      
            lastFileName = new Path(lastFileName + "_" + suffixId);
          }
      
          if (plan instanceof InsertNode) {
            InsertNode createTableNode = (InsertNode) plan;
            appender = StorageManager.getStorageManager(context.getConf()).getAppender(meta,
                createTableNode.getTableSchema(), context.getOutputPath());
          } else {
            appender = StorageManager.getStorageManager(context.getConf()).getAppender(meta, outSchema, lastFileName);
          }
      
          appender.enableStats();
          appender.init();
      
          if (suffixId > 0) {
            LOG.info(prevFile + " exceeds " + SessionVars.MAX_OUTPUT_FILE_SIZE.keyname() + " (" + maxPerFileSize + " MB), " +
                "The remain output will be written into " + lastFileName.toString());
          }
        }
      

      Especially, context.getOutputPath() should be lastFileName in the below part.

          if (plan instanceof InsertNode) {
            InsertNode createTableNode = (InsertNode) plan;
            appender = StorageManager.getStorageManager(context.getConf()).getAppender(meta,
                createTableNode.getTableSchema(), context.getOutputPath());
          } else {
            appender = StorageManager.getStorageManager(context.getConf()).getAppender(meta, outSchema, lastFileName);
          }
      

      Attachments

        Activity

          People

            blrunner JaeHwa Jung
            hyunsik Hyunsik Choi
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: