Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-931

Output file can be punctuated depending on the file size.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Physical Operator
    • Labels:
      None

      Description

      There are some file formats (e.g., Parquet) which are not splittable. They can usually span multiple HDFS blocks if one file is very large. It causes remote HDFS access and limits the parallel degree, resulting in significant performance degradation.

      We can solve this problem if StoreTableExec or

      {Col|SortBased}

      PartitionStoreExec can punctuate the final output file according to the written size.

      In addition, we need to support a session variable to determine the per file size of final output files. So, TAJO-928 blocks this issue.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hyunsik Hyunsik Choi
                Reporter:
                hyunsik Hyunsik Choi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: