[TAJO-931] Output file can be punctuated depending on the file size.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Component/s: Physical Operator
• Assignee: Hyunsik Choi
• Reporter: Hyunsik Choi
• Created: 10/Jul/14 11:31
• Updated: 11/Oct/14 21:41
• Resolved: 11/Oct/14 21:41

Description

There are some file formats (e.g., Parquet) which are not splittable. They can usually span multiple HDFS blocks if one file is very large. It causes remote HDFS access and limits the parallel degree, resulting in significant performance degradation.

We can solve this problem if StoreTableExec or

{Col|SortBased}

PartitionStoreExec can punctuate the final output file according to the written size.

In addition, we need to support a session variable to determine the per file size of final output files. So, TAJO-928 blocks this issue.

Issue Links

• blocks: TAJO-935 (ColPartition)StoreTableExec should use per-file block size config for file creations.
• is blocked by: TAJO-928 Session variables should override query configs in TajoConf.
• is blocked by: TAJO-933 Fork some classes of Parquet as builtin third-party classes