[KYLIN-4026] Avoid too many file append operations in HiveProducer of hive metrics reporter

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v3.0.0-alpha2
• Assignee: Unassigned
• Reporter: Shaohui Liu
• Created: 31/May/19 11:29
• Updated: 06/Jul/20 08:37
• Resolved: 17/Jul/19 07:11

Description

Currently  for each write in HiveProducer, there will be a hdfs append operation, which is heavy for HDFS.

A improvement is to keep a FSDataOutputStream in  HiveProducer and write data to it continuously

 

 

Issue Links

• Blocked: KYLIN-4059 metrics write to hive error
• is duplicated by: KYLIN-4605 HiveProducer write metrics less when one host running some kylin servers
• is related to: KYLIN-4573 Add option to indicate whether to close file for every append for Hive Producer
• links to: GitHub Pull Request #671