-
Type:
Improvement
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: 1.2.0
-
Fix Version/s: 1.3.0
-
Component/s: Statistics
-
Labels:None
Currently, stats files are created in S3. Later as a part of FSStatsAggregator, it reads this file and populates MS again.
2016-09-23 05:57:46,772 INFO [main]: fs.FSStatsPublisher (FSStatsPublisher.java:init(49)) - created : s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001 2016-09-23 05:57:46,773 DEBUG [main]: fs.FSStatsAggregator (FSStatsAggregator.java:connect(53)) - About to read stats from : s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
Instead of this, stats can be written directly on to HDFS and read locally instead of S3, which would help in reducing couple of calls to S3.
- is related to
-
HIVE-13925 ETL optimizations
-
- Open
-