[HDFS-13752] fs.Path stores file path in java.net.URI causes big memory waste

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.6
• Component/s: fs

Environment

Hive 2.1.1 and hadoop 2.7.6

• Assignee: Barnabas Maidics
• Reporter: Barnabas Maidics
• Created: 20/Jul/18 09:23
• Updated: 02/Oct/19 17:12

Description

I was looking at HiveServer2 memory usage, and a big percentage of this was because of org.apache.hadoop.fs.Path, where you store file paths in a java.net.URI object. The URI implementation stores the same string in 3 different objects (see the attached image). In Hive when there are many partitions this cause a big memory usage. In my particular case 42% of memory was used by java.net.URI so it could be reduced to 14%. 

I wonder if the community is open to replace it with a more memory efficient implementation and what other things should be considered here? It can be a huge memory improvement for Hadoop and for Hive as well.