[HIVE-17237] HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: HiveServer2
• Assignee: Misha Dmitriev
• Reporter: Misha Dmitriev
• Created: 03/Aug/17 01:00
• Updated: 22/May/18 23:57
• Resolved: 22/Aug/17 15:41

Description

I've analyzed a heap dump from a production Hive installation using jxray (www.jxray.com) It turns out that there are a lot of duplicate strings in memory, that waste 26.4% of the heap. Most of them come from HashMaps referenced by org.apache.hadoop.hive.metastore.api.Partition.parameters. Below is the relevant section of the jxray report.

Looking at Partition.java, I see that in the past somebody has already added code to intern keys and values in the parameters table when it's first set up. However, when more key-value pairs are added, they are not interned, and that probably explains the reason for all these duplicate strings. Also when a Partition instance is deserialized, no interning of parameters is currently done.

6. DUPLICATE STRINGS

Total strings: 3,273,557  Unique strings: 460,390  Duplicate values: 110,232  Overhead: 3,220,458K (26.4%)
....

===================================================

7. REFERENCE CHAINS FOR DUPLICATE STRINGS

  2,326,150K (19.1%), 597058 dup strings (36386 unique), 597058 dup backing arrays:
39949 of "-1", 39088 of "true", 28959 of "8", 20987 of "1", 18437 of "10", 9583 of "9", 5908 of "269664", 5691 of "174528", 4598 of "133980", 4598 of "BgUGBQgFCAYFCgYIBgUEBgQHBgUGCwYGBwYHBgkKBwYGBggIBwUHBgYGCgUJCQUG ...[length 3560]"
... and 419200 more strings, of which 36376 are unique
Also contains one-char strings: 217 of "6", 147 of "7", 91 of "4", 28 of "5", 28 of "2", 21 of "0"
     <--  {j.u.HashMap}.values <-- org.apache.hadoop.hive.metastore.api.Partition.parameters <--  {j.u.ArrayList} <-- org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success <-- Java Local (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result) [@6e33618d8,@6eedb9a80,@6eedbad68,@6eedbc788] ... and 3 more GC roots
  463,060K (3.8%), 119644 dup strings (34075 unique), 119644 dup backing arrays:
7914 of "true", 7912 of "-1", 6578 of "8", 5606 of "1", 2302 of "10", 1626 of "174528", 1223 of "9", 970 of "171680", 837 of "269664", 657 of "133980"
... and 84009 more strings, of which 34065 are unique
Also contains one-char strings: 42 of "7", 31 of "6", 20 of "4", 8 of "5", 5 of "2", 3 of "0"
     <--  {j.u.HashMap}.values <-- org.apache.hadoop.hive.metastore.api.Partition.parameters <--  {j.u.TreeMap}.values <-- Java Local (j.u.TreeMap) [@6f084afa0,@73aac9e68]
  233,384K (1.9%), 64601 dup strings (27295 unique), 64601 dup backing arrays:
4472 of "true", 4173 of "-1", 3798 of "1", 3591 of "8", 813 of "174528", 684 of "10" ... and 44568 more strings, of which 27285 are unique
Also contains one-char strings: 305 of "7", 301 of "0", 277 of "4", 146 of "6", 29 of "2", 23 of "5", 19 of "9", 2 of "3"
     <--  {j.u.HashMap}.values <-- org.apache.hadoop.hive.metastore.api.Partition.parameters <--  {j.u.ArrayList} <-- Java Local (j.u.ArrayList) [@4f4cfbd10,@536122408,@726616778]
...
  52,916K (0.4%), 597058 dup strings (16 unique), 597058 dup backing arrays:
     <--  {j.u.HashMap}.keys <-- org.apache.hadoop.hive.metastore.api.Partition.parameters <--  {j.u.ArrayList} <-- org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success <-- Java Local (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result) [@6e33618d8,@6eedb9a80,@6eedbad68,@6eedbc788] ... and 3 more GC roots

Issue Links

• is duplicated by: HIVE-16489 HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters
• relates to: HIVE-16489 HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters