[ACCUMULO-2905] Seeing short hold times and no MinC's

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.5.1, 1.6.0
• Fix Version/s: 1.5.2, 1.6.1, 1.7.0
• Component/s: tserver

Environment

continuous ingest test

• Assignee: Eric C. Newton
• Reporter: Eric C. Newton
• Created: 13/Jun/14 19:14
• Updated: 25/Oct/14 06:28
• Resolved: 07/Aug/14 18:39

Description

Started continuous ingest on a single node cluster to investigate oscillating ingest rates. Set the split size to 200M, and fired up 3 ingesters. I increase the major/minor compactors to 24/12 respectively.
The native map was set to 4G.

When the table got to 256 tablets, ingest was constantly showing hold times. MinC would run, but memory would constantly fill before new MinCs would start. Here's some output from the log:

$ tail -F ../../../logs/tserver_host.debug.log  | egrep 'Up|Got|held|MinC \['
2014-06-13 14:51:56,553 [tablet.DatafileManager] TABLET_HIST: 2;5d104;5cf07 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003kc/F0000qtk.rf
2014-06-13 14:51:56,695 [tablet.DatafileManager] TABLET_HIST: 2;11cb1;11ab MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eww/F0000qtl.rf
2014-06-13 14:51:56,861 [tablet.DatafileManager] TABLET_HIST: 2;0e4a3;0e2a3 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ery/F0000qtm.rf
2014-06-13 14:51:56,927 [tablet.DatafileManager] TABLET_HIST: 2;44907;44504 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008q9/F0000qtn.rf
2014-06-13 14:51:57,019 [tablet.DatafileManager] TABLET_HIST: 2;4a0fa;49cfd MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00001lg/F0000qto.rf
2014-06-13 14:51:57,535 [tablet.DatafileManager] TABLET_HIST: 2;4c0ff;4bd03 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000ut/F0000qtq.rf
2014-06-13 14:51:58,443 [tablet.DatafileManager] TABLET_HIST: 2;770cf;76ed2f MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lv/F0000qts.rf
2014-06-13 14:51:58,448 [tserver.TabletServerResourceManager] DEBUG: Commits held for   2.18 secs
2014-06-13 14:51:58,449 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 225 in 1.669s, at=[0 0 0.00 1] ft=1.668s(pt=1.667s lt=0.001s ct=0.000s)
2014-06-13 14:51:58,790 [tablet.DatafileManager] TABLET_HIST: 2;4ad04b;4a902 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f05/F0000qtt.rf
2014-06-13 14:51:59,113 [tablet.DatafileManager] TABLET_HIST: 2;23af3;238f4 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei6/F0000qtw.rf
2014-06-13 14:52:00,545 [tablet.DatafileManager] TABLET_HIST: 2;4d906e;4d508 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008s0/F0000qtx.rf
2014-06-13 14:52:00,599 [tablet.DatafileManager] TABLET_HIST: 2;1bcb4f;1b8b6 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f9v/F0000qty.rf
2014-06-13 14:52:00,640 [tablet.DatafileManager] TABLET_HIST: 2;1d8be;1d4c1 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008j7/F0000qtz.rf
2014-06-13 14:52:00,648 [tablet.DatafileManager] TABLET_HIST: 2;40ce3;408de MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etz/F0000qu0.rf
2014-06-13 14:52:01,040 [tablet.DatafileManager] TABLET_HIST: 2;2c13;2bf32 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000uc/F0000qu5.rf
2014-06-13 14:52:01,939 [tablet.DatafileManager] TABLET_HIST: 2;5fedd;5fce3 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fir/F0000qu7.rf
2014-06-13 14:52:02,063 [tablet.DatafileManager] TABLET_HIST: 2;26d0a;26907 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eum/F0000qu8.rf
2014-06-13 14:52:02,067 [tserver.TabletServerResourceManager] DEBUG: Commits held for   3.19 secs
2014-06-13 14:52:02,069 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 288 in 2.617s, at=[0 0 0.00 1] ft=2.616s(pt=2.614s lt=0.001s ct=0.001s)
2014-06-13 14:52:02,604 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58940 279,538 in 9.656s, at=[0 1 0.00 899] ft=9.274s(pt=7.516s lt=1.099s ct=0.659s)
2014-06-13 14:52:02,700 [tablet.DatafileManager] TABLET_HIST: 2;758e6;756e8 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008py/F0000qub.rf
2014-06-13 14:52:02,765 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58937 279,538 in 6.735s, at=[0 1 0.00 899] ft=6.446s(pt=4.446s lt=1.359s ct=0.641s)
2014-06-13 14:52:02,821 [tablet.DatafileManager] TABLET_HIST: 2;53b18;53918 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000kk7/F0000quc.rf
2014-06-13 14:52:03,503 [tablet.DatafileManager] TABLET_HIST: 2;116af;114af MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jnw/F0000qud.rf
2014-06-13 14:52:03,554 [tablet.DatafileManager] TABLET_HIST: 2;76ed2f;76cd7 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ju3/F0000que.rf
2014-06-13 14:52:04,197 [tablet.DatafileManager] TABLET_HIST: 2;7da4e;7d851 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000igh/F0000quf.rf
2014-06-13 14:52:04,603 [tablet.DatafileManager] TABLET_HIST: 2;38d03;388fe MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000evr/F0000quh.rf
2014-06-13 14:52:05,119 [tablet.DatafileManager] TABLET_HIST: 2;156d1f;154d1e MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jqx/F0000quj.rf
2014-06-13 14:52:05,577 [tablet.DatafileManager] TABLET_HIST: 2;3fada;3f8db MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pre/F0000qul.rf
2014-06-13 14:52:05,582 [tserver.TabletServerResourceManager] DEBUG: Commits held for   3.29 secs
2014-06-13 14:52:05,583 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 252 in 2.512s, at=[0 0 0.00 1] ft=2.511s(pt=2.510s lt=0.001s ct=0.000s)
2014-06-13 14:52:06,206 [tablet.DatafileManager] TABLET_HIST: 2;4550e;4510b MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ex0/F0000qum.rf
2014-06-13 14:52:06,360 [tablet.DatafileManager] TABLET_HIST: 2;7fe06;7fc0b MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000hyp/F0000qun.rf
2014-06-13 14:52:06,642 [tablet.DatafileManager] TABLET_HIST: 2;238f4;236f4 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008i1/F0000quq.rf
2014-06-13 14:52:08,626 [tablet.DatafileManager] TABLET_HIST: 2;5931a;59119 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fy8/F0000qus.rf
2014-06-13 14:52:08,682 [tablet.DatafileManager] TABLET_HIST: 2;4a902;4a5 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008kd/F0000quu.rf
2014-06-13 14:52:08,740 [tablet.DatafileManager] TABLET_HIST: 2;1dcbe;1d8be MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etq/F0000quv.rf
2014-06-13 14:52:08,748 [tablet.DatafileManager] TABLET_HIST: 2;23ef2;23cf2 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei9/F0000qut.rf
2014-06-13 14:52:09,215 [tablet.DatafileManager] TABLET_HIST: 2;0fa97;0f899 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fgk/F0000quy.rf
2014-06-13 14:52:09,623 [tablet.DatafileManager] TABLET_HIST: 2;0101f;00e1df MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003ka/F0000qv0.rf
2014-06-13 14:52:09,626 [tserver.TabletServerResourceManager] DEBUG: Commits held for   3.81 secs
2014-06-13 14:52:09,628 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 250 in 3.043s, at=[0 0 0.00 1] ft=3.042s(pt=3.041s lt=0.000s ct=0.001s)
2014-06-13 14:52:10,256 [tablet.DatafileManager] TABLET_HIST: 2;29d21;2991f MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f6f/F0000qv2.rf
2014-06-13 14:52:10,388 [tablet.DatafileManager] TABLET_HIST: 2;6f6f3;6f4f4 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pr5/F0000qv4.rf
2014-06-13 14:52:10,879 [tablet.DatafileManager] TABLET_HIST: 2;16cc5;168c8 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb5/F0000qv5.rf
2014-06-13 14:52:11,379 [tablet.DatafileManager] TABLET_HIST: 2;4b104;4ad04b MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lg/F0000qv7.rf
2014-06-13 14:52:11,879 [tablet.DatafileManager] TABLET_HIST: 2;6f8f;6f6f3 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008mo/F0000qv9.rf
2014-06-13 14:52:12,395 [tablet.DatafileManager] TABLET_HIST: 2;3f4e1;3f2e3e MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb1/F0000qva.rf

Perhaps the server should be more aggressive with MinC's to get better ingest performance?

Issue Links

• relates to: ACCUMULO-3260 TreeMap usage in LargestFirstMemoryManager could preclude certain tablets
• links to: ACCUMULO-2905.patch01
• links to: patch #, stands alone