[PHOENIX-1596] Turning tracing on causes region servers to crash

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.3.0, 4.2.3
• Fix Version/s: 4.3.0, 4.2.3
• Assignee: Samarth Jain
• Reporter: Samarth Jain
• Created: 20/Jan/15 01:36
• Updated: 21/Nov/15 02:15
• Resolved: 30/Jan/15 23:19

Description

After setting trace collection frequency to always by setting the following in hbase-site.xml, I noticed that it created way too many traces in the trace table.

<property>
<name>phoenix.trace.frequency</name>
<value>always</value>
</property>

------------------------------------------

COUNT(1)

------------------------------------------

1283

------------------------------------------
1 row selected (1.104 seconds)
0: jdbc:phoenix:localhost> select count from system.tracing_stats;
------------------------------------------

COUNT(1)

------------------------------------------

4051

------------------------------------------
1 row selected (1.058 seconds)
0: jdbc:phoenix:localhost> select count from system.tracing_stats;
------------------------------------------

COUNT(1)

------------------------------------------

10668

------------------------------------------
1 row selected (1.105 seconds)
0: jdbc:phoenix:localhost> select count from system.tracing_stats;
------------------------------------------

COUNT(1)

------------------------------------------

11361

------------------------------------------
1 row selected (1.046 seconds)
0: jdbc:phoenix:localhost> select count from system.tracing_stats;
------------------------------------------

COUNT(1)

------------------------------------------

193119

------------------------------------------

------------------------------------------

COUNT(1)

------------------------------------------

1283

------------------------------------------
1 row selected (1.104 seconds)
0: jdbc:phoenix:localhost> select count from system.tracing_stats;
------------------------------------------

COUNT(1)

------------------------------------------

4051

------------------------------------------
1 row selected (1.058 seconds)
0: jdbc:phoenix:localhost> select count from system.tracing_stats;
------------------------------------------

COUNT(1)

------------------------------------------

10668

------------------------------------------
1 row selected (1.105 seconds)
0: jdbc:phoenix:localhost> select count from system.tracing_stats;
------------------------------------------

COUNT(1)

------------------------------------------

11361

------------------------------------------
1 row selected (1.046 seconds)
0: jdbc:phoenix:localhost> select count from system.tracing_stats;
------------------------------------------

COUNT(1)

------------------------------------------

193119

------------------------------------------
1 row selected (6.737 seconds)

0: jdbc:phoenix:localhost> select count from system.tracing_stats;
15/01/19 17:26:57 WARN client.HConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it

Even though the only query that was being executed was the select count to get the number of rows in the trace table, it ended up creating way too many traces than I had expected.

On my mac, it in fact ended up killing the local hbase cluster altogether!

Issue Links

• is related to: HTRACE-92 Thread local storing the currentSpan is never cleared