Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-1596

Turning tracing on causes region servers to crash

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 4.3.0, 4.2.3
    • 4.3.0, 4.2.3
    • None
    • None

    Description

      After setting trace collection frequency to always by setting the following in hbase-site.xml, I noticed that it created way too many traces in the trace table.

      <property>
      <name>phoenix.trace.frequency</name>
      <value>always</value>
      </property>

      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      1283

      ------------------------------------------
      1 row selected (1.104 seconds)
      0: jdbc:phoenix:localhost> select count from system.tracing_stats;
      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      4051

      ------------------------------------------
      1 row selected (1.058 seconds)
      0: jdbc:phoenix:localhost> select count from system.tracing_stats;
      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      10668

      ------------------------------------------
      1 row selected (1.105 seconds)
      0: jdbc:phoenix:localhost> select count from system.tracing_stats;
      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      11361

      ------------------------------------------
      1 row selected (1.046 seconds)
      0: jdbc:phoenix:localhost> select count from system.tracing_stats;
      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      193119

      ------------------------------------------

      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      1283

      ------------------------------------------
      1 row selected (1.104 seconds)
      0: jdbc:phoenix:localhost> select count from system.tracing_stats;
      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      4051

      ------------------------------------------
      1 row selected (1.058 seconds)
      0: jdbc:phoenix:localhost> select count from system.tracing_stats;
      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      10668

      ------------------------------------------
      1 row selected (1.105 seconds)
      0: jdbc:phoenix:localhost> select count from system.tracing_stats;
      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      11361

      ------------------------------------------
      1 row selected (1.046 seconds)
      0: jdbc:phoenix:localhost> select count from system.tracing_stats;
      ------------------------------------------

      COUNT(1)

      ------------------------------------------

      193119

      ------------------------------------------
      1 row selected (6.737 seconds)

      0: jdbc:phoenix:localhost> select count from system.tracing_stats;
      15/01/19 17:26:57 WARN client.HConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it

      Even though the only query that was being executed was the select count to get the number of rows in the trace table, it ended up creating way too many traces than I had expected.

      On my mac, it in fact ended up killing the local hbase cluster altogether!

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            samarthjain Samarth Jain
            samarthjain Samarth Jain
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment