Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-4701

Write client-side metrics asynchronously to SYSTEM.LOG

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.14.0, 5.0.0
    • Labels:
      None

      Description

      Rather than inventing a new, different set of client-side metrics to persist, we should just persist our client metrics in the SYSTEM.LOG. The metrics captures all the same information as your QueryLogInfo (and much more), rolls all the information up to a single set of metrics for each Phoenix statement (aggregating/merging parallel scans, etc), and can emits a single log line (which could be written in a single upsert statement). At SFDC, we emit this information into a file system log in a layer above (and use Splunk to produce nifty dashboard for monitoring), but this could easily be emitted directly in Phoenix and go through your asynchronous write path (and then use Phoenix queries to produce the same kind of dashboards). The only piece would be to add the concept of a log level to each metric to enable statically controlling which metrics are output.

      With this approach, the SYSTEM.LOG table could be declared immutable and use our dense storage format with a single byte for column encoding and get a 3-5x perf gain. This would also open the door for users to potentially add secondary indexes on the table. See schema identified in the wip2 patch.

        Attachments

        1. PHOENIX-4701_addendum2.patch
          1 kB
          James R. Taylor
        2. PHOENIX-4701_addendum.patch
          19 kB
          Ankit Singhal
        3. PHOENIX-4701_v2.patch
          112 kB
          Ankit Singhal
        4. PHOENIX-4701_master.patch
          107 kB
          Ankit Singhal
        5. PHOENIX-4701.patch
          102 kB
          Ankit Singhal
        6. PHOENIX-4701_wip3.patch
          102 kB
          Ankit Singhal
        7. PHOENIX-4701_wip2.patch
          22 kB
          James R. Taylor
        8. PHOENIX-4701_wip1.patch
          3 kB
          James R. Taylor

          Activity

            People

            • Assignee:
              ankit@apache.org Ankit Singhal
              Reporter:
              jamestaylor James R. Taylor
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: