Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-15132

Add temporal graph query to the nodes Streaming Expression

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Resolved
    • None
    • 8.9
    • streaming expressions
    • None

    Description

      The nodes Streaming Expression performs a breadth first graph traversal. This ticket will add a window parameter to allow the nodes expression to traverse the graph within a window of time.

      To take advantage of this feature you must index the content with a String field which is an ISO timestamp truncated at ten seconds. Then the window parameter can be applied to walk the graph within a window prior to a specific ten second window and perform aggregations.

      The main use cases for this feature are event correlation and root cause analysis. This is useful in many different fields.

      Here is an example using Solr logs to answer the following question:

      What types of log events occur most frequently in the 30 second window prior to 10 second windows with the most slow queries:

      nodes(logs,
            facet(logs, q="qtime_s:[5000 TO *]", buckets="time_ten_seconds", rows="25"),
            walk="time_ten_seconds->time_ten_seconds",
            window="3",
            gather="type_s",
            count(*))
      

      This ticket is phase 1. Phase 2 will auto-detect different ISO Timestamp truncations so that increments of one second, one minute, one day etc... can also be traversed using the same query syntax. There will be a follow-on ticket for that after this ticket is completed. This will create a more general purpose time graph.

      Attachments

        1. SOLR-15132.patch
          5 kB
          Joel Bernstein
        2. SOLR-15132.patch
          5 kB
          Joel Bernstein
        3. SOLR-15132.patch
          12 kB
          Joel Bernstein
        4. SOLR-15132.patch
          14 kB
          Joel Bernstein

        Activity

          houston Houston Putman added a comment -

          If this is just looking at the last n buckets when window=n, then why does the bucket field need to be an ISO timestamp in a string field?

          houston Houston Putman added a comment - If this is just looking at the last n buckets when window=n , then why does the bucket field need to be an ISO timestamp in a string field?
          jbernste Joel Bernstein added a comment - - edited

          We need to subtract the 10 seconds window each time and create the previous timestamps. The walk is a join. So instead of joining on just the matching ISO timestamps, it will add the prior window timestamps as well. So we're expanding the window backwards for each matching timestamp.

          jbernste Joel Bernstein added a comment - - edited We need to subtract the 10 seconds window each time and create the previous timestamps. The walk is a join. So instead of joining on just the matching ISO timestamps, it will add the prior window timestamps as well. So we're expanding the window backwards for each matching timestamp.
          houston Houston Putman added a comment -

          my question is, does "nodes()" have any idea what "time_ten_seconds" is, or is it just walking the previous 3 buckets that are returned from the facet? is the window not just a number of buckets at the end of the day?

          houston Houston Putman added a comment - my question is, does "nodes()" have any idea what "time_ten_seconds" is, or is it just walking the previous 3 buckets that are returned from the facet? is the window not just a number of buckets at the end of the day?
          jbernste Joel Bernstein added a comment -

          Nodes will take each matching timestamp, convert to a Date, subtract 10 seconds, convert back to a timestamp, and include in the join. Atleast that's the current plan.

          jbernste Joel Bernstein added a comment - Nodes will take each matching timestamp, convert to a Date, subtract 10 seconds, convert back to a timestamp, and include in the join. Atleast that's the current plan.
          jbernste Joel Bernstein added a comment - - edited

          So it doesn't care about the previous timestamps returned by the facet.

          jbernste Joel Bernstein added a comment - - edited So it doesn't care about the previous timestamps returned by the facet.
          jbernste Joel Bernstein added a comment -

          You can think of this as a time window join.

          jbernste Joel Bernstein added a comment - You can think of this as a time window join.
          jbernste Joel Bernstein added a comment -

          First patch, tests to follow.

          jbernste Joel Bernstein added a comment - First patch, tests to follow.

          Commit 4a42ecd9364efea9867976295cd0342c96875786 in lucene-solr's branch refs/heads/master from Joel Bernstein
          [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4a42ecd ]

          SOLR-15132: Add temporal graph query to the nodes Streaming Expression

          jira-bot ASF subversion and git services added a comment - Commit 4a42ecd9364efea9867976295cd0342c96875786 in lucene-solr's branch refs/heads/master from Joel Bernstein [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4a42ecd ] SOLR-15132 : Add temporal graph query to the nodes Streaming Expression

          Commit aadaad4a0909a1fc66746d56d86853417ff395ef in lucene-solr's branch refs/heads/branch_8x from Joel Bernstein
          [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=aadaad4 ]

          SOLR-15132: Add temporal graph query to the nodes Streaming Expression

          jira-bot ASF subversion and git services added a comment - Commit aadaad4a0909a1fc66746d56d86853417ff395ef in lucene-solr's branch refs/heads/branch_8x from Joel Bernstein [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=aadaad4 ] SOLR-15132 : Add temporal graph query to the nodes Streaming Expression

          Commit 53deb6f7351ae9cbc5a8ff9138643b01ccffc86a in lucene-solr's branch refs/heads/master from Joel Bernstein
          [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=53deb6f ]

          SOLR-15132: Update CHANGES.txt

          jira-bot ASF subversion and git services added a comment - Commit 53deb6f7351ae9cbc5a8ff9138643b01ccffc86a in lucene-solr's branch refs/heads/master from Joel Bernstein [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=53deb6f ] SOLR-15132 : Update CHANGES.txt

          Commit a1d77589cd85cd6a22aa45f049146107bee736eb in lucene-solr's branch refs/heads/branch_8x from Joel Bernstein
          [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a1d7758 ]

          SOLR-15132: Update CHANGES.txt

          jira-bot ASF subversion and git services added a comment - Commit a1d77589cd85cd6a22aa45f049146107bee736eb in lucene-solr's branch refs/heads/branch_8x from Joel Bernstein [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a1d7758 ] SOLR-15132 : Update CHANGES.txt

          Closing after the 8.9.0 release

          mayya Mayya Sharipova added a comment - Closing after the 8.9.0 release

          People

            Unassigned Unassigned
            jbernste Joel Bernstein
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: