Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13621

Visual log parsing and data loading with Streaming Expressions

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • streaming expressions
    • None

    Description

      Streaming Expressions and Math Expressions are now mature on the query side. This includes the ability to query, transform, analyze and visualize data. 

      It's now time to build the data loading and log parsing capabilities to apply the full suite of mathematics and visualizations over log data and CSV files.

      The design is to have stream sources that read from a file system and stream decorators that parse different file and log formats. The data can be further transformed and joined with other data by stream decorators and sent to any Solr Cloud collection with the update Stream.

      This design also allows Streaming Expressions to perform regex filtering, aggregations, statistical analysis and visualization directly over CSV files and log files before the data is loaded to Solr. Because of Streaming Expressions built in paralyzation capabilities this allows Solr Cloud to behave like a massively parallel grep engine. 

      It also allows users to visualize data using Apache Zeppelin as part of loading process, to make it easier to understand the data before it's loaded into an index.

      This ticket will track the sub-tickets for the different log formats that will be supported.

      Attachments

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. SOLR-13623 Add SyslogStream Streaming Expression

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. SOLR-13624 Add CommonLogStream Streaming Expression

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. SOLR-13622 Add FileStream Streaming Expression

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SOLR-13625 Add CsvStream, TsvStream Streaming Expressions and supporting Stream Evaluators

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SOLR-13644 Add dateFormat Stream Evaluator

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. SOLR-13667 Add upper, lower, trim and split Stream Evaluators

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SOLR-13635 Add SolrLogStream Streaming Expression

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SOLR-13636 Add CommonEventFormatStream (CEF) Streaming Expression

          • Major - Major loss of function.
          • Open

          Activity

            People

              jbernste Joel Bernstein
              jbernste Joel Bernstein
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: