Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-423

Add document store to Samza

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      At the moment Samza only has a key-value storage engine (LevelDB/RocksDB/in-memory), but Samza's state abstraction is designed to be more general than that. In particular, we've discussed integrating Lucene in order to support full-text indexes.

      There are two modes of using Lucene which would make sense in a stream processing system:

      • Treat incoming messages as documents to be added to an index. This would be akin to the key-value storage model, but with much richer indexing capabilities. It would enable joining streams not just on a single key, but on more complex criteria (arbitrary boolean expressions for joins) and stuff like deduplicating similar documents (which you may want e.g. if implementing a web crawler).
      • Match incoming messages against a set of queries, where the queries are more or less fixed (perhaps the queries are updated via another input stream). In this case, the message isn't added to a persistent index, but it's only analysed and matched against a query as it flows through the stream processor. This is useful for monitoring an activity stream for events of interest ("notify me whenever a news article mentions my company name"). It's perhaps comparable to ElasticSearch's percolator.

      I was chatting to romseygeek yesterday, a committer on Lucene/Solr. We have a vague plan to hack on a proof of concept to see what an integration of Lucene and Samza could look like. This is a placeholder ticket for collecting any stuff related to that experiment.

      Attachments

        1. DESIGN-SAMZA-423-0.pdf
          452 kB
          Robert Žuljević
        2. DESIGN-SAMZA-423-1.pdf
          565 kB
          Robert Žuljević
        3. SAMZA-423.0.patch
          60 kB
          Robert Žuljević

        Issue Links

          Activity

            People

              rzuljevic Robert Žuljević
              martinkl Martin Kleppmann
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: