Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9240

Support parallel ETL with the topic expression

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Resolved
    • None
    • 6.2
    • None
    • None

    Description

      It would be useful for SolrCloud to support large scale Extract, Transform and Load work loads with streaming expressions. Instead of using MapReduce for ETL, the topic expression can be used which allows SolrCloud to be treated like a distributed message queue filled with data to be processed. The topic expression works in batches and supports retrieval of stored fields, so large scale text ETL will work well with this approach.

      This ticket makes two changes to the topic() expression that makes parallel ETL possible:

      1) Changes the topic expression so it can operate in parallel.
      2) Adds the initialCheckpoint parameter to the topic expression so a topic can start pulling records from anywhere in the queue.

      Daemons can be sent to worker nodes that each work on processing a partition of the data from the same topic. The daemon() function's natural behavior is perfect for iteratively calling a topic until all records in the topic have been processed.

      The sample code below pulls all records from one collection and indexes them into another collection. A Transform function could be wrapped around the topic() to transform the records before loading. Custom functions can also be built to load the data in parallel to any outside system.

      parallel(
               workerCollection, 
               workers="2", 
               sort="DaemonOp desc", 
               daemon(
                      update(
                            updateCollection, 
                            batchSize=200, 
                            topic(
                                  checkpointCollection,
                                  topicCollection, 
                                  q=*:*, 
                                  id="topic1",
                                  fl="id, to , from, body", 
                                  partitionKeys="id",
                                  initialCheckpoint="0")), 
                      runInterval="1000", 
                      id="daemon1"))
      

      Attachments

        1. SOLR-9240.patch
          22 kB
          Joel Bernstein
        2. SOLR-9240.patch
          19 kB
          Joel Bernstein

        Issue Links

          Activity

            People

              jbernste Joel Bernstein
              jbernste Joel Bernstein
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: