Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9905

Add NullStream to isolate the performance of the ExportWriter

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Resolved
    • None
    • 6.4, 7.0
    • None
    • None

    Description

      The NullStream is a utility function to test the raw performance of the ExportWriter. This is a nice utility to have to diagnose bottlenecks in streaming MapReduce operations. The NullStream will allow developers to test the performance of the shuffling (Sorting, Partitioning, Exporting) in isolation from the reduce operation (Rollup, Join, Group, etc..).

      The NullStream simply iterates it's internal stream and eats the tuples. It returns a single Tuple from each worker with the number of Tuples processed. The idea is to iterate the stream without additional overhead so the performance of the underlying stream can be isolated.

      Sample syntax:

      parallel(collection2, workers=7, sort="nullCount desc", 
            null(search(collection1, 
                         q=*:*, 
                         fl="id", 
                         sort="id desc", 
                         qt="/export", 
                         wt="javabin", 
                         partitionKeys=id)))
      

      In the example above the NullStream is sent to 7 workers. Each worker will iterate the search() expression and the NullStream will eat the tuples so the raw performance of the search() can be understood.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            jbernste Joel Bernstein
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment