Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9905

Add NullStream to isolate the performance of the ExportWriter

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Resolved
    • Affects Version/s: None
    • Fix Version/s: 6.4, 7.0
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      The NullStream is a utility function to test the raw performance of the ExportWriter. This is a nice utility to have to diagnose bottlenecks in streaming MapReduce operations. The NullStream will allow developers to test the performance of the shuffling (Sorting, Partitioning, Exporting) in isolation from the reduce operation (Rollup, Join, Group, etc..).

      The NullStream simply iterates it's internal stream and eats the tuples. It returns a single Tuple from each worker with the number of Tuples processed. The idea is to iterate the stream without additional overhead so the performance of the underlying stream can be isolated.

      Sample syntax:

      parallel(collection2, workers=7, sort="nullCount desc", 
            null(search(collection1, 
                         q=*:*, 
                         fl="id", 
                         sort="id desc", 
                         qt="/export", 
                         wt="javabin", 
                         partitionKeys=id)))
      

      In the example above the NullStream is sent to 7 workers. Each worker will iterate the search() expression and the NullStream will eat the tuples so the raw performance of the search() can be understood.

      1. SOLR-9905.patch
        9 kB
        Joel Bernstein

        Activity

        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 7dcb557ab73da7fb7af0e8f698895e28dde4bbca in lucene-solr's branch refs/heads/master from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7dcb557 ]

        SOLR-9905: Add NullStream to isolate the performance of the ExportWriter

        Show
        jira-bot ASF subversion and git services added a comment - Commit 7dcb557ab73da7fb7af0e8f698895e28dde4bbca in lucene-solr's branch refs/heads/master from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7dcb557 ] SOLR-9905 : Add NullStream to isolate the performance of the ExportWriter
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 00723827ff5ad5c129d3d8487d2c64469ea03239 in lucene-solr's branch refs/heads/master from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0072382 ]

        SOLR-9905: Update CHANGES.txt

        Show
        jira-bot ASF subversion and git services added a comment - Commit 00723827ff5ad5c129d3d8487d2c64469ea03239 in lucene-solr's branch refs/heads/master from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0072382 ] SOLR-9905 : Update CHANGES.txt
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 1a1b3af78d1ce147f5be3da09edc27729578d744 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1a1b3af ]

        SOLR-9905: Add NullStream to isolate the performance of the ExportWriter

        Show
        jira-bot ASF subversion and git services added a comment - Commit 1a1b3af78d1ce147f5be3da09edc27729578d744 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1a1b3af ] SOLR-9905 : Add NullStream to isolate the performance of the ExportWriter
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 0d830a7656e9b741970286dce5cb56d60df004f4 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0d830a7 ]

        SOLR-9905: Update CHANGES.txt

        Show
        jira-bot ASF subversion and git services added a comment - Commit 0d830a7656e9b741970286dce5cb56d60df004f4 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=0d830a7 ] SOLR-9905 : Update CHANGES.txt

          People

          • Assignee:
            Unassigned
            Reporter:
            joel.bernstein Joel Bernstein
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development