Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7197

Can solr EntityProcessor implement curosrs

    Details

    • Type: Wish
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.0
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      Prod

      Description

      package org.apache.solr.handler.dataimport;

      class SolrEntityProcessor

      protected SolrDocumentList doQuery(int start) {
      ....

      SolrQuery solrQuery = new SolrQuery(queryString);
      solrQuery.setRows(rows);
      solrQuery.setStart(start);
      if (fields != null) {
      for (String field : fields)

      { solrQuery.addField(field); }

      }
      solrQuery.setRequestHandler(requestHandler);
      solrQuery.setFilterQueries(filterQueries);
      solrQuery.setTimeAllowed(timeout * 1000);

      QueryResponse response = null;
      try

      { response = solrClient.query(solrQuery); }

      catch (SolrServerException e) {
      if (ABORT.equals(onError))

      { wrapAndThrow(SEVERE, e); }

      else if (SKIP.equals(onError))

      { wrapAndThrow(DataImportHandlerException.SKIP_ROW, e); }

      }

      -------------------

      If the do Query variant can be implemented with cursor, then it helps with any heavy lifting (bulk processing) with entity processor. That really helps.

      If permitted I can contribute the fix. Currently I am using 4.10 and see the performance issues and planning the work around. If the cursor is available then it really helps.

      1. DIH_SEP_CURSOR_SOLR_7197.patch
        2 kB
        Raveendra Yerraguntl
      2. lucene_ant_test_op
        753 kB
        Raveendra Yerraguntl

        Issue Links

          Activity

          Hide
          shalinmangar Shalin Shekhar Mangar added a comment -

          Hi Raveendra, a patch would be great. Please see http://wiki.apache.org/solr/HowToContribute on how to create a patch for inclusion. The patch should be against the trunk branch which will then be backported to 5.x branch once ready.

          Show
          shalinmangar Shalin Shekhar Mangar added a comment - Hi Raveendra, a patch would be great. Please see http://wiki.apache.org/solr/HowToContribute on how to create a patch for inclusion. The patch should be against the trunk branch which will then be backported to 5.x branch once ready.
          Hide
          Raveendra Raveendra added a comment -

          Shalin, I started on this, I need to test deep paging in the testFramework. Any pointers are appreciated.

          Show
          Raveendra Raveendra added a comment - Shalin, I started on this, I need to test deep paging in the testFramework. Any pointers are appreciated.
          Hide
          Raveendra Raveendra Yerraguntl added a comment -

          Attaching the path file. The 'ant test' was successful for this test, though it failed for replication test. Attached the redirected output from ant test.

          Show
          Raveendra Raveendra Yerraguntl added a comment - Attaching the path file. The 'ant test' was successful for this test, though it failed for replication test. Attached the redirected output from ant test.
          Hide
          Raveendra Raveendra Yerraguntl added a comment -

          Will be doing a bench mark test soon before and after the fix for a couple of million records.

          Show
          Raveendra Raveendra Yerraguntl added a comment - Will be doing a bench mark test soon before and after the fix for a couple of million records.

            People

            • Assignee:
              Unassigned
              Reporter:
              Raveendra Raveendra Yerraguntl
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development