Solr
  1. Solr
  2. SOLR-2200

DIH DocBuilder - Improve perf. on large delta deletes

    Details

      Description

      In collectDelta, the procedure that collects the PKs for the documents that should be updated or deleted for an entity, iterates over the entire deltaSet for every deleted document. This is very expensive when you are updating and deleting millions of documents in one delta-import.
      Considering that the comparison between deleted and delta is on the PK, lets build the deltaSet as a HashMap instead of a HashSet to enable quick key lookups and remove the need for repeated iterations.

        Issue Links

          Activity

          Hide
          Mark Waddle added a comment -

          Uploading patch to improve performance for delta-imports with a significant number of deletions.

          Show
          Mark Waddle added a comment - Uploading patch to improve performance for delta-imports with a significant number of deletions.
          Hide
          Robert Muir added a comment -

          Mark, thanks for your contribution.

          Seems like a no-brainer to me, and all tests pass with the patch.

          I'd like to commit this unless anyone has objections.

          Show
          Robert Muir added a comment - Mark, thanks for your contribution. Seems like a no-brainer to me, and all tests pass with the patch. I'd like to commit this unless anyone has objections.
          Hide
          Robert Muir added a comment -

          Committed revisions 1029325 (trunk), 1029328 (3x).

          Thanks Mark!

          Show
          Robert Muir added a comment - Committed revisions 1029325 (trunk), 1029328 (3x). Thanks Mark!
          Hide
          Grant Ingersoll added a comment -

          Bulk close for 3.1.0 release

          Show
          Grant Ingersoll added a comment - Bulk close for 3.1.0 release
          Hide
          Mark Waddle added a comment -

          Hi Robert,

          I apologize for my ignorance, but why can't I see these changes in the current dev/trunk? Am I looking in the wrong place?

          Mark

          Show
          Mark Waddle added a comment - Hi Robert, I apologize for my ignorance, but why can't I see these changes in the current dev/trunk? Am I looking in the wrong place? Mark
          Hide
          Robert Muir added a comment -

          click the subversion commits tab.

          Show
          Robert Muir added a comment - click the subversion commits tab.

            People

            • Assignee:
              Robert Muir
              Reporter:
              Mark Waddle
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development