Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9478

Do delta import will loss some documents, when the documents added in the duration of delta import.

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • 5.5, 5.5.2, 6.0
    • 5.5.2
    • SolrCloud

    Description

      hello guys,
      I met a problem when i using the solrcloud mode. When the solr instance run delta-import, it may take
      some time to be finished( my data source is mysql database). So during this time, the new added documents
      will loss, the deltaQuery, i use SUBDATE(${dih.last_index_time}, INTERVAL 2 MINUTE),
      let it run the delta-import 2 mins earlier than the last_index_time, if the delta-import's duration is 5 mins, it will loss the records at the first 3 mins.
      Our servers doesn't use solr cloud mode before, we deal with this issue is tring to rewrite dataimport.properties file,
      query the max(sys_time_stamp), which will help to record the max time stamp, and let the solr can run delta import standing
      by the time found in the file, of course, it will never miss docuements.
      But now, we use solrcloud, the dataimport.properties is on the zookeeper, and we may have multiple collections for the
      same core.how can i update the dataimport.properties file now in colleciton now? Do you have any solution to help record
      the max(sys_time_stamp) in dataimport.properties, rather than using the time of delta-import start to run?

      Cheers

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            tedorz ted zhu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment