Solr
  1. Solr
  2. SOLR-2729

DIH status: successful zero-document delta-import missing "" field

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.2
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      Description

      If you have a successful delta-import that happens to process zero documents, the <str name=""> field is not present in the status. I've run into this situation when the SQL query results in an empty set. A workaround for the problem is to instead look for the "Time taken " field ... but if you don't happen to notice that this field has an extraneous space in the name, that won't work either.

      A full-import that processes zero documents has the field present as expected:

      <str name="">Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.</str>

      1. SOLR-2729.patch
        1 kB
        Shawn Heisey

        Issue Links

          Activity

          Hide
          Robert Muir added a comment -

          3.4 -> 3.5

          Show
          Robert Muir added a comment - 3.4 -> 3.5
          Hide
          Hoss Man added a comment -

          Bulk changing fixVersion 3.6 to 4.0 for any open issues that are unassigned and have not been updated since March 19.

          Email spam suppressed for this bulk edit; search for hoss20120323nofix36 to identify all issues edited

          Show
          Hoss Man added a comment - Bulk changing fixVersion 3.6 to 4.0 for any open issues that are unassigned and have not been updated since March 19. Email spam suppressed for this bulk edit; search for hoss20120323nofix36 to identify all issues edited
          Hide
          Shawn Heisey added a comment -

          Found it. In solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DocBuilder.java:

          // Do not commit unnecessarily if this is a delta-import and no documents were created or deleted
          if (!requestParameters.clean) {
            if (importStatistics.docCount.get() > 0 || importStatistics.deletedDocCount.get() > 0) {
              finish(lastIndexTimeProps);
            }
          } else {
            // Finished operation normally, commit now
            finish(lastIndexTimeProps);
          }
          

          The method named finish is where the status message gets updated with the status that says how many documents were added/updated.

          A fix that would take care of the immediate problem is to move the code that populates the "" part of statusMessages into its own method that is called by finish, then add an else clause to the inner if statement above which calls that method. Does that sound at all reasonable?

          Show
          Shawn Heisey added a comment - Found it. In solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DocBuilder.java: // Do not commit unnecessarily if this is a delta- import and no documents were created or deleted if (!requestParameters.clean) { if (importStatistics.docCount.get() > 0 || importStatistics.deletedDocCount.get() > 0) { finish(lastIndexTimeProps); } } else { // Finished operation normally, commit now finish(lastIndexTimeProps); } The method named finish is where the status message gets updated with the status that says how many documents were added/updated. A fix that would take care of the immediate problem is to move the code that populates the "" part of statusMessages into its own method that is called by finish, then add an else clause to the inner if statement above which calls that method. Does that sound at all reasonable?
          Hide
          Shawn Heisey added a comment -

          Something that will require a separate issue, perhaps two: I really think that "" is not a good name for the place where this stuff goes, and that "Time Taken " should also be fixed so it has no trailing space. Perhaps the entire status response needs some TLC. Making these changes will break a lot of user code, but it specifically says in the status output that the format is experimental and may change.

          Show
          Shawn Heisey added a comment - Something that will require a separate issue, perhaps two: I really think that "" is not a good name for the place where this stuff goes, and that "Time Taken " should also be fixed so it has no trailing space. Perhaps the entire status response needs some TLC. Making these changes will break a lot of user code, but it specifically says in the status output that the format is experimental and may change.
          Hide
          James Dyer added a comment -

          I agree the status messages should be better (Fix typos, no blank names, etc). I don't think we should worry too much about breaking people's code (mine included). Really, there should be a better way for automated schedulers to be able to check DIH status (JMX maybe?). This is probably more of a long-term wish though. In any case, I think the focus on the existing status page should be human-readability.

          Show
          James Dyer added a comment - I agree the status messages should be better (Fix typos, no blank names, etc). I don't think we should worry too much about breaking people's code (mine included). Really, there should be a better way for automated schedulers to be able to check DIH status (JMX maybe?). This is probably more of a long-term wish though. In any case, I think the focus on the existing status page should be human-readability.
          Hide
          Shawn Heisey added a comment -

          I created a method named putFinishStatus(). If this doesn't sound reasonable, or if this is not a good way to go, let me know and I'll fix it if I can. When it's all ironed out, I'll make a patch for 3x as well.

          All solr tests pass with this applied.

          Show
          Shawn Heisey added a comment - I created a method named putFinishStatus(). If this doesn't sound reasonable, or if this is not a good way to go, let me know and I'll fix it if I can. When it's all ironed out, I'll make a patch for 3x as well. All solr tests pass with this applied.
          Hide
          Hoss Man added a comment -

          James: Could you please triage this for 4.0? commit if you think it's ready, otherwise remove the fix version?

          Show
          Hoss Man added a comment - James: Could you please triage this for 4.0? commit if you think it's ready, otherwise remove the fix version?
          Hide
          Hoss Man added a comment -

          bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment

          Show
          Hoss Man added a comment - bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment
          Hide
          Robert Muir added a comment -

          rmuir20120906-bulk-40-change

          Show
          Robert Muir added a comment - rmuir20120906-bulk-40-change
          Hide
          Robert Muir added a comment -

          moving all 4.0 issues not touched in a month to 4.1

          Show
          Robert Muir added a comment - moving all 4.0 issues not touched in a month to 4.1
          Hide
          Shawn Heisey added a comment -

          Closing this issue because it's a small piece of what SOLR-3319 is trying to do. Will update notes on SOLR-3319 to indicate that a patch is available on this issue.

          This is part of an effort to close old issues that I have reported. Search tag: elyograg2013springclean

          Show
          Shawn Heisey added a comment - Closing this issue because it's a small piece of what SOLR-3319 is trying to do. Will update notes on SOLR-3319 to indicate that a patch is available on this issue. This is part of an effort to close old issues that I have reported. Search tag: elyograg2013springclean

            People

            • Assignee:
              James Dyer
              Reporter:
              Shawn Heisey
            • Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development