Solr
  1. Solr
  2. SOLR-4921

Support for Adding Documents via the Solr UI

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.4, 6.0
    • Component/s: web gui
    • Labels:
      None

      Description

      For demos and prototyping, it would be nice if we could add documents via the admin UI.

      Various things to support:
      1. Uploading XML, JSON, CSV, etc.
      2. Optionally also do file upload

      1. SOLR-4921.patch
        38 kB
        Grant Ingersoll
      2. SOLR-4921.patch
        37 kB
        Grant Ingersoll
      3. SOLR-4921.patch
        37 kB
        Grant Ingersoll
      4. SOLR-4921.patch
        37 kB
        Grant Ingersoll
      5. SOLR-4921.patch
        36 kB
        Grant Ingersoll
      6. SOLR-4921.patch
        36 kB
        Grant Ingersoll
      7. SOLR-4921.patch
        36 kB
        Grant Ingersoll
      8. SOLR-4921.patch
        35 kB
        Grant Ingersoll
      9. SOLR-4921.patch
        34 kB
        Grant Ingersoll
      10. SOLR-4921.patch
        34 kB
        Grant Ingersoll
      11. SOLR-4921.patch
        32 kB
        Grant Ingersoll
      12. SOLR-4921.patch
        30 kB
        Grant Ingersoll
      13. SOLR-4921.patch
        29 kB
        Grant Ingersoll
      14. SOLR-4921.patch
        19 kB
        Grant Ingersoll
      15. SOLR-4921.patch
        17 kB
        Grant Ingersoll
      16. SOLR-4921.patch
        16 kB
        Grant Ingersoll
      17. SOLR-4921.patch
        14 kB
        Grant Ingersoll
      18. SOLR-4921.patch
        12 kB
        Grant Ingersoll
      19. SOLR-4921.patch
        10 kB
        Grant Ingersoll
      20. SOLR-4921.patch
        9 kB
        Grant Ingersoll

        Issue Links

          Activity

          Hide
          Grant Ingersoll added a comment -

          Here's a start. Still need to figure out the Javascript for submitting the form, etc.

          Show
          Grant Ingersoll added a comment - Here's a start. Still need to figure out the Javascript for submitting the form, etc.
          Hide
          Grant Ingersoll added a comment -

          Minor tweaks

          Show
          Grant Ingersoll added a comment - Minor tweaks
          Hide
          Grant Ingersoll added a comment -

          A little bit more progress, but not much. Still getting my head around sammy.js.

          Show
          Grant Ingersoll added a comment - A little bit more progress, but not much. Still getting my head around sammy.js.
          Hide
          Grant Ingersoll added a comment -

          Actually posts the documents now.

          Things to do:

          1. Need an icon for the documents tab
          2. Success/Error handling
          3. Test other content types
          4. File upload
          5. Document creation wizard
          6. Better layout of the form, etc.
          7. Other things I'm sure I forgot.
          Show
          Grant Ingersoll added a comment - Actually posts the documents now. Things to do: Need an icon for the documents tab Success/Error handling Test other content types File upload Document creation wizard Better layout of the form, etc. Other things I'm sure I forgot.
          Hide
          Grant Ingersoll added a comment - - edited

          Also, still need to handle the command parameters like commitWithin, etc. Also, handling incremental updates, etc. would be nice

          Show
          Grant Ingersoll added a comment - - edited Also, still need to handle the command parameters like commitWithin, etc. Also, handling incremental updates, etc. would be nice
          Hide
          Grant Ingersoll added a comment -

          Can add JSON or XML now, a few other tweaks.

          Show
          Grant Ingersoll added a comment - Can add JSON or XML now, a few other tweaks.
          Hide
          Grant Ingersoll added a comment -

          CSV support, Raw Solr Command support.
          commitWithin, overwrite support.

          Boost for JSON support.

          Show
          Grant Ingersoll added a comment - CSV support, Raw Solr Command support. commitWithin, overwrite support. Boost for JSON support.
          Hide
          Grant Ingersoll added a comment -

          Patch has the following improvements

          1. Better Layout
          2. Result Reporting, including errors
          3. Various other little fixes

          You should be able to submit a variety of document types at this point and see the response.

          Left to do:

          1. Icon for Collection drop down
          2. Wizard implementation
          3. General cleanup, comments
          4. File Upload
          5. Other things I've forgotten
          Show
          Grant Ingersoll added a comment - Patch has the following improvements Better Layout Result Reporting, including errors Various other little fixes You should be able to submit a variety of document types at this point and see the response. Left to do: Icon for Collection drop down Wizard implementation General cleanup, comments File Upload Other things I've forgotten
          Hide
          Grant Ingersoll added a comment -

          Here's a start on file upload. It kind of works right now if you hit the submit button twice (after changing the "QT" option to /update/extract). There seems to be some oddities with variable bindings for creating the document_url based off of the handler path.

          Show
          Grant Ingersoll added a comment - Here's a start on file upload. It kind of works right now if you hit the submit button twice (after changing the "QT" option to /update/extract). There seems to be some oddities with variable bindings for creating the document_url based off of the handler path.
          Hide
          Grant Ingersoll added a comment -

          File upload should now work after making some tweaks to the original file upload code to support immediate uploading as opposed to attaching the functionality to the submit button (since we already have done that). One thing to note: based on the response we get back from the file upload, the output looks diff. than for other docs.

          Patch also includes various other minor fixes.

          Left to do:

          1. Documents icon
          2. Clean up of code (comments, functions)
          3. Wizard functionality

          This is getting closer to committing.

          Show
          Grant Ingersoll added a comment - File upload should now work after making some tweaks to the original file upload code to support immediate uploading as opposed to attaching the functionality to the submit button (since we already have done that). One thing to note: based on the response we get back from the file upload, the output looks diff. than for other docs. Patch also includes various other minor fixes. Left to do: Documents icon Clean up of code (comments, functions) Wizard functionality This is getting closer to committing.
          Hide
          Grant Ingersoll added a comment -

          Soliciting ideas on what the Wizard should do. I'm thinking it would be cool to have the ability to select fields (or create new ones) and then fill in what that value is and see the JSON get built on the fly.

          Show
          Grant Ingersoll added a comment - Soliciting ideas on what the Wizard should do. I'm thinking it would be cool to have the ability to select fields (or create new ones) and then fill in what that value is and see the JSON get built on the fly.
          Hide
          Grant Ingersoll added a comment -

          Added:

          1. Example documents for several of the selectors
          2. Documents icon in the collection drop down
          Show
          Grant Ingersoll added a comment - Added: Example documents for several of the selectors Documents icon in the collection drop down
          Hide
          Shalin Shekhar Mangar added a comment -

          I'm thinking it would be cool to have the ability to select fields (or create new ones) and then fill in what that value is and see the JSON get built on the fly

          +1

          Show
          Shalin Shekhar Mangar added a comment - I'm thinking it would be cool to have the ability to select fields (or create new ones) and then fill in what that value is and see the JSON get built on the fly +1
          Hide
          Grant Ingersoll added a comment -

          A start on the wizard.

          Show
          Grant Ingersoll added a comment - A start on the wizard.
          Hide
          Grant Ingersoll added a comment -

          Rethought the Wizard a bit. Should be working now.

          Steps:

          1. Select the Wizard from the Doc Type drop down
          2. Choose your field
          3. Fill in text area
          4. Hit "Add Field"
          5. repeat until done
          6. Hit submit document when the doc is complete

          This uses the same pathway as the JSON option when submitting.

          Show
          Grant Ingersoll added a comment - Rethought the Wizard a bit. Should be working now. Steps: Select the Wizard from the Doc Type drop down Choose your field Fill in text area Hit "Add Field" repeat until done Hit submit document when the doc is complete This uses the same pathway as the JSON option when submitting.
          Hide
          Grant Ingersoll added a comment -

          A bit more cleanup and fixed some issues with the Wizard.

          Renamed the Wizard to be "Document Builder"

          Added the plus button to the add field section.

          Ready for review, as I think this about done.

          Show
          Grant Ingersoll added a comment - A bit more cleanup and fixed some issues with the Wizard. Renamed the Wizard to be "Document Builder" Added the plus button to the add field section. Ready for review, as I think this about done.
          Hide
          Erick Erickson added a comment -

          Sweet! This'll be a great tool for giving people a better understanding of what all that stuff on the wiki is about!

          Minor comments:
          1> When doing a "file upload" and adding a literal.id=doc1 it took me a couple of minutes to figure out that I had to prepend with &. Perhaps a check before appending the params to the URL? Or just an example on the label for the entry field with the '&'?

          2> maybe a similar thing for entering the Request-handler for uploading structured docs, pdfs and the like, or maybe some kind of button or checkbox or radio button for "upload structured doc". Or maybe the document type drop down with another choice "upload/extract doc (e.g. pdf, jpg)" that auto-filled the request handler? I kind of like this last, using "extract" directs them to the extracting request handler. And auto-fill the request handler with just "/update" when they clicked something else.

          3> If I actually specify a version on the "document builder" section I get an error. Should we not allow them to specify a version? I can also argue that having the version actually really good so people get a clue not to mess with it <G>... Possibly display version (expert) in the drop down?

          But these are all minor issues compared to how cool this is!

          Show
          Erick Erickson added a comment - Sweet! This'll be a great tool for giving people a better understanding of what all that stuff on the wiki is about! Minor comments: 1> When doing a "file upload" and adding a literal.id=doc1 it took me a couple of minutes to figure out that I had to prepend with &. Perhaps a check before appending the params to the URL? Or just an example on the label for the entry field with the '&'? 2> maybe a similar thing for entering the Request-handler for uploading structured docs, pdfs and the like, or maybe some kind of button or checkbox or radio button for "upload structured doc". Or maybe the document type drop down with another choice "upload/extract doc (e.g. pdf, jpg)" that auto-filled the request handler? I kind of like this last, using "extract" directs them to the extracting request handler. And auto-fill the request handler with just "/update" when they clicked something else. 3> If I actually specify a version on the "document builder" section I get an error. Should we not allow them to specify a version? I can also argue that having the version actually really good so people get a clue not to mess with it <G>... Possibly display version (expert) in the drop down? But these are all minor issues compared to how cool this is!
          Hide
          Grant Ingersoll added a comment -

          Fixes #1 from Erick Erickson comments

          Show
          Grant Ingersoll added a comment - Fixes #1 from Erick Erickson comments
          Hide
          Grant Ingersoll added a comment -

          2> maybe a similar thing for entering the Request-handler

          Is there an easy way to know which Req. Handlers to use since it doesn't have to be named /update/extract?

          3> If I actually specify a version...

          Probably makes sense to filter the version

          Show
          Grant Ingersoll added a comment - 2> maybe a similar thing for entering the Request-handler Is there an easy way to know which Req. Handlers to use since it doesn't have to be named /update/extract? 3> If I actually specify a version... Probably makes sense to filter the version
          Hide
          Grant Ingersoll added a comment -

          Filters version out of the Doc Builder.

          Show
          Grant Ingersoll added a comment - Filters version out of the Doc Builder.
          Hide
          Grant Ingersoll added a comment -

          Cleaned up the TODOs.

          Planning to do a bit more testing, than I will commit either later today or tomorrow.

          Show
          Grant Ingersoll added a comment - Cleaned up the TODOs. Planning to do a bit more testing, than I will commit either later today or tomorrow.
          Hide
          Shalin Shekhar Mangar added a comment -

          This is really nice to have. Thanks Grant!

          1. Can we add the sample doc back to the textarea if you click on it and then remove focus without writing anything? The only way to restore the example in the textarea is to reload the "Documents" tab.
          2. If I choose "Document Builder" and add a bunch of fields but click on the auto-generated "Document(s)" textarea the whole textarea is wiped clean. Why is that? We can allow the user to edit it or else we can make the textarea read-only.
          3. Once a field has been added the textarea should be emptied
          Show
          Shalin Shekhar Mangar added a comment - This is really nice to have. Thanks Grant! Can we add the sample doc back to the textarea if you click on it and then remove focus without writing anything? The only way to restore the example in the textarea is to reload the "Documents" tab. If I choose "Document Builder" and add a bunch of fields but click on the auto-generated "Document(s)" textarea the whole textarea is wiped clean. Why is that? We can allow the user to edit it or else we can make the textarea read-only. Once a field has been added the textarea should be emptied
          Hide
          Grant Ingersoll added a comment - - edited

          Can we add the sample doc back to the textarea if you click on it and then remove focus without writing anything? The only way to restore the example in the textarea is to reload the "Documents" tab.

          Yeah, we can make that happen, although I don't know if it is a big deal. The whole idea of the sample document is just to give an idea of what it looks like

          If I choose "Document Builder" and add a bunch of fields but click on the auto-generated "Document(s)" textarea the whole textarea is wiped clean. Why is that? We can allow the user to edit it or else we can make the textarea read-only.

          Good catch. I thought about making it read-only, but thought not to in case someone wanted to edit it by hand (since that is the only way to remove mistakes). Probably best to have the "focusin" event only remove the text if we aren't in Doc builder mode

          Once a field has been added the textarea should be emptied

          Consider it done.

          Show
          Grant Ingersoll added a comment - - edited Can we add the sample doc back to the textarea if you click on it and then remove focus without writing anything? The only way to restore the example in the textarea is to reload the "Documents" tab. Yeah, we can make that happen, although I don't know if it is a big deal. The whole idea of the sample document is just to give an idea of what it looks like If I choose "Document Builder" and add a bunch of fields but click on the auto-generated "Document(s)" textarea the whole textarea is wiped clean. Why is that? We can allow the user to edit it or else we can make the textarea read-only. Good catch. I thought about making it read-only, but thought not to in case someone wanted to edit it by hand (since that is the only way to remove mistakes). Probably best to have the "focusin" event only remove the text if we aren't in Doc builder mode Once a field has been added the textarea should be emptied Consider it done.
          Hide
          Erick Erickson added a comment -

          bq: Is there an easy way to know which Req. Handlers to use since it doesn't have to be named /update/extract?

          Good point, I wasn't thinking about anything complex, update/extract is the default though, makes it easy OOB but would be less convenient for a custom setup. No big deal either way though.

          Show
          Erick Erickson added a comment - bq: Is there an easy way to know which Req. Handlers to use since it doesn't have to be named /update/extract? Good point, I wasn't thinking about anything complex, update/extract is the default though, makes it easy OOB but would be less convenient for a custom setup. No big deal either way though.
          Hide
          Stefan Matheis (steffkes) added a comment -

          Looks good Grant!

          One Bug:

          • If you change the Type to Document Builder, the list of fields is fine - after switching (back) to CSV and again to Document Builder the list of fields contains all fields twice (i guess they are just appended w/o clearing the select?)

          Two Suggestions:

          • i'd move the large Document(s) Textarea to the right? Following the typical layout (like we have it on the Query Page)?
          • Selecting the Document Builder the select + related textarea could be shown below the textarea? to make it consistent, where the main area is displayed
          Show
          Stefan Matheis (steffkes) added a comment - Looks good Grant! One Bug: If you change the Type to Document Builder , the list of fields is fine - after switching (back) to CSV and again to Document Builder the list of fields contains all fields twice (i guess they are just appended w/o clearing the select?) Two Suggestions: i'd move the large Document(s) Textarea to the right? Following the typical layout (like we have it on the Query Page)? Selecting the Document Builder the select + related textarea could be shown below the textarea? to make it consistent, where the main area is displayed
          Hide
          Grant Ingersoll added a comment -

          Fixes Shalin's items #2 and #3.

          Fixes Stefan's bug

          Still evaluating Stefan's other issues.

          Show
          Grant Ingersoll added a comment - Fixes Shalin's items #2 and #3. Fixes Stefan's bug Still evaluating Stefan's other issues.
          Hide
          Grant Ingersoll added a comment -

          i'd move the large Document(s) Textarea to the right? Following the typical layout (like we have it on the Query Page)?

          I'm not following the "typical layout" part. AFAICT, on the query page, the right hand column is reserved for the response, which is the same pattern I've followed here.

          Selecting the Document Builder the select + related textarea could be shown below the textarea? to make it consistent, where the main area is displayed

          Not following the consistency issue (or my version of consistency is diff. from yours). Not claiming to know what is best here, but the current layout for all doc types is:

          1. Doc Type selector
          2. The Document (however it is built, either as an upload, by the Documents Text Area or via the Wizard
          3. Optional attributes that modify the command (commit within, boost, etc.) (I will fix the File Upload options, to follow this)

          To me, if we move the Builder text area below the "Documents" text area, people will try to edit that space first and will miss the value of the builder.

          Show
          Grant Ingersoll added a comment - i'd move the large Document(s) Textarea to the right? Following the typical layout (like we have it on the Query Page)? I'm not following the "typical layout" part. AFAICT, on the query page, the right hand column is reserved for the response, which is the same pattern I've followed here. Selecting the Document Builder the select + related textarea could be shown below the textarea? to make it consistent, where the main area is displayed Not following the consistency issue (or my version of consistency is diff. from yours). Not claiming to know what is best here, but the current layout for all doc types is: Doc Type selector The Document (however it is built, either as an upload, by the Documents Text Area or via the Wizard Optional attributes that modify the command (commit within, boost, etc.) (I will fix the File Upload options, to follow this) To me, if we move the Builder text area below the "Documents" text area, people will try to edit that space first and will miss the value of the builder.
          Hide
          Grant Ingersoll added a comment -

          Fix the location of the Extr. Req. Handlers params.

          Show
          Grant Ingersoll added a comment - Fix the location of the Extr. Req. Handlers params.
          Hide
          Grant Ingersoll added a comment -

          Try to clean up some of the interactions between different doc types.

          Show
          Grant Ingersoll added a comment - Try to clean up some of the interactions between different doc types.
          Hide
          Grant Ingersoll added a comment -

          I plan on committing this soon. The functionality works, AFAICT, and the layout is decent. Layout can be changed.

          Show
          Grant Ingersoll added a comment - I plan on committing this soon. The functionality works, AFAICT, and the layout is decent. Layout can be changed.
          Hide
          Steve Rowe added a comment -

          Bulk close resolved 4.4 issues

          Show
          Steve Rowe added a comment - Bulk close resolved 4.4 issues

            People

            • Assignee:
              Grant Ingersoll
              Reporter:
              Grant Ingersoll
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development