Solr
  1. Solr
  2. SOLR-1065

Add a ContentStreamDataSource to DIH to accept post data

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4
    • Labels:
      None

      Description

      It is a common requirement to push data to DIH. Currently it is not possible . If we have a ContentStreamDataSource it can easily solve this problem

      sample configuration

      <dataSource type="ContentStreamDataSource"/>
      

      This datasource does not need any extra configuration. Make a normal POST request with the data as the body. The params remain same.

      1. SOLR-1065.patch
        3 kB
        Noble Paul
      2. SOLR-1065.patch
        33 kB
        Noble Paul
      3. SOLR-1065.patch
        35 kB
        Shalin Shekhar Mangar

        Activity

        Hide
        Grant Ingersoll added a comment -

        Bulk close Solr 1.4 issues

        Show
        Grant Ingersoll added a comment - Bulk close Solr 1.4 issues
        Hide
        Shalin Shekhar Mangar added a comment -

        Is there a reason why this can't be added to the core update handler?

        I'm not sure I follow you. We need to post to DIH for the various transforms/joins specified in data-config to take effect.

        Show
        Shalin Shekhar Mangar added a comment - Is there a reason why this can't be added to the core update handler? I'm not sure I follow you. We need to post to DIH for the various transforms/joins specified in data-config to take effect.
        Hide
        Otis Gospodnetic added a comment -

        regular update handler can only handle xml in the standard format. With DIH you can post any xml or any other file . Moreover DIH lets you have custom transformations to the data.

        It is also possible to mix the uploaded data with other DatSources (DB) before creating the documents

        Is there a reason why this can't be added to the core update handler?

        Show
        Otis Gospodnetic added a comment - regular update handler can only handle xml in the standard format. With DIH you can post any xml or any other file . Moreover DIH lets you have custom transformations to the data. It is also possible to mix the uploaded data with other DatSources (DB) before creating the documents Is there a reason why this can't be added to the core update handler?
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 755141.

        The committed patch also has changes to build.xml to add jetty to the classpath for testing the ContentStreamDataSource.

        Thanks Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 755141. The committed patch also has changes to build.xml to add jetty to the classpath for testing the ContentStreamDataSource. Thanks Noble!
        Hide
        Shalin Shekhar Mangar added a comment -
        1. Removed slurping the stream on close
        2. Minor javadocs updates
        3. Removed s.o.p from the test

        I'll commit this shortly.

        Show
        Shalin Shekhar Mangar added a comment - Removed slurping the stream on close Minor javadocs updates Removed s.o.p from the test I'll commit this shortly.
        Hide
        Noble Paul added a comment - - edited

        regular update handler can only handle xml in the standard format. With DIH you can post any xml or any other file . Moreover DIH lets you have custom transformations to the data.

        It is also possible to mix the uploaded data with other DatSources (DB) before creating the documents

        Show
        Noble Paul added a comment - - edited regular update handler can only handle xml in the standard format. With DIH you can post any xml or any other file . Moreover DIH lets you have custom transformations to the data. It is also possible to mix the uploaded data with other DatSources (DB) before creating the documents
        Hide
        Grant Ingersoll added a comment -

        Huh? Why would I need to post to DIH? Why not just post to the regular update handler?

        Show
        Grant Ingersoll added a comment - Huh? Why would I need to post to DIH? Why not just post to the regular update handler?
        Hide
        Noble Paul added a comment -

        with testcase

        Show
        Noble Paul added a comment - with testcase
        Hide
        Noble Paul added a comment -

        idea as code. I will test it soon

        Show
        Noble Paul added a comment - idea as code. I will test it soon

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Noble Paul
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development