Solr
  1. Solr
  2. SOLR-7376

Allow DocTransformers to write directly to the response (support raw json/xml fields)

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.2, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      In order to return raw json when wt=json (and raw xml when wt=xml), it would be great if we could output directly to the response.

      I propose we can put a new `WriteableValue` in the response using DocumentTransformer – when a TextResponseWriter finds this type, it will let the `WriteableValue` figure out what to do.

        Issue Links

          Activity

          Hide
          Ryan McKinley added a comment -

          Here is a link to possible changes:
          https://github.com/voyagersearch/lucene-solr-for-voyager/commit/d1150c2c4ed5324d6830cfd996e7a595a4fd9fb6

          If there are no strong objections, i'll get that into patch form

          Show
          Ryan McKinley added a comment - Here is a link to possible changes: https://github.com/voyagersearch/lucene-solr-for-voyager/commit/d1150c2c4ed5324d6830cfd996e7a595a4fd9fb6 If there are no strong objections, i'll get that into patch form
          Hide
          Yonik Seeley added a comment -

          If it's a JSON string, we probably only want it written out verbatim if the JSON writer is being used (and be escaped normally otherwise)?

          Another approach would be to have a json field type that is validated at index time and spit back directly at query time when using wt=json

          Show
          Yonik Seeley added a comment - If it's a JSON string, we probably only want it written out verbatim if the JSON writer is being used (and be escaped normally otherwise)? Another approach would be to have a json field type that is validated at index time and spit back directly at query time when using wt=json
          Hide
          Ryan McKinley added a comment -

          If it's a JSON string, we probably only want it written out verbatim if the JSON writer

          Exactly, check:
          https://github.com/voyagersearch/lucene-solr-for-voyager/blob/d1150c2c4ed5324d6830cfd996e7a595a4fd9fb6/solr/core/src/java/org/apache/solr/response/transform/RawValueTransformerFactory.java#L55

          this checks if it is a JSON writer or does nothing. The other use case I am looking at is doing one thing if it is XML and another if it is JSON (GeoJSON vs KML etc)

          Show
          Ryan McKinley added a comment - If it's a JSON string, we probably only want it written out verbatim if the JSON writer Exactly, check: https://github.com/voyagersearch/lucene-solr-for-voyager/blob/d1150c2c4ed5324d6830cfd996e7a595a4fd9fb6/solr/core/src/java/org/apache/solr/response/transform/RawValueTransformerFactory.java#L55 this checks if it is a JSON writer or does nothing. The other use case I am looking at is doing one thing if it is XML and another if it is JSON (GeoJSON vs KML etc)
          Hide
          Ryan McKinley added a comment -

          Here is a patch with tests – this adds two default document transformers:

          +    defaultFactories.put( "json", new RawValueTransformerFactory("json") );
          +    defaultFactories.put( "xml", new RawValueTransformerFactory("xml") );
          

          The values are only unescaped when using the appropriate response writer.

          So if you have:
          &fl=id,name,json_s:[json],xml_s:[xml]&wt=json
          the xml_s value would be escaped, but json_s would be a raw value.

          Show
          Ryan McKinley added a comment - Here is a patch with tests – this adds two default document transformers: + defaultFactories.put( "json" , new RawValueTransformerFactory( "json" ) ); + defaultFactories.put( "xml" , new RawValueTransformerFactory( "xml" ) ); The values are only unescaped when using the appropriate response writer. So if you have: &fl=id,name,json_s: [json] ,xml_s: [xml] &wt=json the xml_s value would be escaped, but json_s would be a raw value.
          Hide
          ASF subversion and git services added a comment -

          Commit 1673647 from Ryan McKinley in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1673647 ]

          SOLR-7376: Return raw XML or JSON

          Show
          ASF subversion and git services added a comment - Commit 1673647 from Ryan McKinley in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1673647 ] SOLR-7376 : Return raw XML or JSON
          Hide
          ASF subversion and git services added a comment -

          Commit 1673649 from Ryan McKinley in branch 'dev/trunk'
          [ https://svn.apache.org/r1673649 ]

          Merged revision(s) 1673647 from lucene/dev/branches/branch_5x:
          SOLR-7376: Return raw XML or JSON
          ........

          Show
          ASF subversion and git services added a comment - Commit 1673649 from Ryan McKinley in branch 'dev/trunk' [ https://svn.apache.org/r1673649 ] Merged revision(s) 1673647 from lucene/dev/branches/branch_5x: SOLR-7376 : Return raw XML or JSON ........
          Hide
          Shalin Shekhar Mangar added a comment -

          This commit is causing compilation errors. See http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/12139/

          Show
          Shalin Shekhar Mangar added a comment - This commit is causing compilation errors. See http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/12139/
          Hide
          ASF subversion and git services added a comment -

          Commit 1673654 from Ryan McKinley in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1673654 ]

          SOLR-7376: adding missing fields

          Show
          ASF subversion and git services added a comment - Commit 1673654 from Ryan McKinley in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1673654 ] SOLR-7376 : adding missing fields
          Hide
          ASF subversion and git services added a comment -

          Commit 1673655 from Ryan McKinley in branch 'dev/trunk'
          [ https://svn.apache.org/r1673655 ]

          Merged revision(s) 1673654 from lucene/dev/branches/branch_5x:
          SOLR-7376: adding missing files
          ........

          Show
          ASF subversion and git services added a comment - Commit 1673655 from Ryan McKinley in branch 'dev/trunk' [ https://svn.apache.org/r1673655 ] Merged revision(s) 1673654 from lucene/dev/branches/branch_5x: SOLR-7376 : adding missing files ........
          Hide
          ASF subversion and git services added a comment -

          Commit 1673665 from Tommaso Teofili in branch 'dev/trunk'
          [ https://svn.apache.org/r1673665 ]

          SOLR-7376 - commented code using old IndexableField API, just using Object#toString

          Show
          ASF subversion and git services added a comment - Commit 1673665 from Tommaso Teofili in branch 'dev/trunk' [ https://svn.apache.org/r1673665 ] SOLR-7376 - commented code using old IndexableField API, just using Object#toString
          Hide
          Tommaso Teofili added a comment -

          I've committed a trivial fix for the compilation error (it seemed to be due to usage of old IndexableField API), Ryan McKinley feel free to revert / change it as you like, just wanted to have the build in good state.

          Show
          Tommaso Teofili added a comment - I've committed a trivial fix for the compilation error (it seemed to be due to usage of old IndexableField API), Ryan McKinley feel free to revert / change it as you like, just wanted to have the build in good state.
          Hide
          ASF subversion and git services added a comment -

          Commit 1673731 from Tommaso Teofili in branch 'dev/trunk'
          [ https://svn.apache.org/r1673731 ]

          SOLR-7376 - adjusted RawValueTransformerFactoryimpl to use StorableField#stringValue

          Show
          ASF subversion and git services added a comment - Commit 1673731 from Tommaso Teofili in branch 'dev/trunk' [ https://svn.apache.org/r1673731 ] SOLR-7376 - adjusted RawValueTransformerFactoryimpl to use StorableField#stringValue
          Hide
          Ryan McKinley added a comment -

          thank you Tommaso Teofili!

          Show
          Ryan McKinley added a comment - thank you Tommaso Teofili !
          Hide
          Anshum Gupta added a comment -

          Bulk close for 5.2.0.

          Show
          Anshum Gupta added a comment - Bulk close for 5.2.0.

            People

            • Assignee:
              Ryan McKinley
              Reporter:
              Ryan McKinley
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development