Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3
    • Component/s: clients - php, search
    • Labels:
      None

      Description

      It would be useful to have a PHP response writer that returns an array to be eval-ed directly. This is especially true for PHP4.x installs, where there is no built in support for JSON.

      This issue attempts to address this.

      1. SOLR-196-PHPResponseWriter.patch
        31 kB
        Pieter Berkel
      2. SOLR-192-php-responsewriter.patch
        10 kB
        Paul Borgermans

        Issue Links

          Activity

          Hide
          Paul Borgermans added a comment -

          Attached first version of my patch to provide a PHP response writer (patch made against trunk rev 516287)

          This relies on a correct schema.xml config file with respect to multi valued fields!! Otherwise, duplicate keys are generated.

          Show
          Paul Borgermans added a comment - Attached first version of my patch to provide a PHP response writer (patch made against trunk rev 516287) This relies on a correct schema.xml config file with respect to multi valued fields!! Otherwise, duplicate keys are generated.
          Hide
          Yonik Seeley added a comment -

          Thanks Paul,
          Perhaps introducing some protected variables in the base class like
          ARRAY_START, ARRAY_SEP, ARRAY_END, etc, would allow us to eliminate some code duplication?

          Show
          Yonik Seeley added a comment - Thanks Paul, Perhaps introducing some protected variables in the base class like ARRAY_START, ARRAY_SEP, ARRAY_END, etc, would allow us to eliminate some code duplication?
          Hide
          Paul Borgermans added a comment -

          Hi Yonik

          Yes, I was looking in this direction as well, but I did not want to touch the Python and Ruby response writers (yet).

          My next priority however is to release a set of PHP classes to interact with Solr. If these protected variables appear, the PHP response writer will be immediately adapted.

          Regards
          Paul

          Show
          Paul Borgermans added a comment - Hi Yonik Yes, I was looking in this direction as well, but I did not want to touch the Python and Ruby response writers (yet). My next priority however is to release a set of PHP classes to interact with Solr. If these protected variables appear, the PHP response writer will be immediately adapted. Regards Paul
          Hide
          Tristan Vittorio added a comment -

          Hi Paul,

          Thanks for your work on the PHP response writer, I've applied your patch to the current svn trunk (revsion 549649) and manage to get it working only one minor change to PHPResponseWriter.java on line 23:

          • import org.apache.solr.util.NamedList;
            + import org.apache.solr.common.util.NamedList;

          Have you managed to get any further in developing this response writer or the Solr client classes you mention in your previous comment? I have been looking for a response writer that created serialized php data rather than eval-able code, perhaps it would be easy to adapt the code in this patch to support this? (I will take a look however my Java skill are not too strong yet).

          cheers,
          Tristan

          Show
          Tristan Vittorio added a comment - Hi Paul, Thanks for your work on the PHP response writer, I've applied your patch to the current svn trunk (revsion 549649) and manage to get it working only one minor change to PHPResponseWriter.java on line 23: import org.apache.solr.util.NamedList; + import org.apache.solr.common.util.NamedList; Have you managed to get any further in developing this response writer or the Solr client classes you mention in your previous comment? I have been looking for a response writer that created serialized php data rather than eval-able code, perhaps it would be easy to adapt the code in this patch to support this? (I will take a look however my Java skill are not too strong yet). cheers, Tristan
          Hide
          Pieter Berkel added a comment -

          This patch updates the PHPResponseWriter original written by Paul Borgermans and integrates the serialized PHP response writer (renamed to PHPSerializedResponseWriter to avoid name clashes) originally authored by Nick Jenkin in SOLR-275. See http://www.nabble.com/PHP-Response-Writer-for-Solr-tf4140580.html for some discussion on this implementation.

          I've made minimal code changes to JSONwriter in order to reducing the amount of code-duplication, specifically replacing all static writes of array and map structure tokens with methods:

          public void writeMapOpener(int size) throws IOException, IllegalArgumentException { writer.write('

          {'); }

          public void writeMapSeparator() throws IOException

          { writer.write(','); }
          public void writeMapCloser() throws IOException { writer.write('}'); }

          public void writeArrayOpener(int size) throws IOException, IllegalArgumentException { writer.write('['); }
          public void writeArraySeparator() throws IOException { writer.write(','); }

          public void writeArrayCloser() throws IOException

          { writer.write(']'); }

          The size parameter has been introduced specifically for PHPSerializedWriter (where the output format explicitly requires the size of the array / map to be set) and is currently ignored by all other response writers. In cases where the size is not trivial to calculate (e.g. an Iterable object), it is set to -1. Classes extending JSONwriter that require a valid (non-negative) size value must overload certain methods (i.e. writeArray() and writeDoc()) to calculate size correctly. It would also be a good idea to check for invalid size values in writeMapOpener() and writeArrayOpener() and throw a IllegalArgumentException if so.

          Some other changes I've made to the PHPWriter code from SOLR-196:

          1) Removed a lot of code duplicated from JSONwriter.
          2) Updated writeStr() to use StringBuilder.

          Some other changes I've made to the PHPSerializedWriter code from SOLR-275:

          1) Removed some uneccessary duplicate code.
          2) Changed key type written by writeArray() from String to int (since they are suppose to be numeric indicies).
          3) Updated writeStr() - serialized php strings don't need to be escaped (it seems to rely only on the specified string size value) and size needs be specified in bytes not characters (some Unicode characters were causing problems when using String.length() to calculate size, can someone please sanity check this code?).

          I've tested both PHPWriter & PHPSerializedWriter and they both seem to output valid PHP data, it would be great if people could also test them to ensure they work in their environments. JSONWriter also seems to be fine and although I didn't test the Python or Ruby writers, I assume they are unaffected (can anyone confirm?).

          Additionally, I've moved PythonWriter and RubyWriter from JSONResponseWriter.java to PythonResponseWriter.java and RubyResponseWriter.java respectively.

          I noticed that while each Writer specifies a content type value (e.g. CONTENT_TYPE_JSON_UTF8, CONTENT_TYPE_PYTHON_ASCII) the value returned by getContentType() is generally CONTENT_TYPE_TEXT_UTF8 or CONTENT_TYPE_TEXT_ASCII. This is not a big deal and I guessed this allows the output to be easily displayed in a browser, however it would be quite useful to have the actual content type value set so that client applications can determine the response format encoding and process it accordingly without relying on access to the original "wt" query paramater.

          Show
          Pieter Berkel added a comment - This patch updates the PHPResponseWriter original written by Paul Borgermans and integrates the serialized PHP response writer (renamed to PHPSerializedResponseWriter to avoid name clashes) originally authored by Nick Jenkin in SOLR-275 . See http://www.nabble.com/PHP-Response-Writer-for-Solr-tf4140580.html for some discussion on this implementation. I've made minimal code changes to JSONwriter in order to reducing the amount of code-duplication, specifically replacing all static writes of array and map structure tokens with methods: public void writeMapOpener(int size) throws IOException, IllegalArgumentException { writer.write(' {'); } public void writeMapSeparator() throws IOException { writer.write(','); } public void writeMapCloser() throws IOException { writer.write('}'); } public void writeArrayOpener(int size) throws IOException, IllegalArgumentException { writer.write('['); } public void writeArraySeparator() throws IOException { writer.write(','); } public void writeArrayCloser() throws IOException { writer.write(']'); } The size parameter has been introduced specifically for PHPSerializedWriter (where the output format explicitly requires the size of the array / map to be set) and is currently ignored by all other response writers. In cases where the size is not trivial to calculate (e.g. an Iterable object), it is set to -1. Classes extending JSONwriter that require a valid (non-negative) size value must overload certain methods (i.e. writeArray() and writeDoc()) to calculate size correctly. It would also be a good idea to check for invalid size values in writeMapOpener() and writeArrayOpener() and throw a IllegalArgumentException if so. Some other changes I've made to the PHPWriter code from SOLR-196 : 1) Removed a lot of code duplicated from JSONwriter. 2) Updated writeStr() to use StringBuilder. Some other changes I've made to the PHPSerializedWriter code from SOLR-275 : 1) Removed some uneccessary duplicate code. 2) Changed key type written by writeArray() from String to int (since they are suppose to be numeric indicies). 3) Updated writeStr() - serialized php strings don't need to be escaped (it seems to rely only on the specified string size value) and size needs be specified in bytes not characters (some Unicode characters were causing problems when using String.length() to calculate size, can someone please sanity check this code?). I've tested both PHPWriter & PHPSerializedWriter and they both seem to output valid PHP data, it would be great if people could also test them to ensure they work in their environments. JSONWriter also seems to be fine and although I didn't test the Python or Ruby writers, I assume they are unaffected (can anyone confirm?). Additionally, I've moved PythonWriter and RubyWriter from JSONResponseWriter.java to PythonResponseWriter.java and RubyResponseWriter.java respectively. I noticed that while each Writer specifies a content type value (e.g. CONTENT_TYPE_JSON_UTF8, CONTENT_TYPE_PYTHON_ASCII) the value returned by getContentType() is generally CONTENT_TYPE_TEXT_UTF8 or CONTENT_TYPE_TEXT_ASCII. This is not a big deal and I guessed this allows the output to be easily displayed in a browser, however it would be quite useful to have the actual content type value set so that client applications can determine the response format encoding and process it accordingly without relying on access to the original "wt" query paramater.
          Hide
          Pieter Berkel added a comment -

          This latest patch incorporates and deprecates code supplied in SOLR-275.

          Show
          Pieter Berkel added a comment - This latest patch incorporates and deprecates code supplied in SOLR-275 .
          Hide
          Yonik Seeley added a comment -

          Thanks Pieter,
          Regarding the content-type, I found it more useful to be able to actually see the result in a browser.
          Is there a content-type we can use for JSON that can achieve both goals for firefox and IE at least?

          Show
          Yonik Seeley added a comment - Thanks Pieter, Regarding the content-type, I found it more useful to be able to actually see the result in a browser. Is there a content-type we can use for JSON that can achieve both goals for firefox and IE at least?
          Hide
          Nick Jenkin added a comment -

          Excellent stuff, thanks Pieter!

          Show
          Nick Jenkin added a comment - Excellent stuff, thanks Pieter!
          Hide
          Yonik Seeley added a comment -

          Thanks guys!
          I just committed this.

          Show
          Yonik Seeley added a comment - Thanks guys! I just committed this.
          Hide
          Pieter Berkel added a comment -

          Great! I'll try to add some documentation to the wiki in the next few days.

          >Regarding the content-type, I found it more useful to be able to actually see the result in a browser.
          >Is there a content-type we can use for JSON that can achieve both goals for firefox and IE at least?

          I couldn't find any suitable mime types that would achieve this goal so it's probably better to leave the content-types unchanged for the moment.

          Show
          Pieter Berkel added a comment - Great! I'll try to add some documentation to the wiki in the next few days. >Regarding the content-type, I found it more useful to be able to actually see the result in a browser. >Is there a content-type we can use for JSON that can achieve both goals for firefox and IE at least? I couldn't find any suitable mime types that would achieve this goal so it's probably better to leave the content-types unchanged for the moment.
          Hide
          Pieter Berkel added a comment -

          Hmm, it doesn't look like the two new files from the patch were added properly during the latest commit:
          /src/java/org/apache/solr/request/PHPResponseWriter.java
          /src/java/org/apache/solr/request/PHPSerializedResponseWriter.java
          We won't get very far without those!

          Show
          Pieter Berkel added a comment - Hmm, it doesn't look like the two new files from the patch were added properly during the latest commit: /src/java/org/apache/solr/request/PHPResponseWriter.java /src/java/org/apache/solr/request/PHPSerializedResponseWriter.java We won't get very far without those!
          Hide
          Yonik Seeley added a comment -

          Got it... thanks.
          How I wish there were an svn patch that would add files for me

          -Yonik

          Show
          Yonik Seeley added a comment - Got it... thanks. How I wish there were an svn patch that would add files for me -Yonik
          Hide
          Hoss Man added a comment -

          This bug was modified as part of a bulk update using the criteria...

          • Marked "Resolved" and "Fixed"
          • Had no "Fix Version" versions
          • Was listed in the CHANGES.txt for 1.3 as of today 2008-03-15

          The Fix Version for all 29 issues found was set to 1.3, email notification was suppressed to prevent excessive email.

          For a list of all the issues modified, search jira comments for this (hopefully) unique string: batch20070315hossman1

          Show
          Hoss Man added a comment - This bug was modified as part of a bulk update using the criteria... Marked "Resolved" and "Fixed" Had no "Fix Version" versions Was listed in the CHANGES.txt for 1.3 as of today 2008-03-15 The Fix Version for all 29 issues found was set to 1.3, email notification was suppressed to prevent excessive email. For a list of all the issues modified, search jira comments for this (hopefully) unique string: batch20070315hossman1
          Hide
          Peter Wolanin added a comment - - edited

          This PHP writer is inconsistent with the JSON if you use php 5's decode_json, maps come back as objects.

          Show
          Peter Wolanin added a comment - - edited This PHP writer is inconsistent with the JSON if you use php 5's decode_json, maps come back as objects.
          Hide
          Andrew McCombe added a comment - - edited

          I'm finding occasionally that the return from the php response writer fails to unserialize in PHP5.

          I have a number of fields in the 'fl=' parameter of my query and it works fine until I add the field name 'recommended'. It then fails. If I remove all other field names and leave fl=recommended it still fails.

          I can see the output and it contains the results, but fails on unserializing in PHP.

          Is 'recommended' some sort of reserved word?

          Show
          Andrew McCombe added a comment - - edited I'm finding occasionally that the return from the php response writer fails to unserialize in PHP5. I have a number of fields in the 'fl=' parameter of my query and it works fine until I add the field name 'recommended'. It then fails. If I remove all other field names and leave fl=recommended it still fails. I can see the output and it contains the results, but fails on unserializing in PHP. Is 'recommended' some sort of reserved word?
          Hide
          Yonik Seeley added a comment -

          Perhaps escaping isn't correct for this particular field?
          Can you post the output from fl=recommended?
          Is it particular values of that field that cause the problem, or any document retrieved with that field? If particular values, perhaps you could narrow it down and just post the problematic value?

          Show
          Yonik Seeley added a comment - Perhaps escaping isn't correct for this particular field? Can you post the output from fl=recommended? Is it particular values of that field that cause the problem, or any document retrieved with that field? If particular values, perhaps you could narrow it down and just post the problematic value?
          Hide
          Andrew McCombe added a comment -

          I think the issue I have is with Solr and not the PHP output. My 'recommended' field is an integer in the solr schema.xml but when documents are added it is converting 0 and 1 to boolean values, even though I have some values at 0,1,2, 10 and 11.

          Hence the output is showing as:

          {s:2:"id";s:6:"538366";s:11:"recommended";i:false;}

          I think that PHP's unserialize function may be failing due to the wrong value for the datatype.

          Show
          Andrew McCombe added a comment - I think the issue I have is with Solr and not the PHP output. My 'recommended' field is an integer in the solr schema.xml but when documents are added it is converting 0 and 1 to boolean values, even though I have some values at 0,1,2, 10 and 11. Hence the output is showing as: {s:2:"id";s:6:"538366";s:11:"recommended";i:false;} I think that PHP's unserialize function may be failing due to the wrong value for the datatype.

            People

            • Assignee:
              Unassigned
              Reporter:
              Paul Borgermans
            • Votes:
              3 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 3h
                3h
                Remaining:
                Remaining Estimate - 3h
                3h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development