Solr
  1. Solr
  2. SOLR-3852

Admin UI - Cloud Tree ArrayIndexOutOfBoundsException if binary files anywhere in ZK tree

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0-BETA
    • Fix Version/s: 4.5, 5.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Tomcat 6, external zookeeper-3.3.5

      Description

      Original bug description indicated that when using Solr with embedded ZK everything was fine, but with an external ZK you'd get an ArrayIndexOutOfBoundsException.

      Crux of the problem is some bad assumptions about any ZK node containing data – the ZookeeperInfoServlet powering the tree view of the Cloud Admin UI assumed that any data would be utf8 text.

      If you are using extenral ZK, and other systems are writing data into ZK, then you are more likely to see this problem, because those other systems might be writing binary data in to ZK nodes – if you are using ZK embedded in solr, or using solr with it's own private (external) ZK instance, then you would only see this problem if you explicitly put binary files into solr configs and upconfig them into ZK.


      One workarround for people encountering this problem when using Solr with a ZK instance shared by other tools is to make sure you use a "chroot" patch when pointing Solr at ZK, so that it won't know about any other paths in your ZK tree that might have binary data...

      https://wiki.apache.org/solr/SolrCloud#Zookeeper_chroot

      If you are having this problem because you put binay files into your own config dir (ie: images for velocity or something like that) then there is no straight forward workarround.

      Example stack trace for this bug...

      43242 [qtp965223859-14] WARN  org.eclipse.jetty.servlet.ServletHandler /solr/zookeeper
      java.lang.ArrayIndexOutOfBoundsException: 213
              at org.apache.lucene.util.UnicodeUtil.UTF8toUTF16(UnicodeUtil.java:620)
              at org.apache.lucene.util.BytesRef.utf8ToString(BytesRef.java:168)
              at org.apache.solr.servlet.ZookeeperInfoServlet$ZKPrinter.printTree(ZookeeperInfoServlet.java:303)
              at org.apache.solr.servlet.ZookeeperInfoServlet$ZKPrinter.printTree(ZookeeperInfoServlet.java:339)
              at org.apache.solr.servlet.ZookeeperInfoServlet$ZKPrinter.printTree(ZookeeperInfoServlet.java:339)
      ...
      org.apache.solr.servlet.ZookeeperInfoServlet$ZKPrinter.print(ZookeeperInfoServlet.java:228)
              at org.apache.solr.servlet.ZookeeperInfoServlet.doGet(ZookeeperInfoServlet.java:106)
      

        Activity

        Hide
        Stefan Matheis (steffkes) added a comment -

        Mark Miller would you mind having a look here? With the given response the UI is not able to display that Tree, that's pretty sure .. but no idea what causes the ArrayIndexOutOfBounds?

        Show
        Stefan Matheis (steffkes) added a comment - Mark Miller would you mind having a look here? With the given response the UI is not able to display that Tree, that's pretty sure .. but no idea what causes the ArrayIndexOutOfBounds?
        Hide
        Mark Miller added a comment -

        Woah - i got pinged in email because you mentioned my name - never saw that before. Cool.

        Yeah, I can take a look.

        Show
        Mark Miller added a comment - Woah - i got pinged in email because you mentioned my name - never saw that before. Cool. Yeah, I can take a look.
        Hide
        Shikhar Bhushan added a comment -

        We've run into this. Was an ArrayIndexOutOfBoundsException arising out of: https://github.com/apache/lucene-solr/blob/4ce168a/solr/core/src/java/org/apache/solr/servlet/ZookeeperInfoServlet.java#L303

        We have some znodes storing binary data but that bit above assumes that if a znode has data, it'll be a UTF-8 encoded string.

        That block doesn't actually do anything post-decode, so maybe it should just be removed.

        Show
        Shikhar Bhushan added a comment - We've run into this. Was an ArrayIndexOutOfBoundsException arising out of: https://github.com/apache/lucene-solr/blob/4ce168a/solr/core/src/java/org/apache/solr/servlet/ZookeeperInfoServlet.java#L303 We have some znodes storing binary data but that bit above assumes that if a znode has data, it'll be a UTF-8 encoded string. That block doesn't actually do anything post-decode, so maybe it should just be removed.
        Hide
        Hoss Man added a comment -

        This has popped up on the mailing list again recently, so i took a look...

        As Shikhar noted, the particular exception is happening in a bit of dead code which is easy enough to remove – but that's only part of hte problem.

        the dead code is part of the tree walk of every ZK node that happens anytime you load the tree view page at all – so having any non utf8 files anywhere in your ZK tree is going to cause that exception as soon as you try to look at the Solr ZK UI – to reproduce just copy favicon.ico into your collection1/conf dir for example.

        but even if we purge that dead code, a similar bit of logic exists in the function for fetching the details about a node if someone clicks on it.

        Looking through the history of the file, it seems like this problem was introduced back in r1298010 when the code was switched to use BytesRef – the existing logic to hex escape anything that wasn't in unicode wasn't preserved – but from what i can tell, i don't think that ever worked (catching UnsupportedEncodingException from "new String(byte[],String)" doesn't seem right there according to the docs) and i'm not sure what use that would be anyway.

        In the attached patch, i purged the dead code, added some error handling around the remaining attempt to serialize the "data" of any node the user clicks on, and in the event that the data can't be converted to String add a new "dataNote" property explaining why "data" is empty (even though dataLengh is non zero)

        If no one sees any problems with this, i'll commit ASAP ... if we want to revist adding teh hex encoding output of bin conent we should probably hash that out in a new issue as an improvement, since i'm not convinced it's a good idea to just blindly do that in the same "data" field of the response ... could confuse people.

        Show
        Hoss Man added a comment - This has popped up on the mailing list again recently, so i took a look... As Shikhar noted, the particular exception is happening in a bit of dead code which is easy enough to remove – but that's only part of hte problem. the dead code is part of the tree walk of every ZK node that happens anytime you load the tree view page at all – so having any non utf8 files anywhere in your ZK tree is going to cause that exception as soon as you try to look at the Solr ZK UI – to reproduce just copy favicon.ico into your collection1/conf dir for example. but even if we purge that dead code, a similar bit of logic exists in the function for fetching the details about a node if someone clicks on it. Looking through the history of the file, it seems like this problem was introduced back in r1298010 when the code was switched to use BytesRef – the existing logic to hex escape anything that wasn't in unicode wasn't preserved – but from what i can tell, i don't think that ever worked (catching UnsupportedEncodingException from "new String(byte[],String)" doesn't seem right there according to the docs) and i'm not sure what use that would be anyway. In the attached patch, i purged the dead code, added some error handling around the remaining attempt to serialize the "data" of any node the user clicks on, and in the event that the data can't be converted to String add a new "dataNote" property explaining why "data" is empty (even though dataLengh is non zero) If no one sees any problems with this, i'll commit ASAP ... if we want to revist adding teh hex encoding output of bin conent we should probably hash that out in a new issue as an improvement, since i'm not convinced it's a good idea to just blindly do that in the same "data" field of the response ... could confuse people.
        Hide
        Hoss Man added a comment -

        updated description with some notes and workaround for common trigger of underlying problem.

        (thanks to joeo's comments on the solr mailing list for pointing out hte work arround)

        Show
        Hoss Man added a comment - updated description with some notes and workaround for common trigger of underlying problem. (thanks to joeo's comments on the solr mailing list for pointing out hte work arround)
        Hide
        ASF subversion and git services added a comment -

        Commit 1519763 from hossman@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1519763 ]

        SOLR-3852: Fixed ZookeeperInfoServlet so that the SolrCloud Admin UI pages will work even if ZK contains nodes with data which are not utf8 text

        Show
        ASF subversion and git services added a comment - Commit 1519763 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1519763 ] SOLR-3852 : Fixed ZookeeperInfoServlet so that the SolrCloud Admin UI pages will work even if ZK contains nodes with data which are not utf8 text
        Hide
        ASF subversion and git services added a comment -

        Commit 1519779 from hossman@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1519779 ]

        SOLR-3852: Fixed ZookeeperInfoServlet so that the SolrCloud Admin UI pages will work even if ZK contains nodes with data which are not utf8 text (merge r1519763)

        Show
        ASF subversion and git services added a comment - Commit 1519779 from hossman@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1519779 ] SOLR-3852 : Fixed ZookeeperInfoServlet so that the SolrCloud Admin UI pages will work even if ZK contains nodes with data which are not utf8 text (merge r1519763)
        Hide
        Adrien Grand added a comment -

        4.5 release -> bulk close

        Show
        Adrien Grand added a comment - 4.5 release -> bulk close

          People

          • Assignee:
            Hoss Man
            Reporter:
            Vadim Kisselmann
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development