Solr
  1. Solr
  2. SOLR-4729

Using a copyField with * as the source doesn't work with LukeRequestHandler - breaks admin UI

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.2
    • Fix Version/s: 4.3, 6.0
    • Component/s: Schema and Analysis
    • Labels:
      None

      Description

      It seems you can no longer use a wildcard as the source when defining a copyField. I don't believe that this was fixed as part of SOLR-4650 since I've tested it with the 4/17 nightly build and it doesn't work.

      I'm using the following line: <copyField source="*" dest="text"/>

      If I index something, this line is ignored. If I go to the Analysis tab, the fields aren't populated and I see the error: 'org.apache.solr.common.SolrException: undefined field: "*"' in the log.

      This worked correctly in 4.0, but I didn't test it in 4.1.

        Issue Links

          Activity

          Hide
          Hoss Man added a comment -

          Adam: we're going to need more details on exactly what svn branch & revision you're testing, what exactly your schema looks like, and how exactly you generated that Exception: what exactly did you do in the analysis tab, and what appeared in solr's logs arround that exception (eg: the underlying request to solr made by the UI, the log messages from that request, and the full stack of hte exception.)

          I just committed a test demonstrating that a source="*" copyfield works, so i'm fairly certain that SOLR-4650 fixed this – but if you're still getting errors then there may be some edge case here we're not understanding...

          Committed revision 1469529.
          Committed revision 1469533.
          Committed revision 1469534.

          Show
          Hoss Man added a comment - Adam: we're going to need more details on exactly what svn branch & revision you're testing, what exactly your schema looks like, and how exactly you generated that Exception: what exactly did you do in the analysis tab, and what appeared in solr's logs arround that exception (eg: the underlying request to solr made by the UI, the log messages from that request, and the full stack of hte exception.) I just committed a test demonstrating that a source="*" copyfield works, so i'm fairly certain that SOLR-4650 fixed this – but if you're still getting errors then there may be some edge case here we're not understanding... Committed revision 1469529. Committed revision 1469533. Committed revision 1469534.
          Hide
          Adam Hahn added a comment -

          I've just tested again and I'm still seeing the error using the solr-4.1-2013-04-22_10-31-16 nightly build. This should include the fix right?

          Environment: I've tested in both Windows and Linux. Solr is launched using the copy of Jetty included.

          Steps taken:
          1. Unzipped Solr
          2. Modified the example/solr/collection1/conf/schema.xml file to add the line
          <copyField source="*" dest="text"/> after the other copyField lines and just before the start of the "types" block
          3. Opened up the Solr Admin page, selected the collection1 core from the dropdown and clicked on Analysis.

          At this point the "Analyse Fieldname / FieldType" shows "Loading ...". The log file shows the error mentioned in the description. The log only contains this one line and not a full stack trace. I do see a stack trace in the terminal where I started Solr. This is shown below. Using Firebug, it appears that the call that's causing this is: http://localhost:8983/solr/collection1/admin/luke?wt=json&show=schema&_=1366655832433

          I've also tried commenting out the other copyFields in case that was causing an issue, but it didn't resolve it.

          Stack trace from terminal:
          8447 [qtp811887233-14] INFO org.apache.solr.servlet.SolrDispatchFilter â [admin] webapp=null path=/admin/cores params=

          {indexInfo=false&_=1366655616048&wt=json}

          status=0 QTime=1
          8477 [qtp811887233-14] INFO org.apache.solr.core.SolrCore â [collection1] webapp=/solr path=/admin/system params={_=1366655616075&wt=json} status=0 QTime=7
          8526 [qtp811887233-14] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: undefined field: "*"
          at org.apache.solr.schema.IndexSchema.getField(IndexSchema.java:1142)
          at org.apache.solr.schema.IndexSchema.getCopySources(IndexSchema.java:1242)
          at org.apache.solr.handler.admin.LukeRequestHandler.populateFieldInfo(LukeRequestHandler.java:528)
          at org.apache.solr.handler.admin.LukeRequestHandler.getSchemaInfo(LukeRequestHandler.java:415)
          at org.apache.solr.handler.admin.LukeRequestHandler.handleRequestBody(LukeRequestHandler.java:157)
          at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
          at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
          at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:642)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
          at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
          at org.eclipse.jetty.server.Server.handle(Server.java:365)
          at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
          at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
          at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
          at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
          at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
          at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
          at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
          at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
          at java.lang.Thread.run(Thread.java:722)

          8527 [qtp811887233-14] INFO org.apache.solr.core.SolrCore â [collection1] webapp=/solr path=/admin/luke params={_=1366655616125&show=schema&wt=json} status=400 QTime=6

          Show
          Adam Hahn added a comment - I've just tested again and I'm still seeing the error using the solr-4.1-2013-04-22_10-31-16 nightly build. This should include the fix right? Environment: I've tested in both Windows and Linux. Solr is launched using the copy of Jetty included. Steps taken: 1. Unzipped Solr 2. Modified the example/solr/collection1/conf/schema.xml file to add the line <copyField source="*" dest="text"/> after the other copyField lines and just before the start of the "types" block 3. Opened up the Solr Admin page, selected the collection1 core from the dropdown and clicked on Analysis. At this point the "Analyse Fieldname / FieldType" shows "Loading ...". The log file shows the error mentioned in the description. The log only contains this one line and not a full stack trace. I do see a stack trace in the terminal where I started Solr. This is shown below. Using Firebug, it appears that the call that's causing this is: http://localhost:8983/solr/collection1/admin/luke?wt=json&show=schema&_=1366655832433 I've also tried commenting out the other copyFields in case that was causing an issue, but it didn't resolve it. Stack trace from terminal: 8447 [qtp811887233-14] INFO org.apache.solr.servlet.SolrDispatchFilter â [admin] webapp=null path=/admin/cores params= {indexInfo=false&_=1366655616048&wt=json} status=0 QTime=1 8477 [qtp811887233-14] INFO org.apache.solr.core.SolrCore â [collection1] webapp=/solr path=/admin/system params={_=1366655616075&wt=json} status=0 QTime=7 8526 [qtp811887233-14] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: undefined field: "*" at org.apache.solr.schema.IndexSchema.getField(IndexSchema.java:1142) at org.apache.solr.schema.IndexSchema.getCopySources(IndexSchema.java:1242) at org.apache.solr.handler.admin.LukeRequestHandler.populateFieldInfo(LukeRequestHandler.java:528) at org.apache.solr.handler.admin.LukeRequestHandler.getSchemaInfo(LukeRequestHandler.java:415) at org.apache.solr.handler.admin.LukeRequestHandler.handleRequestBody(LukeRequestHandler.java:157) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:642) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:365) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:722) 8527 [qtp811887233-14] INFO org.apache.solr.core.SolrCore â [collection1] webapp=/solr path=/admin/luke params={_=1366655616125&show=schema&wt=json} status=400 QTime=6
          Hide
          Hoss Man added a comment -

          Adam: thanks for following up...

          I've just tested again and I'm still seeing the error using the solr-4.1-2013-04-22_10-31-16 nightly build. This should include the fix right?

          FYI: I opened LUCENE-4949 to address the problem in how the nightly builds are being named

          3. Opened up the Solr Admin page, selected the collection1 core from the dropdown and clicked on Analysis. ...

          thank you for that specific piece of info and the included stack trace – i can in fact reproduce this error you are getting. there is definitely a bug in the LukeRequestHandler that will need addressed that prevents it (and thus the analysis screen) from working properly with this type of a copyField wildcard.

          A possible work around until this bug can be fixed is to add a dynimcField matching "*" like below to your schema.xml – although this will prevent errors from being reported if you inadvertently spell a field name incorrectly when indexing/quering, so it may be best to only do this in dev setups where you need to test out analysis options...

             <dynamicField name="*" type="ignored" />
          

          For the moment however i'm much more concerned by thiscomment from your original bug report (and general summary of your bug report) that i don't understand...

          If I index something, this line is ignored.

          ("copyField with * as the source doesn't work")

          As far as i can tell, there is no actual bug here preventing the copyField from working properly when indexing documents, all values from all fields do in fact get added to the "text" field – it's only the LukeRequestHandler & admin UI that seem to have problems... correct? (or are their other errors you are seeing that you haven't mentioned yet?)

          Show
          Hoss Man added a comment - Adam: thanks for following up... I've just tested again and I'm still seeing the error using the solr-4.1-2013-04-22_10-31-16 nightly build. This should include the fix right? FYI: I opened LUCENE-4949 to address the problem in how the nightly builds are being named 3. Opened up the Solr Admin page, selected the collection1 core from the dropdown and clicked on Analysis. ... thank you for that specific piece of info and the included stack trace – i can in fact reproduce this error you are getting. there is definitely a bug in the LukeRequestHandler that will need addressed that prevents it (and thus the analysis screen) from working properly with this type of a copyField wildcard. A possible work around until this bug can be fixed is to add a dynimcField matching "*" like below to your schema.xml – although this will prevent errors from being reported if you inadvertently spell a field name incorrectly when indexing/quering, so it may be best to only do this in dev setups where you need to test out analysis options... <dynamicField name="*" type="ignored" /> For the moment however i'm much more concerned by thiscomment from your original bug report (and general summary of your bug report) that i don't understand... If I index something, this line is ignored. ("copyField with * as the source doesn't work") As far as i can tell, there is no actual bug here preventing the copyField from working properly when indexing documents, all values from all fields do in fact get added to the "text" field – it's only the LukeRequestHandler & admin UI that seem to have problems... correct? (or are their other errors you are seeing that you haven't mentioned yet?)
          Hide
          Adam Hahn added a comment -

          Thanks for the quick response. I'm glad to see that I wasn't crazy.

          As for the part about it ignoring the copyField line, that seems to be my mistake. Once I saw the error with the analysis page using my custom schema, I switched to the collection1 schema for testing. Unfortunately, I didn't pay attention to the fact that "text" is indexed, but not stored. Obviously this makes it hard to return the value in a query.

          I've edited the description to remove the false error. Do you want me to edit this issue to make it specific to the LukeRequestHandler or create a new task?

          Thanks for your help...and also for the workaround. I just tried it and it will work perfect for what I need.

          -Adam

          Show
          Adam Hahn added a comment - Thanks for the quick response. I'm glad to see that I wasn't crazy. As for the part about it ignoring the copyField line, that seems to be my mistake. Once I saw the error with the analysis page using my custom schema, I switched to the collection1 schema for testing. Unfortunately, I didn't pay attention to the fact that "text" is indexed, but not stored. Obviously this makes it hard to return the value in a query. I've edited the description to remove the false error. Do you want me to edit this issue to make it specific to the LukeRequestHandler or create a new task? Thanks for your help...and also for the workaround. I just tried it and it will work perfect for what I need. -Adam
          Hide
          Steve Rowe added a comment -

          The problem was this added code as part of SOLR-4503 in IndexSchema.getCopySources():

              for (DynamicCopy dynamicCopy : dynamicCopyFields) {
                if (dynamicCopy.getDestFieldName().equals(destField)) {
                  sf.add(getField(dynamicCopy.getRegex()));
                }
              }
           

          (I added this code because the javadocs said that dynamic fields were returned but that wasn't happening.)

          The problem was that getField("(glob)") will fail if "(glob)" isn't also a separately defined dynamic field.

          The attached patch fixes the error and adds a test that triggers it.

          Committing shortly.

          Show
          Steve Rowe added a comment - The problem was this added code as part of SOLR-4503 in IndexSchema.getCopySources(): for (DynamicCopy dynamicCopy : dynamicCopyFields) { if (dynamicCopy.getDestFieldName().equals(destField)) { sf.add(getField(dynamicCopy.getRegex())); } } (I added this code because the javadocs said that dynamic fields were returned but that wasn't happening.) The problem was that getField("(glob)") will fail if "(glob)" isn't also a separately defined dynamic field. The attached patch fixes the error and adds a test that triggers it. Committing shortly.
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] sarowe
          http://svn.apache.org/viewvc?view=revision&revision=1470820

          SOLR-4729: LukeRequestHandler: Using a dynamic copyField source that is not also a dynamic field triggers error message 'undefined field: "(glob)"'

          Show
          Commit Tag Bot added a comment - [trunk commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1470820 SOLR-4729 : LukeRequestHandler: Using a dynamic copyField source that is not also a dynamic field triggers error message 'undefined field: "(glob)"'
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] sarowe
          http://svn.apache.org/viewvc?view=revision&revision=1470821

          SOLR-4729: LukeRequestHandler: Using a dynamic copyField source that is not also a dynamic field triggers error message 'undefined field: "(glob)"' (merged trunk r1470820)

          Show
          Commit Tag Bot added a comment - [branch_4x commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1470821 SOLR-4729 : LukeRequestHandler: Using a dynamic copyField source that is not also a dynamic field triggers error message 'undefined field: "(glob)"' (merged trunk r1470820)
          Hide
          Steve Rowe added a comment -

          Committed to trunk r1470820 and branch_4x r1470821.

          Thanks Hoss and Adam!

          Show
          Steve Rowe added a comment - Committed to trunk r1470820 and branch_4x r1470821. Thanks Hoss and Adam!
          Hide
          Hoss Man added a comment -

          updating summary to reflect scope of issue

          Show
          Hoss Man added a comment - updating summary to reflect scope of issue
          Hide
          Steve Rowe added a comment -

          I forgot to include this in the respin - I'll commit to lucene_solr_4_3, hopefully Simon hasn't yet started an RC.

          Show
          Steve Rowe added a comment - I forgot to include this in the respin - I'll commit to lucene_solr_4_3, hopefully Simon hasn't yet started an RC.
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] sarowe
          http://svn.apache.org/viewvc?view=revision&revision=1476491

          SOLR-4729: Move CHANGES.txt entry to 4.3.0

          Show
          Commit Tag Bot added a comment - [trunk commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1476491 SOLR-4729 : Move CHANGES.txt entry to 4.3.0
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] sarowe
          http://svn.apache.org/viewvc?view=revision&revision=1476492

          SOLR-4729: Move CHANGES.txt entry to 4.3.0 (merged trunk r1476491)

          Show
          Commit Tag Bot added a comment - [branch_4x commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1476492 SOLR-4729 : Move CHANGES.txt entry to 4.3.0 (merged trunk r1476491)
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] sarowe
          http://svn.apache.org/viewvc?view=revision&revision=1476493

          SOLR-4729: Remove CHANGES.txt entry from 4.4.0

          Show
          Commit Tag Bot added a comment - [trunk commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1476493 SOLR-4729 : Remove CHANGES.txt entry from 4.4.0
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] sarowe
          http://svn.apache.org/viewvc?view=revision&revision=1476495

          SOLR-4729: Remove CHANGES.txt entry from 4.4.0 (merge trunk r1476493)

          Show
          Commit Tag Bot added a comment - [branch_4x commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1476495 SOLR-4729 : Remove CHANGES.txt entry from 4.4.0 (merge trunk r1476493)
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] sarowe
          http://svn.apache.org/viewvc?view=revision&revision=1476496

          SOLR-4729: exit loop after finding expected dynamic field definition in LukeRequestHandlerTest.testCatchAllCopyField()

          Show
          Commit Tag Bot added a comment - [trunk commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1476496 SOLR-4729 : exit loop after finding expected dynamic field definition in LukeRequestHandlerTest.testCatchAllCopyField()
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] sarowe
          http://svn.apache.org/viewvc?view=revision&revision=1476497

          SOLR-4729: exit loop after finding expected dynamic field definition in LukeRequestHandlerTest.testCatchAllCopyField() (merged trunk r1476491)

          Show
          Commit Tag Bot added a comment - [branch_4x commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1476497 SOLR-4729 : exit loop after finding expected dynamic field definition in LukeRequestHandlerTest.testCatchAllCopyField() (merged trunk r1476491)
          Hide
          Commit Tag Bot added a comment -

          [lucene_solr_4_3 commit] sarowe
          http://svn.apache.org/viewvc?view=revision&revision=1476498

          SOLR-4729: LukeRequestHandler: Using a dynamic copyField source that is not also a dynamic field triggers error message 'undefined field: "(glob)"' (merged trunk r1470820, r1476491, r1476493, and 1476496)

          Show
          Commit Tag Bot added a comment - [lucene_solr_4_3 commit] sarowe http://svn.apache.org/viewvc?view=revision&revision=1476498 SOLR-4729 : LukeRequestHandler: Using a dynamic copyField source that is not also a dynamic field triggers error message 'undefined field: "(glob)"' (merged trunk r1470820, r1476491, r1476493, and 1476496)
          Hide
          Steve Rowe added a comment -

          Committed to lucene_solr_4_3 and changed fix version from 4.4 to 4.3.

          Show
          Steve Rowe added a comment - Committed to lucene_solr_4_3 and changed fix version from 4.4 to 4.3.
          Hide
          Uwe Schindler added a comment -

          Closed after release.

          Show
          Uwe Schindler added a comment - Closed after release.

            People

            • Assignee:
              Steve Rowe
              Reporter:
              Adam Hahn
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development