Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.0
    • Component/s: clients - java, search
    • Labels:
      None

      Description

      This ticket provides support for executing Parallel SQL queries across SolrCloud collections. The SQL engine will be built on top of the Streaming API (SOLR-7082), which provides support for parallel relational algebra and real-time map-reduce.

      Basic design:

      1) A new SQLHandler will be added to process SQL requests. The SQL statements will be compiled to live Streaming API objects for parallel execution across SolrCloud worker nodes.

      2) SolrCloud collections will be abstracted as Relational Tables.

      3) The Presto SQL parser will be used to parse the SQL statements.

      4) A JDBC thin client will be added as a Solrj client.

      This ticket will focus on putting the framework in place and providing basic SELECT support and GROUP BY aggregate support.

      Future releases will build on this framework to provide additional SQL features.

      1. SOLR-7560.calcite.patch
        29 kB
        Joel Bernstein
      2. SOLR-7560.patch
        156 kB
        Joel Bernstein
      3. SOLR-7560.patch
        125 kB
        Joel Bernstein
      4. SOLR-7560.patch
        118 kB
        Joel Bernstein
      5. SOLR-7560.patch
        99 kB
        Joel Bernstein

        Issue Links

          Activity

          Hide
          noble.paul Noble Paul added a comment -

          Looking forward to this

          Can you post some example queries

          Show
          noble.paul Noble Paul added a comment - Looking forward to this Can you post some example queries
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Still working on the details of the queries. I will post more information with the patch.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Still working on the details of the queries. I will post more information with the patch.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Initial patch implements parallel Group By with aggregates. Very basic tests included. It shows the strategy of compiling SQL statements to Streaming API objects.

          SQL Tables are mapped to SolrCloud collections using a simple table mapping configured in the SolrConfig.

          Sample Group By:

          select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from mytable where text='XXXX' group by str_s order by sum(field_i) asc limit 2
          

          The RollupStream and metric package have been added to the Streaming API to perform the Map/Reduce aggregations.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Initial patch implements parallel Group By with aggregates. Very basic tests included. It shows the strategy of compiling SQL statements to Streaming API objects. SQL Tables are mapped to SolrCloud collections using a simple table mapping configured in the SolrConfig. Sample Group By: select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from mytable where text='XXXX' group by str_s order by sum(field_i) asc limit 2 The RollupStream and metric package have been added to the Streaming API to perform the Map/Reduce aggregations.
          Hide
          dpgove Dennis Gove added a comment -

          Possible expression syntax for the RollupStream

          rollup(
            someStream(....),
            over="fieldA, fieldB, fieldC",
            min(fieldA),
            max(fieldA),
            min(fieldB),
            mean(fieldD),
            sum(fieldC)
          )
          

          This would require making the *Metric types Expressible but I think that ends up as a good thing. Would make it real easy to support other options on metrics like excluding outliers, for example find the sum of values within 3 standard deviations from the mean could be

          sum(fieldC, limit=standardDev(3))
          

          (note, how that particular calculation could be implemented is left as an exercise for the reader, I'm just using it as an example of adding additional options on a relatively simple metric).
          Another option example is what to do with null values. For example, in some cases a null should not impact a mean but in others it should. You could express those as

          mean(fieldA, replace(null, 0))  // replace null values with 0 thus leading to an impact on the mean
          mean(fieldA, includeNull="true") // nulls are counted in the denominator but nothing added to numerator
          mean(fieldA, includeNull="false") // nulls neither counted in denominator nor added to numerator
          mean(fieldA, replace(null, fieldB), includeNull="true") // if fieldA is null replace it with fieldB, include null fieldB in mean
          

          so on and so forth.

          Show
          dpgove Dennis Gove added a comment - Possible expression syntax for the RollupStream rollup( someStream(....), over= "fieldA, fieldB, fieldC" , min(fieldA), max(fieldA), min(fieldB), mean(fieldD), sum(fieldC) ) This would require making the *Metric types Expressible but I think that ends up as a good thing. Would make it real easy to support other options on metrics like excluding outliers, for example find the sum of values within 3 standard deviations from the mean could be sum(fieldC, limit=standardDev(3)) (note, how that particular calculation could be implemented is left as an exercise for the reader, I'm just using it as an example of adding additional options on a relatively simple metric). Another option example is what to do with null values. For example, in some cases a null should not impact a mean but in others it should. You could express those as mean(fieldA, replace( null , 0)) // replace null values with 0 thus leading to an impact on the mean mean(fieldA, includeNull= " true " ) // nulls are counted in the denominator but nothing added to numerator mean(fieldA, includeNull= " false " ) // nulls neither counted in denominator nor added to numerator mean(fieldA, replace( null , fieldB), includeNull= " true " ) // if fieldA is null replace it with fieldB, include null fieldB in mean so on and so forth.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Looks great!

          A little background on this discussion. Dennis Gove is describing the Streaming Expression (SOLR-7377) syntax for the RollupStream which is being added in this ticket to support the SQL aggregate functions. The RollupStream does Map/Reduce style aggregations where group by fields are sorted first and the aggregates are rolled up one group at a time. This technique will be very strong for time series rollups and aggregating high cardinality fields.

          There will be other aggregation streams added in the future that tap into Solr faceting directly.

          Show
          joel.bernstein Joel Bernstein added a comment - Looks great! A little background on this discussion. Dennis Gove is describing the Streaming Expression ( SOLR-7377 ) syntax for the RollupStream which is being added in this ticket to support the SQL aggregate functions. The RollupStream does Map/Reduce style aggregations where group by fields are sorted first and the aggregates are rolled up one group at a time. This technique will be very strong for time series rollups and aggregating high cardinality fields. There will be other aggregation streams added in the future that tap into Solr faceting directly.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          New patch with support for complex boolean WHERE clauses and accompanying tests.

          Simple SELECT statements (without aggregates functions) are also now supported.

          More tests added for the GROUP BY (with aggregate functions) queries that demonstrate parallel time series aggregation.

          I believe this patch is now functionally complete.

          Thought about adding the HAVING clause and ALIASES but I think this is functional enough to be useful for an initial release.

          Some more cleanup work to follow.

          Show
          joel.bernstein Joel Bernstein added a comment - New patch with support for complex boolean WHERE clauses and accompanying tests. Simple SELECT statements ( without aggregates functions) are also now supported. More tests added for the GROUP BY ( with aggregate functions) queries that demonstrate parallel time series aggregation. I believe this patch is now functionally complete. Thought about adding the HAVING clause and ALIASES but I think this is functional enough to be useful for an initial release. Some more cleanup work to follow.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          New Patch adding support for the HAVING clause of GROUP BY statements.

          Sample syntax:

          select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from mytable where text='XXXX' group by str_s having ((sum(field_i) = 19) AND (min(field_i) = 8))
          
          Show
          joel.bernstein Joel Bernstein added a comment - New Patch adding support for the HAVING clause of GROUP BY statements. Sample syntax: select str_s, count(*), sum(field_i), min(field_i), max(field_i), avg(field_i) from mytable where text='XXXX' group by str_s having ((sum(field_i) = 19) AND (min(field_i) = 8))
          Hide
          joel.bernstein Joel Bernstein added a comment -

          New patch with all tests and precommit passing.

          Show
          joel.bernstein Joel Bernstein added a comment - New patch with all tests and precommit passing.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          There is still some more work to do before it's ready for a release but I believe this ticket is far enough along to commit to trunk.

          The main goals have been accomplished:

          Two SQL constructs are supported:

          1) SELECT statements with complex nested WHERE clauses, ORDER BY and LIMIT.
          2) GROUP BY statements with aggregate functions, complex nested WHERE and HAVING clauses, ORDER BY and LIMIT. The functions supported are SUM, COUNT, AVG, MIN, MAX.

          The GROUP BY aggregations can be run in parallel across a SolrCloud worker collection, providing a true Parallel SQL framework for SolrCloud.

          A new /sql request (SQLHandler.java) has been created to handle SQL requests and stream back the results.

          Initial test cases demonstrate the functionality described above.

          The excellent Presto SQL parser has been integrated into the SQLHandler and all the licenses for new dependencies have been added to the project.

          The JDBC driver has not been built as part of this ticket so the SQLHandler's HTTP interface will need to be accessed directly.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited There is still some more work to do before it's ready for a release but I believe this ticket is far enough along to commit to trunk . The main goals have been accomplished: Two SQL constructs are supported: 1) SELECT statements with complex nested WHERE clauses, ORDER BY and LIMIT. 2) GROUP BY statements with aggregate functions, complex nested WHERE and HAVING clauses, ORDER BY and LIMIT. The functions supported are SUM, COUNT, AVG, MIN, MAX. The GROUP BY aggregations can be run in parallel across a SolrCloud worker collection, providing a true Parallel SQL framework for SolrCloud. A new /sql request (SQLHandler.java) has been created to handle SQL requests and stream back the results. Initial test cases demonstrate the functionality described above. The excellent Presto SQL parser has been integrated into the SQLHandler and all the licenses for new dependencies have been added to the project. The JDBC driver has not been built as part of this ticket so the SQLHandler's HTTP interface will need to be accessed directly.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1685497 from Joel Bernstein in branch 'dev/trunk'
          [ https://svn.apache.org/r1685497 ]

          SOLR-7560: Parallel SQL Support

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1685497 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1685497 ] SOLR-7560 : Parallel SQL Support
          Hide
          cpoerschke Christine Poerschke added a comment -

          Hello, was experiencing 'ant resolve' difficulties related to the com.facebook.presto dependency that was added above.

          We have tried a few things and just adding conf="compile->*" attributes for the new dependencies in solr/core/ivy.xml makes it work (this is in combination with customised lucene/ivy-settings.xml though).

          -    <dependency org="com.facebook.presto" name="presto-parser" rev="${/com.facebook.presto/presto-parser}"/>
          -    <dependency org="org.antlr" name="antlr4-runtime" rev="${/org.antlr/antlr4-runtime}"/>
          -    <dependency org="io.airlift" name="slice" rev="${/io.airlift/slice}"/>
          +    <dependency org="com.facebook.presto" name="presto-parser" rev="${/com.facebook.presto/presto-parser}" conf="compile->*"/>
          +    <dependency org="org.antlr" name="antlr4-runtime" rev="${/org.antlr/antlr4-runtime}" conf="compile->*"/>
          +    <dependency org="io.airlift" name="slice" rev="${/io.airlift/slice}" conf="compile->*"/>
          
          Show
          cpoerschke Christine Poerschke added a comment - Hello, was experiencing 'ant resolve' difficulties related to the com.facebook.presto dependency that was added above. We have tried a few things and just adding conf="compile->*" attributes for the new dependencies in solr/core/ivy.xml makes it work (this is in combination with customised lucene/ivy-settings.xml though). - <dependency org= "com.facebook.presto" name= "presto-parser" rev= "${/com.facebook.presto/presto-parser}" /> - <dependency org= "org.antlr" name= "antlr4-runtime" rev= "${/org.antlr/antlr4-runtime}" /> - <dependency org= "io.airlift" name= "slice" rev= "${/io.airlift/slice}" /> + <dependency org= "com.facebook.presto" name= "presto-parser" rev= "${/com.facebook.presto/presto-parser}" conf= "compile->*" /> + <dependency org= "org.antlr" name= "antlr4-runtime" rev= "${/org.antlr/antlr4-runtime}" conf= "compile->*" /> + <dependency org= "io.airlift" name= "slice" rev= "${/io.airlift/slice}" conf= "compile->*" />
          Hide
          joel.bernstein Joel Bernstein added a comment -

          It doesn't seem to be breaking jenkins. I'll clear my ivy cache and see if I can reproduce the errors.

          Show
          joel.bernstein Joel Bernstein added a comment - It doesn't seem to be breaking jenkins. I'll clear my ivy cache and see if I can reproduce the errors.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Wasn't able to reproduce. I blew away the ivy caches, did a fresh trunk pull, and the presto dependancies were pulled right down during an "ant compile".

          I'll do a little more research ...

          Show
          joel.bernstein Joel Bernstein added a comment - Wasn't able to reproduce. I blew away the ivy caches, did a fresh trunk pull, and the presto dependancies were pulled right down during an "ant compile". I'll do a little more research ...
          Hide
          cpoerschke Christine Poerschke added a comment -

          Looked into and learnt some more on this. The original 'ant resolve' error was

          [ivy:retrieve]          ::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:retrieve]          ::          UNRESOLVED DEPENDENCIES         ::
          [ivy:retrieve]          ::::::::::::::::::::::::::::::::::::::::::::::
          [ivy:retrieve]          :: com.facebook.presto#presto-parser;0.107: configuration not found in com.facebook.presto#presto-parser;0.107: 'master'. It was required from org.apache.solr#core;working@hostname compile.hadoop
          [ivy:retrieve]          ::::::::::::::::::::::::::::::::::::::::::::::
          

          and i found http://wrongnotes.blogspot.co.uk/2014/02/simplest-explanation-of-ivy.html useful to understand the meaning of the error and what the conf="compile->*" attribute addition means (still in combination with customised lucene/ivy-settings.xml though).

          Following on from that, i cleared my ivy cache and tried this alternative, shorter dependency declaration

          -    <dependency org="com.facebook.presto" name="presto-parser" rev="${/com.facebook.presto/presto-parser}"/>
          -    <dependency org="org.antlr" name="antlr4-runtime" rev="${/org.antlr/antlr4-runtime}"/>
          -    <dependency org="io.airlift" name="slice" rev="${/io.airlift/slice}"/>
          +    <dependency org="com.facebook.presto" name="presto-parser" rev="${/com.facebook.presto/presto-parser}" conf="compile.hadoop->*"/>
          

          and that also worked but since the https://repo1.maven.org/maven2/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.pom file has additional dependencies apart from org.antlr antlr4-runtime and io.airlift slice the shorter alternative is perhaps/probably not equivalent to the original dependency trio.

          Show
          cpoerschke Christine Poerschke added a comment - Looked into and learnt some more on this. The original 'ant resolve' error was [ivy:retrieve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:retrieve] :: UNRESOLVED DEPENDENCIES :: [ivy:retrieve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:retrieve] :: com.facebook.presto#presto-parser;0.107: configuration not found in com.facebook.presto#presto-parser;0.107: 'master'. It was required from org.apache.solr#core;working@hostname compile.hadoop [ivy:retrieve] :::::::::::::::::::::::::::::::::::::::::::::: and i found http://wrongnotes.blogspot.co.uk/2014/02/simplest-explanation-of-ivy.html useful to understand the meaning of the error and what the conf="compile->*" attribute addition means (still in combination with customised lucene/ivy-settings.xml though). Following on from that, i cleared my ivy cache and tried this alternative, shorter dependency declaration - <dependency org= "com.facebook.presto" name= "presto-parser" rev= "${/com.facebook.presto/presto-parser}" /> - <dependency org= "org.antlr" name= "antlr4-runtime" rev= "${/org.antlr/antlr4-runtime}" /> - <dependency org= "io.airlift" name= "slice" rev= "${/io.airlift/slice}" /> + <dependency org= "com.facebook.presto" name= "presto-parser" rev= "${/com.facebook.presto/presto-parser}" conf= "compile.hadoop->*" /> and that also worked but since the https://repo1.maven.org/maven2/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.pom file has additional dependencies apart from org.antlr antlr4-runtime and io.airlift slice the shorter alternative is perhaps/probably not equivalent to the original dependency trio.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Can you paste the steps you're using to reproduce the error into the ticket?

          Show
          joel.bernstein Joel Bernstein added a comment - Can you paste the steps you're using to reproduce the error into the ticket?
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Just blew away the ivy cache, did a fresh trunk checkout and 'ant compile'. The compile was successful but I did get these warnings:

          ivy:retrieve] :::: WARNINGS
          [ivy:retrieve] problem while downloading module descriptor: http://repo1.maven.org/maven2/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.pom: invalid sha1: expected=7f2f854a24db4cc9b5c26c3e9c6e46da37abbed1 computed=661fee4927f5b356f91880c308e5bf586568ff3c (180ms)
          [ivy:retrieve] problem while downloading module descriptor: http://repo1.maven.org/maven2/com/facebook/presto/presto-root/0.107/presto-root-0.107.pom: invalid sha1: expected=3e2a2729fd8b728fdc29c4062f5dc9a138c433cd computed=2cdf6c93caab02a26e6280862ed25452b862e5bf (218ms)
          [ivy:retrie

          I'll take a look and see what the issue is.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Just blew away the ivy cache, did a fresh trunk checkout and 'ant compile'. The compile was successful but I did get these warnings: ivy:retrieve] :::: WARNINGS [ivy:retrieve] problem while downloading module descriptor: http://repo1.maven.org/maven2/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.pom: invalid sha1: expected=7f2f854a24db4cc9b5c26c3e9c6e46da37abbed1 computed=661fee4927f5b356f91880c308e5bf586568ff3c (180ms) [ivy:retrieve] problem while downloading module descriptor: http://repo1.maven.org/maven2/com/facebook/presto/presto-root/0.107/presto-root-0.107.pom: invalid sha1: expected=3e2a2729fd8b728fdc29c4062f5dc9a138c433cd computed=2cdf6c93caab02a26e6280862ed25452b862e5bf (218ms) [ivy:retrie I'll take a look and see what the issue is.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited
          Show
          joel.bernstein Joel Bernstein added a comment - - edited It appears that sha1 files are incorrect for: http://repo1.maven.org/maven2/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.pom http://repo1.maven.org/maven2/com/facebook/presto/presto-root/0.107/presto-root-0.107.pom Wondering if this is causing your errors? I'm just seeing warnings.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          It looks like the issue needs to be fixed on the Presto project. I will notify the project.

          Show
          joel.bernstein Joel Bernstein added a comment - It looks like the issue needs to be fixed on the Presto project. I will notify the project.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Here is the ticket on the presto-parser project:
          https://github.com/facebook/presto/issues/3126

          Show
          joel.bernstein Joel Bernstein added a comment - Here is the ticket on the presto-parser project: https://github.com/facebook/presto/issues/3126
          Hide
          cpoerschke Christine Poerschke added a comment -

          Thanks Joel Bernstein, i will follow the presto-parser project ticket then from here on.

          (Hadn't mentioned before but changing lucene/ivy-versions.properties's /com.facebook.presto/presto-parser from 0.107 to 0.105 worked i.e. only 0.107 and 0.106 caused the error and 0.105 worked without 'conf' attribute addition i.e. retrospectively that would then also point towards a difference/issue w.r.t. presto-parser project.)

          Show
          cpoerschke Christine Poerschke added a comment - Thanks Joel Bernstein , i will follow the presto-parser project ticket then from here on. (Hadn't mentioned before but changing lucene/ivy-versions.properties 's /com.facebook.presto/presto-parser from 0.107 to 0.105 worked i.e. only 0.107 and 0.106 caused the error and 0.105 worked without 'conf' attribute addition i.e. retrospectively that would then also point towards a difference/issue w.r.t. presto-parser project.)
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Christine Poerschke, thanks for the comments and the link was very informative. Just tried another fresh pull and the warnings are still there so the SHA1 files haven't been updated yet.

          A couple things I'd like to understand better based on your comments:

          1) How come your builds were failing, while mine were successful with warnings? Are you building with Maven?

          2) Before a release it will be important to nail down the correct ivy conf. What should the scope be and whether to include all the Presto Parser decencies or just the ones being used in this ticket. This ticket uses about 10% of the full capabilities of the Presto Parser so I didn't want to have un-needed jars in the release, but possibly that was not the right decision.

          Show
          joel.bernstein Joel Bernstein added a comment - Christine Poerschke , thanks for the comments and the link was very informative. Just tried another fresh pull and the warnings are still there so the SHA1 files haven't been updated yet. A couple things I'd like to understand better based on your comments: 1) How come your builds were failing, while mine were successful with warnings? Are you building with Maven? 2) Before a release it will be important to nail down the correct ivy conf. What should the scope be and whether to include all the Presto Parser decencies or just the ones being used in this ticket. This ticket uses about 10% of the full capabilities of the Presto Parser so I didn't want to have un-needed jars in the release, but possibly that was not the right decision.
          Hide
          cpoerschke Christine Poerschke added a comment -

          It was the 'ant resolve' (or 'ant compile' or 'ant dist') that was failing. ant -verbose resolve gave this output (for presto-parser 0.107)

          [ivy:retrieve] == resolving dependencies org.apache.solr#core;working@hostname->com.facebook.presto#presto-parser;0.107 [compile->master]
          [ivy:retrieve] bb-default: Checking cache for: dependency: com.facebook.presto#presto-parser;0.107 {compile=[master], compile.hadoop=[master], test=[master], test.DfsMiniCluster=[master], test.MiniKdc=[master]}
          [ivy:retrieve] don't use cache for com.facebook.presto#presto-parser;0.107: checkModified=true
          [ivy:retrieve]          tried /mydirectory/.ivy2/local/com.facebook.presto/presto-parser/0.107/ivys/ivy.xml
          [ivy:retrieve]          tried /mydirectory/.ivy2/local/com.facebook.presto/presto-parser/0.107/jars/presto-parser.jar
          [ivy:retrieve]  local: no ivy file nor artifact found for com.facebook.presto#presto-parser;0.107
          [ivy:retrieve]          tried http://.../artifactory/repo1/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.pom
          [ivy:retrieve] CLIENT ERROR: Conflict url=http://.../artifactory/repo1/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.pom
          [ivy:retrieve]          tried http://.../artifactory/repo1/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.jar
          [ivy:retrieve]  bb-art-main: no ivy file found for com.facebook.presto#presto-parser;0.107: using default data
          [ivy:retrieve]  found com.facebook.presto#presto-parser;0.107 in bb-art-main
          

          using customised lucene/ivy-settings.xml so that the machine running the build need not directly access external repositories. This is what the customisation patch looks like:

          -  <settings defaultResolver="default"/>
          +  <settings defaultResolver="bb-default"/>
          ...
             <resolvers>
          -    <ibiblio name="sonatype-releases" root="https://oss.sonatype.org/content/repositories/releases" m2compatible="true" />
          +
          +    <ibiblio name="bb-art-main" root="http://.../artifactory/repo1" m2compatible="true" />
          +    <ibiblio name="bb-art-jcenter" root="http://.../artifactory/jcenter" m2compatible="true" />
          +    <ibiblio name="bb-art-cloudera" root="http://.../artifactory/cloudera.repo" m2compatible="true" />
          +
          +    <ibiblio name="bb-art-sonatype-releases" root="http://.../artifactory/sonatype-public" m2compatible="true" />
          +    <ibiblio name="bb-art-maven-restlet" root="http://.../artifactory/maven-restlet" m2compatible="true" />
          +
          +    <ibiblio name="bb-nexus-main" root="http://.../content/repositories/repo1" m2compatible="true" />
          +    <ibiblio name="bb-nexus-central" root="http://.../content/repositories/central" m2compatible="true" />
          +    <ibiblio name="bb-nexus-cloudera" root="http://.../content/repositories/cloudera-public" m2compatible="true" />
          +    <ibiblio name="bb-nexus-sonatype-releases" root="http://.../content/repositories/sonatype-public" m2compatible="true" />
          +
          +    <chain name="bb-default" returnFirst="true" checkmodified="true" changingPattern=".*SNAPSHOT">
          +      <resolver ref="local"/>
          +      <resolver ref="bb-art-main"/>
          +      <resolver ref="bb-art-jcenter"/>
          +      <resolver ref="bb-art-cloudera"/>
          +    </chain>
          +
          +    <ibiblio name="sonatype-releases" root="http://oss.sonatype.org/content/repositories/releases" m2compatible="true" />
          ...
          
          Show
          cpoerschke Christine Poerschke added a comment - It was the 'ant resolve' (or 'ant compile' or 'ant dist') that was failing. ant -verbose resolve gave this output (for presto-parser 0.107) [ivy:retrieve] == resolving dependencies org.apache.solr#core;working@hostname->com.facebook.presto#presto-parser;0.107 [compile->master] [ivy:retrieve] bb- default : Checking cache for : dependency: com.facebook.presto#presto-parser;0.107 {compile=[master], compile.hadoop=[master], test=[master], test.DfsMiniCluster=[master], test.MiniKdc=[master]} [ivy:retrieve] don't use cache for com.facebook.presto#presto-parser;0.107: checkModified= true [ivy:retrieve] tried /mydirectory/.ivy2/local/com.facebook.presto/presto-parser/0.107/ivys/ivy.xml [ivy:retrieve] tried /mydirectory/.ivy2/local/com.facebook.presto/presto-parser/0.107/jars/presto-parser.jar [ivy:retrieve] local: no ivy file nor artifact found for com.facebook.presto#presto-parser;0.107 [ivy:retrieve] tried http: //.../artifactory/repo1/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.pom [ivy:retrieve] CLIENT ERROR: Conflict url=http: //.../artifactory/repo1/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.pom [ivy:retrieve] tried http: //.../artifactory/repo1/com/facebook/presto/presto-parser/0.107/presto-parser-0.107.jar [ivy:retrieve] bb-art-main: no ivy file found for com.facebook.presto#presto-parser;0.107: using default data [ivy:retrieve] found com.facebook.presto#presto-parser;0.107 in bb-art-main using customised lucene/ivy-settings.xml so that the machine running the build need not directly access external repositories. This is what the customisation patch looks like: - <settings defaultResolver= " default " /> + <settings defaultResolver= "bb- default " /> ... <resolvers> - <ibiblio name= "sonatype-releases" root= "https: //oss.sonatype.org/content/repositories/releases" m2compatible= " true " /> + + <ibiblio name= "bb-art-main" root= "http: //.../artifactory/repo1" m2compatible= " true " /> + <ibiblio name= "bb-art-jcenter" root= "http: //.../artifactory/jcenter" m2compatible= " true " /> + <ibiblio name= "bb-art-cloudera" root= "http: //.../artifactory/cloudera.repo" m2compatible= " true " /> + + <ibiblio name= "bb-art-sonatype-releases" root= "http: //.../artifactory/sonatype- public " m2compatible= " true " /> + <ibiblio name= "bb-art-maven-restlet" root= "http: //.../artifactory/maven-restlet" m2compatible= " true " /> + + <ibiblio name= "bb-nexus-main" root= "http: //.../content/repositories/repo1" m2compatible= " true " /> + <ibiblio name= "bb-nexus-central" root= "http: //.../content/repositories/central" m2compatible= " true " /> + <ibiblio name= "bb-nexus-cloudera" root= "http: //.../content/repositories/cloudera- public " m2compatible= " true " /> + <ibiblio name= "bb-nexus-sonatype-releases" root= "http: //.../content/repositories/sonatype- public " m2compatible= " true " /> + + <chain name= "bb- default " returnFirst= " true " checkmodified= " true " changingPattern= ".*SNAPSHOT" > + <resolver ref= "local" /> + <resolver ref= "bb-art-main" /> + <resolver ref= "bb-art-jcenter" /> + <resolver ref= "bb-art-cloudera" /> + </chain> + + <ibiblio name= "sonatype-releases" root= "http: //oss.sonatype.org/content/repositories/releases" m2compatible= " true " /> ...
          Hide
          cpoerschke Christine Poerschke added a comment -

          Hi Joel Bernstein - just wanted to mention that the 0.108 presto-parser release that came out late yesterday didn't have the difficulties that 0.107 (and 0.106) had. I haven't yet looked what the changes between 0.107 and 0.108 are, but 'ant resolve' and 'ant dist' certainly work again now (still with the above customised lucene/ivy-settings.xml).

          Show
          cpoerschke Christine Poerschke added a comment - Hi Joel Bernstein - just wanted to mention that the 0.108 presto-parser release that came out late yesterday didn't have the difficulties that 0.107 (and 0.106 ) had. I haven't yet looked what the changes between 0.107 and 0.108 are, but 'ant resolve' and 'ant dist' certainly work again now (still with the above customised lucene/ivy-settings.xml ).
          Hide
          joel.bernstein Joel Bernstein added a comment -

          ok, great. I'll switch over to the 0.108 parser.

          thanks!

          Show
          joel.bernstein Joel Bernstein added a comment - ok, great. I'll switch over to the 0.108 parser. thanks!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1687603 from Joel Bernstein in branch 'dev/trunk'
          [ https://svn.apache.org/r1687603 ]

          SOLR-7560: Bump Presto Parser version to 0.108

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1687603 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1687603 ] SOLR-7560 : Bump Presto Parser version to 0.108
          Hide
          cpoerschke Christine Poerschke added a comment -

          great, thanks for bumping the version up to 0.108 - when refreshing my local branch of trunk i noticed that it acquired a solr/licenses/presto-parser-0.108.jar.sha1 file as part of the version increase, hadn't noticed that file before - question (please excuse my lack of knowledge on this): can the solr/licenses/presto-parser-0.107.jar.sha1 be removed because the 0.108 equivalent replaces it or is there a need or obligation to keep it around with perhaps only solr releases containing the latest version's .sha1 files?

          Show
          cpoerschke Christine Poerschke added a comment - great, thanks for bumping the version up to 0.108 - when refreshing my local branch of trunk i noticed that it acquired a solr/licenses/presto-parser-0.108.jar.sha1 file as part of the version increase, hadn't noticed that file before - question (please excuse my lack of knowledge on this): can the solr/licenses/presto-parser-0.107.jar.sha1 be removed because the 0.108 equivalent replaces it or is there a need or obligation to keep it around with perhaps only solr releases containing the latest version's .sha1 files?
          Hide
          joel.bernstein Joel Bernstein added a comment -

          I think we only need the latest version. I'll remove presto-parser-0.107.jar.sha1 shortly.

          Show
          joel.bernstein Joel Bernstein added a comment - I think we only need the latest version. I'll remove presto-parser-0.107.jar.sha1 shortly.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1687623 from Joel Bernstein in branch 'dev/trunk'
          [ https://svn.apache.org/r1687623 ]

          SOLR-7560: Removing Presto 0.107 SHA1

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1687623 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1687623 ] SOLR-7560 : Removing Presto 0.107 SHA1
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Just wrapped up the work on SOLR-7441 which really makes the SQL interface much more robust and user friendly when errors occur.

          I think the SQL work is ready to backport to the 5x branch.

          Show
          joel.bernstein Joel Bernstein added a comment - Just wrapped up the work on SOLR-7441 which really makes the SQL interface much more robust and user friendly when errors occur. I think the SQL work is ready to backport to the 5x branch.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1692319 from Joel Bernstein in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1692319 ]

          SOLR-7560: Parallel SQL Support

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1692319 from Joel Bernstein in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1692319 ] SOLR-7560 : Parallel SQL Support
          Hide
          steve_rowe Steve Rowe added a comment -

          Joel, your branch_5x commit won't compile under java7 for me:

          common.compile-core:
              [mkdir] Created dir: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/build/solr-core/classes/java
              [javac] Compiling 835 source files to /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/build/solr-core/classes/java
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/ExpressionFormatter.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/src/java/org/apache/solr/handler/SQLHandler.java:54: error: cannot find symbol
              [javac] import java.util.Optional;
              [javac]                 ^
              [javac]   symbol:   class Optional
              [javac]   location: package java.util
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/parser/SqlParser.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/AstVisitor.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/SortItem.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Expression.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Node.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Unnest.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Query.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/QuerySpecification.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/ComparisonExpression.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Select.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/SingleColumn.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/AllColumns.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Table.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/AliasedRelation.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Values.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Relation.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/LogicalBinaryExpression.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/NotExpression.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Statement.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/SortItem$Ordering.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/LogicalBinaryExpression$Type.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/src/java/org/apache/solr/handler/SQLHandler.java:501: error: cannot access Optional
              [javac]       if(node.getWith().isPresent()) {
              [javac]                      ^
              [javac]   class file for java.util.Optional not found
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/With.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/WithQuery.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/TableSubquery.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/QueryBody.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/SelectItem.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/QualifiedName.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/ComparisonExpression$Type.class): major version 52 is newer than 51, the highest major version supported by this compiler.
              [javac]   It is recommended that the compiler be upgraded.
              [javac] Note: Some input files use or override a deprecated API.
              [javac] Note: Recompile with -Xlint:deprecation for details.
              [javac] Note: Some input files use unchecked or unsafe operations.
              [javac] Note: Recompile with -Xlint:unchecked for details.
              [javac] 2 errors
              [javac] 29 warnings
          
          Show
          steve_rowe Steve Rowe added a comment - Joel, your branch_5x commit won't compile under java7 for me: common.compile-core: [mkdir] Created dir: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/build/solr-core/classes/java [javac] Compiling 835 source files to /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/build/solr-core/classes/java [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/ExpressionFormatter.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/src/java/org/apache/solr/handler/SQLHandler.java:54: error: cannot find symbol [javac] import java.util.Optional; [javac] ^ [javac] symbol: class Optional [javac] location: package java.util [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/parser/SqlParser.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/AstVisitor.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/SortItem.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Expression.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Node.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Unnest.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Query.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/QuerySpecification.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/ComparisonExpression.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Select.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/SingleColumn.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/AllColumns.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Table.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/AliasedRelation.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Values.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Relation.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/LogicalBinaryExpression.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/NotExpression.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/Statement.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/SortItem$Ordering.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/LogicalBinaryExpression$Type.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/src/java/org/apache/solr/handler/SQLHandler.java:501: error: cannot access Optional [javac] if(node.getWith().isPresent()) { [javac] ^ [javac] class file for java.util.Optional not found [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/With.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/WithQuery.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/TableSubquery.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/QueryBody.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/SelectItem.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/QualifiedName.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] warning: /Users/sarowe/svn/lucene/dev/branches/branch_5x/solr/core/lib/presto-parser-0.108.jar(com/facebook/presto/sql/tree/ComparisonExpression$Type.class): major version 52 is newer than 51, the highest major version supported by this compiler. [javac] It is recommended that the compiler be upgraded. [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 2 errors [javac] 29 warnings
          Show
          steve_rowe Steve Rowe added a comment - Same error here: http://jenkins.sarowe.net/job/Lucene-Solr-tests-5.x-Java7/903/
          Hide
          elyograg Shawn Heisey added a comment -

          The requirements on the page for facebook/presto (which is the parent for presto-parser) show Java 8 as a requirement:

          https://github.com/facebook/presto

          Show
          elyograg Shawn Heisey added a comment - The requirements on the page for facebook/presto (which is the parent for presto-parser) show Java 8 as a requirement: https://github.com/facebook/presto
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Ok, looks like I'll need to revert. I'll start the process.

          Show
          joel.bernstein Joel Bernstein added a comment - Ok, looks like I'll need to revert. I'll start the process.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1692331 from Joel Bernstein in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1692331 ]

          SOLR-7560: Reverting 5x commit

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1692331 from Joel Bernstein in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1692331 ] SOLR-7560 : Reverting 5x commit
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Ok, I've reverted everything I believe. Bummer. I need to think about how to get this released now.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Ok, I've reverted everything I believe. Bummer. I need to think about how to get this released now.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Investigating using the Apache Calcite SQL parser rather the Presto SQL parser. Calcite is Java 7 compatible and used by Hive and Drill, so we'll be in good company.

          Show
          joel.bernstein Joel Bernstein added a comment - Investigating using the Apache Calcite SQL parser rather the Presto SQL parser. Calcite is Java 7 compatible and used by Hive and Drill, so we'll be in good company.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Trunk patch that works with Apache Calcite's SQL parser instead of Presto.

          The existing SQL tests pass with this patch. More work to be done as far as removing Presto and properly adding Calcite to the build.

          Show
          joel.bernstein Joel Bernstein added a comment - Trunk patch that works with Apache Calcite's SQL parser instead of Presto. The existing SQL tests pass with this patch. More work to be done as far as removing Presto and properly adding Calcite to the build.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1694905 from Joel Bernstein in branch 'dev/trunk'
          [ https://svn.apache.org/r1694905 ]

          SOLR-7560: Updated CHANGES.txt

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1694905 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1694905 ] SOLR-7560 : Updated CHANGES.txt
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Slated for Solr 6.

          Show
          joel.bernstein Joel Bernstein added a comment - Slated for Solr 6.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Going to stick with the Presto Parser and release this ticket with Solr 6. There is a patch on this ticket that uses the Calcite SQL parser. In the future we can refer to that patch if we decide to switch to Calcite.

          Show
          joel.bernstein Joel Bernstein added a comment - Going to stick with the Presto Parser and release this ticket with Solr 6. There is a patch on this ticket that uses the Calcite SQL parser. In the future we can refer to that patch if we decide to switch to Calcite.
          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Hi,

          I can help to test the Parallel SQL Support feature which is very useful for analytical purpose. Can I get some info on setting up SQLHandler / some instructions to get started.

          Thanks,
          Susheel

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Hi, I can help to test the Parallel SQL Support feature which is very useful for analytical purpose. Can I get some info on setting up SQLHandler / some instructions to get started. Thanks, Susheel
          Hide
          joel.bernstein Joel Bernstein added a comment -

          I will spend some time working on the documentation. So far the docs mostly covers how to form queries.

          The link to docs is here: https://cwiki.apache.org/confluence/display/solr/Parallel+SQL+Interface

          I'll post back to this ticket when I've added the sections on sending the query, configuration and the architecture.

          If you want to get a jump on things the sample configs in trunk have the request handlers already setup. You can take a look at the test cases to see how to send a SQL query:

          https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/core/src/test/org/apache/solr/handler/TestSQLHandler.java

          Show
          joel.bernstein Joel Bernstein added a comment - I will spend some time working on the documentation. So far the docs mostly covers how to form queries. The link to docs is here: https://cwiki.apache.org/confluence/display/solr/Parallel+SQL+Interface I'll post back to this ticket when I've added the sections on sending the query, configuration and the architecture. If you want to get a jump on things the sample configs in trunk have the request handlers already setup. You can take a look at the test cases to see how to send a SQL query: https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/core/src/test/org/apache/solr/handler/TestSQLHandler.java
          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Thanks, Joel. Found the SQLHandler config and I did checkout the trunk and compiled using 'ant compile' but getting error when starting solr. Will look into classpath which may be causing the issue.

          ./bin/solr status

          Found 1 Solr nodes:

          Solr process 11789 running on port 8983
          Error: Could not find or load main class org.apache.solr.util.SolrCLI.

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Thanks, Joel. Found the SQLHandler config and I did checkout the trunk and compiled using 'ant compile' but getting error when starting solr. Will look into classpath which may be causing the issue. ./bin/solr status Found 1 Solr nodes: Solr process 11789 running on port 8983 Error: Could not find or load main class org.apache.solr.util.SolrCLI.
          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Thanks, Eric for pointing server dist target. Now I am able to run basic SQL. Will start looking into deeper.

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Thanks, Eric for pointing server dist target. Now I am able to run basic SQL. Will start looking into deeper.
          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Hi Joel,

          I started with two basic tests on my local box
          a) add field alias e.g.
          select id,name as product_name from techproducts limit 20

          which currently fails as data returned contains still "name" as the field/column key than product_name

          b) I wanted to get additional field returned from SQL e.g
          select id,name,manu,mul(price,weight) from techproducts limit 20
          which currently fails with error "Aggregate functions only supported with group by queries." while actually I just want to have additional calculated field based on some function/formula for every document. I checked SQLHandler.java which currently throws this out due to presence of parenthesis without any group by/aggregate function.

          Please let me know your suggestion on this.

          Thanks,
          Susheel

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Hi Joel, I started with two basic tests on my local box a) add field alias e.g. select id,name as product_name from techproducts limit 20 which currently fails as data returned contains still "name" as the field/column key than product_name b) I wanted to get additional field returned from SQL e.g select id,name,manu,mul(price,weight) from techproducts limit 20 which currently fails with error "Aggregate functions only supported with group by queries." while actually I just want to have additional calculated field based on some function/formula for every document. I checked SQLHandler.java which currently throws this out due to presence of parenthesis without any group by/aggregate function. Please let me know your suggestion on this. Thanks, Susheel
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Hi Susheel,

          The two queries you tried are not yet supported. They are both high priorities but might not be in the initial release. The documentation link (https://cwiki.apache.org/confluence/display/solr/Parallel+SQL+Interface) describes the types of queries currently supported in trunk.

          Joel

          Show
          joel.bernstein Joel Bernstein added a comment - Hi Susheel, The two queries you tried are not yet supported. They are both high priorities but might not be in the initial release. The documentation link ( https://cwiki.apache.org/confluence/display/solr/Parallel+SQL+Interface ) describes the types of queries currently supported in trunk. Joel
          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Thanks, Joel. I'll try to write tests around the supported features. Do you have any list of items/tickets for future releases or If I can help to maintain. Also i will try to understand how presto is being plugged with Solr but if you have any pointers please let me know.

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Thanks, Joel. I'll try to write tests around the supported features. Do you have any list of items/tickets for future releases or If I can help to maintain. Also i will try to understand how presto is being plugged with Solr but if you have any pointers please let me know.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          SOLR-7903 and SOLR-7905 are currently being worked on. Field aliases are the next thing I'd like to tackle, but there isn't a ticket for it yet.

          Presto is currently only used to parse the SQL queries. The code for this is in the SQLHandler.

          One area that might be interesting for you to think about is the JDBC driver. If you wanted to create a ticket for that I'd be happy to help with the design.

          Show
          joel.bernstein Joel Bernstein added a comment - SOLR-7903 and SOLR-7905 are currently being worked on. Field aliases are the next thing I'd like to tackle, but there isn't a ticket for it yet. Presto is currently only used to parse the SQL queries. The code for this is in the SQLHandler. One area that might be interesting for you to think about is the JDBC driver. If you wanted to create a ticket for that I'd be happy to help with the design.
          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Hi Joel,

          I have created a JIRA SOLR-8002 for field aliases. Also noticed the JIRA SOLR-7986 you have created and design thoughts on creating Connection, Statement & ResultSet classes.

          Thanks,
          Susheel

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Hi Joel, I have created a JIRA SOLR-8002 for field aliases. Also noticed the JIRA SOLR-7986 you have created and design thoughts on creating Connection, Statement & ResultSet classes. Thanks, Susheel

            People

            • Assignee:
              Unassigned
              Reporter:
              joel.bernstein Joel Bernstein
            • Votes:
              2 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development