Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8680

Distribute JDBC driver as a separate jar

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently the JDBC driver comes included with the Solrj libraries. As the JDBC driver matures it will be useful to distribute a separate jar which includes all of the dependancies, rather then having to include all the Solrj dependancies separately. This will make it much easier to install and ship with products like JasperSoft, Spotfire and Tableau.

      1. SOLR-8680.patch
        0.7 kB
        Kevin Risden

        Issue Links

          Activity

          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          I could always be wrong about this but I suspect most people using Solrj will be including it through maven. The jdbc clients though will be picking through the directory structure hunting for the jar. So anything that can help this process will be good. But if you feel strongly about leaving jdbc out of the file name than we can leave it out and document / symlink.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited I could always be wrong about this but I suspect most people using Solrj will be including it through maven. The jdbc clients though will be picking through the directory structure hunting for the jar. So anything that can help this process will be good. But if you feel strongly about leaving jdbc out of the file name than we can leave it out and document / symlink.
          Hide
          steve_rowe Steve Rowe added a comment - - edited

          But as Uwe indicated above, there are other situations where a Solrj uber-jar would be useful, and including "jdbc" in the name would be confusing for people who want such a thing. Maybe documentation and/or a jdbc/ directory with a symlink to the uber-jar?

          Show
          steve_rowe Steve Rowe added a comment - - edited But as Uwe indicated above, there are other situations where a Solrj uber-jar would be useful, and including "jdbc" in the name would be confusing for people who want such a thing. Maybe documentation and/or a jdbc/ directory with a symlink to the uber-jar?
          Hide
          joel.bernstein Joel Bernstein added a comment -

          I think having JDBC in name would be helpful. It helps to clarify that this is the jar that is needed for JDBC clients.

          Show
          joel.bernstein Joel Bernstein added a comment - I think having JDBC in name would be helpful. It helps to clarify that this is the jar that is needed for JDBC clients.
          Hide
          steve_rowe Steve Rowe added a comment -

          If I'm reading the patch right, there is nothing specific to JDBC in the produced uber-jar. If that doesn't change, I don't think we should include jdbc in the name. solrj-shaded-<version>.jar maybe?

          Show
          steve_rowe Steve Rowe added a comment - If I'm reading the patch right, there is nothing specific to JDBC in the produced uber-jar. If that doesn't change, I don't think we should include jdbc in the name. solrj-shaded-<version>.jar maybe?
          Hide
          joel.bernstein Joel Bernstein added a comment -

          I think the best way to approach this is to build the uber jar (Licenses included), and place it a separate location in the release where it won't be added to the Solrj class path. Perhaps a new directory called jdbc somewhere in the build.

          Show
          joel.bernstein Joel Bernstein added a comment - I think the best way to approach this is to build the uber jar (Licenses included), and place it a separate location in the release where it won't be added to the Solrj class path. Perhaps a new directory called jdbc somewhere in the build.
          Hide
          risdenk Kevin Risden added a comment -

          Thanks Uwe Schindler for the explanation.

          Show
          risdenk Kevin Risden added a comment - Thanks Uwe Schindler for the explanation.
          Hide
          thetaphi Uwe Schindler added a comment -

          That is not allowed in the way it is done. All licenses are missing. So I'd suggest to got the correct Apache way and package the Uber JAR correctly. The attached patch just copying files into one JAR is a no-go.

          Show
          thetaphi Uwe Schindler added a comment - That is not allowed in the way it is done. All licenses are missing. So I'd suggest to got the correct Apache way and package the Uber JAR correctly. The attached patch just copying files into one JAR is a no-go.
          Hide
          risdenk Kevin Risden added a comment -

          I don't think this should even go on Maven Central. Users of maven can just pull in solr-solrj and it will work correctly if they are building java applications.

          The only reason to have this uber jar is to support 3rd party JDBC clients which need to have all of the solrj dependent jars on the classpath. Currently there are ~11 jars that need to be added to the classpath and its cumbersome to do so. Its more of a convenience to only have to add 1 jar to the classpath.

          Show
          risdenk Kevin Risden added a comment - I don't think this should even go on Maven Central. Users of maven can just pull in solr-solrj and it will work correctly if they are building java applications. The only reason to have this uber jar is to support 3rd party JDBC clients which need to have all of the solrj dependent jars on the classpath. Currently there are ~11 jars that need to be added to the classpath and its cumbersome to do so. Its more of a convenience to only have to add 1 jar to the classpath.
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          In fact this is a complete solrj.jar file with all dependencies munged together without respecting packages and licenses. This is not good practise to have on Maven Central for several reasons:

          • it leads to class path hell, because the user cannot figure out class duplicates (so if he has slf4j or httpclient in his classpath, by adding the uber-jar, he would have 2 versions of all class files on classpath, possibly incompatible versions without knowing). So Uber JARs should use Maven Shade plugin or the jarjar Ant plugin to rewrite all package names in dependecies to something like org.apache.solr.3rdparty.*; e.g. forbiddenapis does this to bundle ASM (which is a typical candidate that always breaks in Uber-JARs, because versions are incompatible to eac other, see https://github.com/policeman-tools/forbidden-apis/blob/master/build.xml#L361-L380 how to do this). If you do this you can name this file solr-solrj-uber.jar and also deploy on Maven Central without problems. But you have to place all additional LICENSE files in the META-INF folder (not only the ASF license).
          • wont work with Java 9 anyways once modules are there, if you dont rewrite package names

          Just to conclude here: It may also be a good idea for other people to have an solrj uber JAR on Maven central (which would have no dependencies), because this helps people who get into conflicts e.g. with httpclient in their classpath. I have seen this quite often for SolrJ (solrj needs httpclient4 but another lib requires version 3 -> boom).

          Those people could just user solr-solrj-uber.jar and have a solr client without any dependencies and because all packages are rewritten it wont conflict with other jars in classpath,

          Show
          thetaphi Uwe Schindler added a comment - - edited In fact this is a complete solrj.jar file with all dependencies munged together without respecting packages and licenses. This is not good practise to have on Maven Central for several reasons: it leads to class path hell, because the user cannot figure out class duplicates (so if he has slf4j or httpclient in his classpath, by adding the uber-jar, he would have 2 versions of all class files on classpath, possibly incompatible versions without knowing). So Uber JARs should use Maven Shade plugin or the jarjar Ant plugin to rewrite all package names in dependecies to something like org.apache.solr.3rdparty.* ; e.g. forbiddenapis does this to bundle ASM (which is a typical candidate that always breaks in Uber-JARs, because versions are incompatible to eac other, see https://github.com/policeman-tools/forbidden-apis/blob/master/build.xml#L361-L380 how to do this). If you do this you can name this file solr-solrj-uber.jar and also deploy on Maven Central without problems. But you have to place all additional LICENSE files in the META-INF folder (not only the ASF license). wont work with Java 9 anyways once modules are there, if you dont rewrite package names Just to conclude here: It may also be a good idea for other people to have an solrj uber JAR on Maven central (which would have no dependencies), because this helps people who get into conflicts e.g. with httpclient in their classpath. I have seen this quite often for SolrJ (solrj needs httpclient4 but another lib requires version 3 -> boom). Those people could just user solr-solrj-uber.jar and have a solr client without any dependencies and because all packages are rewritten it wont conflict with other jars in classpath,
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Thanks Uwe Schindler and Steve Rowe, we'll definitely need guidance on how do this properly. The goal is to have the JDBC driver be as easy as possible to install in clients like Apache Zeppelin. Most of these types of clients don't pull from maven. And once maven gets involved the dependancies are managed anyway so I'm not sure a maven artifact is needed.
          So perhaps it makes sense to start with a packaged jar that is distributed with the Solr binaries and a separate link from the Solr home page.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Thanks Uwe Schindler and Steve Rowe , we'll definitely need guidance on how do this properly. The goal is to have the JDBC driver be as easy as possible to install in clients like Apache Zeppelin. Most of these types of clients don't pull from maven. And once maven gets involved the dependancies are managed anyway so I'm not sure a maven artifact is needed. So perhaps it makes sense to start with a packaged jar that is distributed with the Solr binaries and a separate link from the Solr home page.
          Hide
          thetaphi Uwe Schindler added a comment -

          Very important: Please don't deploy this artifact on Maven Central! If you really want a UBER-JAR file, use Maven Shade plugin and rewrite package names.

          Show
          thetaphi Uwe Schindler added a comment - Very important: Please don't deploy this artifact on Maven Central! If you really want a UBER-JAR file, use Maven Shade plugin and rewrite package names.
          Hide
          risdenk Kevin Risden added a comment -

          The fully assembled jar is really a full SolrJ jar and not just for JDBC. It includes all the solrj-libs and solr-solrj jar. Maybe it should be renamed from solr-solrj-jdbc to something else?

          Show
          risdenk Kevin Risden added a comment - The fully assembled jar is really a full SolrJ jar and not just for JDBC. It includes all the solrj-libs and solr-solrj jar. Maybe it should be renamed from solr-solrj-jdbc to something else?
          Hide
          risdenk Kevin Risden added a comment -

          I just tested the solr-solrj-jdbc*.jar file that is created with the above patch seems to include everything required for JDBC to work. It gets packaged up in the tar.gz when running "ant package" as well.

          Show
          risdenk Kevin Risden added a comment - I just tested the solr-solrj-jdbc*.jar file that is created with the above patch seems to include everything required for JDBC to work. It gets packaged up in the tar.gz when running "ant package" as well.
          Hide
          steve_rowe Steve Rowe added a comment -

          Joel Bernstein, Kevin Risden, I'll take a look.

          Show
          steve_rowe Steve Rowe added a comment - Joel Bernstein , Kevin Risden , I'll take a look.
          Hide
          risdenk Kevin Risden added a comment -

          No idea if this is the right approach at all. It seems to build a solr-solrj-jdbc jar properly. Will test here shortly to make sure it has the right pieces.

          Show
          risdenk Kevin Risden added a comment - No idea if this is the right approach at all. It seems to build a solr-solrj-jdbc jar properly. Will test here shortly to make sure it has the right pieces.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Steve Rowe, I was wondering if you had some ideas about the best way to package up and distribute the jdbc driver as a separate jar? Basically whats needed is a single jar that contains all the Solrj dependancies.

          Show
          joel.bernstein Joel Bernstein added a comment - Steve Rowe , I was wondering if you had some ideas about the best way to package up and distribute the jdbc driver as a separate jar? Basically whats needed is a single jar that contains all the Solrj dependancies.
          Hide
          risdenk Kevin Risden added a comment -

          Linked to JIRAs that are related and this would help them.

          Show
          risdenk Kevin Risden added a comment - Linked to JIRAs that are related and this would help them.
          Hide
          risdenk Kevin Risden added a comment -

          If this gets implemented then documentation from SOLR-8521 would be a lot easier.

          Show
          risdenk Kevin Risden added a comment - If this gets implemented then documentation from SOLR-8521 would be a lot easier.
          Hide
          risdenk Kevin Risden added a comment -

          This would be great since it would eliminate having to copy all the jars from solrj-lib and the solrj-solrj jar. Currently its a bit of a pain to test releases with each build requiring a lot of copying to the right place.

          Show
          risdenk Kevin Risden added a comment - This would be great since it would eliminate having to copy all the jars from solrj-lib and the solrj-solrj jar. Currently its a bit of a pain to test releases with each build requiring a lot of copying to the right place.

            People

            • Assignee:
              Unassigned
              Reporter:
              joel.bernstein Joel Bernstein
            • Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development