Tika
  1. Tika
  2. TIKA-490

Support for adding language profiles dynamically

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.8
    • Component/s: languageidentifier
    • Labels:
      None

      Description

      Currently the Tika LanguageIdentifier loads language profiles thorugh a hardcoded static block in the java code.

      It would be better to make this configurable, so you could add your own languages without recompiling.

      Suggested approach:
      Remove the static code block loading all languages. Instead look for a tika.languageidentification.properties file on classpath.
      Now the user can simply make his/her own (additional) language profile files, put them on the classpath together with a properties file and off you go!

      Also, once you make it configurable, there might be an issue of having the profiles as static members, as you will force the same behaviour for the whole VM. A static Map of Maps could solve this.

      1. TIKA-490.janhoy.092010.patch
        14 kB
        Jan Høydahl
      2. TIKA-490.janhoy.092010.patch
        14 kB
        Jan Høydahl
      3. TIKA-490.janhoy.082310.patch
        12 kB
        Jan Høydahl
      4. TIKA-490.janhoy.082310.patch
        12 kB
        Jan Høydahl
      5. TIKA-490.Mattmann.082210.2.patch.txt
        11 kB
        Chris A. Mattmann
      6. TIKA-490.Mattmann.082210.patch.txt
        10 kB
        Chris A. Mattmann
      7. TIKA-490.patch
        8 kB
        Jan Høydahl
      8. TIKA-490.patch
        8 kB
        Jan Høydahl

        Activity

        Jan Høydahl created issue -
        Jan Høydahl made changes -
        Field Original Value New Value
        Attachment TIKA-490.patch [ 12452709 ]
        Jan Høydahl made changes -
        Attachment TIKA-490.patch [ 12452719 ]
        Chris A. Mattmann made changes -
        Assignee Chris A. Mattmann [ chrismattmann ]
        Chris A. Mattmann made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Chris A. Mattmann made changes -
        Comment [ Comment from Jan on the ML:

        {quote}
        My rationale for the override part is as follows:

        The default properties file will be embedded within tika-xx.jar
        I assume most people are not keen to unpack and repack JARs to make a config change.
        We COULD put a similar named properties file at another location, but then the user
        needs to make sure that location is EARLIER in classpath than the JAR file.
        In the case of e.g. Solr (Tomcat, Jetty..) it is not obvious how to ensure this,
        and to avoid any confusion about class-loader peculiarities, it's more straight-forward
        to look for an override file.

        Take the Solr example. The user would then put the properties file along with
        his new language profiles in a folder $SOLR_HOME/lib/org/apache/tika/language/

        {quote} ]
        Chris A. Mattmann made changes -
        Attachment TIKA-490.Mattmann.082210.patch.txt [ 12452744 ]
        Chris A. Mattmann made changes -
        Attachment TIKA-490.Mattmann.082210.2.patch.txt [ 12452745 ]
        Jan Høydahl made changes -
        Attachment TIKA-490.janhoy.082310.patch [ 12452760 ]
        Jan Høydahl made changes -
        Attachment TIKA-490.janhoy.082310.patch [ 12452761 ]
        Jan Høydahl made changes -
        Attachment TIKA-490.janhoy.092010.patch [ 12455040 ]
        Jan Høydahl made changes -
        Attachment TIKA-490.janhoy.092010.patch [ 12455041 ]
        Chris A. Mattmann made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Chris A. Mattmann
            Reporter:
            Jan Høydahl
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development