Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-2440

Phonetic strings handling for multilingual environments.

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 1.17
    • parser
    • None
    • Patch

    Description

      Hi there,

      I would like to propose an idea to improve phonetic strings handling for multilingual environments. I believe Tika should not concatenate phonetic strings because text with phonetic strings is recognized as noisy text in most situations of natural language processing.

      Excel files include phonetic strings in some languages such as Japanese, Chinese and so on. Apache POI concatenates phonetic strings onto the shared strings when Tika extract text from Excel files.

      Recent Apache POI has an switch flag for phonetic strings concatination as follows:
      https://poi.apache.org/apidocs/org/apache/poi/xssf/eventusermodel/ReadOnlySharedStringsTable.html#ReadOnlySharedStringsTable(org.apache.poi.openxml4j.opc.OPCPackage,%20boolean)

      Tika should set the 2nd argument "includePhoneticRuns" as false. Here is the simple patch for my idea.

      $ diff -ru XSSFExcelExtractorDecorator.java ./tika/tika-1.15/tika-parsers/src/main/java/org/apache/tika/parser/microsoft/ooxml/XSSFExcelExtractorDecorator.java
      --- XSSFExcelExtractorDecorator.java    2017-06-10 19:13:33.355412625 +0900
      +++ ./tika/tika-1.15/tika-parsers/src/main/java/org/apache/tika/parser/microsoft/ooxml/XSSFExcelExtractorDecorator.java 2017-06-10 19:14:30.452411830 +0900
      @@ -130,7 +130,7 @@
                   styles = xssfReader.getStylesTable();
      
                   iter = (XSSFReader.SheetIterator) xssfReader.getSheetsData();
      -            strings = new ReadOnlySharedStringsTable(container);
      +            strings = new ReadOnlySharedStringsTable(container,false);
               } catch (InvalidFormatException e) {
                   throw new XmlException(e);
               } catch (OpenXML4JException oe) {
      
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            Takahiro Takahiro Ochi
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment