Tika
  1. Tika
  2. TIKA-704

PDF and Outlook docs embedded in MS Word documents not parsed

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9
    • Fix Version/s: 0.10
    • Component/s: parser
    • Labels:
      None
    • Environment:

      Windows 7 64-bit

      Description

      Currently there appear to be issues with embedded pdf's and outlook Msg files contained in MS Word documents. I'll attach a sample for each and my recursive parser (incase the problem lies in there).

      From what I see, when these embedded objects are parsed, they're initially identified as vnd.openxmlformats-officedocument.oleObject in the metadata's Content-Type field. After a call to the RecurciveParsers super parse class the Content-Types update to the following:

      PDF's: application/vnd.ms-works
      .MSG: application/x-tika-msoffice

      The internal AutoDetectParser is unable to properly identify these PDF's and therfore does not call the PDFParser on them.

      1. TestWithPdf.docx
        3.80 MB
        Jeremy Anderson
      2. TestWithOutlook.docx
        3.76 MB
        Jeremy Anderson
      3. recursiveUsage.txt
        3 kB
        Jeremy Anderson
      4. LicensedTestWithPdf.docx
        3.80 MB
        Jeremy Anderson
      5. LicensedTestWithOutlook.docx
        111 kB
        Jeremy Anderson

        Activity

        Hide
        Jeremy Anderson added a comment -

        Sample files:

        •RecursiveParser being used.
        •Word document with a embedded .msg.
        •Word document with some embedded pdf's.

        Show
        Jeremy Anderson added a comment - Sample files: •RecursiveParser being used. •Word document with a embedded .msg. •Word document with some embedded pdf's.
        Hide
        Jukka Zitting added a comment -

        Thanks for bringing this up! Fixed in revision 1164578 with better handling of embedded OLE objects.

        Show
        Jukka Zitting added a comment - Thanks for bringing this up! Fixed in revision 1164578 with better handling of embedded OLE objects.
        Hide
        Jukka Zitting added a comment -

        See also revisions 1165230 and 1165259 for followup work.

        Jeremy, I notice you didn't select the "Grant license to ASF for inclusion in ASF works (as per the Apache License §5)" when uploading the test documents. Was this on purpose, or is it OK if we include at least the testWithPdf.docx document as a test case in Tika?

        The email attachment in testWithOutlook.docx contains a PDF under Yamaha copyright, so we probably can't use that in any case. It would be great if you could create an alternative test file without any external content that we could include in Tika.

        Show
        Jukka Zitting added a comment - See also revisions 1165230 and 1165259 for followup work. Jeremy, I notice you didn't select the "Grant license to ASF for inclusion in ASF works (as per the Apache License §5)" when uploading the test documents. Was this on purpose, or is it OK if we include at least the testWithPdf.docx document as a test case in Tika? The email attachment in testWithOutlook.docx contains a PDF under Yamaha copyright, so we probably can't use that in any case. It would be great if you could create an alternative test file without any external content that we could include in Tika.
        Hide
        Jeremy Anderson added a comment -

        Thanks for the fast attention for the fix. Didn't realize that these type of test files can or are desired to be included... I'll either change the license on those already loaded (MSG example) and/or re-uupload (a better PDF example).

        Thanks again!!!

        Show
        Jeremy Anderson added a comment - Thanks for the fast attention for the fix. Didn't realize that these type of test files can or are desired to be included... I'll either change the license on those already loaded (MSG example) and/or re-uupload (a better PDF example). Thanks again!!!
        Hide
        Jeremy Anderson added a comment -

        These are licensed versions... No yamaha manual this time

        Show
        Jeremy Anderson added a comment - These are licensed versions... No yamaha manual this time
        Hide
        Jukka Zitting added a comment -

        Thanks! I added the test cases in revision 1167052.

        Show
        Jukka Zitting added a comment - Thanks! I added the test cases in revision 1167052.
        Hide
        Jukka Zitting added a comment -

        Hmm, there was still a hidden copy of the Yamaha manual in the test file. I removed that in revision 1167056, which also brought down the size of the file from 3.9MB to a more comfortable 100KB.

        Show
        Jukka Zitting added a comment - Hmm, there was still a hidden copy of the Yamaha manual in the test file. I removed that in revision 1167056, which also brought down the size of the file from 3.9MB to a more comfortable 100KB.

          People

          • Assignee:
            Jukka Zitting
            Reporter:
            Jeremy Anderson
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development