[TIKA-2578] Mails not recognized when unknown X-headers are present

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.17, 1.18, 2.0.0
• Fix Version/s: 1.18, 2.0.0
• Component/s: detector, mime
• Assignee: Tim Allison
• Reporter: Andreas Meier
• Created: 20/Feb/18 08:52
• Updated: 29/Oct/19 15:52
• Resolved: 26/Feb/18 22:31

Description

Found some mails with leading X-headers.

These mails are recognized as text/plain.

One example is CISCOs IronPort, which might add "X-IronPort-AV" to the beginning of mails.

Therefore I would like to discuss if and how TIKA shall handle these cases.

In my opinion TIKA should try to detect files with x-headers and preprocess them to get a valid mail.

Suggestion:

<mime-type type="text/x-tika-x-header">
  <magic priority="50">
    <match value="X-" type="string" offset="0">
      <match value="Message-ID:" type="string" offset="0:8192"/>
      <match value="From:" type="stringignorecase" offset="0:8192"/>
      <match value="To:" type="stringignorecase" offset="0:8192"/>
      <match value="Subject:" type="string" offset="0:8192"/>
      <match value="MIME-Version:" type="stringignorecase" offset="0:8192"/>
    </match>
  </magic>
  <sub-class-of type="text/x-tika-text-based-message"/>
</mime-type>

See also: RFC6648

Attached an example file.

Regards

Andreas

Issue Links

• duplicates: TIKA-879 Detection problem: message/rfc822 file is detected as text/plain.
• is related to: TIKA-2688 MBOX not recognized when unknown X-headers are present
• relates to: TIKA-2587 DKIM signed mails recognized as text/plain