Fixed in 72d2d88b381ba75942ae791042ef54af33ee1f38 - your test file is now detected as mbox even without the filename

FAILURE: Integrated in tika-2.x #124 (See https://builds.apache.org/job/tika-2.x/124/)
TIKA-2042 MBOX magic and detection unit test (nick: rev 65cc9bcecdc6b86294a88f3b2b6b26017f356ae5)

• tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml
• tika-app/src/test/java/org/apache/tika/mime/TestMimeTypes.java

FAILURE: Integrated in Tika-trunk #1085 (See https://builds.apache.org/job/Tika-trunk/1085/)
TIKA-2042 MBOX magic and detection unit test (nick: rev 72d2d88b381ba75942ae791042ef54af33ee1f38)

• tika-parsers/src/test/java/org/apache/tika/mime/TestMimeTypes.java
• tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml

FAILURE: Integrated in tika-2.x-windows #28 (See https://builds.apache.org/job/tika-2.x-windows/28/)
TIKA-2042 MBOX magic and detection unit test (nick: rev 65cc9bcecdc6b86294a88f3b2b6b26017f356ae5)

• tika-app/src/test/java/org/apache/tika/mime/TestMimeTypes.java
• tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml

I'd like to ask for this issue to be reopened. Around half the MBOX files in our corpus are being detected as text/html. My guess is that there are two reasons for this:

1) the files have no extension - the filenames are literally "mbox" rather than "*.mbox" (I think this is the way they're generated or used to be generated on Macs - they're in an *.mbox container directory, but the meat is within an mbox file contained within that directory);

2) the headers don't fall within the 256 byte offset specified by the matcher in the mimetypes XML file.

Header attached with identifying information stripped out. This file is detected as text/html instead of application/mbox.

Matthew Caruana Galizia I've added some more patterns in 0277fbb92c4714361949d59708db2a8734f1b1f2 based on the mbox you've supplied. Could you try that, and let us have redacted copies of any remaining mbox files you have which that doesn't fix things for?

FAILURE: Integrated in Jenkins build Tika-trunk #1331 (See https://builds.apache.org/job/Tika-trunk/1331/)
TIKA-2042 Add a few more mbox patterns, based on file supplied by (nick: https://github.com/apache/tika/commit/0277fbb92c4714361949d59708db2a8734f1b1f2)

• (edit) tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml

Nick Burch thank you - that fixes the detection of at least one of the MBOX files. Now the problem is that that when the email streams get passed to the delegate parser by the ParsingEmbeddedDocumentExtractor implementation, they're detected as text/html instead of message/rfc822.

I've attached a sample of one of the message sections from the MBOX. Detected as text/html instead of message/rfc822.

This problem is very very recurrent. I think we should search for some of the eml magics in a larger range.

See Tika-879. Looks like widening the magic search helped to detect more emls in the test corpus. Tim Allison do you remember if that resulted in lots of false positives?

Matthew Caruana Galizia I've added some more rfc822 magic, which I think should solve it for this most recent file. If I trim off everything up the mbox From, then I can detect the outer file as mbox, and the embedded one as rfc822.

Do please let us know if there's any more you come across!

FAILURE: Integrated in Jenkins build Tika-trunk #1335 (See https://builds.apache.org/job/Tika-trunk/1335/)
Two more EML header magics from Matthew Caruana Galizia from TIKA-2042 (nick: https://github.com/apache/tika/commit/0579efe59c35a4db248d8a5547af05fbee3caad4)

• (edit) tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml