Added a custom detector and an updates tika-mimetypes.xml file

Did you mean to close this? Is there a matching pull request or patch that needs to be applied to implement the changes? And what file types are you working with?

The updates are for identifying .cdf , .xar , . wordperfect and application/zerosize files. A pull request will be generated today in regards to this

re-opening per comments

please provide the pull request Adesh Gupta

Any luck with the pull request?

Pull request has been created. Sorry for the delay.

Pull request has been created. Sorry for the delay.

Adesh Gupta you need to update your PR to only include your specific changes. Please rebase and review your PR.

ping Adesh Gupta

Github user adeshgupta closed the pull request at:

https://github.com/apache/tika/pull/89

GitHub user adeshgupta opened a pull request:

https://github.com/apache/tika/pull/115

fix for TIKA-1885

Hello @chrismattmann , I guess this is what you were looking for. Thanks and do update on the same.
@aishward @radhikachandwadkar

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/adeshgupta/tika fixTIKA-1885

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/tika/pull/115.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #115

commit 96ec428ccf20edf4eecf995c48129367ffc8bbe1
Author: adeshgupta <adeshgup@usc.edu>
Date: 2016-05-03T04:08:54Z

fix for TIKA-1885

Hello,

I am on vacation and will return to work Monday May 9, 2016. In my absence, the following folks are responsible POCs for my work:

1. DARPA Memex / DARPA XDATA / DHS OSS

• Paul Ramirez (Paul.M.Ramirez@jpl.nasa.gov)
• Wayne Burke (Wayne.M.Burke@jpl.nasa.gov)

2. Celgene

• Paul Ramirez (Paul.M.Ramirez@jpl.nasa.gov)
• Robert Tapella (Rob.Tapella@jpl.nasa.gov)

3. AIST SciSpark

• Brian Wilson (bdwilson@jpl.nasa.gov)
• Paul Ramirez (Paul.M.Ramirez@jpl.nasa.gov)

4. NASA ESDSWG

• Paul Ramirez (Paul.M.Ramirez@jpl.nasa.gov)
• Maziyar Boustani (Maziyar.Boustani@jpl.nasa.gov)
• Wayne Burke (Wayne.M.Burke@jpl.nasa.gov)

If someone needs to reach me text would be best, and/or email, however in an emergency I have my cell phone with me (626-755-6564).

I look forward to keeping abreast of the situations and will have email access during this time. Thank you all for your support.

Cheers!

Chris

Hi Adesh Gupta - Just reviewing the pull request. Do you have any tests with this change? If not, I can drop some in and assuming they pass include this in 1.13.

Have incorporated this code as to not block TIKA-1955.

Ended up making the following changes:

• Renamed the detector to ZeroSizeFileDetector
• Moved it from tika-parsers into tika-core under org.apache.detect package
• Added a test class in tika-core
• Changed mime type to application/x-zerosize
• Added in the ASF header to all files.

Adesh Gupta - there was no tika-mimetypes.xml updates in the PR. Has this been done elsewhere?

FAILURE: Integrated in tika-trunk-jdk1.7 #979 (See https://builds.apache.org/job/tika-trunk-jdk1.7/979/)
TIKA-1885: Addition of ZeroSizeFileDetector based on Pull Request from (dmeikle: rev d447193f29531df3022f5137b8f0ec1c73e58cc8)

• tika-core/src/main/java/org/apache/tika/mime/MediaType.java
• tika-core/src/main/java/org/apache/tika/mime/MediaTypeRegistry.java
• tika-core/src/test/java/org/apache/tika/detect/ZeroSizeFileDetectorTest.java
• tika-core/src/main/java/org/apache/tika/detect/ZeroSizeFileDetector.java
  Added CHANGE information for TIKA-1885 and TIKA-1965 (dmeikle: rev 5f0e9303929b34b018e6857c32bed87c80f0c9d2)
• CHANGES.txt

OK - checking your GitHub account there appears to be nothing newer there, so I am going to roll the 1.13 release.

If I am wrong, just shout and we can incorporate in a second RC.

FAILURE: Integrated in tika-trunk-jdk1.7 #980 (See https://builds.apache.org/job/tika-trunk-jdk1.7/980/)
TIKA-1885: Updated test to specify charset in getBytes() (dmeikle: rev eede044eab86b5a380ddd8a585e6dda563dc42d3)

• tika-core/src/test/java/org/apache/tika/detect/ZeroSizeFileDetectorTest.java

Code committed in d447193f29531df3022f5137b8f0ec1c73e58cc8

I was looking at the code for ZeroSizeFileDetector and noticed the use of InputStream.available. I don't think that method is a very reliable way to see if a stream is empty. The java doc says "Returns an estimate of the number of bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream" and the default implementation returns 0. Also wouldn’t “application/x-empty” be better then “application/x-zerovalue“ its what the Linux file command returns.

Good point re Stream. Checking for -1 from read() will be more accurate.

Re mimetype, not sure if this matters to you and the team from USC Chris A. Mattmann?

Happy to use the Linux file command one

I think we need to check with a read. Blocking won't matter, as any of the other detectors would block too. x-empty feels better to me for the mimetype, and if that's what others already use all the better!

Just saw the changes and noticed a bug; you need to add a mark(1) and reset call like the other detectors so it doesn't actually consume the byte. (Can't wait for 1.13 )

SUCCESS: Integrated in tika-trunk-jdk1.7 #985 (See https://builds.apache.org/job/tika-trunk-jdk1.7/985/)
TIKA-1885: Updated MimeType to application/x-empty to match Unix file (dmeikle: rev 114b6044a74fe463b90712345ea8e3b2cb085b62)

• tika-core/src/main/java/org/apache/tika/mime/MediaType.java
• tika-core/src/main/java/org/apache/tika/detect/ZeroSizeFileDetector.java
• tika-core/src/main/java/org/apache/tika/mime/MediaTypeRegistry.java
• tika-core/src/test/java/org/apache/tika/detect/ZeroSizeFileDetectorTest.java

Github user asfgit closed the pull request at:

https://github.com/apache/tika/pull/115