Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.14
    • Component/s: parser
    • Labels:
      None

      Description

      Create "ImageRecognitionParser" which can have pluggable implementation for core recognition logic.

      As the name suggests, this parser should detect objects in the images, and support many implementations + models (similar to what NamedEntityParser did for text).

      Supply a default implementation based on Tensorflow with the current state-of-the-art model [1].

      Links:
      [1] https://www.tensorflow.org/versions/r0.8/tutorials/image_recognition/index.html#usage-with-python-api

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user thammegowda opened a pull request:

          https://github.com/apache/tika/pull/125

          TIKA-1993: ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation

          Summary of changes:

          • Fixed TIKA-2002 : ExternalParser.check() empties stdout and stderr buffers so no more hanging is expected
          • Added ObjectRecognitionParser, ObjectRecogniser, RecognisedObject - A parser, interface and a model class respectively
          • implemented TensorFlowImageRecParser - an `ExternalParser` which (if missing) downloads and calls tensorflow `image_classify.py` script (the script then downloads Inception-v3 model)

            1. Quick Setup and Test
            1. Demos
              Compile package : `mvn clean install` # `-DskipTests` if you dont like to wait for tests

          Lets check

          / /NOTE:
          1. The most efficient way to make use of tensorflow would be to use C++ api via JNI. I didn't have a chance to learn that stuff so far so help needed to make this efficient. Or else we may wait for tensorflow folks to offer Java bindings! Right now, the image recognition model is loaded and unloaded every time by the script (200MB of disk-read per parse call, very inefficient!).
          2. The very first call takes plenty of time as the model is downloaded lazily
          3. Only `image/jpeg` is supported. PNG coming later

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/thammegowda/tika TIKA-1993

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tika/pull/125.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #125


          commit 9b5dc7fae4456b12b75ec21d050b9439e6527c47
          Author: Thamme Gowda <thammegowda@apache.org>
          Date: 2016-06-12T02:04:08Z

          External Parser now have consumer for ignored lines, Fix TIKA-2002

          commit eccc15387f0d4a5c62d8d12e6579878dba2f52a8
          Author: Thamme Gowda <thammegowda@apache.org>
          Date: 2016-06-12T02:04:28Z

          Added an utility to load and insatiate classes

          commit 2184e2c2c2a0e507be6be4f9692e0fab5b38a476
          Author: Thamme Gowda <thammegowda@apache.org>
          Date: 2016-06-12T02:04:49Z

          Object recognition parser, tensorflow based implementation, and test cases for these

          commit 0305cfb402f5d5e289533411d5737e1e832888ac
          Author: Thamme Gowda <thammegowda@apache.org>
          Date: 2016-06-12T02:43:07Z

          Explicit Locale


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user thammegowda opened a pull request: https://github.com/apache/tika/pull/125 TIKA-1993 : ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation Summary of changes: Fixed TIKA-2002 : ExternalParser.check() empties stdout and stderr buffers so no more hanging is expected Added ObjectRecognitionParser, ObjectRecogniser, RecognisedObject - A parser, interface and a model class respectively implemented TensorFlowImageRecParser - an `ExternalParser` which (if missing) downloads and calls tensorflow `image_classify.py` script (the script then downloads Inception-v3 model) — Quick Setup and Test Install tensor flow using pip - https://www.tensorflow.org/versions/r0.9/get_started/os_setup.html#pip-installation Checkout the test case `tika-parsers/src/test/java/org/apache/tika/parser/recognition/ObjectRecognitionParserTest.java` Demos Compile package : `mvn clean install` # `-DskipTests` if you dont like to wait for tests Lets check (for animal lovers,) on a cat's image at https://raw.githubusercontent.com/apache/tika/master/tika-parsers/src/test/resources/test-documents/testJPEG.jpg ``` java -jar tika-app/target/tika-app-1.14-SNAPSHOT.jar \ --config=tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow.xml \ tika-parsers/src/test/resources/test-documents/testJPEG.jpg ``` ```xml <meta name="English foxhound" content="0.02759"/> <meta name="Egyptian cat" content="0.09168"/> <meta name="collie" content="0.02982"/> <meta name="bluetick" content="0.06043"/> <meta name="Border collie" content="0.07553"/> ``` (For law-keepers) On a rifle at https://upload.wikimedia.org/wikipedia/commons/thumb/3/38/US_Navy_100714-N-4965F-174_Chief_Mass_Communication_Specialist_Paula_Ludwick%2C_assigned_to_Fleet_Combat_Camera_Group_Pacific%2C_shoots_at_a_target_during_a_Navy_Rifle_Qualification_Course.jpg/220px-thumbnail.jpg ```xml <meta name="projectile, missile" content="0.00034"/> <meta name="military uniform" content="0.00763"/> <meta name="bulletproof vest" content="0.00489"/> <meta name="assault rifle, assault gun" content="0.92418"/> <meta name="rifle" content="0.04343"/> ``` (for law-keepers) On a revolver at https://upload.wikimedia.org/wikipedia/commons/8/8d/Glock17.jpg ```xml <meta name="power drill" content="0.00470"/> <meta name="revolver, six-gun, six-shooter" content="0.69355"/> <meta name="holster" content="0.21180"/> <meta name="assault rifle, assault gun" content="0.01513"/> <meta name="rifle" content="0.01053"/> ``` (for car enthusiasts) On a car at http://www.trbimg.com/img-57226a08/turbine/ct-tesla-model-3-unveiling-20160404/650/650x366 ```xml <meta name="car wheel" content="0.02527"/> <meta name="convertible" content="0.01338"/> <meta name="sports car, sport car" content="0.87855"/> <meta name="beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon" content="0.00903"/> <meta name="minivan" content="0.01217"/> ``` / /NOTE: 1. The most efficient way to make use of tensorflow would be to use C++ api via JNI. I didn't have a chance to learn that stuff so far so help needed to make this efficient. Or else we may wait for tensorflow folks to offer Java bindings! Right now, the image recognition model is loaded and unloaded every time by the script (200MB of disk-read per parse call, very inefficient!). 2. The very first call takes plenty of time as the model is downloaded lazily 3. Only `image/jpeg` is supported. PNG coming later You can merge this pull request into a Git repository by running: $ git pull https://github.com/thammegowda/tika TIKA-1993 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tika/pull/125.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #125 commit 9b5dc7fae4456b12b75ec21d050b9439e6527c47 Author: Thamme Gowda <thammegowda@apache.org> Date: 2016-06-12T02:04:08Z External Parser now have consumer for ignored lines, Fix TIKA-2002 commit eccc15387f0d4a5c62d8d12e6579878dba2f52a8 Author: Thamme Gowda <thammegowda@apache.org> Date: 2016-06-12T02:04:28Z Added an utility to load and insatiate classes commit 2184e2c2c2a0e507be6be4f9692e0fab5b38a476 Author: Thamme Gowda <thammegowda@apache.org> Date: 2016-06-12T02:04:49Z Object recognition parser, tensorflow based implementation, and test cases for these commit 0305cfb402f5d5e289533411d5737e1e832888ac Author: Thamme Gowda <thammegowda@apache.org> Date: 2016-06-12T02:43:07Z Explicit Locale
          Hide
          chrismattmann Chris A. Mattmann added a comment -

          LOVE IT! I will try and review right away.

          Show
          chrismattmann Chris A. Mattmann added a comment - LOVE IT! I will try and review right away.
          Hide
          chrismattmann Chris A. Mattmann added a comment -

          so I've reviewed this patch looks good! Let me test it out too tomorrow then I think we're good to commit!

          Show
          chrismattmann Chris A. Mattmann added a comment - so I've reviewed this patch looks good! Let me test it out too tomorrow then I think we're good to commit!
          Hide
          tgowdan@gmail.com Thamme Gowda added a comment -

          I just updated the PR with the changes made to TIKA-1508, it is ready to merge

          Show
          tgowdan@gmail.com Thamme Gowda added a comment - I just updated the PR with the changes made to TIKA-1508 , it is ready to merge
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tika/pull/125

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tika/pull/125
          Hide
          chrismattmann Chris A. Mattmann added a comment -

          Resolved! Thanks everyone! Thamme Gowda

          Show
          chrismattmann Chris A. Mattmann added a comment - Resolved! Thanks everyone! Thamme Gowda
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Tika-trunk #1092 (See https://builds.apache.org/job/Tika-trunk/1092/)
          TIKA-1993 – High throughput Tensorflow Inception based image (thammegowda: rev 9e0a87e68a0a6b45a2f740a58b0db99d9eeecb4a)

          • (add) tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/InceptionRestDockerfile
          • (add) tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow-addon.xml
          • (edit) tika-parsers/src/main/java/org/apache/tika/parser/recognition/ObjectRecognitionParser.java
          • (edit) tika-parsers/src/main/java/org/apache/tika/parser/recognition/tf/TensorflowImageRecParser.java
          • (add) tika-parsers/src/main/java/org/apache/tika/parser/recognition/tf/TensorflowRESTRecogniser.java
          • (edit) tika-core/src/main/java/org/apache/tika/utils/ServiceLoaderUtils.java
          • (add) tika-parsers/src/main/java/org/apache/tika/parser/recognition/tf/TensorflowGrpcRecogniser.java
          • (edit) tika-core/src/main/java/org/apache/tika/config/TikaConfig.java
          • (edit) tika-core/src/main/java/org/apache/tika/utils/AnnotationUtils.java
          • (add) tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/inceptionapi.py
          • (edit) tika-parsers/src/test/java/org/apache/tika/parser/recognition/ObjectRecognitionParserTest.java
          • (add) tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow-rest.xml
            TIKA-1993 – improve usability of docker container (thammegowda: rev 3deea1b7262f772e7b084a32b78f4005811f89a6)
          • (edit) tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/InceptionRestDockerfile
          • (edit) tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/inceptionapi.py
            Update changes for TIKA-1508, TIKA-1993, TIKA-1986. (mattmann: rev da82df5e9def9698fd32f85fe706660641d7c31f)
          • (edit) CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Tika-trunk #1092 (See https://builds.apache.org/job/Tika-trunk/1092/ ) TIKA-1993 – High throughput Tensorflow Inception based image (thammegowda: rev 9e0a87e68a0a6b45a2f740a58b0db99d9eeecb4a) (add) tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/InceptionRestDockerfile (add) tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow-addon.xml (edit) tika-parsers/src/main/java/org/apache/tika/parser/recognition/ObjectRecognitionParser.java (edit) tika-parsers/src/main/java/org/apache/tika/parser/recognition/tf/TensorflowImageRecParser.java (add) tika-parsers/src/main/java/org/apache/tika/parser/recognition/tf/TensorflowRESTRecogniser.java (edit) tika-core/src/main/java/org/apache/tika/utils/ServiceLoaderUtils.java (add) tika-parsers/src/main/java/org/apache/tika/parser/recognition/tf/TensorflowGrpcRecogniser.java (edit) tika-core/src/main/java/org/apache/tika/config/TikaConfig.java (edit) tika-core/src/main/java/org/apache/tika/utils/AnnotationUtils.java (add) tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/inceptionapi.py (edit) tika-parsers/src/test/java/org/apache/tika/parser/recognition/ObjectRecognitionParserTest.java (add) tika-parsers/src/test/resources/org/apache/tika/parser/recognition/tika-config-tflow-rest.xml TIKA-1993 – improve usability of docker container (thammegowda: rev 3deea1b7262f772e7b084a32b78f4005811f89a6) (edit) tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/InceptionRestDockerfile (edit) tika-parsers/src/main/resources/org/apache/tika/parser/recognition/tf/inceptionapi.py Update changes for TIKA-1508 , TIKA-1993 , TIKA-1986 . (mattmann: rev da82df5e9def9698fd32f85fe706660641d7c31f) (edit) CHANGES.txt
          Hide
          githubbot ASF GitHub Bot added a comment -

          kkrgithub commented on issue #125: TIKA-1993: ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation
          URL: https://github.com/apache/tika/pull/125#issuecomment-285954307

          Hi Mr @thammegowda & Mr @chrismattmann
          I am a student looking to participate in GSOC2017 . So I would like to contributing to your reported project @Supporting Image-to-Text (Image Captioning) in Tika for Image MIME Types .
          As far as I understood in this present thread you discussed about improving this module through usage of c++ api for tensorflow , so i am wondering should I do that(improvement) to get started for the actual project .
          If gRPC is enough , should I consider fixing any bugs in deeplearning4j . I have gone through all the resources you have quoted at the "apache Gsoc2017 ideas " site and I completed the quick start tutorial on deeplearning4j.
          So please help me to get on further on this .

          Thanks and Regards
          M kranthi kumar reddy
          student @IIIT Gwalior India.

          ----------------------------------------------------------------
          This is an automated message from the Apache Git Service.
          To respond to the message, please log on GitHub and use the
          URL above to go to the specific comment.

          For queries about this service, please contact Infrastructure at:
          users@infra.apache.org

          Show
          githubbot ASF GitHub Bot added a comment - kkrgithub commented on issue #125: TIKA-1993 : ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation URL: https://github.com/apache/tika/pull/125#issuecomment-285954307 Hi Mr @thammegowda & Mr @chrismattmann I am a student looking to participate in GSOC2017 . So I would like to contributing to your reported project @Supporting Image-to-Text (Image Captioning) in Tika for Image MIME Types . As far as I understood in this present thread you discussed about improving this module through usage of c++ api for tensorflow , so i am wondering should I do that(improvement) to get started for the actual project . If gRPC is enough , should I consider fixing any bugs in deeplearning4j . I have gone through all the resources you have quoted at the "apache Gsoc2017 ideas " site and I completed the quick start tutorial on deeplearning4j. So please help me to get on further on this . Thanks and Regards M kranthi kumar reddy student @IIIT Gwalior India. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
          Hide
          githubbot ASF GitHub Bot added a comment -

          agibsonccc commented on issue #125: TIKA-1993: ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation
          URL: https://github.com/apache/tika/pull/125#issuecomment-285978068

          Hey folks, If there is a problem with deeplearning4j, we have a gitter channel:
          https://gitter.im/deeplearning4j/deeplearning4j

          The release is coming out which will have the model import. @chrismattmann 's and teams example is based on snapshot. Please communicate with us if there are issues.

          Thanks!

          ----------------------------------------------------------------
          This is an automated message from the Apache Git Service.
          To respond to the message, please log on GitHub and use the
          URL above to go to the specific comment.

          For queries about this service, please contact Infrastructure at:
          users@infra.apache.org

          Show
          githubbot ASF GitHub Bot added a comment - agibsonccc commented on issue #125: TIKA-1993 : ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation URL: https://github.com/apache/tika/pull/125#issuecomment-285978068 Hey folks, If there is a problem with deeplearning4j, we have a gitter channel: https://gitter.im/deeplearning4j/deeplearning4j The release is coming out which will have the model import. @chrismattmann 's and teams example is based on snapshot. Please communicate with us if there are issues. Thanks! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
          Hide
          githubbot ASF GitHub Bot added a comment -

          kkrgithub commented on issue #125: TIKA-1993: ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation
          URL: https://github.com/apache/tika/pull/125#issuecomment-285980044

          hello @agibsonccc thanks for the concern but I think you got me wrong I am discussing about this project
          https://issues.apache.org/jira/browse/TIKA-2262?filter=12339687Supporting Image-to-Text (Image Captioning) in Tika for Image MIME Types
          I think the above project is about deeplearning4j or is it about TIKA , I may be wrong so any clarification is greatly appriciated
          thank you !

          ----------------------------------------------------------------
          This is an automated message from the Apache Git Service.
          To respond to the message, please log on GitHub and use the
          URL above to go to the specific comment.

          For queries about this service, please contact Infrastructure at:
          users@infra.apache.org

          Show
          githubbot ASF GitHub Bot added a comment - kkrgithub commented on issue #125: TIKA-1993 : ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation URL: https://github.com/apache/tika/pull/125#issuecomment-285980044 hello @agibsonccc thanks for the concern but I think you got me wrong I am discussing about this project https://issues.apache.org/jira/browse/TIKA-2262?filter=12339687 Supporting Image-to-Text (Image Captioning) in Tika for Image MIME Types I think the above project is about deeplearning4j or is it about TIKA , I may be wrong so any clarification is greatly appriciated thank you ! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
          Hide
          githubbot ASF GitHub Bot added a comment -

          thammegowda commented on issue #125: TIKA-1993: ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation
          URL: https://github.com/apache/tika/pull/125#issuecomment-286135857

          Hi @kkrgithub ! Please go to https://issues.apache.org/jira/browse/TIKA-2262 and comment that you're are interested, that will add you to watcher list and thus notify you if any further action is taken on that issue.

          To contribute to Tika, please look for open issues in JIRA at https://issues.apache.org/jira/browse/TIKA/?selectedTab=com.atlassian.jira.jira-projects-plugin:issues-panel and start digging!

          ----------------------------------------------------------------
          This is an automated message from the Apache Git Service.
          To respond to the message, please log on GitHub and use the
          URL above to go to the specific comment.

          For queries about this service, please contact Infrastructure at:
          users@infra.apache.org

          Show
          githubbot ASF GitHub Bot added a comment - thammegowda commented on issue #125: TIKA-1993 : ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation URL: https://github.com/apache/tika/pull/125#issuecomment-286135857 Hi @kkrgithub ! Please go to https://issues.apache.org/jira/browse/TIKA-2262 and comment that you're are interested, that will add you to watcher list and thus notify you if any further action is taken on that issue. To contribute to Tika, please look for open issues in JIRA at https://issues.apache.org/jira/browse/TIKA/?selectedTab=com.atlassian.jira.jira-projects-plugin:issues-panel and start digging! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org
          Hide
          githubbot ASF GitHub Bot added a comment -

          thammegowda commented on issue #125: TIKA-1993: ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation
          URL: https://github.com/apache/tika/pull/125#issuecomment-286138166

          @agibsonccc Sure thing.

          > The release is coming out which will have the model import. @chrismattmann 's and teams example is based on snapshot. Please communicate with us if there are issues.

          We are eagerly waiting for it. Can't wait to integrate it to tika and run it on Spark clusters to process huge datasets of images.

          FYI, we are also exploring [Image2Text](https://github.com/tensorflow/models/blob/master/im2txt/im2txt/show_and_tell_model.py) and other OCR models to run them using deeplearning4j.

          ----------------------------------------------------------------
          This is an automated message from the Apache Git Service.
          To respond to the message, please log on GitHub and use the
          URL above to go to the specific comment.

          For queries about this service, please contact Infrastructure at:
          users@infra.apache.org

          Show
          githubbot ASF GitHub Bot added a comment - thammegowda commented on issue #125: TIKA-1993 : ObjectRecognitionParser + Tensorflow image recognition with Inception-V3 model as default implementation URL: https://github.com/apache/tika/pull/125#issuecomment-286138166 @agibsonccc Sure thing. > The release is coming out which will have the model import. @chrismattmann 's and teams example is based on snapshot. Please communicate with us if there are issues. We are eagerly waiting for it. Can't wait to integrate it to tika and run it on Spark clusters to process huge datasets of images. FYI, we are also exploring [Image2Text] ( https://github.com/tensorflow/models/blob/master/im2txt/im2txt/show_and_tell_model.py ) and other OCR models to run them using deeplearning4j. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org

            People

            • Assignee:
              chrismattmann Chris A. Mattmann
              Reporter:
              tgowdan@gmail.com Thamme Gowda
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development