Tika
  1. Tika
  2. TIKA-849

Identify and parse the Apple iBooks format

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.1
    • Fix Version/s: None
    • Component/s: mime, parser
    • Labels:
      None

      Description

      With the release of iBooks Author 1.0, Apple have created a new eBook format very similar to ePub. Tika could be extended to identify and parse this new format, re-using the existing ePub code wherever possible.

      I have created an initial patch, which I will attach to this issue.

      1. ibooks-support.patch
        1.04 MB
        Andrew Jackson

        Activity

        Hide
        Andrew Jackson added a comment -

        Having thought about it, I think your proposed sub-type relationship is a more sensible approach, and my interpretation of the relationship was overly strict.

        Show
        Andrew Jackson added a comment - Having thought about it, I think your proposed sub-type relationship is a more sensible approach, and my interpretation of the relationship was overly strict.
        Hide
        Nick Burch added a comment -

        We might be able to use the same handler, but it'd need one of our XML experts to weigh in and say for sure!

        I've fixed a bug in r1235215 where start/end document was being called for each html/xhtml file, which solves the use of --metadata from TikaCLI but doesn't affect the text extraction

        For the mimetype subtype, I thought it made sense as both use the same structure and a very similar format. If you think that's wrong, we could maybe invent a common supertype for the two, to show their relation

        Show
        Nick Burch added a comment - We might be able to use the same handler, but it'd need one of our XML experts to weigh in and say for sure! I've fixed a bug in r1235215 where start/end document was being called for each html/xhtml file, which solves the use of --metadata from TikaCLI but doesn't affect the text extraction For the mimetype subtype, I thought it made sense as both use the same structure and a very similar format. If you think that's wrong, we could maybe invent a common supertype for the two, to show their relation
        Hide
        Andrew Jackson added a comment - - edited

        By the way, I notice you added a

            <sub-class-of type="application/epub+zip" />
        

        to the iBooks identification information. I'm not sure this is correct, as I was under the impression that this was a strict conformance hierarchy. This would imply that every x-ibooks+zip also conforms to the epub+zip format, but this is not the case, because the new MIME type declaration means it violates the ePub standard.

        Show
        Andrew Jackson added a comment - - edited By the way, I notice you added a <sub-class-of type= "application/epub+zip" /> to the iBooks identification information. I'm not sure this is correct, as I was under the impression that this was a strict conformance hierarchy. This would imply that every x-ibooks+zip also conforms to the epub+zip format, but this is not the case, because the new MIME type declaration means it violates the ePub standard.
        Hide
        Andrew Jackson added a comment -

        I'm not that familiar with the content handling infrastructure of Tika at present, so I'm having trouble working out how to do this. I can see the ePubParser delegating the content handling to the ePubContentParser which uses an XHTMLContentHandler, but I don't really understand that code well enough to see why it discards the <object> elements and lets the others through. Or perhaps that filtering is done at a later stage?

        In any case, I suppose we would need a new iBooksParser and a new iBooksContentParser, based on the ePub code, but using a different ContentHandler?

        Show
        Andrew Jackson added a comment - I'm not that familiar with the content handling infrastructure of Tika at present, so I'm having trouble working out how to do this. I can see the ePubParser delegating the content handling to the ePubContentParser which uses an XHTMLContentHandler, but I don't really understand that code well enough to see why it discards the <object> elements and lets the others through. Or perhaps that filtering is done at a later stage? In any case, I suppose we would need a new iBooksParser and a new iBooksContentParser, based on the ePub code, but using a different ContentHandler?
        Hide
        Nick Burch added a comment -

        Test and parser change committed in r1234904, thanks

        It looks like all the text is stored within things like
        <object type="application/x-ibooks+shape" id="textShape-89">

        It's all in regular html div tags inside those objects, though sometimes that object is nested inside another of type "application/x-ibooks+flowhead"

        Images seem to be within svg tags

        I guess we'll need to do something for the ibooks case to treat these objects as regular xhtml?

        Show
        Nick Burch added a comment - Test and parser change committed in r1234904, thanks It looks like all the text is stored within things like <object type="application/x-ibooks+shape" id="textShape-89"> It's all in regular html div tags inside those objects, though sometimes that object is nested inside another of type "application/x-ibooks+flowhead" Images seem to be within svg tags I guess we'll need to do something for the ibooks case to treat these objects as regular xhtml?
        Hide
        Nick Burch added a comment -

        Sample file committed in r1234886, along with a unit test for Zip Container Aware detection of epub files (epub seems to use the same scheme as the Open Document Format does)

        Show
        Nick Burch added a comment - Sample file committed in r1234886, along with a unit test for Zip Container Aware detection of epub files (epub seems to use the same scheme as the Open Document Format does)
        Hide
        Andrew Jackson added a comment - - edited

        This patch identifies *.ibooks files, and passes them through the ePub parser. Unfortunately, the content of the eBook is not extracted successfully. I'm not sure why this is, and feedback would be appreciated here. I think the content*.xhtml files are actually HTML5 and it seems that the current XHTML-based parser is unable to parse them. A unit test and test file is included.

        Show
        Andrew Jackson added a comment - - edited This patch identifies *.ibooks files, and passes them through the ePub parser. Unfortunately, the content of the eBook is not extracted successfully. I'm not sure why this is, and feedback would be appreciated here. I think the content*.xhtml files are actually HTML5 and it seems that the current XHTML-based parser is unable to parse them. A unit test and test file is included.

          People

          • Assignee:
            Unassigned
            Reporter:
            Andrew Jackson
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development