Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-721

UTF16-LE not detected

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • parser
    • None

    Description

      I have a test file encoded in UTF16-LE, but Tika fails to detect it.

      Note that it is missing the BOM, which is not allowed (for UTF16-BE
      the BOM is optional).

      Not sure we can realistically fix this; I have no idea how...

      Here's what Tika detects:

      windows-1250:   confidence=9
      windows-1250:   confidence=7
      windows-1252:   confidence=7
      windows-1252:   confidence=6
      windows-1252:   confidence=5
      IBM420_ltr:     confidence=4
      windows-1252:   confidence=3
      windows-1254:   confidence=2
      windows-1250:   confidence=2
      windows-1252:   confidence=2
      IBM420_rtl:     confidence=1
      windows-1253:   confidence=1
      windows-1250:   confidence=1
      windows-1252:   confidence=1
      windows-1252:   confidence=1
      windows-1252:   confidence=1
      windows-1252:   confidence=1
      windows-1252:   confidence=1
      

      The test file decodes fine as UTF16-LE; eg in Python just run this:

      import codecs
      codecs.getdecoder('utf_16_le')(open('Chinese_Simplified_utf16.txt').read())
      

      Attachments

        1. Chinese_Simplified_utf16.txt
          28 kB
          Michael McCandless
        2. TIKA-721.patch
          14 kB
          Michael McCandless

        Issue Links

          Activity

            People

              mikemccand Michael McCandless
              mikemccand Michael McCandless
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: