Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-3992

Add common missing mimes based on Common Crawl data

    XMLWordPrintableJSON

Details

    • Task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      In the latest Common Crawl crawl, there are ~600k 'octet-stream' files as detected by Tika. It would be useful to extract those (even if truncated) and run 'file' and 'siegfried' against those file types that are unknown to Tika. We can prioritize the most common file formats as identified by file and siegfried for addition to our mime-types.xml.

      Separately, we might also want to do the same thing for `application/zip`...there are likely zip-based file types that we could do a better job on.

      Thanks to snagel for a dump of stats on the most recent crawl.

      Attachments

        1. mimes.zip
          27.67 MB
          Tim Allison

        Issue Links

          supercedes

          Sub-task - The sub-task of the issue TIKA-3995 image/x-3ds

          • Major - Major loss of function.
          • Resolved
          1.
          		 Improve audio/mpeg detection Sub-task Resolved Unassigned
          2.
          		 image/x-3ds Sub-task Resolved Unassigned
          3.
          		 audio/x-sap Sub-task Resolved Unassigned
          4.
          		 audio/x-s3m Sub-task Resolved Unassigned
          5.
          		 application/zstd Sub-task Resolved Unassigned
          6.
          		 audio/xm audio/x-mod Sub-task Open Unassigned
          7.
          		 application/vnd.msa-disk-image Sub-task Resolved Unassigned
          8.
          		 audio/x-sap Sub-task Closed Unassigned
          9.
          		 application/vnd.tcpdump.pcapng Sub-task Resolved Unassigned
          10.
          		 application/vnd.isac.fcs Sub-task Resolved Unassigned
          11.
          		 font/otf application/vnd.ms-opentype Sub-task Resolved Unassigned
          12.
          		 application/x-endnote-style Sub-task Resolved Unassigned
          13.
          		 application/x-cdf Sub-task Resolved Unassigned
          14.
          		 Improve detection of text files Sub-task Open Unassigned
          15.
          		 Add various file identifications to reduce application/octet-stream Sub-task Resolved Unassigned
          16.
          		 Add file extension .rmd160 to tika-mimetypes.xml Sub-task Open Unassigned
          17.
          		 Add magic to audio/aac in tika-mimetypes.xml Sub-task Resolved Unassigned
          18.
          		 Add magic for ASPRS Lidar data Sub-task Reopened Unassigned
          19.
          		 Add magic for FAT Disk Image format Sub-task Open Unassigned
          20.
          		 Add magic for Mach-O format Sub-task Open Unassigned
          21.
          		 Add magic for MS-DOS Compression Format (SZDD Variant) Sub-task Open Unassigned
          22.
          		 Add magic for Jigdo Download Template format Sub-task Open Unassigned
          23.
          		 Add magic for Atari Floppy Disk Image Format Sub-task Open Unassigned
          24.
          		 Add magic for Guitar Pro format Sub-task Open Unassigned
          25.
          		 Add magic for TeX Virtual Font format Sub-task Open Unassigned
          26.
          		 Add magic for GRAPPA Database RADX File Sub-task Open Unassigned
          27.
          		 Add magic for Touhou Project Replay File format Sub-task Open Unassigned
          28.
          		 Add magic for Modified Maximum Method Digisonde Portable Sounder File format Sub-task Open Unassigned
          29.
          		 Add magic for IDL Binary Format Save File format Sub-task Open Unassigned
          30.
          		 Add magic for Planetary Data System Version 3 format Sub-task Open Unassigned
          31.
          		 Add magic for Planetary Data System Version 2 format Sub-task Open Unassigned
          32.
          		 Add magic for ZIM format Sub-task Open Unassigned
          33.
          		 Add magic for ClamAV CDiff files Sub-task Open Unassigned
          34.
          		 Add magic for SquashFS Format Sub-task Open Unassigned
          35.
          		 Add magic for Unreal Engine Package format Sub-task Open Unassigned
          36.
          		 Add magic for X-Moto Replay format Sub-task Open Unassigned
          37.
          		 Add magic for Warcraft III Map format Sub-task Open Unassigned
          38.
          		 Add magic for SEG Y format Sub-task Open Unassigned
          39.
          		 Add magic for Teeworlds/DDRace Map Format Sub-task Open Unassigned
          40.
          		 Add magic for SolidWorks eDrawing Electronic Assembly Data File format Sub-task Open Unassigned

          Activity

            People

              Unassigned Unassigned
              tallison Tim Allison
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: