[TIKA-2449] Enabling extraction of standard references from text

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.17
• Component/s: handler
• Labels: handler
• Flags: Patch
• External issue URL: https://github.com/apache/tika/pull/204
• Assignee: Giuseppe Totaro
• Reporter: Giuseppe Totaro
• Created: 29/Aug/17 19:22
• Updated: 14/Sep/17 01:59
• Resolved: 14/Sep/17 00:37

Description

Apache Tika currently provides many ContentHandler which help to de-obfuscate some information from text. For instance, the PhoneExtractingContentHandler is used to extract phone numbers while parsing.

This improvement adds the StandardsExtractingContentHandler to Tika, a new ContentHandler that relies on regular expressions in order to identify and extract standard references from text.
Basically, a standard reference is just a reference to a norm/convention/requirement (i.e., a standard) released by a standard organization. This work is maily focused on identifying and extracting the references to the standards already cited within a given document (e.g., SOW/PWS) so the references can be stored and provided to the user as additional metadata in case the StandardExtractingContentHandler is used.

In addition to the patch, the first version of the StandardsExtractingContentHandler along with an example class to easily execute the handler is available on GitHub. The following sections provide more in detail how the StandardsExtractingHandler has been developed.

Background

From a technical perspective, a standard reference is a string that is usually composed of two parts:

1. the name of the standard organization;
2. the alphanumeric identifier of the standard within the organization.
   Specifically, the first part can include the acronym or the full name of the standard organization or even both, and the second part can include an alphanumeric string, possibly containing one or more separation symbols (e.g., "-", "_", ".") depending on the format adopted by the organization, representing the identifier of the standard within the organization.

Furthermore, the standard references are usually reported within the "Applicable Documents" or "References" section of a SOW, and they can be cited also within sections that include in the header the word "standard", "requirement", "guideline", or "compliance".

Consequently, the citation of standard references within a SOW/PWS document can be summarized by the following rules:

• RULE #1: standard references are usually reported within the section named "Applicable Documents" or "References".
• RULE #2: standard references can be cited also within sections including the word "compliance" or another semantically-equivalent word in their name.
• RULE #3: standard references is composed of two parts:
  • Name of the standard organization (acronym, full name, or both).
  • Alphanumeric identifier of the standard within the organization.
• RULE #4: The name of the standard organization includes the acronym or the full name or both. The name must belong to the set of standard organizations S = O U V, where O represents the set of open standard organizations (e.g., ANSI) and V represents the set of vendor-specific standard organizations (e.g., Motorola).
• RULE #5: A separation symbol (e.g., "-", "_", "." or whitespace) can be used between the name of the standard organization and the alphanumeric identifier.
• RULE #6: The alphanumeric identifier of the standard is composed of alphabetic and numeric characters, possibly split in two or more parts by a separation symbol (e.g., "-", "_", ".").

On the basis of the above rules, here are some examples of formats used for reporting standard references within a SOW/PWS:

• <ORGANIZATION_ACRONYM><SEPARATION_SYMBOL><ALPHANUMERIC_IDENTIFIER>
• <ORGANIZATION_ACRONYM><SEPARATION_SYMBOL>(<ORGANIZATION_FULL_NAME>)<SEPARATION_SYMBOL><ALPHANUMERIC_IDENTIFIER>
• <ORGANIZATION_FULL_NAME><SEPARATION_SYMBOL>(<ORGANIZATION_FULL_NAME>)<SEPARATION_SYMBOL><ALPHANUMERIC_IDENTIFIER>

Moreover, some standards are sometimes released by two standard organizations. In this case, the standard reference can be reported as follows:

• <MAIN_ORGANIZATION_ACRONYM>/<SECOND_ORGANIZATION_ACRONYM><SEPARATION_SYMBOL><ALPHANUMERIC_IDENTIFIER>

Regular Expressions

The StandardsExtractingContentHandler uses a helper class named StandardsText that relies on Java regular expressions and provides some methods to identify headers and standard references, and determine the score of the references found within the given text.

Here are the main regular expressions used within the StandardsText class:

• REGEX_HEADER: regular expression to match only uppercase headers.

    (\d+\.(\d+\.?)*)\p{Blank}+([A-Z]+(\s[A-Z]+)*){5,}
    

• REGEX_APPLICABLE_DOCUMENTS: regular expression to match the the header of "APPLICABLE DOCUMENTS" and equivalent sections.

    (?i:.*APPLICABLE\sDOCUMENTS|REFERENCE|STANDARD|REQUIREMENT|GUIDELINE|COMPLIANCE.*)
    

• REGEX_FALLBACK: regular expression to match a string that is supposed to be a standard reference.

    \(?(?<mainOrganization>[A-Z]\w+)\)?((\s?(?<separator>\/)\s?)(\w+\s)*\(?(?<secondOrganization>[A-Z]\w+)\)?)?(\s(Publication|Standard))?(-|\s)?(?<identifier>([0-9]{3,}|([A-Z]+(-|_|\.)?[0-9]{2,}))((-|_|\.)?[A-Z0-9]+)*)
    

• REGEX_STANDARD: regular expression to match the standard organization within a string potentially representing a standard reference.
  This regular expression is obtained by using a helper class named StandardOrganizations that provides a list of the most important standard organizations reported on Wikipedia. Basically, the list is composed of International standard organizations, Regional standard organizations, and American and British among Nationally-based standard organizations. Other lists of standard organizations are reported on OpenStandards and IBR Standards Portal.

How To Use The Standards Extraction Capability

The standard references identification performed by using the StandardsExtractingContentHandler is based on the following steps (see also the flow chart in attachment):

1. searches for headers;
2. searches for patterns that are supposed to be standard references (basically, every string mostly composed of uppercase letters followed by an alphanumeric characters);
3. each potential standard reference starts with score equal to 0.25;
4. increases by 0.50 the score of references which include the name of a known standard organization;
5. increases by 0.25 the score of references which have been found within "Applicable Documents" and equivalent sections;
6. returns the standard references along with scores;
7. adds the standard references as additional metadata.

The unit test is implemented within the StandardsExtractingContentHandlerTest class and extracts the standard references from a SoW downloaded from the FOIA Library. This SoW is also provided as PDF in attachment.

The StandardsExtractionExample is a class to demonstrate how to use the StandardsExtractingContentHandler to get a list of the standard references from every file in a directory.

The patch in attachment includes all the changes to add the support for standards extraction.

Issue Links

• links to: GitHub Pull Request #204