Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Documentation
    • Labels:

      Description

      The documentation is lacking a section about the detokenizer API.

        Activity

        Hide
        prakash111287@gmail.com Prakash Mathiyalagan added a comment - - edited

        Create instance of SimpleTokenizer.

        String sentence = "He said \"This is a test\".";
        SimpleTokenizer instance = SimpleTokenizer.INSTANCE;

        Tokenize the sentence using tokenize(String str) method from SimpleTokenizer

        String tokens[] = instance.tokenize(sentence);

        The operations array must have the same number of operation name as tokens array. Basically array length should be equal.
        Store the operation name N-times (tokens.length times) into operation array.

        Operation operations[] = new Operation[tokens.length];
        String oper = "MOVE_RIGHT"; // please refer above list for the list of operations
        for (int i = 0; i < tokens.length; i++)

        { operations[i] = Operation.parse(oper); }

        System.out.println(operations.length);
        Here the operation array length will be equal to the tokens array length.

        Now create an instance of DetokenizationDictionary by passing tokens and operations arrays to the constructor.

        DetokenizationDictionary detokenizeDict = new DetokenizationDictionary(tokens, operations);

        Pass DetokenizationDictionary instance to the DictionaryDetokenizer class to detokenize the tokens.

        DictionaryDetokenizer dictDetokenize = new DictionaryDetokenizer(detokenizeDict);

        DictionaryDetokenizer.detokenize requires two parameters. a). tokens array and b). split marker
        String st = dictDetokenize.detokenize(tokens, " ");

        Output:
        ---------

        He said " This is a test " .

        Show
        prakash111287@gmail.com Prakash Mathiyalagan added a comment - - edited Create instance of SimpleTokenizer. String sentence = "He said \"This is a test\"."; SimpleTokenizer instance = SimpleTokenizer.INSTANCE; Tokenize the sentence using tokenize(String str) method from SimpleTokenizer String tokens[] = instance.tokenize(sentence); The operations array must have the same number of operation name as tokens array. Basically array length should be equal. Store the operation name N-times (tokens.length times) into operation array. Operation operations[] = new Operation [tokens.length] ; String oper = "MOVE_RIGHT"; // please refer above list for the list of operations for (int i = 0; i < tokens.length; i++) { operations[i] = Operation.parse(oper); } System.out.println(operations.length); Here the operation array length will be equal to the tokens array length. Now create an instance of DetokenizationDictionary by passing tokens and operations arrays to the constructor. DetokenizationDictionary detokenizeDict = new DetokenizationDictionary(tokens, operations); Pass DetokenizationDictionary instance to the DictionaryDetokenizer class to detokenize the tokens. DictionaryDetokenizer dictDetokenize = new DictionaryDetokenizer(detokenizeDict); DictionaryDetokenizer.detokenize requires two parameters. a). tokens array and b). split marker String st = dictDetokenize.detokenize(tokens, " "); Output: --------- He said " This is a test " .

          People

          • Assignee:
            Unassigned
            Reporter:
            joern Joern Kottmann
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development