[ASTERIXDB-1208] ngram tokenizer failure with negative length

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: CLUS - Cluster management
• Assignee: Taewoo Kim
• Reporter: Wenhai Li
• Created: 01/Dec/15 22:25
• Updated: 18/Jan/17 01:21
• Resolved: 18/Jan/17 01:21

Description

Schemas

drop dataverse test if exists;
create dataverse test;
use dataverse test;
create type DBLPOpenType as open {
  id: int64,
  dblpid: string,
  authors: string,
  misc: string
}
create dataset DBLPOpen(DBLPOpenType) primary key id;
insert into dataset DBLPOpen { "id": 93, "dblpid": "journals/iandc/IbarraJCR91", "authors": "Some Classes of Languages in NC¹", "misc": "2006-04-25 86-106 Inf. Comput. January 1991 90 1 db/journals/iandc/iandc90.html#IbarraJCR91" }

Query

use dataverse test;
set import-private-functions 'true'
for $d in dataset DBLPOpen
where similarity-jaccard(gram-tokens("",3,false),gram-tokens($d.title,3,false)) >= 0.5
return {"rec": $d}