[KYLIN-2956] building trie dictionary blocked on value of length over 4095

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v2.3.0
• Component/s: Job Engine
• Assignee: Wang, Gang
• Reporter: Wang, Gang
• Created: 23/Oct/17 05:37
• Updated: 02/Apr/18 13:42
• Resolved: 21/Dec/17 10:18

Description

In the new release, Kylin will check the value length when building trie dictionary, in class TrieDictionaryBuilder method buildTrieBytes, through method:

private void positiveShortPreCheck(int i, String fieldName) {
if (!BytesUtil.isPositiveShort)

{ throw new IllegalStateException(fieldName + " is not positive short, usually caused by too long dict value."); }

}
public static boolean isPositiveShort(int i)

{ return (i & 0xFFFF7000) == 0; }

And 0xFFFF7000 in binary: 1111 1111 1111 1111 0111 0000 0000 0000, so the value length should be less than 0000 0000 0000 0000 0001 0000 0001 1111, values 4095 in decimalism.

I wonder why is 0xFFFF7000, should 0xFFFF8000 (1111 1111 1111 1111 1000 0000 0000 0000), support max length: 0000 0000 0000 0000 0111 1111 1111 1111 (32767)
be what you want?
Or 32767 may be too large, I prefer use 0xFFFFE000, 0xFFFFE000 (1111 1111 1111 1111 1110 0000 0000 0000), support max length: 0000 0000 0000 0000 0001 1111 1111 1111 (8191)

Issue Links

• Blocked: KYLIN-2907 Check if a number is a positive integer