[CAMEL-19094] Tokenizer ignores includeTokens

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.20.1
• Fix Version/s: 3.21.0, 4.0-M2, 4.0.0
• Component/s: camel-core
• Estimated Complexity: Unknown
• Assignee: Claus Ibsen
• Reporter: Pasquale Congiusti
• Created: 27/Feb/23 14:19
• Updated: 27/Feb/23 18:08
• Resolved: 27/Feb/23 18:08

Description

Hi folks,
maybe I am using it in a wrong way, but it seems to me that the Tokenizer is not working as it should when we try to use IncludeTokens option. I'm testing the following:

        TokenizeLanguage lan = new TokenizeLanguage();
        lan.setIncludeTokens(true);
        lan.setToken("el");

        from("timer:java?period={{time:1000}}").routeId("java")
            .setBody()
                .simple("Hello Camel from ${routeId}")
            //.split(body().tokenize("el"))
            .split(lan.createExpression())
            .log("${body}");

But I get

2023-02-27 15:13:27.462  INFO 596799 --- [ - timer://java] Test.java:22                        : H
2023-02-27 15:13:27.464  INFO 596799 --- [ - timer://java] Test.java:22                        : lo Cam
2023-02-27 15:13:27.464  INFO 596799 --- [ - timer://java] Test.java:22                        :  from java
2023-02-27 15:13:28.428  INFO 596799 --- [ - timer://java] Test.java:22                        : H
2023-02-27 15:13:28.430  INFO 596799 --- [ - timer://java] Test.java:22                        : lo Cam
2023-02-27 15:13:28.431  INFO 596799 --- [ - timer://java] Test.java:22                        :  from java
2023-02-27 15:13:29.428  INFO 596799 --- [ - timer://java] Test.java:22                        : H
2023-02-27 15:13:29.430  INFO 596799 --- [ - timer://java] Test.java:22                        : lo Cam
2023-02-27 15:13:29.431  INFO 596799 --- [ - timer://java] Test.java:22                        :  from java

I'd expect there to get "el" token as well.