Velocity
  1. Velocity
  2. VELOCITY-631

VTL Creates parse error, but shouldn't

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.6-beta1
    • Fix Version/s: 1.6
    • Component/s: Engine
    • Labels:
      None

      Description

      The following creates a parse exception, but should not:

      #macro(test $x)$x#set($i=1)#end

      Exception:

      Threw: org.apache.velocity.exception.ParseErrorException
      msg: Encountered "=" at /foo.vm[line 2, column 25]
      Was expecting one of:
      <RPAREN> ...
      <WHITESPACE> ...
      <WHITESPACE> ...

      1. fix631.patch
        127 kB
        Byron Foster
      2. Velocity631TestCase.java
        1 kB
        Nathan Bubna

        Activity

        Hide
        Nathan Bubna added a comment -

        Here's a test case with an even simpler template to demonstrate the problem.

        $a#set($b = 1)

        is all it takes.

        Show
        Nathan Bubna added a comment - Here's a test case with an even simpler template to demonstrate the problem. $a#set($b = 1) is all it takes.
        Hide
        Nathan Bubna added a comment -

        I'm messing around with this off and on today, though i'm anything but an expert on the parser. It looks like putting a space between the reference and the #set is a perfect workaround, as #set then gobbles that space up. Here's some perhaps useful debug output from my latest version of the test (with ParserTokenManager.debugPrint = true). If anyone more parser-savvy happens to read this and have an insight, that would be swell...

        Running org.apache.velocity.test.issues.Velocity631TestCase
        [info] Expectation: $a
        [info] Template: $a #set($b = 1)
        $ : going to 5 (0) pushing cur state : 3
        REF_TERM : stack pop (0) : lparen=0 newstate=3
        #set : going to 0 (0) pushing cur state : 3
        $ : going to 5 (1) pushing cur state : 0
        REF_TERM : stack pop (1) : lparen=1 newstate=0
        stack pop (0) : lparen=0 newstate=3
        [info] Result: $a

        [info] Expectation: $a
        [info] Template: $a#set($b = 1)
        $ : going to 5 (0) pushing cur state : 3
        stack pop (0) : lparen=0 newstate=3

        1. : going to 0 (0) pushing cur state : 3
          $ : going to 5 (1) pushing cur state : 0
          REF_TERM : stack pop (1) : lparen=1 newstate=0
          [error] Parser Exception: $a#set($b = 1)
          Encountered "=" at line 1, column 11.
          Was expecting:
          <RPAREN> ...

        org.apache.velocity.runtime.parser.ParseException: Encountered "=" at line 1, column 11.
        Was expecting:
        <RPAREN> ...

        at org.apache.velocity.runtime.parser.Parser.generateParseException(Parser.java:3437)
        at org.apache.velocity.runtime.parser.Parser.jj_consume_token(Parser.java:3316)
        at org.apache.velocity.runtime.parser.Parser.Directive(Parser.java:833)
        at org.apache.velocity.runtime.parser.Parser.Statement(Parser.java:373)
        at org.apache.velocity.runtime.parser.Parser.process(Parser.java:311)
        at org.apache.velocity.runtime.parser.Parser.parse(Parser.java:105)

        I can see that the $a#set version is not recognizing the #set, but i don't know how to fix that...

        Show
        Nathan Bubna added a comment - I'm messing around with this off and on today, though i'm anything but an expert on the parser. It looks like putting a space between the reference and the #set is a perfect workaround, as #set then gobbles that space up. Here's some perhaps useful debug output from my latest version of the test (with ParserTokenManager.debugPrint = true). If anyone more parser-savvy happens to read this and have an insight, that would be swell... Running org.apache.velocity.test.issues.Velocity631TestCase [info] Expectation: $a [info] Template: $a #set($b = 1) $ : going to 5 (0) pushing cur state : 3 REF_TERM : stack pop (0) : lparen=0 newstate=3 #set : going to 0 (0) pushing cur state : 3 $ : going to 5 (1) pushing cur state : 0 REF_TERM : stack pop (1) : lparen=1 newstate=0 stack pop (0) : lparen=0 newstate=3 [info] Result: $a [info] Expectation: $a [info] Template: $a#set($b = 1) $ : going to 5 (0) pushing cur state : 3 stack pop (0) : lparen=0 newstate=3 : going to 0 (0) pushing cur state : 3 $ : going to 5 (1) pushing cur state : 0 REF_TERM : stack pop (1) : lparen=1 newstate=0 [error] Parser Exception: $a#set($b = 1) Encountered "=" at line 1, column 11. Was expecting: <RPAREN> ... org.apache.velocity.runtime.parser.ParseException: Encountered "=" at line 1, column 11. Was expecting: <RPAREN> ... at org.apache.velocity.runtime.parser.Parser.generateParseException(Parser.java:3437) at org.apache.velocity.runtime.parser.Parser.jj_consume_token(Parser.java:3316) at org.apache.velocity.runtime.parser.Parser.Directive(Parser.java:833) at org.apache.velocity.runtime.parser.Parser.Statement(Parser.java:373) at org.apache.velocity.runtime.parser.Parser.process(Parser.java:311) at org.apache.velocity.runtime.parser.Parser.parse(Parser.java:105) I can see that the $a#set version is not recognizing the #set, but i don't know how to fix that...
        Hide
        Nathan Bubna added a comment -

        Would be nice to fix this for 1.6...

        Show
        Nathan Bubna added a comment - Would be nice to fix this for 1.6...
        Hide
        Byron Foster added a comment -

        I looked into it, but it's kind of nasty. It appears to be a problem in the grammer. the LOOKAHEAD(2) for a Reference is forcing the tokenizer to interpret the next token after the reference as a WORD instead of a SET_DIRECTIVE (This is happening when the parser is testing for the DOT token). Because it's interpreted as a WORD the rest is interpreted as a directive, which of course doesn't allow "=". I don't know enough about javacc to make the changes, and this kind of stuff is delicate Maybe someone more familiar with javacc and the grammer can pick this up.

        Show
        Byron Foster added a comment - I looked into it, but it's kind of nasty. It appears to be a problem in the grammer. the LOOKAHEAD(2) for a Reference is forcing the tokenizer to interpret the next token after the reference as a WORD instead of a SET_DIRECTIVE (This is happening when the parser is testing for the DOT token). Because it's interpreted as a WORD the rest is interpreted as a directive, which of course doesn't allow "=". I don't know enough about javacc to make the changes, and this kind of stuff is delicate Maybe someone more familiar with javacc and the grammer can pick this up.
        Hide
        Byron Foster added a comment - - edited

        Ok, here's a fix. the issue from the comments I added to Parser.jjt:

        I added the lexical states REFERENCE, REFMODIFIER, REFMOD2 to the SET_DIRECTIVE. With SET_DIRECTIVE only in the DEFAULT lexical state the following VTL fails "a$#set(b$ = 1)" because the Reference token uses LOOKAHEAD(2) combined with the fact that we explicity set the lex state to REFERENCE with the $ token, which means we would never evaulate the SET_DIRECTIVE token during the look ahead. This general issue is disscussed here:

        http://www.engr.mun.ca/~theo/JavaCC-FAQ/javacc-faq-ie.htm#tth_sEc3.12

        The patch looks scarier then it really is because most of it is regenerated files. The only explicit changes is the addition of the attached TestCase, and a change to src/parser/Parser.jjt

        Show
        Byron Foster added a comment - - edited Ok, here's a fix. the issue from the comments I added to Parser.jjt: I added the lexical states REFERENCE, REFMODIFIER, REFMOD2 to the SET_DIRECTIVE. With SET_DIRECTIVE only in the DEFAULT lexical state the following VTL fails "a$#set(b$ = 1)" because the Reference token uses LOOKAHEAD(2) combined with the fact that we explicity set the lex state to REFERENCE with the $ token, which means we would never evaulate the SET_DIRECTIVE token during the look ahead. This general issue is disscussed here: http://www.engr.mun.ca/~theo/JavaCC-FAQ/javacc-faq-ie.htm#tth_sEc3.12 The patch looks scarier then it really is because most of it is regenerated files. The only explicit changes is the addition of the attached TestCase, and a change to src/parser/Parser.jjt
        Hide
        Nathan Bubna added a comment -

        Awesome. Thanks, Byron!

        Show
        Nathan Bubna added a comment - Awesome. Thanks, Byron!

          People

          • Assignee:
            Unassigned
            Reporter:
            Byron Foster
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development