James Server
  1. James Server
  2. JAMES-1375

IMap server fails to search for mail using SearchTerms which contain USER flags

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0-beta3
    • Fix Version/s: None
    • Component/s: IMAPServer
    • Labels:
      None

      Description

      The code below will fail

      public void testWithSearchTerm() throws Exception {
      	Properties props = System.getProperties();
      	props.setProperty("mail.store.protocol", "imap");
      	props.setProperty("mail.debug", "true");
      
      	Session session = Session.getDefaultInstance(props);
      	URLName urlName = new URLName(
      				"imap://oleg%40oleg-2.local:password@oleg-2.local/INBOX");
      	Store store = session.getStore(urlName);
      	if (!store.isConnected()) {
      		store.connect();
      		Folder folder = store.getFolder(urlName);
      		folder.open(2);
      		NotTerm notJunk = new NotTerm(new FlagTerm(new Flags("foo"), true));
      		folder.search(notJunk);
      	}
      }
      

      with

      Caused by: com.sun.mail.iap.BadCommandException: A4 BAD SEARCH failed. Illegal arguments.
      	at com.sun.mail.iap.Protocol.handleResult(Protocol.java:346)
      	at com.sun.mail.imap.protocol.IMAPProtocol.issueSearch(IMAPProtocol.java:1693)
      	at com.sun.mail.imap.protocol.IMAPProtocol.search(IMAPProtocol.java:1599)
      	at com.sun.mail.imap.protocol.IMAPProtocol.search(IMAPProtocol.java:1586)
      	at com.sun.mail.imap.IMAPFolder.search(IMAPFolder.java:1649)
      

      However setting the user flag works fine and it can be seen in the incoming message

      1. james-1375.txt
        2 kB
        Oleg Zhurakousky

        Activity

        Hide
        Oleg Zhurakousky added a comment -

        more stack trace on the server side:

        jvm 1 | org.apache.james.imap.decode.DecodingException: Invalid character: ')'
        jvm 1 | at org.apache.james.imap.decode.ImapRequestLineReader.consumeWord(ImapRequestLineReader.java:330)
        jvm 1 | at org.apache.james.imap.decode.ImapRequestLineReader.consumeWord(ImapRequestLineReader.java:316)
        jvm 1 | at org.apache.james.imap.decode.ImapRequestLineReader.atom(ImapRequestLineReader.java:183)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.keyword(SearchCommandParser.java:370)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.searchKey(SearchCommandParser.java:97)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.addUntilParen(SearchCommandParser.java:140)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.paren(SearchCommandParser.java:131)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.searchKey(SearchCommandParser.java:71)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.not(SearchCommandParser.java:539)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.n(SearchCommandParser.java:331)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.searchKey(SearchCommandParser.java:103)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.decode(SearchCommandParser.java:863)
        jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.decode(SearchCommandParser.java:997)
        jvm 1 | at org.apache.james.imap.decode.parser.AbstractUidCommandParser.decode(AbstractUidCommandParser.java:36)
        jvm 1 | at org.apache.james.imap.decode.base.AbstractImapCommandParser.parse(AbstractImapCommandParser.java:86)
        jvm 1 | at org.apache.james.imap.decode.main.DefaultImapDecoder.decodeCommandNamed(DefaultImapDecoder.java:130)

        Show
        Oleg Zhurakousky added a comment - more stack trace on the server side: jvm 1 | org.apache.james.imap.decode.DecodingException: Invalid character: ')' jvm 1 | at org.apache.james.imap.decode.ImapRequestLineReader.consumeWord(ImapRequestLineReader.java:330) jvm 1 | at org.apache.james.imap.decode.ImapRequestLineReader.consumeWord(ImapRequestLineReader.java:316) jvm 1 | at org.apache.james.imap.decode.ImapRequestLineReader.atom(ImapRequestLineReader.java:183) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.keyword(SearchCommandParser.java:370) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.searchKey(SearchCommandParser.java:97) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.addUntilParen(SearchCommandParser.java:140) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.paren(SearchCommandParser.java:131) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.searchKey(SearchCommandParser.java:71) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.not(SearchCommandParser.java:539) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.n(SearchCommandParser.java:331) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.searchKey(SearchCommandParser.java:103) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.decode(SearchCommandParser.java:863) jvm 1 | at org.apache.james.imap.decode.parser.SearchCommandParser.decode(SearchCommandParser.java:997) jvm 1 | at org.apache.james.imap.decode.parser.AbstractUidCommandParser.decode(AbstractUidCommandParser.java:36) jvm 1 | at org.apache.james.imap.decode.base.AbstractImapCommandParser.parse(AbstractImapCommandParser.java:86) jvm 1 | at org.apache.james.imap.decode.main.DefaultImapDecoder.decodeCommandNamed(DefaultImapDecoder.java:130)
        Hide
        Eric Charles added a comment -

        Hi Oleg,

        Yes, I've got the same stacktrace here. Are you willing to track this down, or will take ownership?
        We start a vote for imap begin next week, it would be great to have this fixed for the vote.

        James is prolly guilty here, but we can't exclude a bug in javamail client library...
        The ideal fix should include a integration test in https://svn.apache.org/repos/asf/james/mailbox-integration-tester/trunk/src/main/resources/org/apache/james/imap/scripts

        Just tell us how you would like to further proceed.
        Thx again, Eric

        Show
        Eric Charles added a comment - Hi Oleg, Yes, I've got the same stacktrace here. Are you willing to track this down, or will take ownership? We start a vote for imap begin next week, it would be great to have this fixed for the vote. James is prolly guilty here, but we can't exclude a bug in javamail client library... The ideal fix should include a integration test in https://svn.apache.org/repos/asf/james/mailbox-integration-tester/trunk/src/main/resources/org/apache/james/imap/scripts Just tell us how you would like to further proceed. Thx again, Eric
        Hide
        Oleg Zhurakousky added a comment -

        I would have no problem doing this, but I am having a hard time understanding James dev environment. The ImapRequestLineReader class seems to be in the different place in the current trunk than it is in beta3 and the code also seems different. Also the latest snapshot downloads of James won't even start etc. So if you can throw some pointers on how to set up a dev env, i 'll be able to help

        Show
        Oleg Zhurakousky added a comment - I would have no problem doing this, but I am having a hard time understanding James dev environment. The ImapRequestLineReader class seems to be in the different place in the current trunk than it is in beta3 and the code also seems different. Also the latest snapshot downloads of James won't even start etc. So if you can throw some pointers on how to set up a dev env, i 'll be able to help
        Hide
        Eric Charles added a comment -

        Hi Oleg,

        I just downloaded a snapshot and indeed it can't start... :/ I will fix that today.

        For the dev env, we tried to help on http://james.apache.org/server/3/dev-build.html.
        You can 'svn co https://svn.apache.org/repos/asf/james/current/' which will pull all you src you need.
        imap is the project to work on. cd imap and try mvn install.
        You can import into eclipse the imap projects, and ImapRequestLineReader is in the message project (https://svn.apache.org/repos/asf/james/imap/trunk/message/src/main/java/org/apache/james/imap/decode/ImapRequestLineReader.java)

        Thx, Eric

        Show
        Eric Charles added a comment - Hi Oleg, I just downloaded a snapshot and indeed it can't start... :/ I will fix that today. For the dev env, we tried to help on http://james.apache.org/server/3/dev-build.html . You can 'svn co https://svn.apache.org/repos/asf/james/current/ ' which will pull all you src you need. imap is the project to work on. cd imap and try mvn install. You can import into eclipse the imap projects, and ImapRequestLineReader is in the message project ( https://svn.apache.org/repos/asf/james/imap/trunk/message/src/main/java/org/apache/james/imap/decode/ImapRequestLineReader.java ) Thx, Eric
        Hide
        Oleg Zhurakousky added a comment -

        Thanks Eric

        The problem is in how ImapRequestLineReader.consumeWord is invoked from ImapRequestLineReader.atom().
        Currently you are invoking:

        public String consumeWord(CharacterValidator validator)

        However the stack originates from SearchComomandParser.addUntilParen(..) which clearly means that you want to consume the word up until you encounter closing paren, so you you should actually invoke:

        private String consumeWord(CharacterValidator validator, boolean stripParen)

        ... with boolean value 'true', so the atom() method should look like this:

        public String atom() throws DecodingException

        { return consumeWord(new ATOM_CHARValidator(), true); }

        And here is the passing test to include in SearchCommandParserNotTest (nothing else failed)

        @Test
        public void testUserFlagsParsing() throws Exception

        { ImapRequestLineReader reader = new ImapRequestStreamLineReader( new ByteArrayInputStream("NOT (KEYWORD bar KEYWORD foo)".getBytes("US-ASCII")), new ByteArrayOutputStream()); SearchKey key = parser.searchKey(null, reader, null, false); List<SearchKey> keys = key.getKeys().get(0).getKeys(); assertEquals(2, keys.size()); assertEquals("bar", keys.get(0).getValue()); assertEquals("foo", keys.get(1).getValue()); }

        I'll try to set up the full dev environment and start James from it and integration test it but so far the problem seems to be straight forward.

        Show
        Oleg Zhurakousky added a comment - Thanks Eric The problem is in how ImapRequestLineReader.consumeWord is invoked from ImapRequestLineReader.atom(). Currently you are invoking: public String consumeWord(CharacterValidator validator) However the stack originates from SearchComomandParser.addUntilParen(..) which clearly means that you want to consume the word up until you encounter closing paren, so you you should actually invoke: private String consumeWord(CharacterValidator validator, boolean stripParen) ... with boolean value 'true', so the atom() method should look like this: public String atom() throws DecodingException { return consumeWord(new ATOM_CHARValidator(), true); } And here is the passing test to include in SearchCommandParserNotTest (nothing else failed) @Test public void testUserFlagsParsing() throws Exception { ImapRequestLineReader reader = new ImapRequestStreamLineReader( new ByteArrayInputStream("NOT (KEYWORD bar KEYWORD foo)".getBytes("US-ASCII")), new ByteArrayOutputStream()); SearchKey key = parser.searchKey(null, reader, null, false); List<SearchKey> keys = key.getKeys().get(0).getKeys(); assertEquals(2, keys.size()); assertEquals("bar", keys.get(0).getValue()); assertEquals("foo", keys.get(1).getValue()); } I'll try to set up the full dev environment and start James from it and integration test it but so far the problem seems to be straight forward.
        Hide
        Eric Charles added a comment -

        In the meantime, I have temporary fixed the snapshot deploy.
        You can download a working version from https://repository.apache.org/content/repositories/snapshots/org/apache/james/apache-james/3.0-beta4-SNAPSHOT/apache-james-3.0-beta4-20120216.141927-575-app.tar.gz

        Tell us if you have issues with your dev environment.

        I will be happy to review and commit your patch

        Show
        Eric Charles added a comment - In the meantime, I have temporary fixed the snapshot deploy. You can download a working version from https://repository.apache.org/content/repositories/snapshots/org/apache/james/apache-james/3.0-beta4-SNAPSHOT/apache-james-3.0-beta4-20120216.141927-575-app.tar.gz Tell us if you have issues with your dev environment. I will be happy to review and commit your patch
        Hide
        Oleg Zhurakousky added a comment -

        Attaching a patch.

        I was able to start James with an updated JAR, send a Message, flag it with USER flag and than use SearchTerm to filter it out based on the USER flag set.

        Show
        Oleg Zhurakousky added a comment - Attaching a patch. I was able to start James with an updated JAR, send a Message, flag it with USER flag and than use SearchTerm to filter it out based on the USER flag set.
        Hide
        Eric Charles added a comment -

        Committed in http://svn.apache.org/viewvc?rev=1245026&view=rev
        Thx Oleg!

        Feel free to close this JIRA.

        Show
        Eric Charles added a comment - Committed in http://svn.apache.org/viewvc?rev=1245026&view=rev Thx Oleg! Feel free to close this JIRA.
        Hide
        Oleg Zhurakousky added a comment -

        When do you think the next snapshot will be available?

        Show
        Oleg Zhurakousky added a comment - When do you think the next snapshot will be available?
        Hide
        Eric Charles added a comment -

        I'm busy fixing builds on our apache builds.apache.org. jobs are in the queue... I will inform when ok.

        Show
        Eric Charles added a comment - I'm busy fixing builds on our apache builds.apache.org. jobs are in the queue... I will inform when ok.
        Hide
        Eric Charles added a comment -

        Oleg, builds are now ok.
        You can download latest snapshot from https://repository.apache.org/content/repositories/snapshots/org/apache/james/apache-james/3.0-beta4-SNAPSHOT/
        Thx, Eric

        Show
        Eric Charles added a comment - Oleg, builds are now ok. You can download latest snapshot from https://repository.apache.org/content/repositories/snapshots/org/apache/james/apache-james/3.0-beta4-SNAPSHOT/ Thx, Eric

          People

          • Assignee:
            Unassigned
            Reporter:
            Oleg Zhurakousky
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development