Lucene - Core
  1. Lucene - Core
  2. LUCENE-4092

Check what's Jenkins pattern for e-mailing log fragments (so that it includes failures).

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: general/test
    • Labels:
      None
    • Lucene Fields:
      New

      Activity

      Hide
      Steve Rowe added a comment -

      Editable Email Notification/Default Content configuration for the Lucene-Solr-tests-only-trunk job:

      Build: ${BUILD_URL}
      
      ${FAILED_TESTS}
      
      Build Log (for compile errors):
      ${BUILD_LOG_REGEX,regex="\\[javac\\]\\s+\\d+ error(s*)\\b",linesBefore=100,linesAfter=0}
      
      Show
      Steve Rowe added a comment - Editable Email Notification / Default Content configuration for the Lucene-Solr-tests-only-trunk job: Build: ${BUILD_URL} ${FAILED_TESTS} Build Log (for compile errors): ${BUILD_LOG_REGEX,regex="\\[javac\\]\\s+\\d+ error(s*)\\b",linesBefore=100,linesAfter=0}
      Hide
      Dawid Weiss added a comment -

      Yeah, but what is $

      {FAILED_TESTS}

      ? I mean – can we edit it to detect "<<< FAILURES!" and report 100 lines before this string?

      Show
      Dawid Weiss added a comment - Yeah, but what is $ {FAILED_TESTS} ? I mean – can we edit it to detect "<<< FAILURES!" and report 100 lines before this string?
      Hide
      Steve Rowe added a comment -

      Yeah, but what is ${FAILED_TESTS}? I mean – can we edit it to detect "<<< FAILURES!" and report 100 lines before this string?

      Jenkins's Content Token Reference

      All arguments are optional. Arguments may be given for each token in the form name="value" for strings and in the form name=value for booleans and numbers. The {'s and }'s may be omitted if there are no arguments.

      Examples: $TOKEN, ${TOKEN}, ${TOKEN, count=100}, ${ENV, var="PATH"}

      Available Tokens

      • ${DEFAULT_SUBJECT} - This is the default email subject that is configured in Jenkins's system configuration page.
      • ${DEFAULT_CONTENT} - This is the default email content that is configured in Jenkins's system configuration page.
      • ${PROJECT_DEFAULT_SUBJECT} - This is the default email subject for this project. The result of using this token in the advanced configuration is what is in the Default Subject field above. WARNING: Do not use this token in the Default Subject or Content fields. Doing this has an undefined result.
      • ${PROJECT_DEFAULT_CONTENT} - This is the default email content for this project. The result of using this token in the advanced configuration is what is in the Default Content field above. WARNING: Do not use this token in the Default Subject or Content fields. Doing this has an undefined result.
      • ${BUILD_LOG, maxLines, escapeHtml} - Displays the end of the build log.
        • maxLines - display at most this many lines of the log.
          Defaults to 250.
        • escapeHtml - If true, HTML is escaped.
          Defaults to false.
      • ${BUILD_LOG_REGEX, regex, linesBefore, linesAfter, maxMatches, showTruncatedLines, substText, escapeHtml, matchedLineHtmlStyle} - Displays lines from the build log that match the regular expression.
        • regex - Lines that match this regular expression are included. See also java.util.regex.Pattern
          Defaults to "(?i)\b(error|exception|fatal|fail(ed|ure)|un(defined|resolved))\b".
        • linesBefore - The number of lines to include before the matching line. Lines that overlap with another match or linesAfter are only included once.
          Defaults to 0.
        • linesAfter - The number of lines to include after the matching line. Lines that overlap with another match or linesBefore are only included once.
          Defaults to 0.
        • maxMatches - The maximum number of matches to include. If 0, all matches will be included.
          Defaults to 0.
        • showTruncatedLines - If true, include ...truncated ### lines... lines.
          Defaults to true.
        • substText - If non-null, insert this text into the email rather than the entire line.
          Defaults to null.
        • escapeHtml - If true, escape HTML.
          Defaults to false.
        • matchedLineHtmlStyle - If non-null, output HTML. matched lines will become <b style="your-style-value">html escaped matched line</b>.
          Defaults to null.
      • ${BUILD_LOG_EXCERPT, start, end} - Displays an excerpt from the build log.
        • start - Regular expression to match the excerpt starting line to be included (exluded). See java.util.regex.Pattern
        • end - Regular expression to match the excerpt ending line to be included (exluded)
      • ${BUILD_NUMBER} - Displays the number of the current build.
      • ${BUILD_STATUS} - Displays the status of the current build. (failing, success, etc...)
      • ${BUILD_URL} - Displays the URL to the current build.
      • ${CHANGES, showPaths, showDependencies, format, pathFormat} - Displays the changes since the last build.
        • showDependencies - if true, changes to projects this build depends on are shown.
          Defaults to false.
        • showPaths - if true, the paths modified by a commit are shown.
          Defaults to false.
        • format - for each commit listed, a string containing %X, where %X is one of %a for author, %d for date, %m for message, %p for paths, or %r for revision. Not all revision systems support %d and %r. If specified, showPaths is ignored.
          Defaults to "[%a] %m\n".
        • pathFormat - a string containing %p to indicate how to print paths.
          Defaults to "\t%p\n".
      • ${CHANGES_SINCE_LAST_SUCCESS, reverse, format, showPaths, changesFormat, pathFormat} - Displays the changes since the last successful build.
        <ul>
        • reverse - indicates that most recent builds should be at the top.
          Defaults to false.
        • format - for each build listed, a string containing %X, where %X is one of %c for changes, or %n for build number.
          Defaults to "Changes for Build #%n\n%c\n".
        • showPaths, changesFormat, pathFormat - defined as showPaths, format, and pathFormat from ${CHANGES}, respectively.
      • ${CHANGES_SINCE_LAST_UNSTABLE, reverse, format, showPaths, changesFormat, pathFormat} - Displays the changes since the last unstable or successful build.
        • reverse - indicates that most recent builds should be at the top.
          Defaults to false.
        • format - for each build listed, a string containing %X, where %X is one of %c for changes, or %n for build number.
          Defaults to "Changes for Build #%n\n%c\n".
        • showPaths, changesFormat, pathFormat - defined as showPaths, format, and pathFormat from ${CHANGES}, respectively.
      • ${ENV, var} - Displays an environment variable.
        • var - the name of the environment variable to display. If "", show all.
          Defaults to "".
      • ${FAILED_TESTS, showStack, maxTests} - Displays failing unit test information, if any tests have failed.
        • showStack - indicates that most recent builds should be at the top.
          Defaults to true.
        • maxTests - display at most this many failing tests.
          No limit is set by default.
      • ${JENKINS_URL} - Displays the URL to the Jenkins server. (You can change this on the system configuration page.)
      • ${HUDSON_URL} - deprecated, please use $JENKINS_URL
      • ${PROJECT_NAME} - Displays the project's name.
      • ${PROJECT_URL} - Displays a URL to the project's page.
      • ${SVN_REVISION} - Displays the subversion revision number.
      • ${CAUSE} - Displays the cause of the build.
      • ${JELLY_SCRIPT, template} - Custom message content generated from a Jelly script template. There are two templates provided: "html" and "text". Custom Jelly templates should be placed in $JENKINS_HOME/email-templates. When using custom templates, the template filename without ".jelly" should be used for the "template" argument.
        • template - the template name.
          Defaults to "html".
      • ${FILE, path} - Includes the content of a specified file.
        • path - The path to the file. Relative to the workspace root.
      • ${TEST_COUNTS, var} - Displays the number of tests.
        • var - Defaults to "total".
          • total - the number of all tests.
          • fail - the number of failed tests.
          • skip - the number of skipped tests.
      • ${SCRIPT, script, template, init} - Custom message content generated from a script using JSR 223. Custom scripts should be placed in $JENKINS_HOME/email-templates. When using custom scripts, the script filename WITH .py/.rb/etc should be used for the "script" argument.
        templates and other items may be loaded using the host.readFile(String fileName) function
        the function will look in the resources for the email-ext plugin first, and then in the $JENKINS_HOME/email-templates directory. No other directories will be searched.
        • script - the script name.
          Defaults to "email-ext.groovy".
        • template - the template filename.
          Defaults to "groovy-html.template"
        • init - true to run the language's init script.
          Defaults to true
        • Available Script Engines
          • ECMAScript - 1.8 (js)
          • Groovy - 1.8.5 (groovy)
      Show
      Steve Rowe added a comment - Yeah, but what is ${FAILED_TESTS}? I mean – can we edit it to detect "<<< FAILURES!" and report 100 lines before this string? Jenkins's Content Token Reference All arguments are optional. Arguments may be given for each token in the form name="value" for strings and in the form name=value for booleans and numbers. The {'s and }'s may be omitted if there are no arguments. Examples: $TOKEN, ${TOKEN}, ${TOKEN, count=100}, ${ENV, var="PATH"} Available Tokens ${DEFAULT_SUBJECT} - This is the default email subject that is configured in Jenkins's system configuration page. ${DEFAULT_CONTENT} - This is the default email content that is configured in Jenkins's system configuration page. ${PROJECT_DEFAULT_SUBJECT} - This is the default email subject for this project. The result of using this token in the advanced configuration is what is in the Default Subject field above. WARNING: Do not use this token in the Default Subject or Content fields. Doing this has an undefined result. ${PROJECT_DEFAULT_CONTENT} - This is the default email content for this project. The result of using this token in the advanced configuration is what is in the Default Content field above. WARNING: Do not use this token in the Default Subject or Content fields. Doing this has an undefined result. ${BUILD_LOG, maxLines , escapeHtml } - Displays the end of the build log. maxLines - display at most this many lines of the log. Defaults to 250. escapeHtml - If true, HTML is escaped. Defaults to false. ${BUILD_LOG_REGEX, regex , linesBefore , linesAfter , maxMatches , showTruncatedLines , substText , escapeHtml , matchedLineHtmlStyle } - Displays lines from the build log that match the regular expression. regex - Lines that match this regular expression are included. See also java.util.regex.Pattern Defaults to "(?i)\b(error|exception|fatal|fail(ed|ure)|un(defined|resolved))\b". linesBefore - The number of lines to include before the matching line. Lines that overlap with another match or linesAfter are only included once. Defaults to 0. linesAfter - The number of lines to include after the matching line. Lines that overlap with another match or linesBefore are only included once. Defaults to 0. maxMatches - The maximum number of matches to include. If 0, all matches will be included. Defaults to 0. showTruncatedLines - If true , include ...truncated ### lines... lines. Defaults to true. substText - If non-null, insert this text into the email rather than the entire line. Defaults to null. escapeHtml - If true, escape HTML. Defaults to false. matchedLineHtmlStyle - If non-null, output HTML. matched lines will become <b style="your-style-value">html escaped matched line</b> . Defaults to null. ${BUILD_LOG_EXCERPT, start , end } - Displays an excerpt from the build log. start - Regular expression to match the excerpt starting line to be included (exluded). See java.util.regex.Pattern end - Regular expression to match the excerpt ending line to be included (exluded) ${BUILD_NUMBER} - Displays the number of the current build. ${BUILD_STATUS} - Displays the status of the current build. (failing, success, etc...) ${BUILD_URL} - Displays the URL to the current build. ${CHANGES, showPaths , showDependencies , format , pathFormat } - Displays the changes since the last build. showDependencies - if true, changes to projects this build depends on are shown. Defaults to false. showPaths - if true, the paths modified by a commit are shown. Defaults to false. format - for each commit listed, a string containing %X, where %X is one of %a for author, %d for date, %m for message, %p for paths, or %r for revision. Not all revision systems support %d and %r. If specified, showPaths is ignored. Defaults to " [%a] %m\n". pathFormat - a string containing %p to indicate how to print paths. Defaults to "\t%p\n". ${CHANGES_SINCE_LAST_SUCCESS, reverse , format , showPaths , changesFormat , pathFormat } - Displays the changes since the last successful build. <ul> reverse - indicates that most recent builds should be at the top. Defaults to false. format - for each build listed, a string containing %X, where %X is one of %c for changes, or %n for build number. Defaults to "Changes for Build #%n\n%c\n". showPaths , changesFormat , pathFormat - defined as showPaths , format , and pathFormat from ${CHANGES}, respectively. ${CHANGES_SINCE_LAST_UNSTABLE, reverse , format , showPaths , changesFormat , pathFormat } - Displays the changes since the last unstable or successful build. reverse - indicates that most recent builds should be at the top. Defaults to false. format - for each build listed, a string containing %X, where %X is one of %c for changes, or %n for build number. Defaults to "Changes for Build #%n\n%c\n". showPaths , changesFormat , pathFormat - defined as showPaths , format , and pathFormat from ${CHANGES}, respectively. ${ENV, var } - Displays an environment variable. var - the name of the environment variable to display. If "", show all. Defaults to "". ${FAILED_TESTS, showStack , maxTests } - Displays failing unit test information, if any tests have failed. showStack - indicates that most recent builds should be at the top. Defaults to true. maxTests - display at most this many failing tests. No limit is set by default. ${JENKINS_URL} - Displays the URL to the Jenkins server. (You can change this on the system configuration page.) ${HUDSON_URL} - deprecated, please use $JENKINS_URL ${PROJECT_NAME} - Displays the project's name. ${PROJECT_URL} - Displays a URL to the project's page. ${SVN_REVISION} - Displays the subversion revision number. ${CAUSE} - Displays the cause of the build. ${JELLY_SCRIPT, template } - Custom message content generated from a Jelly script template. There are two templates provided: "html" and "text". Custom Jelly templates should be placed in $JENKINS_HOME/email-templates. When using custom templates, the template filename without ".jelly" should be used for the "template" argument. template - the template name. Defaults to "html". ${FILE, path } - Includes the content of a specified file. path - The path to the file. Relative to the workspace root. ${TEST_COUNTS, var } - Displays the number of tests. var - Defaults to "total". total - the number of all tests. fail - the number of failed tests. skip - the number of skipped tests. ${SCRIPT, script , template , init } - Custom message content generated from a script using JSR 223. Custom scripts should be placed in $JENKINS_HOME/email-templates. When using custom scripts, the script filename WITH .py/.rb/etc should be used for the "script" argument. templates and other items may be loaded using the host.readFile(String fileName) function the function will look in the resources for the email-ext plugin first, and then in the $JENKINS_HOME/email-templates directory. No other directories will be searched. script - the script name. Defaults to "email-ext.groovy". template - the template filename. Defaults to "groovy-html.template" init - true to run the language's init script. Defaults to true Available Script Engines ECMAScript - 1.8 (js) Groovy - 1.8.5 (groovy)
      Hide
      Dawid Weiss added a comment -

      Ok, so BUILD_LOG_REGEX should do! I don't know Jenkins but any failed test (suite) will have the "<<< FAILURES!" marker attached – feel free to experiment

      Show
      Dawid Weiss added a comment - Ok, so BUILD_LOG_REGEX should do! I don't know Jenkins but any failed test (suite) will have the "<<< FAILURES!" marker attached – feel free to experiment
      Hide
      Steve Rowe added a comment -

      I'll switch it to the following - hopefully it will capture everything (any length multiline) between the suite header and "<<< FAILURES!"

      Build: ${BUILD_URL}
      
      ${FAILED_TESTS}
      
      Build Log (for compile errors):
      ${BUILD_LOG_REGEX,regex="(?s:\\[java4\\]\\s*Suite:.*?<<<\\s*FAILURES!)"}
      
      Show
      Steve Rowe added a comment - I'll switch it to the following - hopefully it will capture everything (any length multiline) between the suite header and "<<< FAILURES!" Build: ${BUILD_URL} ${FAILED_TESTS} Build Log (for compile errors): ${BUILD_LOG_REGEX,regex="(?s:\\[java4\\]\\s*Suite:.*?<<<\\s*FAILURES!)"}
      Hide
      Steve Rowe added a comment -

      regex="(?s:\\[java4\\]\\s*Suite:.*?<<<
      s*FAILURES!)"

      Hmm, that won't work - it'll grab everything from the first "Suite:" to "<<< FAILURES!", including any number of (non-failing) Suite mentions inbetween.

      I guess "<<< FAILURES!" and 100 previous lines will work, but I'd rather get the exact region. I'll work on it.

      Show
      Steve Rowe added a comment - regex="(?s:\\[java4\\]\\s*Suite:.*?<<< s*FAILURES!)" Hmm, that won't work - it'll grab everything from the first "Suite:" to "<<< FAILURES!", including any number of (non-failing) Suite mentions inbetween. I guess "<<< FAILURES!" and 100 previous lines will work, but I'd rather get the exact region. I'll work on it.
      Hide
      Dawid Weiss added a comment -

      This will be a killer regexp, I can feel it

      Show
      Dawid Weiss added a comment - This will be a killer regexp, I can feel it
      Hide
      Robert Muir added a comment -

      Is there also a way we could improve the output of other checks (e.g. the javadocs warnings task, two javadocs checkers in javadocs-lint, and the rat-checker)
      so that if it causes a build failure its included as well, rather than just "No tests ran" or "All tests passed" or whatever it does today?

      Show
      Robert Muir added a comment - Is there also a way we could improve the output of other checks (e.g. the javadocs warnings task, two javadocs checkers in javadocs-lint, and the rat-checker) so that if it causes a build failure its included as well, rather than just "No tests ran" or "All tests passed" or whatever it does today?
      Hide
      Robert Muir added a comment -

      By the way: I'm not suggesting to make the regex even more hairy, i'm just wondering if we can modify these tasks
      so that when they fail, they can include certain symbols/stuff to ensure they are included in the summary...

      Show
      Robert Muir added a comment - By the way: I'm not suggesting to make the regex even more hairy, i'm just wondering if we can modify these tasks so that when they fail, they can include certain symbols/stuff to ensure they are included in the summary...
      Hide
      Steve Rowe added a comment -

      This one seems to work (from Perl against a recent Jenkins log with a failure):

      [^\r\n]*\[junit4\]\s*Suite:.*[\r\n]+[^\r\n]*\[junit4\]\s*(?!Completed)(?!IGNOR)\S(?s:.*?)<<<\s*FAILURES!
      

      I'll change the Lucene-Solr-tests-only-trunk job configuration to use it (after escaping backslashes).

      Show
      Steve Rowe added a comment - This one seems to work (from Perl against a recent Jenkins log with a failure): [^\r\n]*\[junit4\]\s*Suite:.*[\r\n]+[^\r\n]*\[junit4\]\s*(?!Completed)(?!IGNOR)\S(?s:.*?)<<<\s*FAILURES! I'll change the Lucene-Solr-tests-only-trunk job configuration to use it (after escaping backslashes).
      Hide
      Steve Rowe added a comment -

      Is there also a way we could improve the output of other checks (e.g. the javadocs warnings task, two javadocs checkers in javadocs-lint, and the rat-checker)
      so that if it causes a build failure its included as well, rather than just "No tests ran" or "All tests passed" or whatever it does today?

      It may be that alternates can be added to the BUILD_LOG_REGEX - I'll look at the output and see if any changes need to be made to enable that.

      Show
      Steve Rowe added a comment - Is there also a way we could improve the output of other checks (e.g. the javadocs warnings task, two javadocs checkers in javadocs-lint, and the rat-checker) so that if it causes a build failure its included as well, rather than just "No tests ran" or "All tests passed" or whatever it does today? It may be that alternates can be added to the BUILD_LOG_REGEX - I'll look at the output and see if any changes need to be made to enable that.
      Hide
      Steve Rowe added a comment -

      I plan on adding the following (as suggested by Robert) as alternations to the BUILD_LOG_REGEX for all non-Maven Jenkins jobs (some of these things don't run under the Maven jobs, and Maven's output is different enough that it'll require separate treatment):

      the javadocs warnings task

      (?:[^\r\n]*\[javadoc\].*\r?\n)*[^\r\n]*\[javadoc\]\s*[1-9]\d*\s+warnings.*\r?\n
      

      two javadocs checkers in javadocs-lint

      Output from javadocs-lint seems to show up only when there's a problem, so any output from it will always be extracted by the following regex:

      [^\r\n]*javadocs-lint:.*\r?\n(?:[^\r\n]*\[echo\].*\r?\n)*
      

      and the rat-checker

      [^\r\n]*rat-sources:\s+\[echo\].*(?:\r?\n[^\r\n]*\[echo\].*)*\s*[1-9]\d*\s+Unknown\s+Licenses.*\r?\n(?:[^\r\n]*\[echo\].*\r?\n)*
      

      Along with two others:

      1. Compilation failures:
        (?:[^\r\n]*\[javac\].*\r?\n)*[^\r\n]*\[javac\]\s*[1-9]\d*\s*error.*\r?\n
        
      2. Jenkins FATAL errors:
        [^\r\n]*FATAL:(?s:.*)
        
      Show
      Steve Rowe added a comment - I plan on adding the following (as suggested by Robert) as alternations to the BUILD_LOG_REGEX for all non-Maven Jenkins jobs (some of these things don't run under the Maven jobs, and Maven's output is different enough that it'll require separate treatment): the javadocs warnings task (?:[^\r\n]*\[javadoc\].*\r?\n)*[^\r\n]*\[javadoc\]\s*[1-9]\d*\s+warnings.*\r?\n two javadocs checkers in javadocs-lint Output from javadocs-lint seems to show up only when there's a problem, so any output from it will always be extracted by the following regex: [^\r\n]*javadocs-lint:.*\r?\n(?:[^\r\n]*\[echo\].*\r?\n)* and the rat-checker [^\r\n]*rat-sources:\s+\[echo\].*(?:\r?\n[^\r\n]*\[echo\].*)*\s*[1-9]\d*\s+Unknown\s+Licenses.*\r?\n(?:[^\r\n]*\[echo\].*\r?\n)* Along with two others: Compilation failures: (?:[^\r\n]*\[javac\].*\r?\n)*[^\r\n]*\[javac\]\s*[1-9]\d*\s*error.*\r?\n Jenkins FATAL errors: [^\r\n]*FATAL:(?s:.*)
      Hide
      Steve Rowe added a comment -

      I plan on adding the following (as suggested by Robert) as alternations to the BUILD_LOG_REGEX for all non-Maven Jenkins jobs

      Done. Here's the full Editable Email Notification/Default Content configuration:

      Build: ${BUILD_URL}
      
      ${FAILED_TESTS}
      
      Build Log:
      ${BUILD_LOG_REGEX,regex="(?x:
      # Compilation failures
      (?:[^\\r\\n]*\\[javac\\].*\\r?\\n)*[^\\r\\n]*\\[javac\\]\\s*[1-9]\\d*\\s*error.*\\r?\\n
      # Test failures
      |[^\\r\\n]*\\[junit4\\]\\s*Suite:.*[\\r\\n]+[^\\r\\n]*\\[junit4\\]\\s*(?!Completed)(?!IGNOR)\\S(?s:.*?)<<<\\s*FAILURES!
      # License problems
      |[^\\r\\n]*rat-sources:\\s+\\[echo\\].*(?:\\r?\\n[^\\r\\n]*\\[echo\\].*)*\\s*[1-9]\\d*\\s+Unknown\\s+Licenses.*\\r?\\n(?:[^\\r\\n]*\\[echo\\].*\\r?\\n)*
      # Javadocs warnings
      |(?:[^\\r\\n]*\\[javadoc\\].*\\r?\\n)*[^\\r\\n]*\\[javadoc\\]\\s*[1-9]\\d*\\s+warnings.*\\r?\\n
      # Other javadocs problems (broken links and missing javadocs)
      |[^\\r\\n]*javadocs-lint:.*\\r?\\n(?:[^\\r\\n]*\\[echo\\].*\\r?\\n)*
      # Jenkins problems
      |[^\\r\\n]*FATAL:(?s:.*)
      )"}
      
      Show
      Steve Rowe added a comment - I plan on adding the following (as suggested by Robert) as alternations to the BUILD_LOG_REGEX for all non-Maven Jenkins jobs Done. Here's the full Editable Email Notification / Default Content configuration: Build: ${BUILD_URL} ${FAILED_TESTS} Build Log: ${BUILD_LOG_REGEX,regex="(?x: # Compilation failures (?:[^\\r\\n]*\\[javac\\].*\\r?\\n)*[^\\r\\n]*\\[javac\\]\\s*[1-9]\\d*\\s*error.*\\r?\\n # Test failures |[^\\r\\n]*\\[junit4\\]\\s*Suite:.*[\\r\\n]+[^\\r\\n]*\\[junit4\\]\\s*(?!Completed)(?!IGNOR)\\S(?s:.*?)<<<\\s*FAILURES! # License problems |[^\\r\\n]*rat-sources:\\s+\\[echo\\].*(?:\\r?\\n[^\\r\\n]*\\[echo\\].*)*\\s*[1-9]\\d*\\s+Unknown\\s+Licenses.*\\r?\\n(?:[^\\r\\n]*\\[echo\\].*\\r?\\n)* # Javadocs warnings |(?:[^\\r\\n]*\\[javadoc\\].*\\r?\\n)*[^\\r\\n]*\\[javadoc\\]\\s*[1-9]\\d*\\s+warnings.*\\r?\\n # Other javadocs problems (broken links and missing javadocs) |[^\\r\\n]*javadocs-lint:.*\\r?\\n(?:[^\\r\\n]*\\[echo\\].*\\r?\\n)* # Jenkins problems |[^\\r\\n]*FATAL:(?s:.*) )"}
      Hide
      Steve Rowe added a comment -

      I'm going to add one more to the regex:

      # Third-party dependency license/notice problems
      |[^\\r\\n]*validate:.*\\r?\\n[^\\r\\n]*\\[echo\\].*\\r?\\n(?:[^\\r\\n]*\\[licenses\\].*\\r?\\n)*[^\\r\\n]*\\[licenses\\].*[1-9]\\d*\\s+error.*\\r?\\n
      
      Show
      Steve Rowe added a comment - I'm going to add one more to the regex: # Third-party dependency license/notice problems |[^\\r\\n]*validate:.*\\r?\\n[^\\r\\n]*\\[echo\\].*\\r?\\n(?:[^\\r\\n]*\\[licenses\\].*\\r?\\n)*[^\\r\\n]*\\[licenses\\].*[1-9]\\d*\\s+error.*\\r?\\n
      Hide
      Steve Rowe added a comment -

      I'm going to add one more to the regex

      Done - added to the configuration on all non-Maven Jenkins jobs

      Show
      Steve Rowe added a comment - I'm going to add one more to the regex Done - added to the configuration on all non-Maven Jenkins jobs
      Hide
      Robert Muir added a comment -

      awesome! thank you!

      Show
      Robert Muir added a comment - awesome! thank you!
      Hide
      Dawid Weiss added a comment -

      Thanks for working on this, Steve. It'll really be useful.

      Show
      Dawid Weiss added a comment - Thanks for working on this, Steve. It'll really be useful.
      Hide
      Steve Rowe added a comment -

      Two problems:

      1. Spreading the BUILD_LOG_REGEX regex value over multiple lines is not supported by Jenkins's email templating functionality, which is provided by the Jenkins Email Extension Plugin (email-ext) https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin. See the configuration token parsing regexes in ContentBuilder.Tokenizer, in particular the comment over the stringRegex field:
        // Sequence of (1) not \ " CR LF and (2) \ followed by non line terminator
        private static final String stringRegex = "\"([^\\\\\"\\r\\n]|(\\\\.))*\"";

        This could be fixed by allowing line terminators to be escaped:

        // Sequence of (1) not \ " CR LF and (2) \ followed by any non-CR/LF character or (CR)LF
        private static final String stringRegex = "\"([^\\\\\"\\r\\n]|(\\\\(?:.|\r?\n)))*\"";

        I submitted a Jenkins JIRA issue for this: https://issues.jenkins-ci.org/browse/JENKINS-13976.

      2. BuildLogRegexContent, the content parser for BUILD_LOG_REGEX, matches line-by-line, so regexes targeting multiple lines will fail. I can see two possible routes to address this:
        1. The BUILD_LOG_EXCERPT token allows specification of begin/end line regexes, and includes everything inbetween matches. I'm doubtful this will enable capture of the stuff we want, though.
        2. I'll try to add an argument to BUILD_LOG_REGEX to enable multi-line content matching, and make a pull request/JIRA issue to get it included in the next release of the plugin.

      In the mean time, I'll switch the configuration in our Jenkins jobs to the following:

      Build: ${BUILD_URL}
      
      ${FAILED_TESTS}
      
      Build Log:
      ${BUILD_LOG_REGEX,regex="[ \\t]*(?:\\[javac\\]\\s+[1-9]\\d*\\s+error|\\[junit4\\].*<<<\\s+FAILURES!|\\[javadoc\\]\\s+[1-9]\\d*\\s+warning).*",linesBefore=100}
      ${BUILD_LOG_REGEX,regex="[ \\t]*\\[echo\\].*)*\\s*[1-9]\\d*\\s+Unknown\\s+Licenses.*",linesBefore=17,linesAfter=20}
      ${BUILD_LOG_REGEX,regex="[ \\t]*javadocs-lint:.*",linesBefore=0,linesAfter=75}
      ${BUILD_LOG_REGEX,regex=".*FATAL:.*",linesBefore=0,linesAfter=100}
      
      Show
      Steve Rowe added a comment - Two problems: Spreading the BUILD_LOG_REGEX regex value over multiple lines is not supported by Jenkins's email templating functionality, which is provided by the Jenkins Email Extension Plugin (email-ext) https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin . See the configuration token parsing regexes in ContentBuilder.Tokenizer , in particular the comment over the stringRegex field: // Sequence of (1) not \ " CR LF and (2) \ followed by non line terminator private static final String stringRegex = "\" ([^\\\\\ "\\r\\n]|(\\\\.))*\" "; This could be fixed by allowing line terminators to be escaped: // Sequence of (1) not \ " CR LF and (2) \ followed by any non-CR/LF character or (CR)LF private static final String stringRegex = "\" ([^\\\\\ "\\r\\n]|(\\\\(?:.|\r?\n)))*\" "; I submitted a Jenkins JIRA issue for this: https://issues.jenkins-ci.org/browse/JENKINS-13976 . BuildLogRegexContent, the content parser for BUILD_LOG_REGEX, matches line-by-line , so regexes targeting multiple lines will fail. I can see two possible routes to address this: The BUILD_LOG_EXCERPT token allows specification of begin/end line regexes, and includes everything inbetween matches. I'm doubtful this will enable capture of the stuff we want, though. I'll try to add an argument to BUILD_LOG_REGEX to enable multi-line content matching, and make a pull request/JIRA issue to get it included in the next release of the plugin. In the mean time, I'll switch the configuration in our Jenkins jobs to the following: Build: ${BUILD_URL} ${FAILED_TESTS} Build Log: ${BUILD_LOG_REGEX,regex="[ \\t]*(?:\\[javac\\]\\s+[1-9]\\d*\\s+error|\\[junit4\\].*<<<\\s+FAILURES!|\\[javadoc\\]\\s+[1-9]\\d*\\s+warning).*",linesBefore=100} ${BUILD_LOG_REGEX,regex="[ \\t]*\\[echo\\].*)*\\s*[1-9]\\d*\\s+Unknown\\s+Licenses.*",linesBefore=17,linesAfter=20} ${BUILD_LOG_REGEX,regex="[ \\t]*javadocs-lint:.*",linesBefore=0,linesAfter=75} ${BUILD_LOG_REGEX,regex=".*FATAL:.*",linesBefore=0,linesAfter=100}
      Hide
      Michael McCandless added a comment -

      Steve you are a regexp God.

      Show
      Michael McCandless added a comment - Steve you are a regexp God.
      Hide
      Steve Rowe added a comment -

      I'll switch the configuration in our Jenkins jobs to the following ...

      Done.

      Show
      Steve Rowe added a comment - I'll switch the configuration in our Jenkins jobs to the following ... Done.
      Hide
      Steve Rowe added a comment -

      This one had syntax problems (a recent test failure notification email complained about it):

      ${BUILD_LOG_REGEX,regex="[ \\t]*\\[echo\\].*)*\\s*[1-9]\\d*\\s+Unknown\\s+Licenses.*",linesBefore=17,linesAfter=20}
      

      I switched it to the following on all non-Maven Jenkins job configuratinos:

      ${BUILD_LOG_REGEX,regex="[ \\t]*\\[echo\\]\\s+[1-9]\\d*\\s+Unknown\\s+Licenses.*",linesBefore=17,linesAfter=20}
      
      Show
      Steve Rowe added a comment - This one had syntax problems (a recent test failure notification email complained about it): ${BUILD_LOG_REGEX,regex="[ \\t]*\\[echo\\].*)*\\s*[1-9]\\d*\\s+Unknown\\s+Licenses.*",linesBefore=17,linesAfter=20} I switched it to the following on all non-Maven Jenkins job configuratinos: ${BUILD_LOG_REGEX,regex="[ \\t]*\\[echo\\]\\s+[1-9]\\d*\\s+Unknown\\s+Licenses.*",linesBefore=17,linesAfter=20}
      Hide
      Steve Rowe added a comment - - edited

      Spreading the BUILD_LOG_REGEX regex value over multiple lines is not supported by Jenkins's email templating functionality
      [...]
      This could be fixed by allowing line terminators to be escaped:
      [...]
      I submitted a Jenkins JIRA issue for this: https://issues.jenkins-ci.org/browse/JENKINS-13976.

      I forked the email-ext project on github and made a pull request, which has now been incorporated into the upstream project.

      Show
      Steve Rowe added a comment - - edited Spreading the BUILD_LOG_REGEX regex value over multiple lines is not supported by Jenkins's email templating functionality [...] This could be fixed by allowing line terminators to be escaped: [...] I submitted a Jenkins JIRA issue for this: https://issues.jenkins-ci.org/browse/JENKINS-13976 . I forked the email-ext project on github and made a pull request, which has now been incorporated into the upstream project.
      Hide
      Steve Rowe added a comment - - edited

      BuildLogRegexContent, the content parser for BUILD_LOG_REGEX, matches line-by-line, so regexes targeting multiple lines will fail. [...] I'll try to add an argument to BUILD_LOG_REGEX to enable multi-line content matching, and make a pull request/JIRA issue to get it included in the next release of the plugin.

      I made a new content token named BUILD_LOG_MULTILINE_REGEX in my fork of the email-ext-plugin - see JENKINS-14000. My pull request (#40) was merged into the upstream project a few days ago, so the next release will include this change. (Looks like this plugin releases once a month or so, so the wait before we can use it shouldn't be too long, assuming the plugins on the ASF Jenkins instance are kept up-to-date.)

      Show
      Steve Rowe added a comment - - edited BuildLogRegexContent, the content parser for BUILD_LOG_REGEX, matches line-by-line, so regexes targeting multiple lines will fail. [...] I'll try to add an argument to BUILD_LOG_REGEX to enable multi-line content matching, and make a pull request/JIRA issue to get it included in the next release of the plugin. I made a new content token named BUILD_LOG_MULTILINE_REGEX in my fork of the email-ext-plugin - see JENKINS-14000 . My pull request ( #40 ) was merged into the upstream project a few days ago, so the next release will include this change. (Looks like this plugin releases once a month or so, so the wait before we can use it shouldn't be too long, assuming the plugins on the ASF Jenkins instance are kept up-to-date.)
      Hide
      Dawid Weiss added a comment -

      Assigning to you, Steven, since you do the heavy lifting here anyway. Thanks!

      Show
      Dawid Weiss added a comment - Assigning to you, Steven, since you do the heavy lifting here anyway. Thanks!
      Hide
      Steve Rowe added a comment -

      v2.22 of the Jenkins Email-ext plugin was released today, incorporating the new BUILD_LOG_MULTILINE_REGEX content token functionality. Olivier Lamy kindly upgraded the plugin in ASF Jenkins and restarted it.

      I'm going to re-try using the full-log multiline regex I posted in comment-13286757, with modification I've since added to Lucene jobs' configurations (using multiple BUILD_LOG_REGEX content tokens, one per context window a.k.a. linesBefore/linesAfter).

      Hopefully now we'll be able to get exactly the right snippet from the logs for each failure case.

      Show
      Steve Rowe added a comment - v2.22 of the Jenkins Email-ext plugin was released today, incorporating the new BUILD_LOG_MULTILINE_REGEX content token functionality. Olivier Lamy kindly upgraded the plugin in ASF Jenkins and restarted it. I'm going to re-try using the full-log multiline regex I posted in comment-13286757 , with modification I've since added to Lucene jobs' configurations (using multiple BUILD_LOG_REGEX content tokens, one per context window a.k.a. linesBefore/linesAfter). Hopefully now we'll be able to get exactly the right snippet from the logs for each failure case.
      Hide
      Steve Rowe added a comment - - edited

      Here is the Editable Email Notification/Default Content configuration I applied to all non-Maven ASF Jenkins jobs (modulo whitespace in front of the trailing line-continuation backslashes):

      edit: BUILD_LOG_REGEX -> BUILD_LOG_MULTILINE_REGEX

      Build: ${BUILD_URL}
      
      ${FAILED_TESTS}
      
      Build Log:
      ${BUILD_LOG_MULTILINE_REGEX,regex="(?x:                                                                                      \
      # Compilation failures                                                                                             \
      (?:.*\\[javac\\].*\\r?\\n)*.*\\[javac\\]\\s+[1-9]\\d*\\s+error.*\\r?\\n                                            \
      # Test failures                                                                                                    \
      |.*\\[junit4\\]\\s*Suite:.*[\\r\\n]+.*\\[junit4\\]\\s*(?!Completed)(?!IGNOR)\\S(?s:.*?)<<<\\s*FAILURES!            \
      # Source file license problems                                                                                     \
      |.*rat-sources:.*(?:\\r?\\n.*\\[echo\\].*)*\\s+[1-9]\\d*\\s+Unknown\\s+Licenses.*\\r?\\n(?:.*\\[echo\\].*\\r?\\n)* \
      # Third-party dependency license problems - include 2 preceding lines and 1 following line                         \
      |(?:.*\\r?\\n){2}.*\\[licenses\\]\\s+MISSING\\s+sha1(?:.*\\r?\\n){2}                                               \
      # Javadoc warnings                                                                                                 \
      |(?:.*\\[javadoc\\].*\\r?\\n)*.*\\[javadoc\\]\\s*[1-9]\\d*\\s+warnings.*\\r?\\n                                    \
      # Other javadocs problems: broken links and missing javadocs                                                       \
      |.*javadocs-lint:.*\\r?\\n(?:.*\\[echo\\].*\\r?\\n)*                                                               \
      # Thread dumps - include 1 preceding line and the remainder of the log                                             \
      |.*\\r?\\n.*Full thread dump(?s:.*)                                                                                \
      # Jenkins problems - include the remainder of the log                                                              \
      |.*(?:FATAL|ERROR):(?s:.*)                                                                                         \
      # Include the Ant call stack - include the remainder of the log                                                    \
      |.*BUILD\\s+FAILED(?s:.*)                                                                                          \
      )"}
      
      Show
      Steve Rowe added a comment - - edited Here is the Editable Email Notification / Default Content configuration I applied to all non-Maven ASF Jenkins jobs (modulo whitespace in front of the trailing line-continuation backslashes): edit : BUILD_LOG_REGEX -> BUILD_LOG_MULTILINE_REGEX Build: ${BUILD_URL} ${FAILED_TESTS} Build Log: ${BUILD_LOG_MULTILINE_REGEX,regex="(?x: \ # Compilation failures \ (?:.*\\[javac\\].*\\r?\\n)*.*\\[javac\\]\\s+[1-9]\\d*\\s+error.*\\r?\\n \ # Test failures \ |.*\\[junit4\\]\\s*Suite:.*[\\r\\n]+.*\\[junit4\\]\\s*(?!Completed)(?!IGNOR)\\S(?s:.*?)<<<\\s*FAILURES! \ # Source file license problems \ |.*rat-sources:.*(?:\\r?\\n.*\\[echo\\].*)*\\s+[1-9]\\d*\\s+Unknown\\s+Licenses.*\\r?\\n(?:.*\\[echo\\].*\\r?\\n)* \ # Third-party dependency license problems - include 2 preceding lines and 1 following line \ |(?:.*\\r?\\n){2}.*\\[licenses\\]\\s+MISSING\\s+sha1(?:.*\\r?\\n){2} \ # Javadoc warnings \ |(?:.*\\[javadoc\\].*\\r?\\n)*.*\\[javadoc\\]\\s*[1-9]\\d*\\s+warnings.*\\r?\\n \ # Other javadocs problems: broken links and missing javadocs \ |.*javadocs-lint:.*\\r?\\n(?:.*\\[echo\\].*\\r?\\n)* \ # Thread dumps - include 1 preceding line and the remainder of the log \ |.*\\r?\\n.*Full thread dump(?s:.*) \ # Jenkins problems - include the remainder of the log \ |.*(?:FATAL|ERROR):(?s:.*) \ # Include the Ant call stack - include the remainder of the log \ |.*BUILD\\s+FAILED(?s:.*) \ )"}
      Hide
      Steve Rowe added a comment -

      ${BUILD_LOG_REGEX,regex="(?x: [...]

      Ack! I forgot to switch the previous version to use the new BUILD_LOG_MULTILINE_REGEX content token!

      Fixed now...

      Show
      Steve Rowe added a comment - ${BUILD_LOG_REGEX,regex="(?x: [...] Ack! I forgot to switch the previous version to use the new BUILD_LOG_ MULTILINE _REGEX content token! Fixed now...
      Hide
      Steve Rowe added a comment -

      A bug effectively prevented use of the new BUILD_LOG_MULTILINE_REGEX config: JENKINS-14132. The fix on that issue has been incorporated in v2.24.1 of the Email-ext plugin, and Olivier Lamy has kindly installed this version on ASF Jenkins. Uwe helped work out the kinks, and has been running the patched version for a while on his Jenkins instance - upgraded to the release version now.

      I will close as fixed later today once it's clear the ASF configuration is functional.

      Show
      Steve Rowe added a comment - A bug effectively prevented use of the new BUILD_LOG_MULTILINE_REGEX config: JENKINS-14132 . The fix on that issue has been incorporated in v2.24.1 of the Email-ext plugin, and Olivier Lamy has kindly installed this version on ASF Jenkins. Uwe helped work out the kinks, and has been running the patched version for a while on his Jenkins instance - upgraded to the release version now. I will close as fixed later today once it's clear the ASF configuration is functional.
      Hide
      Steve Rowe added a comment -

      Here's the current configuration regex (edited so that the trailing backslashes are lined up):

      Build Log:
      ${BUILD_LOG_MULTILINE_REGEX,regex="(?x:                                                                         \
                                                                                                                      \
      (?:.*\\[javac\\]\\s++(?![1-9]\\d*\\s+error).*\\r?\\n)*+.*\\[javac\\]\\s+[1-9]\\d*\\s+error.*\\r?\\n             \
                                                                                                                      \
      |.*\\[(?:junit4:)?junit4\\]\\s+Suite:.*+\\s++.*\\[(?:junit4:)?junit4\\]\\s++(?!Completed)(?!IGNOR).*\\r?\\n     \
       (?:.*\\[(?:junit4:)?junit4\\]\\s++(?!Completed\\s+.*<<<\\s*FAILURES!).*\\r?\\n)*+                              \
       .*\\[(?:junit4:)?junit4\\]\\s++Completed\\s+.*<<<\\s*FAILURES!\\r?\\n                                          \
                                                                                                                      \
      |.*rat-sources:.*\\r?\\n                                                                                        \
       (?:\\s*+\\[echo\\]\\s*\\r?\\n|\\s*+\\[echo\\]\\s++(?![1-9]\\d*\\s+Unknown\\s+License)\\S.*\\r?\\n)*+           \
       \\s*+\\[echo\\]\\s+[1-9]\\d*\\s+Unknown\\s+License.*\\r?\\n                                                    \
       (?:\\s*+\\[echo\\].*\\r?\\n)*+                                                                                 \
                                                                                                                      \
      |(?:.*\\r?\\n){2}.*\\[licenses\\]\\s+MISSING\\s+sha1(?:.*\\r?\\n){2}                                            \
                                                                                                                      \
      |.*check-licenses:.*\\r?\\n\\s*\\[echo\\].*\\r?\\n\\s*\\[licenses\\]\\s+MISSING\\s+LICENSE.*\\r?\\n             \
       (?:\\s*+\\[licenses\\].*\\r?\\n)++                                                                             \
                                                                                                                      \
      |(?:.*\\[javadoc\\]\\s++(?![1-9]\\d*\\s+warning).+\\r?\\n)*+.*\\[javadoc\\]\\s+[1-9]\\d*\\s+warning.*\\r?\\n    \
                                                                                                                      \
      |.*javadocs-lint:.*\\r?\\n(?:.*\\[exec\\].*\\r?\\n)*+                                                           \
                                                                                                                      \
      |.*check-forbidden-apis:.*\\r?\\n                                                                               \
       (?:\\s*+\\[forbidden-apis\\]\\s*\\r?\\n                                                                        \
        |\\s*+\\[forbidden-apis\\]\\s++(?!Scanned\\s+\\d+\\s+class\\s+file\\(s\\))\\S.*\\r?\\n)*+                     \
       \\s*+\\[forbidden-apis\\]\\s++Scanned\\s+\\d+\\s+class\\s+file\\(s\\).*[1-9]\\d*\\s+error\\(s\\)\\.\\r?\\n     \
                                                                                                                      \
      |.*\\r?\\n.*Full\\s+thread\\s+dump(?s:.*+)                                                                      \
                                                                                                                      \
      |.*(?:FATAL|ERROR):(?s:.*+)                                                                                     \
                                                                                                                      \
      |.*BUILD\\s+FAILED(?s:.*+)                                                                                      \
      )"}
      
      Show
      Steve Rowe added a comment - Here's the current configuration regex (edited so that the trailing backslashes are lined up): Build Log: ${BUILD_LOG_MULTILINE_REGEX,regex="(?x: \ \ (?:.*\\[javac\\]\\s++(?![1-9]\\d*\\s+error).*\\r?\\n)*+.*\\[javac\\]\\s+[1-9]\\d*\\s+error.*\\r?\\n \ \ |.*\\[(?:junit4:)?junit4\\]\\s+Suite:.*+\\s++.*\\[(?:junit4:)?junit4\\]\\s++(?!Completed)(?!IGNOR).*\\r?\\n \ (?:.*\\[(?:junit4:)?junit4\\]\\s++(?!Completed\\s+.*<<<\\s*FAILURES!).*\\r?\\n)*+ \ .*\\[(?:junit4:)?junit4\\]\\s++Completed\\s+.*<<<\\s*FAILURES!\\r?\\n \ \ |.*rat-sources:.*\\r?\\n \ (?:\\s*+\\[echo\\]\\s*\\r?\\n|\\s*+\\[echo\\]\\s++(?![1-9]\\d*\\s+Unknown\\s+License)\\S.*\\r?\\n)*+ \ \\s*+\\[echo\\]\\s+[1-9]\\d*\\s+Unknown\\s+License.*\\r?\\n \ (?:\\s*+\\[echo\\].*\\r?\\n)*+ \ \ |(?:.*\\r?\\n){2}.*\\[licenses\\]\\s+MISSING\\s+sha1(?:.*\\r?\\n){2} \ \ |.*check-licenses:.*\\r?\\n\\s*\\[echo\\].*\\r?\\n\\s*\\[licenses\\]\\s+MISSING\\s+LICENSE.*\\r?\\n \ (?:\\s*+\\[licenses\\].*\\r?\\n)++ \ \ |(?:.*\\[javadoc\\]\\s++(?![1-9]\\d*\\s+warning).+\\r?\\n)*+.*\\[javadoc\\]\\s+[1-9]\\d*\\s+warning.*\\r?\\n \ \ |.*javadocs-lint:.*\\r?\\n(?:.*\\[exec\\].*\\r?\\n)*+ \ \ |.*check-forbidden-apis:.*\\r?\\n \ (?:\\s*+\\[forbidden-apis\\]\\s*\\r?\\n \ |\\s*+\\[forbidden-apis\\]\\s++(?!Scanned\\s+\\d+\\s+class\\s+file\\(s\\))\\S.*\\r?\\n)*+ \ \\s*+\\[forbidden-apis\\]\\s++Scanned\\s+\\d+\\s+class\\s+file\\(s\\).*[1-9]\\d*\\s+error\\(s\\)\\.\\r?\\n \ \ |.*\\r?\\n.*Full\\s+thread\\s+dump(?s:.*+) \ \ |.*(?:FATAL|ERROR):(?s:.*+) \ \ |.*BUILD\\s+FAILED(?s:.*+) \ )"}
      Hide
      Steve Rowe added a comment -

      ASF Jenkins build log excerpts in notification emails are looking good.

      Show
      Steve Rowe added a comment - ASF Jenkins build log excerpts in notification emails are looking good.

        People

        • Assignee:
          Steve Rowe
          Reporter:
          Dawid Weiss
        • Votes:
          0 Vote for this issue
          Watchers:
          4 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development