Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.0
    • Component/s: Startup Shell Scripts
    • Labels:
      None

      Description

      Allow configuring Flink's Java options with the logging prefix and log rotation. For example, this allows the following configurations to write .jfr and .jit files alongside the existing .log and .out files.

      Configuration for Java Flight Recorder
      env.java.opts: "-XX:+UnlockCommercialFeatures -XX:+UnlockDiagnosticVMOptions -XX:+FlightRecorder -XX:+DebugNonSafepoints -XX:FlightRecorderOptions=defaultrecording=true,dumponexit=true,dumponexitpath=${LOG_PREFIX}.jfr"
      
      Configuration for JitWatch
      env.java.opts: "-XX:+UnlockDiagnosticVMOptions -XX:+TraceClassLoading -XX:+LogCompilation -XX:LogFile=${LOG_PREFIX}.jit -XX:+PrintAssembly"
      

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user greghogan opened a pull request:

          https://github.com/apache/flink/pull/3701

          FLINK-6280 [scripts] Allow logging with Java flags

          Evaluate user-defined Java options immediately before starting services and rotate all log files.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/greghogan/flink 6280_allow_logging_with_java_flags

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3701.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3701


          commit 7213d06775a06a017ce2609321b23f59f04c6157
          Author: Greg Hogan <code@greghogan.com>
          Date: 2017-04-07T17:34:16Z

          FLINK-6280 [scripts] Allow logging with Java flags

          Evaluate user-defined Java options immediately before starting services
          and rotate all log files.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/3701 FLINK-6280 [scripts] Allow logging with Java flags Evaluate user-defined Java options immediately before starting services and rotate all log files. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 6280_allow_logging_with_java_flags Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3701.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3701 commit 7213d06775a06a017ce2609321b23f59f04c6157 Author: Greg Hogan <code@greghogan.com> Date: 2017-04-07T17:34:16Z FLINK-6280 [scripts] Allow logging with Java flags Evaluate user-defined Java options immediately before starting services and rotate all log files.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3701

          Can you explain a bit more what this change does exactly? Does it make sure that if you put bash commands into the java opts to compute flags, it evaluates those?

          Can you elaborate why this needs a change in the log rotation logic?

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3701 Can you explain a bit more what this change does exactly? Does it make sure that if you put bash commands into the java opts to compute flags, it evaluates those? Can you elaborate why this needs a change in the log rotation logic?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user greghogan commented on the issue:

          https://github.com/apache/flink/pull/3701

          I had provided some examples in the JIRA, but yes, it allows delaying evaluation of the java opts (with double quotes) until immediately before the daemon is started. This allows creating additional log files such as for JFR or JitWatch (the two use cases I have needed, I expect there are more). In addition the log rotation, Flink may be running multiple TaskManagers on the same instance so reusing Flink's log naming scheme prevents overwriting files.

          Show
          githubbot ASF GitHub Bot added a comment - Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3701 I had provided some examples in the JIRA, but yes, it allows delaying evaluation of the java opts (with double quotes) until immediately before the daemon is started. This allows creating additional log files such as for JFR or JitWatch (the two use cases I have needed, I expect there are more). In addition the log rotation, Flink may be running multiple TaskManagers on the same instance so reusing Flink's log naming scheme prevents overwriting files.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user greghogan commented on the issue:

          https://github.com/apache/flink/pull/3701

          @StephanEwen added some documentation covering use and examples.

          Show
          githubbot ASF GitHub Bot added a comment - Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3701 @StephanEwen added some documentation covering use and examples.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3701

          Okay, got it. The docs and examples did help a lot.

          This looks good to me, +1 to merge

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3701 Okay, got it. The docs and examples did help a lot. This looks good to me, +1 to merge
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3701#discussion_r112444789

          — Diff: docs/monitoring/application_profiling.md —
          @@ -0,0 +1,54 @@
          +---
          +title: "Application Profiling"
          +nav-parent_id: monitoring
          +nav-pos: 15
          +---
          +<!--
          +Licensed to the Apache Software Foundation (ASF) under one
          +or more contributor license agreements. See the NOTICE file
          +distributed with this work for additional information
          +regarding copyright ownership. The ASF licenses this file
          +to you under the Apache License, Version 2.0 (the
          +"License"); you may not use this file except in compliance
          +with the License. You may obtain a copy of the License at
          +
          + http://www.apache.org/licenses/LICENSE-2.0
          +
          +Unless required by applicable law or agreed to in writing,
          +software distributed under the License is distributed on an
          +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
          +KIND, either express or implied. See the License for the
          +specific language governing permissions and limitations
          +under the License.
          +-->
          +
          +* ToC
          +{:toc}
          +
          +## Overview of Custom Logging with Apache Flink
          +
          +The standalone JobManager, TaskManager, HistoryServer, and ZooKeeper daemons by write `stdout` and `stderr` to files
          — End diff –

          the "by" between "daemons by write" should be removed.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/3701#discussion_r112444789 — Diff: docs/monitoring/application_profiling.md — @@ -0,0 +1,54 @@ +--- +title: "Application Profiling" +nav-parent_id: monitoring +nav-pos: 15 +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +* ToC +{:toc} + +## Overview of Custom Logging with Apache Flink + +The standalone JobManager, TaskManager, HistoryServer, and ZooKeeper daemons by write `stdout` and `stderr` to files — End diff – the "by" between "daemons by write" should be removed.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3701

          Feel free to merge...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3701 Feel free to merge...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3701

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3701
          Hide
          greghogan Greg Hogan added a comment -

          Implemented in 1664447677404e66ef842350d30469c82286cb41

          Show
          greghogan Greg Hogan added a comment - Implemented in 1664447677404e66ef842350d30469c82286cb41

            People

            • Assignee:
              greghogan Greg Hogan
              Reporter:
              greghogan Greg Hogan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development