Details
-
Sub-task
-
Status: Closed
-
Blocker
-
Resolution: Fixed
-
None
Description
In order to make the configuration documentation easier to understand for users, I would propose to he these major sections.
- Basic Setup / Getting Started
- Common Setup Options
- Deployment Targets (K8s, Yarn, Mesos)
- State Backends
- Expert/Internal
- Deprecated Options
The details of what to put into these sections is as follows:
Basic Setups
This section can be manual, does not need to be auto-generated. It should also be a bit more verbose and not just list options and descriptions.
- "Master and TaskManager Processes"
- taskmanager.memory.process.size, taskmanager.memory.flink.size, jobmanager.heap.size
- rest.address, rest.port
- jobmanager.rpc.address, jobmanager.rpc.port
- "Parallelism"
- taskmanager.numberOfTaskSlots
- "Checkpointing"
- state.backend
- state.checkpoints.dir
- state.savepoints.dir
- Directories
- io.tmp.dirs
- web.upload.dir
- Miscellaneous
- web.submit.enable
- cluster.evenly-spread-out-slots
Setup and Common Operations
(These are options everyone may need at some point. They are not to alter behavior/debug/tune, but to configure the behavior and interaction with the environment)
Hosts and Ports
- rest.address
- rest.port
- rest.bind-port
- jobmanager.rpc.address
- jobmanager.rpc.port
- taskmanager.host
- taskmanager.rpc.port
- taskmanager.data.port
Memory Configuration
- all the FLIP-49 memory options
Fault Tolerance
- restart-strategy
- restart strategy specific options
State Backends
High Availability
- High Availability (HA)
- ZooKeeper-based HA Mode
Security
- SSL Settings
- ZooKeeper Security
- Kerberos-based Security
Metrics
File Systems
History Server
Python Runtime
Queryable State
Deployment Resource Management
- Kubernetes
- Yarn
- Mesos
State Backends
(these options are mainly for specific setups and performance tuning of the respective backends)
- RocksDB State Backend Options
- RocksDB Configurable Options (rename to RocksDB Native Options)
- RocksDB Native Metrics
Debugging & Expert Setups
(Normally, users should not need to touch any of these options)
Debugging ClassLoading
- Preamble links to https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html
- classloader.resolve-order
- classloader.parent-first-patterns.default
- classloader.parent-first-patterns.additional
Debugging Fault Tolerance and Scheduling
- jobmanager.execution.failover-strategy
- heartbeat.interval
- heartbeat.timeout
- cluster.registration.*
- slot.idle.timeout
- slot.request.timeout
Full Rest Options
Full JobManager Options
Full TaskManager Options
- taskmanager.debug.memory.log
- taskmanager.debug.memory.log-interval
Resource Manager Options
Akka Options
Network Options
- Netty Shuffle Environment
- Network Communication (via Netty)
Deprecated
- DataSet Optimizer
- DataSet Algorithms