[FLINK-21232] Kerberos delegation token framework

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0, 1.16.0
• Fix Version/s: 1.17.0
• Component/s: Connectors / Common, Deployment / Kubernetes, Deployment / YARN
• Labels: security
• Assignee: Gabor Somogyi
• Reporter: jackwangcs
• Created: 01/Feb/21 16:55
• Updated: 04/Jan/23 14:20
• Resolved: 04/Jan/23 14:19

Description

Proper delegation token handling is completely missing from Flink. This jira is an umbrella to track this effort.
Please see the attached FLIP and discussion threads for more details.

Email dicussion threads:

https://lists.apache.org/thread.html/rbedb6e769358a10c6426c4c42b3b51cdbed48a3b6537e4ebde912bc0%40%3Cdev.flink.apache.org%3E
https://lists.apache.org/thread.html/r20d4be431ff2f6faff94129b5321a047fcbb0c71c8e092504cd91183%40%3Cdev.flink.apache.org%3E
https://lists.apache.org/thread/cvwknd5fhohj0wfv8mfwn70jwpjvxrjj

 

Issue Links

• Blocked: FLINK-20714 Hive delegation token is not obtained when using `kinit` to submit Yarn per-job
• links to: FLIP-211
• links to: GitHub Pull Request #14841

Sub-Tasks

• Add pluggable delegation token manager: Closed
• Add periodic kerberos relogin to DelegationTokenManager: Closed
• Upgrade mockito to a version where static function mocking is possible: Closed
• Add periodic kerberos delegation token obtain possibility to DelegationTokenManager: Closed
• security.kerberos.login.principal can be defined wihtout keytab which is wrong configuration: Closed
• Add hadoop filesystems configuration possibility to all deployment targets: Closed
• Move Hadoop FS token obtain functionality into HadoopFSDelegationTokenProvider: Closed
• Make Hadoop FS token renewer configurable: Closed
• KerberosDelegationTokenManager fails to load DelegationTokenProvider due to NoClassDefFoundError in various ITCases: Resolved
• Move HBase token obtain functionality into HBaseDelegationTokenProvider: Closed
• Propagate obtained delegation tokens to TaskManagers: Closed
• Remove old delegation token framework code when new is working fine: Closed
• Simplify HadoopModule: Closed
• E2E test for delegation token framework: Closed
• Replace old kerberos docker image used in secure e2e tests: Closed
• Annotate all the public classes for the delegation token framework: Closed
• Add delegation token framework documentation: Closed