Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6878

AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS default value misconfigured

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.10.5, 1.10.6, 1.10.7, 1.10.8, 1.10.9
    • Fix Version/s: 1.10.10
    • Component/s: configuration
    • Labels:
    • Environment:

      Description

       

      Issue:

      The `kube_client_request_args` config value in the `[kubernetes]` section of the default airflow.cfg file is misconfigured.

      Suggested fix:

      Set the following key value pair in the  [kubernetes] section of the file airflow/config_templates/default_airflow.cfg, like so:

       

       

      kube_client_request_args = {"_request_timeout" : [60,60] }

       

      I'd be really keen to contribute this fix!

      Open Questions:

      • Should we patch older versions, or just include this in the 1.10.10 release?
      • Does someone have a hint of where other code might be affected / have this config error?
      • Any feedback/newbie advice? This this is my first contribution here

      Background:

      In 1.10.4, this configuration was left unset.

       

      kube_client_request_args = 

       

      see https://github.com/apache/airflow/blob/1.10.4/airflow/config_templates/default_airflow.cfg#L779

       

      From 1.10.5 on it is configured like so:

       

      kube_client_request_args = {{"_request_timeout" : [60,60] }}

       

      see https://github.com/apache/airflow/blob/1.10.5/airflow/config_templates/default_airflow.cfg#L805

      To reproduce:

      If we run with the KubernetesExecutor using puckel/docker-airflow repository with an "otherwise working" helm stable/airflow values.yaml, the scheduler and webserver will output an error like this when running initdb (retrieved with kubectl logs):

      69f76c7dc-w9stc   
      waiting 10s...
      executing initdb
      [2020-02-22 02:07:35,810] {{settings.py:253}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=12
      DB: postgresql+psycopg2://postgres:***@airflow-dev-0-postgresql:5432/airflow
      [2020-02-22 02:07:36,332] {{db.py:368}} INFO - Creating tables
      INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
      INFO  [alembic.runtime.migration] Will assume transactional DDL.
      INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
      INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
      INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
      INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_instance
      INFO  [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
      INFO  [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding extra to Log
      INFO  [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, add dagrun
      INFO  [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, task_duration
      INFO  [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, dagrun_config
      INFO  [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, add password column to user
      INFO  [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, dagrun start end
      INFO  [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss
      INFO  [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection
      INFO  [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table
      INFO  [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table
      INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, add TI state index
      INFO  [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, add task fails journal table
      INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, add dag_stats table
      INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
      INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, xcom dag task indices
      INFO  [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance
      INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table
      INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance
      Traceback (most recent call last):
        File "/usr/local/bin/airflow", line 37, in <module>
          args.func(args)
        File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1140, in initdb
          db.initdb(settings.RBAC)
        File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 106, in initdb
          upgradedb()
        File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 376, in upgradedb
          command.upgrade(config, 'heads')
        File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade
          script.run_env()
        File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env
          util.load_python_file(self.dir, "env.py")
        File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
          module = load_module_py(module_id, path)
        File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 173, in load_module_py
          spec.loader.exec_module(module)
        File "<frozen importlib._bootstrap_external>", line 728, in exec_module
        File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
        File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 96, in <module>
          run_migrations_online()
        File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 90, in run_migrations_online
          context.run_migrations()
        File "<string>", line 8, in run_migrations
        File "/usr/local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 846, in run_migrations
          self.get_context().run_migrations(**kw)
        File "/usr/local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 518, in run_migrations
          step.migration_fn(**kw)
        File "/usr/local/lib/python3.7/site-packages/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py", line 70, in upgrade
          dagbag = DagBag(settings.DAGS_FOLDER)
        File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 95, in __init__
          executor = get_default_executor()
        File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 48, in get_default_executor
          DEFAULT_EXECUTOR = _get_executor(executor_name)
        File "/usr/local/lib/python3.7/site-packages/airflow/executors/__init__.py", line 87, in _get_executor
          return KubernetesExecutor()
        File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 702, in __init__
          self.kube_config = KubeConfig()
        File "/usr/local/lib/python3.7/site-packages/airflow/contrib/executors/kubernetes_executor.py", line 283, in __init__
          self.kube_client_request_args = json.loads(kube_client_request_args)
        File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
          return _default_decoder.decode(s)
        File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
          obj, end = self.raw_decode(s, idx=_w(s, 0).end())
        File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode
          obj, end = self.scan_once(s, idx)
      json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
      

       

      On StackOverflow:

       

      This was reported for versions 1.10.8 and 1.10.9 in this stackoverflow issue: https://stackoverflow.com/questions/60130992/airflow-scheduler-fails-to-start-with-kubernetes-executor

       

      Docs:

      You can see that the airflow.apache.org docs are also affected: https://airflow.apache.org/docs/stable/configurations-ref.html#kube-client-request-args

       

        Attachments

          Activity

            People

            • Assignee:
              d.stevenson Daniel Stevenson
              Reporter:
              d.stevenson Daniel Stevenson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 2h
                2h
                Remaining:
                Remaining Estimate - 2h
                2h
                Logged:
                Time Spent - Not Specified
                Not Specified