Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-3871

Allow Jinja templating recursively on object attributes

    XMLWordPrintableJSON

    Details

    • Type: Wish
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.10.0
    • Fix Version/s: 1.10.6
    • Component/s: operators
    • Labels:
      None

      Description

      Some Operator fields can be templated (using Jinja). Template rendering only works for string values (either direct values or values stored in collections).
      But a templated string inside a custom class instance won't be rendered

      Here is my scenario:
      I have a python method transform_data_file which is designed to call a command object. This command object constructor (MyAwesomeDataFileTransformer) has parameters that could be templated. These templated parameters are not rendered so far (see BaseOperator.render_template_from_field method).

      simple_task = PythonOperator(
          task_id='simple_task',
          provide_context=True,
          python_callable=transform_data_file,
          templates_dict={
      	'transformer': MyAwesomeDataFileTransformer(
                  "/data/{{ dag.dag_id }}/{{ ts }}/input_file",
                  "/data/{{ dag.dag_id }}/{{ ts }}/output_file",
              )
          },
          dag=dag
      )
      

      I have 3 alternatives in mind to allow rendering inner attributes:

      1. Either define an Abstract Base Class declaring an abstract method render_template; then my command object would have to extend this Abstract Base Class, and then implement render_template method.
      2. Or use duck typing in BaseOperator.render_template_from_field to call render_template method when it exists on templated custom objects; then my command object would just have to implement render_template method.
      3. Or traverse object attributes when rendering templates and call BaseOperator.render_template recursively; then my command object would not need any change

      My preferred solution is the 3rd one, but I would like to hear about your opinion on this before. Maybe is there a 4th and better solution?

      I would be glad to submit a PR if this functionality is accepted.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                galak75 Galak
                Reporter:
                galak75 Galak
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: