Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-7070

SimpleHttpOperator does not consider extra attributes from Airflow connection

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.10.9
    • Fix Version/s: None
    • Component/s: operators
    • Labels:
      None

      Description

      I have noticed this problem with SimpleHttpOperator in Airflow 1.10.9.

      When we set it up with conn_id, it does not take into account 'extra' parameters from connection.

      Example: 

      Airflow -> Admin -> Connection -> Extra:

      {"proxies":"https":"http://xxxxxxxx:yyyyyyy@1.2.3.4:80"}}

      My python skill is very bad, but I think there are 2 things happening:

      1.

      http_hook.py - function get_conn - line 78

      Context:

       

      if conn.extra:
          try:
              session.headers.update(conn.extra_dejson)
      

      This is updating session header, however header can't accept type dictionary, only string. So first error with this is that the header is incorrectly formatted (type error). 

       

      To me, it makes no sense to use extra field as header source, but if this have to be kept for compatibility reason, I would suggest excluding python-request parameters from pushing to headers. Otherwise It may results also in insecure behavior.

       

      session.proxies.update(extra.get("proxies"))
      // This works for me if I disable header code above. (don't use it in solution)

       

      2.

      *http_hook.py - function *run_and_check - ** lines 176-184

      Extra options here are only taken into consideration from extra_option object, but they they should be taken from session object as well and overwritten by connection extra_option object afterwards. 

      proxies=session.proxies
      // replaced this code, as brute force solution.(don't use it in solution)

       

       PS: This prohibits us from having sensitive information like proxy login in secure place therefore creates security concern.

       

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Juraj Juraj Šanko
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: