Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-892

HIvePartitionSensor and HiveNamedPartitionSensor raise AttributeError

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.8.0
    • Component/s: core
    • Labels:
    • Environment:
      Virtualenv 2.7.13 on Centos 6 / Centos 6 + Docker
    • Flags:
      Patch, Important

      Description

      All HiveParititionSensor and friends using the Meatastore hook throw this exception in 2 installs I've done, one on a clean docker to rule out other causes. install has been done using pip install git+https://github.com/apache/incubator-airflow.git@v1-8-stable

      [2017-02-22 08:44:49,079] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
      [2017-02-22 08:44:49,079] {base_task_runner.py:95} INFO - Subtask: Starting attempt 1 of 1
      [2017-02-22 08:44:49,079] {base_task_runner.py:95} INFO - Subtask: --------------------------------------------------------------------------------
      [2017-02-22 08:44:49,079] {base_task_runner.py:95} INFO - Subtask: 
      [2017-02-22 08:44:49,111] {base_task_runner.py:95} INFO - Subtask: [2017-02-22 08:44:49,110] {models.py:1342} INFO - Executing <Task(HivePartitionSensor): wait_for_***> on 2017-02-18 00:00:00
      [2017-02-22 08:44:49,124] {base_task_runner.py:95} INFO - Subtask: [2017-02-22 08:44:49,123] {sensors.py:373} INFO - Poking for table ****, partition d_date='2017-02-18'
      [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask: [2017-02-22 08:44:49,124] {models.py:1417} ERROR - 
      [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
      [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
      [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask:     result = task_copy.execute(context=context)
      [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/operators/sensors.py", line 76, in execute
      [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask:     while not self.poke(context):
      [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/operators/sensors.py", line 375, in poke
      [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask:     self.hook = airflow.hooks.hive_hooks.HiveMetastoreHook(
      [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/utils/helpers.py", line 436, in __getattr__
      [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask:     raise AttributeError
      [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask: AttributeError
      [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask: [2017-02-22 08:44:49,125] {models.py:1441} INFO - Marking task as FAILED.
      [2017-02-22 08:44:49,163] {base_task_runner.py:95} INFO - Subtask: [2017-02-22 08:44:49,163] {models.py:1462} ERROR - 
      [2017-02-22 08:44:49,163] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
      [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/bin/airflow", line 28, in <module>
      [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask:     args.func(args)
      [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/bin/cli.py", line 427, in run
      [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask:     pool=args.pool,
      [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/utils/db.py", line 54, in wrapper
      [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask:     result = func(*args, **kwargs)
      [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
      [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask:     result = task_copy.execute(context=context)
      [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/operators/sensors.py", line 76, in execute
      [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask:     while not self.poke(context):
      [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/operators/sensors.py", line 375, in poke
      [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask:     self.hook = airflow.hooks.hive_hooks.HiveMetastoreHook(
      [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask:   File "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/utils/helpers.py", line 436, in __getattr__
      [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask:     raise AttributeError
      

      I've worked around the issue adding a 'import airflow.hooks.hive_hooks' in class HiveParititionSensor (because I don't really know what this is about, there surely is a real fix to be done). It fixes the issue for me.

        Attachments

        1. airflow-hivesensor.diff
          0.4 kB
          David Morel

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                dmorel David Morel
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: