Details

    • Type: Task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None

      Description

      These are the pending issues for the Airflow 1.8.0 release:

      Blockers:
      Bolke de Bruin please merge into the next RC and then remove from the list the sub-tasks linked in this JIRA

      Other Issues:

      • High DB Load (~8x more than 1.7) - We are still investigating but it's probably not a blocker for the release - (Theories: Might need execution_date index on dag_run (based on slow process list) OR it might be this query which is long running SELECT union_ti.dag_id AS union_ti_dag_id, union_ti.state AS union_ti_state, count( *) AS count_1
        FR))
      • Front page loading time is a lot slower

      Bolke de Bruin

        Activity

        Hide
        saguziel Alex Guziel added a comment -

        The high DB load looks pretty periodic on our end (ie comes and go every 10 minutes). I did some profiling and I found a lot of areas to fix so I'm working on those.

        Show
        saguziel Alex Guziel added a comment - The high DB load looks pretty periodic on our end (ie comes and go every 10 minutes). I did some profiling and I found a lot of areas to fix so I'm working on those.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit d7d9f883e4db3d864eb88ce510988456e8d468c4 in incubator-airflow's branch refs/heads/master from Bolke de Bruin
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d7d9f88 ]

        AIRFLOW-932]AIRFLOW-932AIRFLOW-921AIRFLOW-910 Do not mark tasks removed when backfilling[

        In a backfill one can specify a specific task to
        execute. We
        create a subset of the orginal tasks in a subdag
        from the original dag.
        The subdag has the same name as the original dag.
        This breaks
        the integrity check of a dag_run as tasks are
        suddenly not in
        scope any more.

        Closes #2122 from bolkedebruin/AIRFLOW-921

        Show
        jira-bot ASF subversion and git services added a comment - Commit d7d9f883e4db3d864eb88ce510988456e8d468c4 in incubator-airflow's branch refs/heads/master from Bolke de Bruin [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d7d9f88 ] AIRFLOW-932 ] AIRFLOW-932 AIRFLOW-921 AIRFLOW-910 Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/ AIRFLOW-921
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit d7d9f883e4db3d864eb88ce510988456e8d468c4 in incubator-airflow's branch refs/heads/master from Bolke de Bruin
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d7d9f88 ]

        AIRFLOW-932]AIRFLOW-932AIRFLOW-921AIRFLOW-910 Do not mark tasks removed when backfilling[

        In a backfill one can specify a specific task to
        execute. We
        create a subset of the orginal tasks in a subdag
        from the original dag.
        The subdag has the same name as the original dag.
        This breaks
        the integrity check of a dag_run as tasks are
        suddenly not in
        scope any more.

        Closes #2122 from bolkedebruin/AIRFLOW-921

        Show
        jira-bot ASF subversion and git services added a comment - Commit d7d9f883e4db3d864eb88ce510988456e8d468c4 in incubator-airflow's branch refs/heads/master from Bolke de Bruin [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d7d9f88 ] AIRFLOW-932 ] AIRFLOW-932 AIRFLOW-921 AIRFLOW-910 Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/ AIRFLOW-921
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit a8f2c27ed44449e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-test from Bolke de Bruin
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ]

        AIRFLOW-932]AIRFLOW-932AIRFLOW-921AIRFLOW-910 Do not mark tasks removed when backfilling[

        In a backfill one can specify a specific task to
        execute. We
        create a subset of the orginal tasks in a subdag
        from the original dag.
        The subdag has the same name as the original dag.
        This breaks
        the integrity check of a dag_run as tasks are
        suddenly not in
        scope any more.

        Closes #2122 from bolkedebruin/AIRFLOW-921

        Show
        jira-bot ASF subversion and git services added a comment - Commit a8f2c27ed44449e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-test from Bolke de Bruin [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ] AIRFLOW-932 ] AIRFLOW-932 AIRFLOW-921 AIRFLOW-910 Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/ AIRFLOW-921
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit a8f2c27ed44449e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-test from Bolke de Bruin
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ]

        AIRFLOW-932]AIRFLOW-932AIRFLOW-921AIRFLOW-910 Do not mark tasks removed when backfilling[

        In a backfill one can specify a specific task to
        execute. We
        create a subset of the orginal tasks in a subdag
        from the original dag.
        The subdag has the same name as the original dag.
        This breaks
        the integrity check of a dag_run as tasks are
        suddenly not in
        scope any more.

        Closes #2122 from bolkedebruin/AIRFLOW-921

        Show
        jira-bot ASF subversion and git services added a comment - Commit a8f2c27ed44449e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-test from Bolke de Bruin [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ] AIRFLOW-932 ] AIRFLOW-932 AIRFLOW-921 AIRFLOW-910 Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/ AIRFLOW-921
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit a8f2c27ed44449e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-stable from Bolke de Bruin
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ]

        AIRFLOW-932]AIRFLOW-932AIRFLOW-921AIRFLOW-910 Do not mark tasks removed when backfilling[

        In a backfill one can specify a specific task to
        execute. We
        create a subset of the orginal tasks in a subdag
        from the original dag.
        The subdag has the same name as the original dag.
        This breaks
        the integrity check of a dag_run as tasks are
        suddenly not in
        scope any more.

        Closes #2122 from bolkedebruin/AIRFLOW-921

        Show
        jira-bot ASF subversion and git services added a comment - Commit a8f2c27ed44449e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-stable from Bolke de Bruin [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ] AIRFLOW-932 ] AIRFLOW-932 AIRFLOW-921 AIRFLOW-910 Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/ AIRFLOW-921
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit a8f2c27ed44449e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-stable from Bolke de Bruin
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ]

        AIRFLOW-932]AIRFLOW-932AIRFLOW-921AIRFLOW-910 Do not mark tasks removed when backfilling[

        In a backfill one can specify a specific task to
        execute. We
        create a subset of the orginal tasks in a subdag
        from the original dag.
        The subdag has the same name as the original dag.
        This breaks
        the integrity check of a dag_run as tasks are
        suddenly not in
        scope any more.

        Closes #2122 from bolkedebruin/AIRFLOW-921

        Show
        jira-bot ASF subversion and git services added a comment - Commit a8f2c27ed44449e6611c7c4a9ec8cf2371cf0987 in incubator-airflow's branch refs/heads/v1-8-stable from Bolke de Bruin [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8f2c27 ] AIRFLOW-932 ] AIRFLOW-932 AIRFLOW-921 AIRFLOW-910 Do not mark tasks removed when backfilling[ In a backfill one can specify a specific task to execute. We create a subset of the orginal tasks in a subdag from the original dag. The subdag has the same name as the original dag. This breaks the integrity check of a dag_run as tasks are suddenly not in scope any more. Closes #2122 from bolkedebruin/ AIRFLOW-921

          People

          • Assignee:
            bolke Bolke de Bruin
            Reporter:
            aoen Dan Davydov
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development