Uploaded image for project: 'Apache Airflow'
  1. Apache Airflow
  2. AIRFLOW-6361

Run LocalTaskJob directly in Celery task

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.6
    • Fix Version/s: None
    • Component/s: executors
    • Labels:

      Description

      Hello,

      Celery runs the CLI first command, which contains LocalTaskJob. LocalTaskJob is responsible for starting the next user-code process. This level of isolation is redundant because LocalTaskJob doesn't execute unsafe code. The first command is run by a new process creation, not by a fork, so this is an expensive operation.

      According to preliminary measurements, this change results in an increase in performance close to 30%.

      I will provide more information in PR.

      Best regards
      Kamil Bregula

      After:
      ```
      real 0m38.394s
      user 0m4.340s
      sys 0m1.600s

      real 0m38.355s
      user 0m4.700s
      sys 0m1.340s

      real 0m38.675s
      user 0m4.760s
      sys 0m1.530s

      real 0m38.488s
      user 0m4.770s
      sys 0m1.280s

      real 0m38.434s
      user 0m4.600s
      sys 0m1.390s

      real 0m38.378s
      user 0m4.500s
      sys 0m1.270s

      real 0m38.106s
      user 0m4.200s
      sys 0m1.100s

      real 0m38.082s
      user 0m4.170s
      sys 0m1.030s

      real 0m38.173s
      user 0m4.290s
      sys 0m1.340s

      real 0m38.161s
      user 0m4.460s
      sys 0m1.370s
      ```

      Before:
      ```
      real 0m53.488s
      user 0m5.140s
      sys 0m1.700s

      real 1m8.288s
      user 0m6.430s
      sys 0m2.200s

      real 0m53.371s
      user 0m5.330s
      sys 0m1.630s

      real 0m58.939s
      user 0m6.470s
      sys 0m1.730s

      real 0m53.255s
      user 0m4.950s
      sys 0m1.640s

      real 0m58.802s
      user 0m5.970s
      sys 0m1.790s

      real 0m58.449s
      user 0m5.380s
      sys 0m1.580s

      real 0m53.308s
      user 0m5.120s
      sys 0m1.430s

      real 0m53.485s
      user 0m5.220s
      sys 0m1.290s

      real 0m53.387s
      user 0m5.020s
      sys 0m1.590s
      ```

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kamil.bregula Kamil Bregula
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: