Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3341

Strict datetime parsing and improve performance of loading datetime values

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.1
    • Fix Version/s: 0.12.0, 0.11.2
    • Component/s: impl
    • Labels:
      None

      Description

      The performance of loading datetime values can be improved by about 25% by moving a single line in ToDate.java:

      public static DateTimeZone extractDateTimeZone(String dtStr) {
      Pattern pattern = Pattern.compile("(Z|(?<=(T[0-9\\.:]

      {0,12}))((\\+|-)
      d{2}(:?
      d{2})?))$");;

      should become:

      static Pattern pattern = Pattern.compile("(Z|(?<=(T[0-9\\.:]{0,12}

      ))((\\+|-)
      d

      {2}(:?
      d{2}

      )?))$");
      public static DateTimeZone extractDateTimeZone(String dtStr) {

      There is no need to recompile the regular expression for every value. I'm not sure if this function is ever called concurrently, but Pattern objects are thread-safe anyways.

      As a test, I created a file of 10M timestamps:

      for i in 0..10000000
      puts '2000-01-01T00:00:00+23'
      end

      I then ran this script:

      grunt> A = load 'data' as (a:datetime); B = filter A by a is null; dump B;

      Before the change it took 160s.
      After the change, the script took 120s.

      ----------------

      Another performance improvement can be made for invalid datetime values. If a datetime value is invalid, an exception is created and thrown, which is a costly way to fail a validity check. To test the performance impact, I created 10M invalid datetime values:

      for i in 0..10000000
      puts '2000-99-01T00:00:00+23'
      end

      In this test, the regex pattern was always recompiled. I then ran this script:

      grunt> A = load 'data' as (a:datetime); B = filter A by a is not null; dump B;

      The script took 190s.

      I understand this could be considered an edge case and might not be worth changing. However, if there are use cases where invalid dates are part of normal processing, then you might consider fixing this.

        Attachments

        1. PIG-3341-1.patch
          14 kB
          Rohini Palaniswamy
        2. PIG-3341-2.patch
          14 kB
          Rohini Palaniswamy
        3. PIG-3341-3.patch
          16 kB
          Rohini Palaniswamy
        4. PIG-3341-3-branch11.patch
          16 kB
          Rohini Palaniswamy

          Issue Links

            Activity

              People

              • Assignee:
                rohini Rohini Palaniswamy
                Reporter:
                patc888 pat chan
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: