Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4546

Incorporate Russian/Moscow timezone changes in to the tz db

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
    • Docs Text:
      Once this bug is resolved, which is not yet as of 13 Feb 2017, the docs should explain something like: "Impala now correctly handles the Moscow time after the 2014 change to year-round Standard Time."
    • Target Version:

      Description

      The issue in short

      select to_utc_timestamp('2016-10-14 00:00:00', 'Europe/Moscow')
      
      Expected result:
      2016-10-13 21:00:00
      
      Actual result:
      2016-10-13 20:00:00
      

      The issue here is [1]

      Most Russians will move to permanent Standard Time, or “winter time”, by setting their clocks 1 hour back on October 26, 2014. This is the same date most Europeans end Daylight Saving Time (DST).
      Current local time in Russia
      But the Russians will only join in the setting back of the clocks this year. From then on they will remain on Standard Time (winter time) and will not be setting their clocks forward again 1 hour along with the rest of Europe in March 2015.
      

      For longterm, we should rely on a standard tz DB rather than letting Impala handle timezone exceptions manually in the code.

      Workaround: Use a custom Java UDF that runs on JDK 8u31 or greater (which has this timezone change embedded.)

      [1] https://www.timeanddate.com/news/time/russia-abandons-permanent-summer-time.html

        Activity

        Hide
        dhecht Dan Hecht added a comment -

        I'm not sure the current tz db can handle rules like this. But we already have some special case code for Europe/Moscow that probably needs fixing up:

        TimezoneDatabase::FindTimezone()
          // The backing database does not capture some subtleties, there are special cases
          if ((tv.date().year() < 2011 || (tv.date().year() == 2011 && tv.date().month() < 4)) &&
              (iequals("Europe/Moscow", tz) || iequals("Moscow", tz) || iequals("MSK", tz))) {
            // We transition in pre April 2011 from using the tz_database_ to a custom rule
            // Russia stopped using daylight savings in 2011, the tz_database_ is
            // set up assuming Russia uses daylight saving every year.
            // Sun, Mar 27, 2:00AM Moscow clocks moved forward +1 hour (a total of GMT +4)
            // Specifically,
            // UTC Time 26 Mar 2011 22:59:59 +0000 ===> Sun Mar 27 01:59:59 MSK 2011
            // UTC Time 26 Mar 2011 23:00:00 +0000 ===> Sun Mar 27 03:00:00 MSK 2011
            // This means in 2011, The database rule will apply DST starting March 26 2011.
            // This will be a correct +4 offset, and the database rule can apply until
            // Oct 31 when tz_database_ will incorrectly attempt to turn clocks backwards 1 hour.
            return TIMEZONE_MSK_PRE_2011_DST;
          }
        
        Show
        dhecht Dan Hecht added a comment - I'm not sure the current tz db can handle rules like this. But we already have some special case code for Europe/Moscow that probably needs fixing up: TimezoneDatabase::FindTimezone() // The backing database does not capture some subtleties, there are special cases if ((tv.date().year() < 2011 || (tv.date().year() == 2011 && tv.date().month() < 4)) && (iequals( "Europe/Moscow" , tz) || iequals( "Moscow" , tz) || iequals( "MSK" , tz))) { // We transition in pre April 2011 from using the tz_database_ to a custom rule // Russia stopped using daylight savings in 2011, the tz_database_ is // set up assuming Russia uses daylight saving every year. // Sun, Mar 27, 2:00AM Moscow clocks moved forward +1 hour (a total of GMT +4) // Specifically, // UTC Time 26 Mar 2011 22:59:59 +0000 ===> Sun Mar 27 01:59:59 MSK 2011 // UTC Time 26 Mar 2011 23:00:00 +0000 ===> Sun Mar 27 03:00:00 MSK 2011 // This means in 2011, The database rule will apply DST starting March 26 2011. // This will be a correct +4 offset, and the database rule can apply until // Oct 31 when tz_database_ will incorrectly attempt to turn clocks backwards 1 hour. return TIMEZONE_MSK_PRE_2011_DST; }
        Hide
        jbapple Jim Apple added a comment -

        Taras Bobrovytsky, this has a target version of 2.9. How is the investigation going?

        Show
        jbapple Jim Apple added a comment - Taras Bobrovytsky , this has a target version of 2.9. How is the investigation going?
        Hide
        tarasbob Taras Bobrovytsky added a comment -

        Jim Apple, I already know how to fix this issue. It's going to be a small patch and I'll try to get it done next week.

        Show
        tarasbob Taras Bobrovytsky added a comment - Jim Apple , I already know how to fix this issue. It's going to be a small patch and I'll try to get it done next week.
        Hide
        jbapple Jim Apple added a comment -

        Taras's patch in review: https://gerrit.cloudera.org/#/c/5969/

        Show
        jbapple Jim Apple added a comment - Taras's patch in review: https://gerrit.cloudera.org/#/c/5969/
        Hide
        tarasbob Taras Bobrovytsky added a comment -
        commit 545bfcfe673dd24356eb8f84b664b03b6230b01e
        Author: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
        Date:   Thu Feb 9 20:53:31 2017 -0800
        
            IMPALA-4546: Fix Moscow timezone conversion after 2014
            
            In 2014 Moscow timezone rules changed from UTC+3 with no DST to UTC+4
            with no DST. A special case has been added to timestamp functions to
            handle this.
            
            Testing:
            Added BE Expr tests.
            
            Change-Id: Id6e3f2c9f6ba29749a26bc1087e664637bc02528
            Reviewed-on: http://gerrit.cloudera.org:8080/5969
            Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
            Tested-by: Impala Public Jenkins
        
        Show
        tarasbob Taras Bobrovytsky added a comment - commit 545bfcfe673dd24356eb8f84b664b03b6230b01e Author: Taras Bobrovytsky <tbobrovytsky@cloudera.com> Date: Thu Feb 9 20:53:31 2017 -0800 IMPALA-4546: Fix Moscow timezone conversion after 2014 In 2014 Moscow timezone rules changed from UTC+3 with no DST to UTC+4 with no DST. A special case has been added to timestamp functions to handle this . Testing: Added BE Expr tests. Change-Id: Id6e3f2c9f6ba29749a26bc1087e664637bc02528 Reviewed-on: http: //gerrit.cloudera.org:8080/5969 Reviewed-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            tarasbob Taras Bobrovytsky
            Reporter:
            bharathv bharath v
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development