Solr
  1. Solr
  2. SOLR-4978

Time is stripped from datetime column when imported into Solr date field

    Details

      Description

      I discovered that all dates I imported into a Solr date field from a MySQL datetime column have the time stripped (ie time portion is always 00:00:00).

      After double checking my DIH config and trying different things, I decided to take a look at the DIH code.

      When I looked at the source code of DIH JdbcDataSource class, I discovered that it is using java.sql.ResultSet and its getDate() method to handle date field. The getDate() method returns java.sql.Date. The java api doc for java.sql.Date

      http://docs.oracle.com/javase/6/docs/api/java/sql/Date.html

      states that:

      "To conform with the definition of SQL DATE, the millisecond values wrapped by a java.sql.Date instance must be 'normalized' by setting the hours, minutes, seconds, and milliseconds to zero in the particular time zone with which the instance is associated."

      I am so surprise at my finding that I think I may not be right. What am I doing wrong here? This is such a big hole in DIH, how could it be possible that no one has noticed this until now?

      Has anyone successfully imported a datetime column into a Solr date field using DIH?

        Activity

        Hide
        Bill Au added a comment -

        we have convertType=true in our dih config.

        Show
        Bill Au added a comment - we have convertType=true in our dih config.
        Hide
        Bill Au added a comment -

        setting convertType=false is a workaround for my problem. But the bug still exists when convertType is set to true. DIH should not call ResultSet.getDate() for solr date field because ResultSet.getDate() returns a java.sql.Date which has not time portion. Since sold date field does have a time portion, DIH should call ResultSet.getTimestamp().

        Show
        Bill Au added a comment - setting convertType=false is a workaround for my problem. But the bug still exists when convertType is set to true. DIH should not call ResultSet.getDate() for solr date field because ResultSet.getDate() returns a java.sql.Date which has not time portion. Since sold date field does have a time portion, DIH should call ResultSet.getTimestamp().
        Hide
        Noble Paul added a comment -

        Do you see any backcompat issues if we change from ResultSet.getDate() to ResultSet.getTimestamp()

        Show
        Noble Paul added a comment - Do you see any backcompat issues if we change from ResultSet.getDate() to ResultSet.getTimestamp()
        Hide
        Bill Au added a comment -

        This will only have an effect if convertType is enabled and will only affect date field type. convertType is disabled by default. I guess most people use the default setting, which is probably why no one has noticed this bug before. The current behavior for date field type is incorrect when converType is enabled. Making the change will fix the bug. So date fields indexed by DIH when convertType is enabled will actually have a time portion.

        Show
        Bill Au added a comment - This will only have an effect if convertType is enabled and will only affect date field type. convertType is disabled by default. I guess most people use the default setting, which is probably why no one has noticed this bug before. The current behavior for date field type is incorrect when converType is enabled. Making the change will fix the bug. So date fields indexed by DIH when convertType is enabled will actually have a time portion.
        Hide
        ASF subversion and git services added a comment -

        Commit 1500666 from shalin@apache.org
        [ https://svn.apache.org/r1500666 ]

        SOLR-4978: Time is stripped from datetime column when imported into Solr date field if convertType=true

        Show
        ASF subversion and git services added a comment - Commit 1500666 from shalin@apache.org [ https://svn.apache.org/r1500666 ] SOLR-4978 : Time is stripped from datetime column when imported into Solr date field if convertType=true
        Hide
        ASF subversion and git services added a comment -

        Commit 1500668 from shalin@apache.org
        [ https://svn.apache.org/r1500668 ]

        SOLR-4978: Time is stripped from datetime column when imported into Solr date field if convertType=true

        Show
        ASF subversion and git services added a comment - Commit 1500668 from shalin@apache.org [ https://svn.apache.org/r1500668 ] SOLR-4978 : Time is stripped from datetime column when imported into Solr date field if convertType=true
        Hide
        Steve Rowe added a comment -

        Bulk close resolved 4.4 issues

        Show
        Steve Rowe added a comment - Bulk close resolved 4.4 issues

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Bill Au
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development