Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-12278

[Rust][DataFusion]Use Timestamp(Nanosecond, None) for SQL TIMESTAMP Type

    XMLWordPrintableJSON

Details

    Description

      1. Rationale
        Running the query `CREATE EXTERNAL TABLE .. (c TIMESTAMP)` today in DataFusion will result in a data type pf "Date64" which means that anything more specific than the date will be ignored.

      This leads to strange behavior such as

      echo "Jorge,2018-12-13T12:12:10.011" >> /tmp/foo.csv
      echo "Andrew,2018-11-13T17:11:10.011" > /tmp/foo.csv
      
      cargo run -p datafusion --bin datafusion-cli
          Finished dev [unoptimized + debuginfo] target(s) in 0.23s
           Running `target/debug/datafusion-cli`
      > CREATE EXTERNAL TABLE t(a varchar, b TIMESTAMP)
      STORED AS CSV
      LOCATION '/tmp/foo.csv';
      
      0 rows in set. Query took 0 seconds.
      > select * from t;
      +--------+------------+
      | a      | b          |
      +--------+------------+
      | Andrew | 2018-11-13 |
      | Jorge  | 2018-12-13 |
      +--------+------------+
      

      (note how it is only a date, not a timestamp)

      Attachments

        Issue Links

          Activity

            People

              alamb Andrew Lamb
              alamb Andrew Lamb
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m