Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-601

Some logical types not supported when loading Parquet

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.2.0
    • Fix Version/s: 0.5.0
    • Component/s: Python
    • Labels:

      Description

      When I try to read a parquet file with some logical types in it, pyarrow says they are not supported:

      table = pq.read_table('t.parquet')
      ---------------------------------------------------------------------------
      ArrowException                            Traceback (most recent call last)
      <ipython-input-14-b7190e66bcb5> in <module>()
      ----> 1 table = pq.read_table('parquet/t')
      
      /opt/conda/lib/python3.5/site-packages/pyarrow/parquet.py in read_table(source, columns, nthreads, metadata)
          113 
          114     pf = ParquetFile(source, metadata=metadata)
      --> 115     return pf.read(columns=columns, nthreads=nthreads)
          116 
          117 
      
      /opt/conda/lib/python3.5/site-packages/pyarrow/parquet.py in read(self, nrows, columns, nthreads)
           78 
           79         return self.reader.read(column_indices=column_indices,
      ---> 80                                 nthreads=nthreads)
           81 
           82 
      
      /opt/conda/lib/python3.5/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.ParquetReader.read (/feedstock_root/build_artefacts/pyarrow_1488133203047/work/arrow-f6924ad83bc95741f003830892ad4815ca3b70fd/python/build/temp.linux-x86_64-3.5/_parquet.cxx:7706)()
      
      /opt/conda/lib/python3.5/site-packages/pyarrow/error.pyx in pyarrow.error.check_status (/feedstock_root/build_artefacts/pyarrow_1488133203047/work/arrow-f6924ad83bc95741f003830892ad4815ca3b70fd/python/build/temp.linux-x86_64-3.5/error.cxx:1197)()
      
      ArrowException: NotImplemented: Unhandled logical type for int32
      

      This is the schema of the parquet file (see attached):

      optional group root {
        optional int64 instant (TIMESTAMP_MILLIS);
        optional int32 time (TIME_MILLIS);
        optional double a-double;
        optional int64 another-int;
        optional binary a-string (UTF8);
        optional group list (LIST) {
          repeated group list {
            optional int64 element;
          }
        }
        optional boolean a-boolean;
        optional group a-group {
          optional boolean bool;
          optional int64 another;
        }
        optional int64 an-int;
        optional int32 a-date (DATE);
      }
      

      I assume this is because not pyarrow doesn't support loading all the parquet logical types yet. Is there someplace I can look (even if it's not documented, just in the codebase), where I can find what types are supported currently and which are not?

        Attachments

        1. t.parquet
          1 kB
          Saul Shanabrook

          Issue Links

            Activity

              People

              • Assignee:
                wesmckinn Wes McKinney
                Reporter:
                saulshanabrook Saul Shanabrook
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: