Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.16.0
Description
FileFragment can be made more directly useful by adding convenience methods.
For example, a FileFragment could allow underlying file/buffer to be opened directly:
def open(self): """ Open a NativeFile of the buffer or file viewed by this fragment. """ cdef: CFileSystem* c_filesystem shared_ptr[CRandomAccessFile] opened NativeFile out = NativeFile() buf = self.buffer if buf is not None: return pa.io.BufferReader(buf) with nogil: c_filesystem = self.file_fragment.source().filesystem() opened = GetResultValue(c_filesystem.OpenInputFile( self.file_fragment.source().path())) out.set_random_access_file(opened) out.is_readable = True return out
Additionally, a ParquetFileFragment's metadata could be introspectable:
@property def metadata(self): from pyarrow._parquet import ParquetReader reader = ParquetReader() reader.open(self.open()) return reader.metadata
Attachments
Issue Links
- links to