[BEAM-12945] DataFrame API fails to import with pandas 1.0.x

• Type: Bug
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.32.0
• Fix Version/s: 2.34.0
• Component/s: dsl-dataframe
• Assignee: Brian Hulette
• Reporter: Brian Hulette
• Created: 23/Sep/21 16:59
• Updated: 13/Apr/23 11:07
• Resolved: 28/Sep/21 00:41

Description

The DataFrame API expects pandas.DataFrame to have the attributes "backfill", "bfill", so it fails to import when pandas 1.0.x is installed:

In [1]: import pandas as pd                                                                                                                                                                                                    [11/1919]
                                                                                                                    
In [2]: pd.__version__                                    
Out[2]: '1.0.5'                                           
                                                                                                                    
In [3]: from apache_beam.dataframe.convert import to_dataframe                                                      
---------------------------------------------------------------------------                                         
AttributeError                            Traceback (most recent call last)                                         
<ipython-input-3-26ab6682f699> in <module>                
----> 1 from apache_beam.dataframe.convert import to_dataframe                                                      
                                                          
~/working_dir/beam/sdks/python/apache_beam/dataframe/convert.py in <module>                                         
     30 from apache_beam.dataframe import frame_base      
     31 from apache_beam.dataframe import schemas         
---> 32 from apache_beam.dataframe import transforms      
     33                                                   
     34 if TYPE_CHECKING:                                 

~/working_dir/beam/sdks/python/apache_beam/dataframe/transforms.py in <module>                                      
     30 from apache_beam import transforms                
     31 from apache_beam.dataframe import expressions     
---> 32 from apache_beam.dataframe import frames  # pylint: disable=unused-import                                   
     33 from apache_beam.dataframe import partitionings   
     34 from apache_beam.utils import windowed_value      

~/working_dir/beam/sdks/python/apache_beam/dataframe/frames.py in <module>                                          
    148                                                   
    149                                                   
--> 150 class DeferredDataFrameOrSeries(frame_base.DeferredFrame):                                                  
    151   def _render_indexes(self):                      
    152     if self.index.nlevels == 1:                   

~/working_dir/beam/sdks/python/apache_beam/dataframe/frames.py in DeferredDataFrameOrSeries()                       
    297   ffill = _fillna_alias('ffill')                  
    298   bfill = _fillna_alias('bfill')                  
--> 299   backfill = _fillna_alias('backfill')            
    300   pad = _fillna_alias('pad')                      
    301                                                   

~/working_dir/beam/sdks/python/apache_beam/dataframe/frames.py in _fillna_alias(method)                             
     94   return frame_base.with_docs_from(pd.DataFrame)(                                                           
     95       frame_base.args_to_kwargs(pd.DataFrame)(    
---> 96           frame_base.populate_defaults(pd.DataFrame)(wrapper)))                                             
     97                                                   
     98                                                   

~/working_dir/beam/sdks/python/apache_beam/dataframe/frame_base.py in wrap(func)                                    
    559   For internal use only. No backwards compatibility guarantees."""                                          
    560   def wrap(func):                                 
--> 561     base_argspec = getfullargspec(unwrap(getattr(base_type, func.__name__)))                                
    562     if not base_argspec.defaults:                 
    563       return func                                 

AttributeError: type object 'DataFrame' has no attribute 'backfill'

Issue Links

• links to: GitHub Pull Request #15573