Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-6682

[C#] Arrow R/C++ hangs reading binary file generated by C#

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.14.1
    • 0.15.0
    • C#

    Description

      I get random hangs on arrow_read in R (windows) when using a very large file (10-12gb). (the file has 37 columns)

      I have memory dumps - All threads seem to be in wait handles.

      Are there debug symbols somewhere? 

      Is there a way to get the C++ code to produce diagnostic logging from R? 

       

      UPDATE: it seems that the hangs are not related to file size, row counts, or # of record batches, but rather the number of columns

      Attachments

        1. script.runner.ps1
          0.3 kB
          Anthony Abate
        2. Generated_4000Batch_50Columns_100Rows_PerBatch.zip
          2.98 MB
          Anthony Abate
        3. Generated_4000Batch_50Columns_100Rows_PerBatch.rar
          60 kB
          Anthony Abate
        4. arrow.benchmark.r
          0.3 kB
          Anthony Abate

        Issue Links

          Activity

            People

              eerhardt Eric Erhardt
              abbot Anthony Abate
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m