Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17736 Über-JIRA: abfs phase III: Hadoop 3.3.2 features & fixes
  3. HADOOP-17195

Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs

    XMLWordPrintableJSON

Details

    Description

      OutOfMemory error due to new ThreadPools being made each time AbfsOutputStream is created. Since threadPool aren't limited a lot of data is loaded in buffer and thus it causes OutOfMemory error.

      Possible fixes:

      • Limit the number of ThreadCounts while performing hdfs copyFromLocal (Using -t property).
      • Reducing OUTPUT_BUFFER_SIZE significantly which would limit the amount of buffer to be loaded in threads.
      • Don't create new ThreadPools each time AbfsOutputStream is created and limit the number of ThreadPools each AbfsOutputStream could create.

      Attachments

        Issue Links

          Activity

            People

              mehakmeetSingh Mehakmeet Singh
              mehakmeetSingh Mehakmeet Singh
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 7h 50m
                  7h 50m