Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-685

Sqoop will fail with OutOfMemory on large tables using mysql

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The default MySQL JDBC client behavior is to buffer the entire ResultSet in the client before allowing the user to use the ResultSet object. On large SELECTs, this can cause OutOfMemory exceptions, even when the client intends to close the ResultSet after reading only a few rows. The MySQL ConnManager should configure its connection to use row-at-a-time delivery of results to the client.

      1. MAPREDUCE-685.patch
        15 kB
        Aaron Kimball
      2. MAPREDUCE-685.patch.2
        17 kB
        Aaron Kimball
      3. MAPREDUCE-685.3.patch
        17 kB
        Aaron Kimball

        Activity

        Chris Douglas made changes -
        Component/s contrib/sqoop [ 12312930 ]
        Tom White made changes -
        Fix Version/s 0.21.0 [ 12314045 ]
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.21.0 [ 12314045 ]
        Resolution Fixed [ 1 ]
        Aaron Kimball made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Aaron Kimball made changes -
        Attachment MAPREDUCE-685.3.patch [ 12412766 ]
        Aaron Kimball made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Aaron Kimball made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Aaron Kimball made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Aaron Kimball made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Aaron Kimball made changes -
        Attachment MAPREDUCE-685.patch.2 [ 12412298 ]
        Aaron Kimball made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Aaron Kimball made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Aaron Kimball made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-685.patch [ 12412206 ]
        Aaron Kimball created issue -

          People

          • Assignee:
            Aaron Kimball
            Reporter:
            Aaron Kimball
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development