Avro
  1. Avro
  2. AVRO-521

Out of memory and other issues with Junit tests for mapreduce

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.4.0
    • Component/s: java
    • Labels:
      None

      Description

      The forthcoming patch fixed the out of memory problem we encounter while running JUnit tests for mapreduce.

      1. AVRO-521.patch
        6 kB
        Thiruvalluvan M. G.

        Activity

        Hide
        Thiruvalluvan M. G. added a comment -

        Committed revision 937307 after adding @SuppressWarnings annotation for JobConf deprecation. Thanks Doug for the help.

        Show
        Thiruvalluvan M. G. added a comment - Committed revision 937307 after adding @SuppressWarnings annotation for JobConf deprecation. Thanks Doug for the help.
        Hide
        Doug Cutting added a comment -

        Thanks for updating these tests. I started with a Hadoop JUnit3-style test and neglected to update them.

        We should suppress these deprecation warnings. This uses the Hadoop 0.20 APIs, which, while deprecated in 0.20, have now since been declared the long-term stable Hadoop APIs. There are new APIs, but they're not yet stable, and also are not present in older releases. I believe this code should work in 0.18 and 0.19, and the forthcoming 0.21, but have not tested these yet.

        Show
        Doug Cutting added a comment - Thanks for updating these tests. I started with a Hadoop JUnit3-style test and neglected to update them. We should suppress these deprecation warnings. This uses the Hadoop 0.20 APIs, which, while deprecated in 0.20, have now since been declared the long-term stable Hadoop APIs. There are new APIs, but they're not yet stable, and also are not present in older releases. I believe this code should work in 0.18 and 0.19, and the forthcoming 0.21, but have not tested these yet.
        Hide
        Thiruvalluvan M. G. added a comment -

        Increasing the max memory to 128 MB (from the default 64 MB) fixes the out of memory problem.

        But there is one more issue. TestWordCountGeneric leaves behind the (empty) output directory and the following test TestWordCountSpecific fails to run complaining that the output directory already exists.

        I fixed that problem by removing the output directory after completing each of these tests. Since I was at it, I moved the JUnit tests toJUnit4 style from JUnit3 style. All other tests are in JUnit4. Also added a couple of @Override annotations.

        There are quite a few warnings because we use mapreduce's JobConf. What is the new way to submit jobs? The documentation says we should use Configuration. I'm not sure how to do that. Can anybody help us with correct usage? Thanks.

        Show
        Thiruvalluvan M. G. added a comment - Increasing the max memory to 128 MB (from the default 64 MB) fixes the out of memory problem. But there is one more issue. TestWordCountGeneric leaves behind the (empty) output directory and the following test TestWordCountSpecific fails to run complaining that the output directory already exists. I fixed that problem by removing the output directory after completing each of these tests. Since I was at it, I moved the JUnit tests toJUnit4 style from JUnit3 style. All other tests are in JUnit4. Also added a couple of @Override annotations. There are quite a few warnings because we use mapreduce's JobConf. What is the new way to submit jobs? The documentation says we should use Configuration. I'm not sure how to do that. Can anybody help us with correct usage? Thanks.

          People

          • Assignee:
            Thiruvalluvan M. G.
            Reporter:
            Thiruvalluvan M. G.
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development