Uploaded image for project: 'Joshua (Retired)'
  1. Joshua (Retired)
  2. JOSHUA-318

scripts/training/run_tuner.py should enable configurable memory usage when invioking joshua-decoder

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 6.0.5
    • 6.2
    • tuner
    • None

    Description

      When I run the run_tuner.py script I can easily run into the following

      [mert-1] rebuilding...
        dep=/usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.en
        dep=/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config [CHANGED]
        dep=tune/model/grammar.gz.packed/slice_00000.source [CHANGED]
        dep=/usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config.final [NOT FOUND]
        cmd=/usr/local/incubator-joshua/scripts/training/run_tuner.py /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.en /usr/local/joshua_resources/russian_experiments/exp3/data/tune/corpus.ru --tunedir /usr/local/joshua_resources/russian_experiments/exp3/tune --tuner mert --decoder /usr/local/joshua_resources/russian_experiments/exp3/tune/decoder_command --decoder-config /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config --decoder-output-file /usr/local/joshua_resources/russian_experiments/exp3/tune/output.nbest --decoder-log-file /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.log --iterations 10 --metric 'BLEU 4 closest'
        JOB FAILED (return code 1)
      Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
      	at org.apache.joshua.decoder.ff.tm.packed.PackedGrammar$PackedSlice.initializeFeatureStructures(PackedGrammar.java:385)
      	at org.apache.joshua.decoder.ff.tm.packed.PackedGrammar$PackedSlice.<init>(PackedGrammar.java:368)
      	at org.apache.joshua.decoder.ff.tm.packed.PackedGrammar.<init>(PackedGrammar.java:153)
      	at org.apache.joshua.decoder.Decoder.initializeTranslationGrammars(Decoder.java:458)
      	at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:389)
      	at org.apache.joshua.decoder.Decoder.<init>(Decoder.java:128)
      	at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
      Traceback (most recent call last):
        File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 553, in <module>
          main(sys.argv)
        File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 536, in main
          run_zmert(opts.tunedir, opts.source, opts.target, opts.decoder, opts.decoder_config, opts.decoder_output_file, opts)
        File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 417, in run_zmert
          opts.metric, opts.iterations or 10)
        File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 399, in setup_configs
          for feature,weight in get_features(config):
        File "/usr/local/incubator-joshua/scripts/training/run_tuner.py", line 351, in get_features
          output = check_output("%s/bin/joshua-decoder -c %s -show-weights -v 0" % (JOSHUA, config_file), shell=True)
        File "/Users/lmcgibbn/miniconda3/lib/python3.5/subprocess.py", line 626, in check_output
          **kwargs).stdout
        File "/Users/lmcgibbn/miniconda3/lib/python3.5/subprocess.py", line 708, in run
          output=stdout, stderr=stderr)
      subprocess.CalledProcessError: Command '/usr/local/incubator-joshua/bin/joshua-decoder -c /usr/local/joshua_resources/russian_experiments/exp3/tune/joshua.config -show-weights -v 0' returned non-zero exit status 1
      

      This is because, by default the joshua-decoder script runs with 4g of memory. The run_runer.py script should be flexible enough to continue with the memory allocation provided when a pipe was initially invoked. This value should then be passed to the joshua-decoder script.

      Attachments

        Activity

          People

            Unassigned Unassigned
            lewismc Lewis John McGibbney
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: