[SPARK-6926] Word2Vec's transform method throw IllegalStateException if a word not in vocabulary, but findSynonyms(word: String, num: Int) method neither try catch exception nor throw exception.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 1.3.0
• Component/s: MLlib
• Labels: mllib word2vec
• Flags: Important
• Assignee: Unassigned
• Reporter: chrispy
• Shepherd: Xiangrui Meng
• Created: 15/Apr/15 06:29
• Updated: 15/Apr/15 08:01
• Resolved: 15/Apr/15 07:38

Description

  val doc = sc.textFile("/xxx/segrestxt", args(3).toInt).map(line => line.split(" ").toSeq)
  val model = new Word2Vec().setVectorSize(10).setSeed(42L).setNumIterations(3).fit(doc)
  val cpseeds = sc.textFile("/xxx/seeds/cp.tag", args(3).toInt)

it will get a IllegalStateException and then exit app when tag is a word not in vocabulary, because findSynonyms method handle the possible IllegalStateException.

  cpseeds.map {
    tag =>
      val syn = model.findSynonyms(tag, 30).map(l => l._1)
      tag + ":" + Joiner.on(" ").join(JavaConversions.asJavaIterator(syn.toIterator))
  }.saveAsTextFile("/xxx/synonyms/cp")

developer need to catch the IllegalStateException to find out what's going on here.

  cpseeds.map {
    tag =>
      try {
        val syn = model.findSynonyms(tag, 30).map(l => l._1)
        tag + ":" + Joiner.on(" ").join(JavaConversions.asJavaIterator(syn.toIterator))
      } catch {
        case e: IllegalStateException => log.error(s"cp tag:$tag", e)
      }
  }.saveAsTextFile("/xxx/synonyms/cp")

Issue Links

• duplicates: SPARK-6925 Word2Vec's transform method throw IllegalStateException if a word not in vocabulary, but findSynonyms(word: String, num: Int) method neither try catch exception nor throw exception.