Lost in (Japanese) translation

A while back I needed to support automated conversion of English text into Japanese script with some Java code. And fortunately, there was a project available that support just such a conversion. So, while not particularly difficult this did require some configuration voodoo and the usual troubleshooting to finally get a library up and running. The steps presented here are pretty simple, but it took a bit longer when setting this up initially (primarily because most of the docs/hints in working with this library were in Japanese)--so hopefully it will save somebody sometime in the future. The library used was the sen java library which is a port of the Mecab C libraries to support translation.

Starting with this page:

http://ultimania.org/sen/hiki.cgi

Here is the Java port of mecab:    https://sen.dev.java.net/
First grab the library at:

wget https://sen.dev.java.net/files/documents/1373/31864/sen-1.2.2.1.zip

unzip sen-1.2.2.1.zip

cd sen-1.2.2.1

then, install the jdk and export JAVA_HOME to jre directory and build with

ant

Attempting to compile--missing the apache logging java libraries:

sen/sen-1.2.2.1/src/java$ javac ProcessorDemo.java

./net/java/sen/StringTagger.java:42: package org.apache.commons.logging does not exist

import org.apache.commons.logging.Log;

These are downloaded from commons.apache.org/logging:

commons-logging-1.1.1-src.tar.gz

tar xvfz commons-logging-1.1.1-src.tar.gz

At this point building will work after pointing the classpath to the apache logging project and the sen jar file.

slioch@build:~/sen_test/sen-1.2.2.1# javac demo/ProcessorDemo.java -cp ./commons-logging-1.1.1-src/src/java;./lib/sen.jar

Now time for the some runtime fun. Create a file to run out command (run). The two arguments to the Demo routine are the english test file as input to the processor, and the character encoding type (utf-8 in this case). The "-Dsen.home=." flag tells the jre to provide the executable with the location to find the Japanese dictionaries.

java -Dsen.home=. -Xmx300m -cp lib/sen.jar:lib/commons-logging.jar:./build/classes/ ProcessorDemo japanese-file.txt utf-8

Which results in the following:

slioch@build:~/sen_test/sen-1.2.2.1# ./run random-jap.txt utf-8

Dec 13, 2010 2:21:58 PM net.java.sen.Dictionary
INFO: token file = ./dic/token.sen
java.lang.IllegalArgumentException: Tokenizer Class: net.java.sen.ja.JapaneseTokenizer is invalid.
at net.java.sen.StringTagger.init(StringTagger.java:158)
at net.java.sen.StringTagger.(StringTagger.java:95)
at net.java.sen.StringTagger.getInstance(StringTagger.java:133)
at net.java.sen.StreamTagger.(StreamTagger.java:92)
at ProcessorDemo.main(ProcessorDemo.java:72)


We're missing the Japanese token file which should be at ./dic/token.sen. This is because the dictionary hasn't been built yet (and for some reason the built at the root of the project doesn't automatically compile this):

cd dic && ant 

This will build the missing token.sen dictionary file needed. Now we are off and running again. 


Dec 14, 2010 9:02:39 PM net.java.sen.Tokenizer loadConnectCost
INFO: time to load connect cost file = 183[ms]
　    記号-空白    0    1    581   
従    未知語      1    2    31577   
（    記号-括弧開  2    3    33354   
じ    助動詞      3    4    43354   
ゅ    未知語      4    5    81354   
）    記号-括弧閉  5    6    82759   
四    名詞-数     6    7    84986   
位    名詞-接尾-助数詞    7    8    86706   
下    名詞-接尾-一般    8    9    89574   
（    記号-括弧開    9    10    90926   
い    動詞-自立      10    11    94046   
の    助詞-連体化    11    12    96216   
げ    名詞-接尾-一般  12    13    101578   
）    記号-括弧閉    13    14    102937   
......




A morpheme per line.