{"ygPerms":{"resourceCapabilityList":[{"resourceType":"GROUP","capabilities":[{"name":"UPDATE"},{"name":"READ"},{"name":"WELCOME_MSG"},{"name":"PENDING_COUNT"},{"name":"ACTIVITY_LOGS"}]},{"resourceType":"PHOTO","capabilities":[{"name":"READ"},{"name":"READ"},{"name":"UPLOAD"},{"name":"UPDATE"},{"name":"DELETE"},{"name":"COPY"},{"name":"MOVE"},{"name":"APPROVE"},{"name":"UPLOADTEMP"},{"name":"GROUPCOVER"}]},{"resourceType":"FILE","capabilities":[{"name":"READ"},{"name":"CREATE"},{"name":"UPDATE"},{"name":"DELETE"},{"name":"AUTO_SEND"}]},{"resourceType":"MEMBER","capabilities":[{"name":"READ"},{"name":"READ"},{"name":"EDIT_MODERATOR"},{"name":"DELETE"},{"name":"CREATE"},{"name":"UPDATE"},{"name":"BAN"}]},{"resourceType":"LINK","capabilities":[{"name":"READ"},{"name":"CREATE"},{"name":"UPDATE"},{"name":"DELETE"},{"name":"MOVE_LINK"}]},{"resourceType":"CALENDAR","capabilities":[{"name":"READ"}]},{"resourceType":"DATABASE","capabilities":[{"name":"CREATE"},{"name":"READ"},{"name":"READ_DATA"}]},{"resourceType":"POLL","capabilities":[{"name":"READ"},{"name":"VOTE"},{"name":"UPDATE"},{"name":"CREATE"},{"name":"DELETE"},{"name":"CLOSE"}]},{"resourceType":"MESSAGE","capabilities":[{"name":"CREATE"},{"name":"READ"},{"name":"DELETE"}]},{"resourceType":"PENDING_MESSAGE","capabilities":[{"name":"READ"},{"name":"UPDATE"},{"name":"DELETE"}]},{"resourceType":"ATTACHMENTS","capabilities":[{"name":"READ"},{"name":"DELETE"}]},{"resourceType":"PHOTOMATIC_ALBUMS","capabilities":[{"name":"READ"},{"name":"UPLOAD"}]},{"resourceType":"MEMBERSHIP_TYPE","capabilities":[{"name":"READ"},{"name":"CREATE"},{"name":"DELETE"},{"name":"UPDATE"}]},{"resourceType":"POST","capabilities":[{"name":"UPDATE"},{"name":"CREATE"},{"name":"READ"},{"name":"DELETE"}]},{"resourceType":"PIN","capabilities":[{"name":"UPDATE"},{"name":"CREATE"},{"name":"READ"},{"name":"DELETE"}]}],"modStatus":"MODERATOR","subStatus":"NORMAL","groupUrl":"groups.yahoo.com","intlCode":"us"},"comscore":"pageview_candidate","ygData":{"userId":465241002,"authorName":"Anne Steinke","from":"&quot;Anne Steinke&quot; &lt;annesteinke@gmx.net&gt;","profile":"kurtschwitters42","replyTo":"LIST","senderId":"JLgIOTgS72MO32QtFdH3EBv_rxUzZ4SSyqdXMwsYzqOTy0IM6O0hCh1AE2bsovroHQWouE-BsAQx2eFJ79giFBPjfkMy12Pq8FSi","spamInfo":{"isSpam":false,"reason":"12"},"subject":"Re: [OmT] word/character separation (Chinese) Tokenizers","postDate":"1284666339","msgId":19021,"canDelete":true,"contentTrasformed":false,"systemMessage":false,"headers":{"messageIdInHeader":"PDcxQTBERjBFM0IyMDRCMUZCNjA1Q0Q2MkMzM0E3NzQ4QEFubmVzTm90ZWJvb2s+","referencesHeader":"PGk2MTU0YStzOGVlQGVHcm91cHMuY29tPiA8QkRFRTIwNTQtNUZBMy00RjU4LUJCMjktNEM0MkUyOURERDY0QGdtYWlsLmNvbT4="},"prevInTopic":0,"nextInTopic":19023,"prevInTime":19020,"nextInTime":19022,"topicId":19021,"numMessagesInTopic":3,"msgSnippet":"... I tested the different Tokenizers: LuceneChineseTokenizer: Dictionary: only 1-character matchings, so as modern Chinese uses mostly 2-character words,","rawEmail":"Return-Path: &lt;annesteinke@gmx.net&gt;\r\nX-Sender: annesteinke@gmx.net\r\nX-Apparently-To: OmegaT@yahoogroups.com\r\nX-Received: (qmail 11257 invoked from network); 16 Sep 2010 19:45:51 -0000\r\nX-Received: from unknown (98.137.34.45)\n  by m2.grp.sp2.yahoo.com with QMQP; 16 Sep 2010 19:45:51 -0000\r\nX-Received: from unknown (HELO mail.gmx.net) (213.165.64.23)\n  by mta2.grp.sp2.yahoo.com with SMTP; 16 Sep 2010 19:45:46 -0000\r\nX-Received: (qmail invoked by alias); 16 Sep 2010 19:45:44 -0000\r\nX-Received: from p549F3AC8.dip.t-dialin.net (EHLO AnnesNotebook) [84.159.58.200]\n  by mail.gmx.net (mp025) with SMTP; 16 Sep 2010 21:45:44 +0200\r\nX-Authenticated: #21279533\r\nX-Provags-ID: V01U2FsdGVkX1/238lx53r272okRwTjdk4Voa23V77ikpjOq4BIm9\n\t3430xAVaZIIn8T\r\nMessage-ID: &lt;71A0DF0E3B204B1FB605CD62C33A7748@AnnesNotebook&gt;\r\nTo: &lt;OmegaT@yahoogroups.com&gt;\r\nReferences: &lt;i6154a+s8ee@eGroups.com&gt; &lt;BDEE2054-5FA3-4F58-BB29-4C42E29DDD64@gmail.com&gt;\r\nDate: Thu, 16 Sep 2010 21:45:39 +0200\r\nMIME-Version: 1.0\r\nX-Priority: 3\r\nX-MSMail-Priority: Normal\r\nX-Mailer: Microsoft Outlook Express 6.00.2900.5931\r\nX-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5994\r\nX-Y-GMX-Trusted: 0\r\nX-Originating-IP: 213.165.64.23\r\nX-eGroups-Msg-Info: 1:12:0:0:0\r\nFrom: &quot;Anne Steinke&quot; &lt;annesteinke@gmx.net&gt;\r\nSubject: Re: [OmT] word/character separation (Chinese) Tokenizers\r\nX-Yahoo-Group-Post: member; u=465241002; y=vKu16Zbke9RHDbawSf0K4p2c4zTTaisgY8GFk6G4gwnDrMobQIyRvPEBjg\r\nX-Yahoo-Profile: kurtschwitters42\r\nContent-Type: text/plain; charset=US-ASCII\r\nContent-Transfer-Encoding: 7bit\r\n\r\n&gt;&gt; My problem now is, that OmegaT treats a whole sequence of characters without spaces inbetween as a word,\n&gt;&gt; and does not find the correct fuzzy matchings, nor look up the glossary. \n\n&gt; I think you need to use the CJK tokenizer.\n\nI tested the different Tokenizers:\n\nLuceneChineseTokenizer: \nDictionary: only 1-character matchings, so as modern Chinese uses mostly 2-character words, there are lots of uninteresting matches.\nGlossary: All entries of different length that match to one or more characters are found.\nTM: Works \n\nLuceneCJKTokenizer: \nDictionary: only 2-character matchings, so the results are better to use than with the LuceneChineseTokenizer, but still Words with more characters are not found. The 1-character results are not shown any more.\nGlossary: only entries with more than 2 characters are displayed.\nTM: Works\n\nLuceneSmartChineseTokenizer:\nDictionary: nothing found at all.\nGlossary: If several entries contain one ore more characters, the some of these entries are sorted out, but the principles of this sorting out are not so clear.\nTM:finds more matchings than the other two tokenizers.\n\nSo it seems that each Tokenizer has its own strength and weak points. For me the best to use seemes to be the CJK one, but it also still could be improved.\n\nWith the ideal Tokenizer all words contained in a sentence should be found in the dictionary/glossary. Then the longest words should be sorted up, while the words already contained in another word should be sorted down or excluded.\n\nNow I wonder, if there are still problems, do they result from the tokenizers or from OmegaT?  \n\nAttached are some Chinese example sentences and an example glossary. \n\nCheers\n\n[Non-text portions of this message have been removed]\n\n\n"}}