{"ygPerms":{"resourceCapabilityList":[{"resourceType":"GROUP","capabilities":[{"name":"UPDATE"},{"name":"READ"},{"name":"WELCOME_MSG"},{"name":"PENDING_COUNT"},{"name":"ACTIVITY_LOGS"}]},{"resourceType":"PHOTO","capabilities":[{"name":"READ"},{"name":"READ"},{"name":"UPLOAD"},{"name":"UPDATE"},{"name":"DELETE"},{"name":"COPY"},{"name":"MOVE"},{"name":"APPROVE"},{"name":"UPLOADTEMP"},{"name":"GROUPCOVER"}]},{"resourceType":"FILE","capabilities":[{"name":"READ"},{"name":"CREATE"},{"name":"UPDATE"},{"name":"DELETE"},{"name":"AUTO_SEND"}]},{"resourceType":"MEMBER","capabilities":[{"name":"READ"},{"name":"READ"},{"name":"EDIT_MODERATOR"},{"name":"DELETE"},{"name":"CREATE"},{"name":"UPDATE"},{"name":"BAN"}]},{"resourceType":"LINK","capabilities":[{"name":"READ"},{"name":"CREATE"},{"name":"UPDATE"},{"name":"DELETE"},{"name":"MOVE_LINK"}]},{"resourceType":"CALENDAR","capabilities":[{"name":"READ"}]},{"resourceType":"DATABASE","capabilities":[{"name":"READ_DATA"},{"name":"CREATE"},{"name":"READ"}]},{"resourceType":"POLL","capabilities":[{"name":"READ"},{"name":"VOTE"},{"name":"UPDATE"},{"name":"CREATE"},{"name":"DELETE"},{"name":"CLOSE"}]},{"resourceType":"MESSAGE","capabilities":[{"name":"CREATE"},{"name":"READ"},{"name":"DELETE"}]},{"resourceType":"PENDING_MESSAGE","capabilities":[{"name":"READ"},{"name":"UPDATE"},{"name":"DELETE"}]},{"resourceType":"ATTACHMENTS","capabilities":[{"name":"READ"},{"name":"DELETE"}]},{"resourceType":"PHOTOMATIC_ALBUMS","capabilities":[{"name":"READ"},{"name":"UPLOAD"}]},{"resourceType":"MEMBERSHIP_TYPE","capabilities":[{"name":"READ"},{"name":"CREATE"},{"name":"DELETE"},{"name":"UPDATE"}]},{"resourceType":"POST","capabilities":[{"name":"DELETE"},{"name":"UPDATE"},{"name":"CREATE"},{"name":"READ"}]},{"resourceType":"PIN","capabilities":[{"name":"DELETE"},{"name":"UPDATE"},{"name":"CREATE"},{"name":"READ"}]}],"modStatus":"MODERATOR","subStatus":"NORMAL","groupUrl":"groups.yahoo.com","intlCode":"us"},"comscore":"pageview_candidate","ygData":{"userId":40439957,"authorName":"Didier Briel","from":"&quot;Didier Briel&quot; &lt;d.briel@free.fr&gt;","profile":"didier_briel","replyTo":"LIST","senderId":"hmeMQig9tp3RvaqjVJ3wHKJO7Wklo-uwMZG07DeZOBNocGJKYa11hgkEycLJLmtmik-tGomvjVx__UQpeiyNYJrLHXiuj7A","spamInfo":{"isSpam":false,"reason":"0"},"subject":"RE: [OmT] tokenizer &lt;&gt; stemmer","postDate":"1290865844","msgId":19991,"canDelete":true,"contentTrasformed":false,"systemMessage":false,"headers":{"messageIdInHeader":"PEFPRUhLQ0tPSU9HUEdDTkxERUtCS0VET0hKQUEuZC5icmllbEBmcmVlLmZyPg==","inReplyToHeader":"PGljcjFmNitxMm92QGVHcm91cHMuY29tPg=="},"prevInTopic":19990,"nextInTopic":20002,"prevInTime":19990,"nextInTime":19992,"topicId":19990,"numMessagesInTopic":5,"msgSnippet":"... smo ... for instance to return cat , when offered cats , catlike , catlike , catty etc. Which is what the Tokenizer plugin in OmegaT is about. Of ","rawEmail":"Return-Path: &lt;d.briel@free.fr&gt;\r\nX-Sender: d.briel@free.fr\r\nX-Apparently-To: OmegaT@yahoogroups.com\r\nX-Received: (qmail 41974 invoked from network); 27 Nov 2010 13:50:53 -0000\r\nX-Received: from unknown (66.196.94.105)\n  by m15.grp.re1.yahoo.com with QMQP; 27 Nov 2010 13:50:53 -0000\r\nX-Received: from unknown (HELO smtp5-g21.free.fr) (212.27.42.5)\n  by mta1.grp.re1.yahoo.com with SMTP; 27 Nov 2010 13:50:53 -0000\r\nX-Received: from Acer (unknown [82.235.41.242])\n\tby smtp5-g21.free.fr (Postfix) with SMTP id 783D3D4816C\n\tfor &lt;OmegaT@yahoogroups.com&gt;; Sat, 27 Nov 2010 14:50:47 +0100 (CET)\r\nTo: &lt;OmegaT@yahoogroups.com&gt;\r\nDate: Sat, 27 Nov 2010 14:50:44 +0100\r\nMessage-ID: &lt;AOEHKCKOIOGPGCNLDEKBKEDOHJAA.d.briel@free.fr&gt;\r\nMIME-Version: 1.0\r\nContent-Type: text/plain;\n\tcharset=&quot;iso-8859-1&quot;\r\nContent-Transfer-Encoding: 7bit\r\nX-Priority: 3 (Normal)\r\nX-MSMail-Priority: Normal\r\nX-Mailer: Microsoft Outlook IMO, Build 9.0.6604 (9.0.2911.0)\r\nX-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5994\r\nIn-Reply-To: &lt;icr1f6+q2ov@eGroups.com&gt;\r\nImportance: Normal\r\nX-Originating-IP: 212.27.42.5\r\nFrom: &quot;Didier Briel&quot; &lt;d.briel@free.fr&gt;\r\nSubject: RE: [OmT] tokenizer &lt;&gt; stemmer\r\nX-Yahoo-Group-Post: member; u=40439957; y=uoL0LWt_-lco8-VipKE61uzxVCf7HlNzFkfFozd6w1RcnzhuHkjk\r\nX-Yahoo-Profile: didier_briel\r\n\r\n-----Original Message-----\n&gt;From: OmegaT@yahoogroups.com [mailto:OmegaT@yahoogroups.com]On Behalf Of\nsmo\n&gt;Sent: Saturday, November 27, 2010 2:36 PM\n&gt;To: Omegat@yahoogroups.com\n&gt;Subject: [OmT] tokenizer &lt;&gt; stemmer\n&gt;\n&gt;Tokenizer is a tool to separate a grammatical unit, for instance the\nfollowing:\n&gt;\n&gt;Twas brillig / And the slithy toves ...\n&gt;\n&gt;into its lexical units\n&gt;\n&gt;&lt;verb&gt;Twas&lt;/verb&gt;\n&gt;&lt;adjective&gt;brillig&lt;/adjective&gt;\n&gt;&lt;conjunction&gt;And&lt;/conjuction&gt;\n&gt;...etc...\n&gt;\n&gt;Tokenizers do not address the question of finding the stem of a given word,\nfor instance to return &quot;cat&quot;, when offered &quot;cats&quot;, &quot;catlike&quot;, &quot;catlike&quot;,\n&quot;catty&quot; etc. Which is what the &quot;Tokenizer&quot; plugin in OmegaT is about. Of\ncourse one needs a tokenizer to get to the &quot;foxy&quot; in the &quot;You gotta be all\nmine, all mine, foxy Lady&quot;, but to do this, any half-decent (RegEx-based)\ntokenizer will do (for instance the one used in OmegaT, just look at the\nsegmentation rules, they are tokenizing rules). What no tokenizer will ever\ndo, however, is get the stem for &quot;foxy&quot;, i.e. the noun/verb &quot;fox&quot;.\n&gt;\n&gt;Stemmer may sound like stammer, but for me this is far less of a problem,\nthan the &quot;tokenizer&quot; misnomer. Let&#39;s think about correcting it, before it\nsinks too deep.\n\n- Tokenizers are based on Lucene tokenizers, which we use as a replacement\nfor the built-in Java *token* function.\n\n- Not all (OmegaT) tokenizers are stemmers.\nIn some languages (CJK), we&#39;re already happy if (thanks to Lucene\ntokenizers) we can get tokens that are more decent than the standard Java\nones (which are often unusable). In those languages, do you really think a\nRegEx-based tokenizer could generate something useful?\nWe have even one language were the only function (unfortunately) of the\ntokenizer is to provide stop words, and which uses the Java tokenizer for\nthe actual tokenization.\n\nThink of stemming as an advanced way of tokenizing, which is provided by\n*some* of the tokenizers.\n\nDidier\n\n\n\n"}}