(This is part of the TextMiningProject)

The idea

There are many WikiTopics in similar fields, but not always linked together. I wanted to create an automatic solution to find the "See also" topics, and display them.

There is an assumption of TextMining that topics with common words are similar in meaning, field. So I downloaded some tools and wrote some scripts that read the contents of a wiki, calculate the "bags of words", and create clusters of the similar topics. The result is put to the the _ClusteringResults topic, and the WikiTalk script in ClusteringResultsBorders displays it, as seen at the AdministratorsGuide or TextMiningProject topics

Right now, for using it, you have to include the following line in the topic:

Implementation

1. FwSync is used to get the actual topics (60 sec)
2. A script prepares the content (30 sec)
1. copies the .wiki files with acceptable names (no special chars)
2. separates the PascalCased words
3. writes down the topic name, summary, keywords and headings more times, to give greater weight to the words found there.
3. Txt2Bow.exe form TextGarden calculates the bag of words model (5 sec)
4. BowKMeans.exe from TextGarden calculates the clusters, the output is an xml file (30 sec)
5. An xslt creates _ClusteringResults (ran by msxsl.exe)
1. Usually there are character coding problems with the xml file, so it needs some workaround
2. The topics' path sould be removed, xsl 1.0 is not capable of this. There is a script for this too.

I'll write a script that does this as a batch job, and can be ran periodically, eg. once a day.

-- SzaMa 2007.01.04.