kiwix/kiwix-apple
1
0
Fork 0
mirror of https://github.com/kiwix/kiwix-apple.git synced 2025-09-16 23:58:37 -04:00
Code Issues Packages Projects Releases Wiki Activity

Updated How Search in 1.6 works (markdown)

Chris Li 2016-06-26 07:38:35 -04:00
parent 1026f3cc1f
commit 6dfbc72d0c
1 changed files with 5 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
How-Search-in-1.6-works.md

@ -30,7 +30,7 @@ You may have noticed a big flaw in our system till this step. Our system gives x
## Step 4: Utilize xapian probability
If xapian give a search result 100% probability value, what does it mean? Well, it means xapian is quite sure this is exactly the article you want. It is very confident. So, we need to give such an article a boost, despite they may have (very) long Levenshtein distance to our search term. On the other hand, if xapian gives an article very low probability, then this article may have nothing to do with out search term. Wen may need to give such articles a penalty, even though they may share great similarity with the search term.
A lot of times, xapian results congregate on the high end of the probability range. In other words, you get a lot of articles with 100%, 99%, etc., but not a lot with 75%, 66%, 45% and things like that. To better differentiate them, we need a non-liner map from probability to a boost/penalty factor.
A lot of times, xapian results congregate on the high end of the probability range. In other words, you get a lot of articles with 100%, 99%, etc., but not a lot with 75%, 66%, 45% and things like that. To better differentiate them, we need a non-liner map from probability to a boost/penalty factor. This also allow us not to give too much penalty on xapian result with low probability.
In 1.6, I use f(x) = ln(n - m * prob) as this boost factor, the default value is derived from:
@ -41,3 +41,7 @@ Solve for m & n, you get m = 6.4524 and n = 7.5576, [see plot here](http://www.w
The first order derivative of f(x) is f'(x) = 1 / (x - 1.17129). f(x) decrease faster when x = 1, slower when x = 0.
Step 5: Calculate rank score
For indexed search results: RankScore = Levenshtein distance * f(x)
For title search results: RankScore = Levenshtein distance * 1
Sort ascending by RankScore gives you final search ranks!