How exactly does lemmy search work for related posts? (programming.dev)

submitted 1 month ago by ICastFist@programming.dev to c/fediverse@lemmy.world

2 comments fedilink hide all child comments

I don't know if this is stance specific, but whenever I create a new post and start typing the title, it'll list some "posts that might be related", limited to the community if it's already selected.

The problem is that the posts are almost never related to anything I'm writing in the title or might even write in the body at all. For instance, I just posted a question on how to poison videos against AI, the "might be related" posts were about Donald Trump and Epstein, with another post being about WoW Mists of Pandaria.

It seems to look for the post body for all the words in the title first instead of looking for any similar title first

you are viewing a single comment's thread
view the rest of the comments

[-] schwim@piefed.zip 9 points 1 month ago

It seems to be a non-common word comparison score, which is often mistaken.

Basically, when a post is submitted, the system creates a word list for the post, strips the common words out of it, then creates an association between the post ID and each remaining word. When a search occurs, each word of the search phrase is checked against the db holding the associations and the post with the most matching associations will be considered the most relevant.

It can be seen how this system could be incredibly incorrect when matching posts to a search phrase but it's really the one that's most often used in software, due to it's ease of implementation and low overhead, resource-wise.