Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: integrate meilisearch for searching #56

Open
wants to merge 44 commits into
base: main
Choose a base branch
from

Conversation

Codycody31
Copy link
Contributor

@Codycody31 Codycody31 commented Apr 9, 2024

Adds support for the use of meilisearch for quick indexing and searching of discuit communities.

Required

  • Move code for updating a communities index used before the API returns info to separate function
  • Rather than placing search code in the core move to internal/search so we can have a package for it and not have to worry about names being the same
  • Integration into the UI for searching (only if it is enabled)
  • When indexing, break up the community name on each capital letter (to fix searching for Dev not matching DiscuitDev, etc)
  • Allow searching on the communities page but default it when blank to the standard API

If Possible

  • Index posts and users
  • Sorting based on members, post creation time, user's account age, etc

General

Need to do testing to ensure this will work correctly, from my own local testing everything works as expected. It also needs PR #49 merged so I can add docker support. And lastly after this is merged, someone who is better a UI stuff then me should probably make the searching actually be nice lol

Known Issues

  • When searching for a community while creating a new post, the picture is not used due to it not being included in the data set to meilisearch

Related Issues:

@Codycody31 Codycody31 changed the title Featureintegrate meilisearch for search endpoint Feature: integrate meilisearch for searching Apr 9, 2024
@Codycody31 Codycody31 marked this pull request as ready for review April 10, 2024 16:58
@Codycody31 Codycody31 marked this pull request as draft April 22, 2024 17:52
@Codycody31
Copy link
Contributor Author

A new issue has been found! batching of data, as currently, it would attempt to dump the entire posts table to the index, however, the posts table has over 27K records sooooo. I'll need to batch the data, good thing is I already have code for it, just need to convert it over.

@Codycody31 Codycody31 marked this pull request as ready for review April 26, 2024 13:54
@Codycody31
Copy link
Contributor Author

Codycody31 commented Apr 26, 2024

@previnder, the indexing logic along with everything else is finished. Now when pulling data to index it takes it in batches of 5000 records (I'll change it if you want it to be higher or lower, probably higher around 10,000 or 15,000). When pushing the data over to meilisearch we chunk it into batches of 50MiB (or less if we don't have that much data). By default, the searching ability is disabled and still makes use of the Google search to allow for the hoster to fully configure everything without having to worry about meilisearch till they are ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add in-site search When searched moving to google Request for Additional Search Engines
1 participant