refind

Refind is a project to store my personnal documents in a Weaviate vector database and use OpenAI vectorizer module and generative search.

Here is a shell recording with asciinema showing the different answers with the same input question from :

1. the current ChatGPt version (using sgpt cli)
2. the generative search with Weaviate containing a sample document from the Enteprise Roadmap to SRE book

Getting started

Create and provide your OpenAI API key :

export OPENAI_APIKEY="..."

Put full-text documents with less than 4097 tokens. If your prompt is 4000 tokens, your completion can be 97 tokens at most.

Let's use it :

# start the weaviate database
docker-compose up -d

# verify it is up and running
curl http://localhost:8080/v1/meta | jq .
docker-compose logs

# import data, DO IT ONCE
go run main.go import

# query data related to software
go run main.go query | jq .

# when done, gracefully shutdown
docker-compose down

In this example, Weaviate is returning software-related entries.

Weaviate operations :

# get the schema
curl -s http://localhost:8080/v1/schema | jq .

# get objects
curl -s http://localhost:8080/v1/objects | jq .

# get one class from the schema
curl -s http://localhost:8080/v1/schema/Chatbot | jq .

# delete a class
curl -s -XDELETE http://localhost:8080/v1/schema/Chatbot | jq .

Goal

The ChatGPT Retrieval plugin.

TODO

1. automatically add more metadata in weaviate when ingesting documents
2. automatically split documents bigger than the OpenAI token limit
3. add query string as a flag when running the query command
4. unit tests, let Copilot write them \o/

Trying langchain

README.md