limits, orders, creation dates.. the annoying stuff

[StephenMiracle]

Hey, I'm starting to use Tiedot now in almost production environment of my project and the randomness of getting the data is starting to get to me. Also, I'm a little concerned about the lack of limits. At some point, it'll suck if someone tries to access some data collection and the query pulls back 10,000,000 rows in there right.. unless there isn't a performance hit for grabbing the documents

anyway, I read in another issue about the hassle / performance hit with ordering data. I know that tiedot is similar to MongoDB, but I was wondering if there was a way to add created_at or updated_at fields?

This way you could pull the content based on some created order and then you could have a limiter if needed.

I don't know much about db development and I know that this doesn't really follow the nosql mongo pattern but I thought it may be a useful solution.

[HouzuoGuo]

Till this day I still regret not being able to give tiedot a new index structure that'll support range queries, I implemented a skip list in the past but it did not perform well at all on disk, and since then my focus shifted away from DB engines.

Regarding query limit - indeed it will become a concern if a query incidentally touches millions of records, I have not yet studied how limit is implemented in other NoSQL databases, but for tiedot, there is a challenge to solve. Nevertheless it is most definitely feasible.

Consider an intersection query that calculates an AND condition based on two lookups, e.g. A=1 and B=2, the query processor looks for document IDs that have A=1 and then intersect with document IDs that have B=2. If an overall limit is placed on the query, then it will help reducing the scope of hash table lookup on condition B=2 - this is a fun coding exercise.

The limit logic will be further challenged by more complicated query, such as A=1 and B=2 or C=3. Anyways, a fun coding exercise.

Lately I've been working on a web server software called laitos and have relatively little time to maintain tiedot, nevertheless I'm sincerely thankful that you're evaluating tiedot in a serious software project, and I will try my best to cater it to your requirements.