Find a way to update item schema in database when changed

[mStirner]

After updating the schema on the backend, some values need to set/deleted in the database.
How can this be done without a "updater" script?

• Handle this via joi schema and set each item?
• Use mongoose version version like the _v

Related:

• TypeError: Cannot read properties of undefined (reading 'map') #238
• Automatically update items in databse when component schema has changed #75

[mStirner]

Run each item through validation progress?
If a item is in the database stored, it must be valid.

When validating it on startup again, missing fields a added.
The updated/added fileds can then be saved with its default values set in the database.

But how to remove deprecated fields?
Add a custom Joi .deprecated/.remove method that removes the field when validated?
Just like .default but instead of setting, it removes it.

---

Its a bit ugly to validate each item on startup and save it again.
But how can this be better done without a separate update script?

[mStirner]

Before this can be implemented, #268 should be fixed.
Then a (schema)-version can be stored in the db, and after a backend update (package.json vesion changed), a comparsion, e.g. package.json version > database stored version, a validation check could be done in

backend/system/component/class.component.js

Lines 72 to 94 in 185a480

	set(obj, prop, value) {
	// ignore all methods/properties
	if (new RegExp(/^\d+$/).test(prop)) {
	// preparation for #75
	// apply missing item properties here
	// verfiy object item, if verfiy fails, return false
	//console.log("Add item: %j", value);
	//let { error } = schema.validate(value);
	//return error ? false : true;
	// save update obj in database?
	//return false;
	}
	obj[prop] = value;
	return true;
	}

Not tested implementation:

            set: async (obj, prop, val) => {

                // ignore all methods/properties
                if (new RegExp(/^\d+$/).test(prop)) {
                    // or use semver.compare() here!
                    if (pkg.version > config.version) {

                        // verfiy object item schema, apply default values
                        let { error, value } = this.schema.validate(val);

                        // good idea to kill everything?
                        // will deprecated fields be a error?
                        // would be better to evalute error.details
                        // https://joi.dev/api/?v=17.9.1#validationerror
                        if (error) {
                            console.log(error);
                            process.exit(1);
                        }

                        // update item/document in database 
                        let { ok } = await this.collection.findOneAndUpdate(new ObjectId(value._id), {
                            $set: value
                        });

                        obj[prop] = value;
                        return ok;
                        
                    }
                }

                obj[prop] = val;
                return true;

            }

[mStirner]

Wouldnt it be better to add a option/config value to enable/disable item validation on startup? Which can then be auto enabled after a update, and disabled after successfull validating.

[mStirner]

Check if a validation completes, even when a error is "thrown" and the returned value object is usable.

If so, a deprecation can be added as custom error, and procced with the item data.

Also when the values object is returned, new default values can be added, and old fields removed when deprecated.

[mStirner]

Playground example:

const Joi = require("joi");

const deprecated = (base, opts = {}) => {
    return Joi.extend((joi) => ({
        type: "deprecated",
        base,
        messages: {
            "deprecated": "{{#label}} is deprecated and will be removed in the future"
        },
        validate: (value, helpers) => {
            return {
                value,
                errors: helpers.error("deprecated", {
                    value
                })
            };
        }
    })).deprecated(opts);
}

const flow = Joi.object({
    name: deprecated(Joi.string()),
    enabled: Joi.boolean().default(true)
});

const schema = Joi.object({
    number: deprecated(Joi.number()).default(123),
    name: deprecated(Joi.string()),
    options: Joi.object({
        highWaterMark: Joi.number().default(1024),
        flow: Joi.array().items(flow),
        emptyBuffer: Joi.boolean().default(false)
    }).required(),
});



// Beispielverwendung
const { error, value } = schema.validate({
    name: "Super duper name",
    options: {
        flow: [{
            name: "Disable flow",
            enabled: false
        }, {
            name: "Enabled flow",
            enabled: true
        }]
    },
    number: 420690815
}, {
    abortEarly: false
});



/*
error.details.filter((error) => {
    return error.type === "deprecated.invalid";
}).forEach((error) => {
    console.error(">", error.message)
});
*/


// check if others error exists than deprecation
const others = error.details.filter(({ type }) => {
    return type !== "deprecated";
});



console.log(others, value);

[mStirner]

Another approach: https://dev.to/nas5w/an-approach-to-javascript-object-schema-migration-1a94