Migration strategy

[benjaminpjones]

We haven't done any DB migrations yet, though migration strategy has been brought up in the past. Let's decide what is the best way to approach these things, so we can migrate and remove code that is used to handle legacy data.

Examples of migrations that we'd like to handle:

• empty players array:

  govariants/packages/server/src/games.ts

  Line 227 in 7dbf985

  async function BACKFILL_addEmptyPlayersArray(game: GameResponse) {

• time control undefined:

  govariants/packages/server/src/time-control/time-handler-sequential.ts

  Lines 70 to 74 in 7dbf985

  	// could happen for old games
  	// TODO: remove? maybe after db migration?
  	if (game.time_control === undefined) {
  	game.time_control = this.initialState(game.variant, config);
  	}

--

Close criteria: After discussion, let's get a short document/README somewhere and link it from CONTRIBUTING.md

[benjaminpjones]

See also #183, some discussion on migrations (but mostly w.r.t. indexes)

---

Personally, as someone who does a lot of the admin stuff, I like that our current solutions don't require any bulk actions from a human (running a script or whatever). It's the type of thing I am liable to forget to do after the code changes land. One thing we might consider is to forget about migrations, but colocate all the "fix" code in one place.

[benjaminpjones]

There are also tools like migrate-mongo. It enforces an "up/down" strategy so that rollbacks are easier

[merowin]

I'm thinking it could work something like this:

On startup, the server checks which migrations have been applied (comparing a migrations history from the DB with a list of migrations in the source code). Then the server applies all new migrations in order (and updates the migration history). The server API is available only after successful migration.
Ideally migrations are atomic (one call to the database), so we don't need to deal with half-applied migrations if something goes wrong.

About Up / Down strategies, how would that look in our case? For example for the player array, I can't imagine how we would reverse that migration.

[benjaminpjones]

On startup, the server checks which migrations have been applied (comparing a migrations history from the DB with a list of migrations in the source code). Then the server applies all new migrations in order (and updates the migration history). The server API is available only after successful migration.
Ideally migrations are atomic (one call to the database), so we don't need to deal with half-applied migrations if something goes wrong.

I think this sounds good, and it's what migrate-mongo basically supports (migrate-mongo up figures out which migrations are still pending and performs them in order). I worry about how much downtime bulk migrations will involve in the future, but for now I think it's going to be tolerable given our relatively small size of both users and data.

About Up / Down strategies, how would that look in our case? For example for the player array, I can't imagine how we would reverse that migration.

I think the downgrade strategy is more important in other scenarios. For example, if we change the name of a variant. If we needed to revert to an older version, we would also want to reverse the name change.

For the players array example, I'm not sure what is best, but I think there are two options:

• Reversal is a no-op because old code can just ignore the players array
  • Pro: less destructive
  • Con: leaves a mess
• Delete the players arrays
  • Pro: cleaner data that matches schema
  • Con: Destructive - if players arrays already got populated, we'll lose that