Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow specifying multiple migration sources in CLI #3177

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Tortoaster
Copy link

@Tortoaster Tortoaster commented Apr 5, 2024

This PR extends the CLI to allow specifying multiple migration sources:

sqlx migrate run --source migrations --source fixtures

It will take the migrations within all specified folders, and consider each of them sorted by version in ascending order. This should be fully backwards compatible.

This change would allow separating migrations that set up table structure from those that create test data. This does not have an issue associated to it, please let me know if I should open one first.


Motivation (rather long)

Use case

When setting up a web project for local development, I like to include some test fixtures in the database, so that I can see how everything looks when populated with actual data. I create that test data from database dumps, so they are simple SQL files, just like migrations. So far, I solved this problem by having two separate directories in my project:

/project
    /migrations
        20240101000000_init.up.sql
        20240101000000_init.down.sql
    /fixtures
        20240101000001_add_a_few_blog_posts.sql
    ...

This way, I can run sqlx migrate run and sqlx migrate run --source fixtures locally to create both the tables and the test data within them. For the production database, I only run sqlx migrate run, so that the test data isn't included there. This works well enough in most cases.

The problem

A problem arises when I need to create a migration for a newer version that changes an existing table. The migration itself should already make sure existing data will conform to the new data model. However, with the above setup, all fixture data is created after the last table migration, so they will have to comply to the new data model from the start.

Example: suppose I add a column with a non-null and foreign key constraint to an existing table. That same migration will have to make sure that all existing rows get updated to have a valid value in that column. Existing production data doesn't have that column yet, but that will be fixed by running the migration. Existing fixtures don't have that column yet either, and attempting to run them will result in an error.

Alternative solutions

To work around this issue, I can choose to either:

  • move all fixture migrations into the /migrations folder (so that they are performed in order of their timestamps), then run sqlx migrate run, and then move them back so I don't accidentally do the same in the production database
  • manually adapt all fixture migrations to conform to the new data model from the get-go

Both of these are undesirable. The former requires manual work every time the database needs to be initialized (or some sort of external tool that synchronizes a single folder with the contents of both other folders). The latter can take quite long depending on the amount of test data, and also requires manual work to undo in case of a sqlx migrate revert.

Allowing to specify multiple sources with the CLI would solve the problem elegantly. I hope you'll consider including it in sqlx!

@Tortoaster Tortoaster changed the title Allow specifying multiple migration sources in CLI feat: Allow specifying multiple migration sources in CLI Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant