Writing state migrations

Table of Contents

• Steps to follow while writing state migrations
• Important note
• Testing state migration locally

If your PR changes the properties of an exploration or state (or other structure), it should also include a migration so that existing entities in the datastore can be migrated smoothly to the new structural format. The following instructions describe how to write such a migration for states of explorations or questions.

Steps to follow while writing state migrations

1. Make the necessary changes to the State class (or its descendant classes) to reflect the post-migration state structure.
2. Increment the CURRENT_STATE_SCHEMA_VERSION in the feconf.py file.
3. Increment the CURRENT_EXP_SCHEMA_VERSION in the exp_domain.py file.
4. Start with writing _convert_states_v{old_state_version}_dict_to_v{old_state_version + 1}_dict method in exp_domain.py under Exploration class and in question_domain.py under Question class. In exp_domain.py, update the Exploration._migrate_to_latest_yaml_version method to use your conversion function.
5. Write a conversion function in draft_upgrade_services.py
   • Write _convert_states_v{old_state_version}_dict_to_v{old_state_version + 1}_dict function that makes appropriate upgrades to data that resides in ExplorationChange lists.
   • Here you have to check if the changes that you do in your exploration conversion function affect the draft_change_list, if yes, then make sure to update the drafts or raise InvalidDraftConversionException which will drop the changes in the drafts (this needs to be done in cases where changes to drafts would be too complicated).
   • If no changes need to be done to drafts you can simply return the draft_change_list.
6. Edit the tests for the previous conversion functions in the exp_domain_test.py to update to the latest schema version. Also, update the schema of core/tests/data/oppia-ThetitleforZIPdownloadhandlertest!-v2-gold.zip file. To update the zip file you would require to unzip it and then make the changes.
7. Create a PR. If the tests fail resolve them.
8. Once your PR is finalized, file a one request for the AuditExplorationMigrationJob and MigrateExplorationJob by following the wiki page. The job tests a migration by running your conversion function on the dicts of existing exploration models and validating that the migration will be successful. Make sure to mention to only run MigrateExplorationJob when the AuditExplorationMigrationJob is successful. After successful testing you can get you PR merged.

Note

These steps are for the migration where one is changing the schema of all existing states, depending on the changes your migration is going to make the steps will be less as you’ll have to change very fewer test files.

If you find new test files where changes needed to be required, try updating the list.

Links to relevant PRs:

• Deprecate math interactions' unsupported rules and update cust arg name: #15271
• Add state migration for new customization_arg catchMisspellings: #16077. Please note that in the PR changes are done in several files and those are done according to the requirements.

Important note:

• Make sure to add a thorough test for the migration function covering each possible case. Check the PR #11466 for reference. Please note that while you are writing tests in the exp_domain_test.py file you should use swap function to swap the CURRENT_STATE_SCHEMA_VERSION to the schema version you are writing, This way in case you have several tests the next person who will write the schema tests do not have to change states_schema_version of every test present. We currently do not have any specific reference to this change, the above PR does not include this as we recently decided to go ahead with this decision. You can always take reference of swap function in the codebase to see the usage.

Testing state migration locally:

• Checkout develop.
• Start the server and go to the admin page.
• Load all demo exploration.
• Create a new exploration, make some changes and save them.
• Checkout the feature branch which contains state migration.
• Go to 0.0.0.0:8000 and flush existing Memcache from the Memcache tab.
• Go to the admin page and assign yourself the role of "release coordinator".
• Go to the Misc tab of /release-coordinator page and flush the cache.
• Run AuditExplorationMigrationJob and wait for the job to get completed. This job will not make any changes to the exploration as this is simply an audit job. We run this job before the actual migration job so that in case anything fails we do not make changes to the datastore. Please make sure the job should be successful without any errors before moving forward.
• Run MigrateExplorationJob and wait for the job to get completed. This will make changes to exploration.
• Check the output of the job and post the screen-shot in your PR.
• Go to the exploration you have created lately, check whether it's working as expected.
• Check demo exploration (note demo exploration ids are 0, 1, 2, etc.)