V12 Data Architecture Investments

[aaclayton]

Database Behaviors

• Add server-side migrations framework that ensures retired source-only migrations are persisted whenever a database is connected #10257
• Improvements to the DatabaseBackend interface #10214
• Expand the database transaction workflow to offer batch-level pre and post functions that can modify, react to, or wholesale reject sets of proposed changes. #9740
• Expand the database transaction workflow to offer batch-level pre and post functions that can modify, react to, or wholesale reject sets of proposed changes. #9740
• Eliminate unnecessary Document#toObject call in server-side batch update and delete transactions which improves performance for database round trips. #10239
• System or module-declared fields requiring sanitization can be missed when embedded records are created or modified as part of a parent document workflow. #10352
• Provide a centralized framework for client-side migrations which systems and modules can use to correct or upgrade source data #8476

Websocket Behaviors

• Support buffering of socket updates that occur between the initial page load and game.ready in order to ensure that changes relative to the initially provided socket data are not lost. #5624

Data Field Changes

• min and max data field option for ArrayField #10186
• Added TypedSchemaField #10192
• Added JavaScriptField #10425
• ArrayField#_cast should support Set as an input value type which is converted to an array #10418
• Improve the behavior of readonly fields to resolve some issues with prepared values and changes. #9665
• Support defining certain data model fields which are only editable by a GM user regardless of other permissions #9613
• Add a StringField subclass for DocumentUUIDField which is validated as a UUID string either to world or compendium content. #9736
• NumberField#clean no longer applies abs() to negative values if positive: true #10393
• Improve validation of the DocumentOwnershipField to allow removing ownership assignment for a user, falling back to the default value #10330
• Add support to DocumentUUIDField to specify a certain document type and whether that document is allowed to be embedded or not. #10316
• Improve DataField#fieldPath to accurately reflect the dot-separated property paths in the data model that contain the field. #10307
• Introduce the DataField#toInput and DataField#toFormGroup methods which allow a field in a data schema to be rendered as an HTMLElement #10306
• Add a new constructor parameter to DataField class which defines field context, supporting the assignment of field name or parent at constructor-time #10238
• Deprecated AngleField#base in favor of AngleField#normalize #9970
• Convert values of ColorField to lowercase #9851
• StringField overrides default initial value defined by subclasses #9656
• textSearch property doesn't work with ArrayField(EmbeddedDataField) #10063

Data Model Changes

• Fixed propagation of options.source in EmbeddedDataField#clean, EmbeddedDataField#validate, TypeDataField#clean, and TypeDataField#validate #10442
• Removed _stats field from TombstoneData #10374
• Document.updateSource with dry run option does not properly restore embedded data model sources. #10395

Document Changes

• Refactor Document data model to store source information in _stats #9567
• Pass update data to Document._onUpdateDocuments #10235
• Users who are banned from the world with role NONE automatically fail permission tests over a Document regardless of its ownership #10362
• Forward database operation events to the system type data model #10410
• Revisit support for null ActorDeltas #10131

Potential Future Improvements

The following issues did not make the prioritization cut for V12 but are related enhancements that we could make in a later version:

• Deprecate the socket.io library in favor of move to native client-side WebSocket interface. #4314
• Explore optimization from sending socket data as binary directly instead of as compressed JSON strings. #9777
• Request: Implement an EmbeddedDocumentField definition which references a specific Embedded Document #9854

[aaclayton]

Closing this as (mostly) done for V12.