v0.11.0 Release

Changelog

DefraDB v0.11 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

To get a full outline of the changes, we invite you to review the official changelog below. This release does include a Breaking Change to existing v0.10.x databases. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.source.network/.

Features

• feat: Add Access Control Policy (#2338)
• feat: Add Defra-Lens support for branching schema (#2421)
• feat: Add P Counter CRDT (#2482)
• feat: Add PatchCollection (#2402)
• feat: Allow mutation of col sources via PatchCollection (#2424)
• feat: Force explicit primary decl. in SDL for one-ones (#2462)
• feat: Lens runtime config (#2497)
• feat: Move relation field properties onto collection (#2529)
• feat: Update corelog to 0.0.7 (#2547)

Fix

• fix: Add check to filter result for logical ops (#2573)
• fix: Allow update when updating non-indexed field (#2511)
• fix: Handle compound filters on related indexed fields (#2575)
• fix: Make all array kinds nillable (#2534)
• fix: Return correct results from one-many indexed filter (#2579)

Documentation

• docs: Add data definition document (#2544)

Refactoring

• refactor: Add NormalValue (#2404)
• refactor: Clean up client/request package (#2443)
• refactor: DB transactions context (#2513)
• refactor: Merge collection UpdateWith and DeleteWith (#2531)
• refactor: Replace logging package with corelog (#2406)
• refactor: Rewrite convertImmutable (#2445)
• refactor: Unify Field Kind and Schema properties (#2414)

Testing

• test: Add flag to skip network tests (#2495)

v0.10.0 Release

Changelog

DefraDB v0.10 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

To get a full outline of the changes, we invite you to review the official changelog below. This release does include a Breaking Change to existing v0.9.x databases. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.source.network/.

Features

• feat: Add JSON scalar (#2254)
• feat: Add case insensitive like operator (#2368)
• feat: Add composite indexes (#2226)
• feat: Add support for views with Lens transforms (#2311)
• feat: Allow setting null values on doc fields (#2273)
• feat: Generate OpenAPI command (#2235)
• feat: Model Col. SchemaVersions and migrations on Cols (#2286)
• feat: Multiple docs with nil value on unique-indexed field (#2276)
• feat: Replace FieldDescription.RelationType with IsPrimary (#2288)
• feat: Reverted order for indexed fields (#2335)
• feat: Rework GetCollection/SchemaByFoo funcs into single (#2319)

Fix

• fix: Add latest image tag for ghcr (#2340)
• fix: Add missing delta payload (#2306)
• fix: Add missing directive definitions (#2369)
• fix: Add validation to JSON fields (#2375)
• fix: Fix compound relational filters in aggregates (#2297)
• fix: Load root dir before loading config (#2266)
• fix: Make peers sync secondary index (#2390)
• fix: Make returned collections respect explicit transactions (#2385)
• fix: Mark docs as deleted when querying in delete mut (#2298)
• fix: Move field id off of schema (#2336)
• fix: Update GetCollections behaviour (#2378)

Refactoring

• refactor: Decouple net config (#2258)
• refactor: Generate field ids using a sequence (#2339)
• refactor: HTTP config (#2278)
• refactor: Make CollectionDescription.Name Option (#2223)
• refactor: Make config internal to CLI (#2310)
• refactor: Node config (#2296)
• refactor: Remove unused Delete field from client.Document (#2275)

v0.9.0 Release

DefraDB v0.9 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

To get a full outline of the changes, we invite you to review the official changelog below. This release does include a Breaking Change to existing v0.8.x databases. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.source.network/.

Changelog

Features

• feat: Add PN Counter CRDT type (#2119)
• feat: Add blob scalar type (#2091)
• feat: Add unique secondary index (#2131)
• feat: Allow users to add Views (#2114)
• feat: Mutation typed input (#2167)
• feat: New cmd for docs auto generation (#2096)

Fix

• fix: Add entropy to counter CRDT type updates (#2186)
• fix: Filtering on unique index if there is no match (#2177)
• fix: Handle multiple nil values on unique indexed fields (#2178)

Refactoring

• refactor: Add strong typing to document creation (#2161)
• refactor: Rename key,id,dockey to docID terminology (#1749)
• refactor: Simplify Merkle CRDT workflow (#2111)

Testing

• test: Add auto-doc generation (#2051)

v0.8.0 Release

DefraDB v0.8 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

To get a full outline of the changes, we invite you to review the official changelog below. This release does include a Breaking Change to existing v0.7.x databases. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.source.network/.

Changelog

Features

• feat: Add CCIP Support (#1896)
• feat: Add OpenAPI route (#1960)
• feat: Add Swagger UI to playground (#1979)
• feat: Add means to fetch schema (#2006)
• feat: Allow setting of default schema version (#1888)
• feat: Change GetCollectionBySchemaFoo funcs to return many (#1984)
• feat: Enable playground in Docker build (#1986)
• feat: Make queries utilise secondary indexes (#1925)
• feat: Remove CollectionDescription.Schema (#1965)
• feat: Remove collection from patch schema (#1957)
• feat: Rename Schema.SchemaID to Schema.Root (#2005)

Fix

• fix: Fix test module relying on closed memory leak (#2037)
• fix: Infinite loop when updating one-one relation (#1915)
• fix: Make return type for FieldKind_INT an int64 (#1982)
• fix: Node private key requires data directory (#1938)
• fix: Remove collection name from schema ID generation (#1920)

Refactoring

• refactor: Add peerstore to multistore (#1980)
• refactor: CLI client interface (#1839)
• refactor: CRDT merge direction (#2016)
• refactor: Deprecate CollectionDescription.Schema (#1939)
• refactor: P2P client interface (#1924)
• refactor: Remove net GRPC API (#1927)
• refactor: Reorganise collection description storage (#1988)

v0.7.0 Release

DefraDB v0.7 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

This release has focused on robustness, testing, and schema management. Some highlight new features include notable expansions to the expressiveness of schema migrations.

To get a full outline of the changes, we invite you to review the official changelog below. This release does include a Breaking Change to existing v0.6.x databases. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.source.network/.

Features

• Allow field indexing by name in PatchSchema (#1810)
• Auto-create relation id fields via PatchSchema (#1807)
• Support PatchSchema relational field kind substitution (#1777)
• Add support for adding of relational fields (#1766)
• Enable downgrading of documents via Lens inverses (#1721)

Fixes

• Correctly handle serialisation of nil field values (#1872)
• Compound filter operators with relations (#1855)
• Only update updated fields via update requests (#1817)
• Error when saving a deleted document (#1806)
• Prevent multiple docs from being linked in one one (#1790)
• Handle the querying of secondary relation id fields (#1768)
• Improve the way migrations handle transactions (#1737)

Tooling

• Add Akash deployment configuration (#1736)

Refactoring

• HTTP client interface (#1776)
• Simplify fetcher interface (#1746)

Testing

• Convert and move out of place explain tests (#1878)
• Update mutation tests to make use of mutation system (#1853)
• Test top level agg. with compound relational filter (#1870)
• Skip unsupported mutation types at test level (#1850)
• Extend mutation tests with col.Update and Create (#1838)
• Add tests for multiple one-one joins (#1793)

v0.6.0 Release

DefraDB v0.6 is a major pre-production release. Until the stable version 1.0 is reached, the SemVer minor patch number will denote notable releases, which will give the project freedom to experiment and explore potentially breaking changes.

There are several new and powerful features, important bug fixes, and notable refactors in this release. Some highlight features include: The initial release of our LensVM based schema migration engine powered by WebAssembly (#1650), newly embedded DefraDB Playround which includes a bundled GraphQL client and schema manager, and last but not least a relation field (_id) alias to improve the developer experience (#1609).

To get a full outline of the changes, we invite you to review the official changelog below. This release does include a Breaking Change to existing v0.5.x databases. If you need help migrating an existing deployment, reach out at hello@source.network or join our Discord at https://discord.source.network/.

Features

• Add _not operator (#1631)
• Schema list API (#1625)
• Add simple data import and export (#1630)
• Playground (#1575)
• Add schema migration get and set cmds to CLI (#1650)
• Allow relation alias on create and update (#1609)
• Make fetcher calculate docFetches and fieldFetches (#1713)
• Add lens migration engine to defra (#1564)
• Add _keys attribute to selectNode simple explain (#1546)
• CLI commands for secondary indexes (#1595)
• Add alias to groupBy related object (#1579)
• Non-unique secondary index (no querying) (#1450)
• Add ability to explain-debug all nodes (#1563)
• Include dockey in doc exists err (#1558)

Fixes

• Better wait in CLI integration test (#1415)
• Return error when relation is not defined on both types (#1647)
• Change core.DocumentMapping to pointer (#1528)
• Fix invalid (badger) datastore state (#1685)
• Discard index and subscription implicit transactions (#1715)
• Remove duplicated peers in peerstore prefix (#1678)
• Return errors from typeJoinOne (#1716)
• Document change detector breaking change (#1531)
• Standardise schema migration CLI errors (#1682)
• Introspection OrderArg returns null inputFields (#1633)
• Avoid duplicated requestable fields (#1621)
• Normalize int field kind (#1619)
• Change the WriteSyncer to use lock when piping (#1608)
• Filter splitting and rendering for related types (#1541)

Documentation

• Improve CLI command documentation (#1505)

Refactoring

• Schema list output to include schemaVersionID (#1706)
• Reuse lens wasm modules (#1641)
• Remove redundant txn param from fetcher start (#1635)
• Remove first CRDT byte from field encoded values (#1622)
• Merge node into net and improve coverage (#1593)
• Fetcher filter and field optimization (#1500)

Testing

• Rework transaction test framework capabilities (#1603)
• Expand backup integration tests (#1699)
• Disable test (#1675)
• Add tests for 1-1 group by id (#1655)
• Remove CLI tests from make test (#1643)
• Bundle test state into single var (#1645)
• Convert explain group tests to new explain setup (#1537)
• Add tests for foo_id field name clashes (#1521)
• Resume wait correctly following test node restart (#1515)
• Require no errors when none expected (#1509)

Continuous integration

• Add workflows to push, pull, and validate docker images (#1676)
• Build mocks using make (#1612)
• Fix terraform plan and merge AMI build + deploy workflow (#1514)
• Reconfigure CodeCov action to ensure stability (#1414)

Chore

• Bump to GoLang v1.20 (#1689)
• Update to ipfs boxo 0.10.0 (#1573)

v0.5.1 Release

DefraDB v0.5.1 is a minor release, primarily focusing on additional/extended features and fixes of items added in the v0.5.0 release.

Features

• Add collection response information on creation (#1499)
• CLI client request from file (#1503)
• Add commits fieldName and fieldId fields (#1451)
• Add allowed origins config (#1408)
• Add descriptions to all system defined GQL stuff (#1387)
• Strongly type Request.Errors (#1364)

Fixes

• Skip new test packages in change detector (#1495)
• Make nested joins work correctly from primary direction (#1491)
• Add reconnection to known peers (#1482)
• Rename commit field input arg to fieldId (#1460)
• Reference collectionID in p2p readme (#1466)
• Handling SIGTERM in CLI start command (#1459)
• Update QL documentation link and replicator command (#1440)
• Fix typo in readme (#1419)
• Limit the size of http request bodies that we handle (#1405)
• Improve P2P event handling (#1388)
• Serialize DB errors to json in http package (#1401)
• Do not commit if errors have been returned (#1390)
• Unlock replicator lock before returning error (#1369)
• Improve NonNull error message (#1362)
• Use ring-buffer for WaitForFoo chans (#1359)
• Guarantee event processing order (#1352)
• Explain of _group with dockeys filter to be []string (#1348)

Refactoring

• Use int32 for proper gql scalar Int parsing (#1493)
• Improve rollback on peer P2P collection error (#1461)
• Improve CLI with test suite and builder pattern (#928)

Testing

• Add DB/Node Restart tests (#1504)
• Provide tests for client introspection query (#1492)
• Convert explain count tests to new explain setup (#1488)
• Convert explain sum tests to new explain setup (#1489)
• Convert explain average tests to new explain setup (#1487)
• Convert explain top-level tests to new explain setup (#1480)
• Convert explain order tests to new explain setup (#1478)
• Convert explain join tests to new explain setup (#1476)
• Convert explain dagscan tests to new explain setup (#1474)
• Add tests to assert schema id order independence (#1456)
• Capitalize all integration schema types (#1445)
• Convert explain limit tests to new explain setup (#1446)
• Improve change detector performance (#1433)
• Convert mutation explain tests to new explain setup (#1416)
• Convert filter explain tests to new explain setup (#1380)
• Retry test doc mutation on transaction conflict (#1366)

Continuous integration

• Remove secret ssh key stuff from change detector wf (#1438)
• Fix the SSH security issue from AMI scan report (#1426)
• Add a separate workflow to run the linter (#1434)
• Allow CI to work from forked repo (#1392)
• Bump go version within packer for AWS AMI (#1344)

Chore

• Enshrine defra logger names (#1410)
• Remove some dead code (#1470)
• Update graphql-go (#1422)
• Improve logging consistency (#1424)
• Makefile tests with shorter timeout and common flags (#1397)
• Move to gofrs/uuid (#1396)
• Move to ipfs boxo (#1393)
• Document collection.txn (#1363)

dev-unstable-12-05-2023

THIS IS A UNSTABLE DEV BUILD. DO NOT USE UNLESS SPECIFICALLY DIRECTED TO.

Built using commit: 8aa317a
Built on date: 12-05-2023.

Unstable pre-release of upcoming 0.5.1

v0.5.0 Release

🎉 Introducing DefraDB v0.5: Open Source Release 🎉

We are excited to announce the release of DefraDB v0.5, a major pre-production milestone for our user-centric database that prioritizes data ownership, personal privacy, and information security. This release marks the first open-source release for DefraDB and includes significant quality of life improvements, bug fixes, and overall cleanliness of the repo, making it ready for public testing and use.

Key Features:

• First iteration of our schema update system, enabling developers to add new fields to schemas using our JSON Patch based DDL
• New DAG based delete system that persists "soft-delete" ops into the CRDT Merkle DAG
• Early prototype for collection-level peer-to-peer synchronization

Important Notice:

This release includes a Breaking Change for existing v0.4.x databases. If you need assistance migrating an existing deployment, please contact us at hello@source.network or join our Discord at https://discord.source.network/.

Explore the Release:

For a complete outline of the changes and updates, check out the official changelog above. We encourage you to try DefraDB v0.5 and provide feedback as we continue to develop this innovative solution for decentralized data and access-control management, user-centric apps, data trustworthiness, and more. Your input is invaluable in helping us shape the future of DefraDB and the Source technologies!

Get started with DefraDB v0.5 now!

You can find the full changelog below:

Features

• Add document delete mechanics (#1263)
• Ability to explain an executed request (#1188)
• Add SchemaPatch CLI command (#1250)
• Add support for one-one mutation from sec. side (#1247)
• Store only key in DAG instead of dockey path (#1245)
• Add collectionId field to commit field (#1235)
• Add field kind substitution for PatchSchema (#1223)
• Add dockey field for commit field (#1216)
• Allow new fields to be added locally to schema (#1139)
• Add like sub-string filter (#1091)
• Add ability for P2P to wait for pushlog by peer (#1098)
• Add P2P collection topic subscription (#1086)
• Add support for schema version id in queries (#1067)
• Add schema version id to commit queries (#1061)
• Persist schema version at time of commit (#1055)
• Add ability to input simple explain type arg (#1039)

Fixes

• API address parameter validation (#1311)
• Improve error message for NonNull GQL types (#1333)
• Handle panics in the rpc server (#1330)
• Handle returned error in select.go (#1329)
• Resolve handful of CLI issues (#1318)
• Only check for events queue on subscription request (#1326)
• Remove client Create/UpdateCollection (#1309)
• CLI to display specific command usage help (#1314)
• Fix P2P collection CLI commands (#1295)
• Dont double up badger file path (#1299)
• Update immutable package (#1290)
• Fix panic on success of Add/RemoveP2PCollections (#1297)
• Fix deadlock on memory-datastore Close (#1273)
• Determine if query is introspection query (#1255)
• Allow newly added fields to sync via p2p (#1226)
• Expose ExplainEnum in the GQL schema (#1204)
• Resolve aggregates' mapping with deep nested subtypes (#1175)
• Make sort stable and handle nil comparison (#1094)
• Change successful schema add status to 200 (#1106)
• Add delay in P2P test util execution (#1093)
• Ensure errors test don't hard expect folder name (#1072)
• Remove potential P2P deadlock (#1056)
• Rework the P2P integration tests (#989)
• Improve DAG sync with highly concurrent updates (#1031)

Documentation

• Update docs for the v0.5 release (#1320)
• Document client interfaces in client/db.go (#1305)
• Document client Description types (#1307)
• Improve security policy (#1240)
• Add security disclosure policy (#1194)
• Correct commits query example in readme (#1172)

Refactoring

• Improve p2p collection operations on peer (#1286)
• Migrate gql introspection tests to new framework (#1211)
• Reorganise client transaction related interfaces (#1180)
• Config-local viper, rootdir, and logger parsing (#1132)
• Migrate mutation-relation tests to new framework (#1109)
• Rework integration test framework (#1089)
• Generate gql types using col. desc (#1080)
• Extract config errors to dedicated file (#1107)
• Change terminology from query to request (#1054)
• Allow db keys to handle multiple schema versions (#1026)
• Extract query schema errors to dedicated file (#1037)
• Extract planner errors to dedicated file (#1034)
• Extract query parser errors to dedicated file (#1035)

Testing

• Remove test reference to DEFRA_ROOTDIR env var (#1328)
• Expand tests for Peer subscribe actions (#1287)
• Fix flaky TestCloseThroughContext test (#1265)
• Add gql introspection tests for patch schema (#1219)
• Explicitly state change detector split for test (#1228)
• Add test for successful one-one create mutation (#1215)
• Ensure that all databases are always closed on exit (#1187)
• Add P2P tests for Schema Update adding field (#1182)
• Migrate P2P/state tests to new framework (#1160)
• Remove sleep from subscription tests (#1156)
• Fetch documents on test execution start (#1163)
• Introduce basic testing for the version module (#1111)
• Boost test coverage for collection_update (#1050)
• Wait between P2P update retry attempts (#1052)
• Exclude auto-generated protobuf files from codecov (#1048)
• Add P2P tests for relational docs (#1042)

Continuous integration

• Add workflow that builds DefraDB AMI upon tag push (#1304)
• Allow PR title to end with a capital letter (#1291)
• Chan...

dev-unstable-21-03-2023

THIS IS A UNSTABLE DEV BUILD. DO NOT USE UNLESS SPECIFICALLY DIRECTED TO.

Built using commit: 8cf6855
Built on date: 21-03-2023.

Unstable pre-release of upcoming 0.5.

The following is a very short summary of the nature of this release:

• Several refactorings have been done to extract errors to dedicated files and to allow database keys to handle multiple schema versions.
• The DAG sync has been improved to handle highly concurrent updates and potential P2P deadlocks have been removed.
• New features have been added to the system, such as the ability to input a simple explain type argument, persist schema version at the time of commit, and add like sub-string filter.
• The system has been updated to include a new workflow that ensures dependencies build properly.
• There have been changes to the test suites, including adding P2P tests for relational docs, boosting test coverage for collection updates, and introducing basic testing for the version module.
• Several bug fixes have been implemented, including reworking P2P integration tests, changing terminology from query to request, and making sort stable and handling nil comparison.