From c461ac41c7771b074bf09708d6dc7f6f63905c99 Mon Sep 17 00:00:00 2001 From: Alex Rosengarten Date: Wed, 4 Mar 2020 17:32:22 -0800 Subject: [PATCH] Squashed commit of the following: commit 17a2ab1277abfbb4af6499e78f41aaf6c3ea59a6 Author: Alex Rosengarten Date: Wed Mar 4 16:30:45 2020 -0800 Fixed indentation commit 9094055ef7e774d50cc62cb4b8fa4a11241e19ae Author: Alex Rosengarten Date: Wed Mar 4 16:03:51 2020 -0800 revising from feedback commit d3508219e0f344e8b8ff0c926674c37d7b9c20e2 Merge: 9054c231 3acb7e7e Author: Alex Rosengarten Date: Wed Mar 4 15:56:55 2020 -0800 Merge branch 'master' of github.com:PolymerLabs/arcs into s2k-schemas commit 3acb7e7e493d5f76fa93c0a32a503df7e08a0c8d Author: Alex Rosengarten Date: Wed Mar 4 15:56:42 2020 -0800 XXS CL to add a hash field to the Schema proto (#4752) commit 9054c2319334fc4a7d93e5a713bba6e4ea03ad7a Merge: d15008eb 337b7b26 Author: Alex Rosengarten Date: Wed Mar 4 15:55:27 2020 -0800 Merge branch 'master' of github.com:PolymerLabs/arcs into s2k-schemas commit 337b7b26500e977e069979c66480e82a38a7e7fb Author: Alex Rosengarten Date: Wed Mar 4 15:55:06 2020 -0800 recipe2plan 2/n: Validate Handles, Tests (#4834) * creating structure of ts r2p script * Created fast CLI rapper for r2p script * WIP figuring out ways to resolve a recipe * fix sp * Passing lint, still WIP * Build macros for recipe2plan work * Lightweight iteration on recipe2plan, stubbed out tests * Revised method to find corresponding create handles * Added type info * Added method to get all handles to manifest + get all handles by Id * fix: no flatMap * - currently, fails since it cannot find associated stores for handles. * uncleaned, but working recipe resolution * Cleaned up for r2p, pt 1 * stubbed out second test (short, short) * Squashed commit of the following: commit 62dcc57fac16e26dd9c724d54be4a6c8ee87bc98 Author: Alex Rosengarten Date: Mon Mar 2 14:38:14 2020 -0800 Fixed test, added TODO * updates to test, can't get two to fail * Squashed commit of the following: commit bede8d95ea4c1d7c5115f9d34bd2476e0deb7fef Author: Alex Rosengarten Date: Mon Mar 2 14:52:02 2020 -0800 Fixed build rule, simplified runtime commit 62dcc57fac16e26dd9c724d54be4a6c8ee87bc98 Author: Alex Rosengarten Date: Mon Mar 2 14:38:14 2020 -0800 Fixed test, added TODO * can't get test to fail... hmm... * Squashed commit of the following: commit f6df54ae7d999c724800c969af59a3c29442995a Author: Alex Rosengarten Date: Mon Mar 2 15:44:32 2020 -0800 tools/sigh lint commit bede8d95ea4c1d7c5115f9d34bd2476e0deb7fef Author: Alex Rosengarten Date: Mon Mar 2 14:52:02 2020 -0800 Fixed build rule, simplified runtime commit 62dcc57fac16e26dd9c724d54be4a6c8ee87bc98 Author: Alex Rosengarten Date: Mon Mar 2 14:38:14 2020 -0800 Fixed test, added TODO * adjusting recipe copying to include triggers * Revert "adjusting recipe copying to include triggers" This reverts commit 9bc3734f * four tests complete * Added test, fixed minor async issue * Squashed commit of the following: commit 4b05a287434dcd3a3622ea9c9ebb499fe1d5bc46 Merge: f4c1d26a 99000c25 Author: Alex Rosengarten Date: Tue Mar 3 11:18:14 2020 -0800 Merge branch 'r2p' of github.com:alxrsngrtn/arcs into r2p commit f4c1d26a893e2d6635642567b6eec8d681bdc252 Merge: 140111f5 5c6310da Author: Alex Rosengarten Date: Tue Mar 3 11:14:35 2020 -0800 Merge branch 'master' of github.com:PolymerLabs/arcs into r2p Fixed lint errors commit 99000c25e79b55be6c4a09df73a3d1a34227e630 Merge: 140111f5 5c6310da Author: Alex Rosengarten Date: Tue Mar 3 11:14:35 2020 -0800 Merge branch 'master' of github.com:PolymerLabs/arcs into r2p Fixed lint errors commit 140111f580adf02c3fe0d2c3a4560dc13c9b2510 Author: Alex Rosengarten Date: Tue Mar 3 11:08:12 2020 -0800 Removed flatMap, added TODOs and link to GH issue commit 5c6310da9d1e372544e816c1dd595ef25e5099bf Author: jblebrun Date: Tue Mar 3 11:02:34 2020 -0800 Simplify service test pattern (#4817) * Make a simple test lifecycle registry, instead of creating empty testactivity * Remove use of `runBlockingTest`: according to https://github.com/Kotlin/kotlinx.coroutines/issues/1222, if the test results in coroutines being finished on other threads, it's possible to receive "This job has not yet completed" exceptions, even though the jobs were properly terminated. Since we don't need the delay-skipping properties of `runBlockingTest`, I think it's OK to use `runBlocking`. commit 472bc841296987e01feee21061781bba0f315921 Author: Alex Rosengarten Date: Tue Mar 3 10:27:56 2020 -0800 Improved build rules commit ca1ebf89390202da9e777f08f7e69f03020d3cf1 Author: Alex Rosengarten Date: Tue Mar 3 10:17:13 2020 -0800 Impl suggestsions for r2p commit b5578ea297e4d3033fdfdefd5f3a325e2cbe503f Author: Jason Feinstein Date: Tue Mar 3 09:54:17 2020 -0800 Add tests for dereferencing references to the HandleManagerTest(s) (#4816) * Add dereferencing tests to the android handle manager test. * Add dereferencing tests to core HandleManager. commit a6957979be506e3b7afdbf077aae4b06942b1bea Author: Jason Feinstein Date: Mon Mar 2 18:14:07 2020 -0800 Create RawEntityDereferencerTest, storage Reference-> CrdtEntity.Reference (#4812) * Create RawEntityDereferencerTest, make storage Reference implement CrdtEntity.Reference. Also: Create ParcelableReference. * Add dep. Also, apparently read/writeBoolean is Q-only. * Just write null if there is no version map. * Just write null if there is no version map. commit ba7a1074ffb487da00625dcfce536763da0b9395 Author: Gogul Balakrishnan Date: Mon Mar 2 17:51:35 2020 -0800 Add a decoder for PrimitiveTypeProto and an option to disable android constraints in BUILD. (#4793) commit f6df54ae7d999c724800c969af59a3c29442995a Author: Alex Rosengarten Date: Mon Mar 2 15:44:32 2020 -0800 tools/sigh lint commit bede8d95ea4c1d7c5115f9d34bd2476e0deb7fef Author: Alex Rosengarten Date: Mon Mar 2 14:52:02 2020 -0800 Fixed build rule, simplified runtime commit 62dcc57fac16e26dd9c724d54be4a6c8ee87bc98 Author: Alex Rosengarten Date: Mon Mar 2 14:38:14 2020 -0800 Fixed test, added TODO * fix ineq * existing tests passing * Updating invalid type test * Squashed commit of the following: commit b3d92d0f69f98571237f06e91ff480209f7aab9d Author: Alex Rosengarten Date: Wed Mar 4 12:17:17 2020 -0800 updated test name commit ba7884b62ccc7af616a9f7f1013475946f3cfb0b Author: Alex Rosengarten Date: Wed Mar 4 12:16:09 2020 -0800 nested unit tests commit ef36fdf11762e2a742f9d1dbd2542d63a2170d5a Author: Alex Rosengarten Date: Wed Mar 4 12:15:00 2020 -0800 renamed to tryResolve commit 5703f0278610f836a0b07f1c577cbcbe107f55ca Author: Alex Rosengarten Date: Wed Mar 4 12:07:18 2020 -0800 rm generator commit 658cb1ad9dd354f97115d8e8cd28a66dc6b032ab Author: Alex Rosengarten Date: Wed Mar 4 12:01:54 2020 -0800 implemented more review suggestions commit 0056ee308c6482aeed05051b9e83ef8954af6e72 Merge: cae89c3c 0f0f82a4 Author: Alex Rosengarten Date: Wed Mar 4 11:54:23 2020 -0800 Merge branch 'master' of github.com:PolymerLabs/arcs into r2p commit 0f0f82a439cd00b2ea451f487bc3f96ff63635f1 Author: Maria Mandlis Date: Wed Mar 4 09:33:29 2020 -0800 add queryable capability (#4794) commit d032c75e6996b213e13e5fe6fb42620c6678d71e Author: Maria Mandlis Date: Wed Mar 4 09:23:07 2020 -0800 add creatimeTimestamp to entities (kt) (#4823) commit 30267fc33bc9f74d12f2d7af6564a3f254fc6eaa Author: jblebrun Date: Wed Mar 4 09:09:39 2020 -0800 Remove some unused deps (#4831) commit c51918146dcced836a4a506edfeb3766bdb01a4a Author: Cameron Silvestrini Date: Wed Mar 4 17:14:07 2020 +1100 Increase size of DatabaseImplTest to medium (#4827) * Increase size of DatabaseImplTest to medium Was a bit flaky. * Check flakiness * Revert presubmit tweak commit fa621c63ee680cb91cace2bf2c310b7534eb4dc5 Author: jblebrun Date: Tue Mar 3 22:05:07 2020 -0800 Fix a race condition when setting up `ServiceStore` message channel (#4826) This seems to be the cause of the flakiness in `AndroidAllocatorTest` and `AndroidHandleManagerTest` (#4781) commit 78bf4f05edf7ee8e3162cc8b307be6057f6abedc Author: Gogul Balakrishnan Date: Tue Mar 3 18:00:49 2020 -0800 Utility to convert type proto to field type (if possible). (#4814) commit cae89c3c18f1bea2d9be8861271ab6de0ac0c0a6 Merge: 0360752f e5875390 Author: Alex Rosengarten Date: Tue Mar 3 15:50:34 2020 -0800 Merge branch 'master' of github.com:PolymerLabs/arcs into r2p commit e587539077b35ef73bbbc46d67118231262a3b65 Author: jblebrun Date: Tue Mar 3 15:07:16 2020 -0800 Apply simplified test pattern tests in `arcs.android.host` (#4821) As in 5c6310da commit 1af1aac716359bda5bdee4b4f4ab62abfe3ad5e9 Author: Joshua Pratt Date: Wed Mar 4 09:48:15 2020 +1100 Multinomials (#4804) * multinomial * added tests * making constant readable again * reindent * added multivariate rearrangement tests * updated comment * pr comments: Co-authored-by: Ragav Sachdeva commit 0360752fdd3e597c750fbaf3e035fa47c705289a Author: Alex Rosengarten Date: Tue Mar 3 11:42:31 2020 -0800 fix, bad comparison op commit 4b05a287434dcd3a3622ea9c9ebb499fe1d5bc46 Merge: f4c1d26a 99000c25 Author: Alex Rosengarten Date: Tue Mar 3 11:18:14 2020 -0800 Merge branch 'r2p' of github.com:alxrsngrtn/arcs into r2p commit f4c1d26a893e2d6635642567b6eec8d681bdc252 Merge: 140111f5 5c6310da Author: Alex Rosengarten Date: Tue Mar 3 11:14:35 2020 -0800 Merge branch 'master' of github.com:PolymerLabs/arcs into r2p Fixed lint errors commit 99000c25e79b55be6c4a09df73a3d1a34227e630 Merge: 140111f5 5c6310da Author: Alex Rosengarten Date: Tue Mar 3 11:14:35 2020 -0800 Merge branch 'master' of github.com:PolymerLabs/arcs into r2p Fixed lint errors commit 140111f580adf02c3fe0d2c3a4560dc13c9b2510 Author: Alex Rosengarten Date: Tue Mar 3 11:08:12 2020 -0800 Removed flatMap, added TODOs and link to GH issue commit 5c6310da9d1e372544e816c1dd595ef25e5099bf Author: jblebrun Date: Tue Mar 3 11:02:34 2020 -0800 Simplify service test pattern (#4817) * Make a simple test lifecycle registry, instead of creating empty testactivity * Remove use of `runBlockingTest`: according to https://github.com/Kotlin/kotlinx.coroutines/issues/1222, if the test results in coroutines being finished on other threads, it's possible to receive "This job has not yet completed" exceptions, even though the jobs were properly terminated. Since we don't need the delay-skipping properties of `runBlockingTest`, I think it's OK to use `runBlocking`. commit 472bc841296987e01feee21061781bba0f315921 Author: Alex Rosengarten Date: Tue Mar 3 10:27:56 2020 -0800 Improved build rules commit ca1ebf89390202da9e777f08f7e69f03020d3cf1 Author: Alex Rosengarten Date: Tue Mar 3 10:17:13 2020 -0800 Impl suggestsions for r2p commit b5578ea297e4d3033fdfdefd5f3a325e2cbe503f Author: Jason Feinstein Date: Tue Mar 3 09:54:17 2020 -0800 Add tests for dereferencing references to the HandleManagerTest(s) (#4816) * Add dereferencing tests to the android handle manager test. * Add dereferencing tests to core HandleManager. commit a6957979be506e3b7afdbf077aae4b06942b1bea Author: Jason Feinstein Date: Mon Mar 2 18:14:07 2020 -0800 Create RawEntityDereferencerTest, storage Reference-> CrdtEntity.Reference (#4812) * Create RawEntityDereferencerTest, make storage Reference implement CrdtEntity.Reference. Also: Create ParcelableReference. * Add dep. Also, apparently read/writeBoolean is Q-only. * Just write null if there is no version map. * Just write null if there is no version map. commit ba7a1074ffb487da00625dcfce536763da0b9395 Author: Gogul Balakrishnan Date: Mon Mar 2 17:51:35 2020 -0800 Add a decoder for PrimitiveTypeProto and an option to disable android constraints in BUILD. (#4793) commit f6df54ae7d999c724800c969af59a3c29442995a Author: Alex Rosengarten Date: Mon Mar 2 15:44:32 2020 -0800 tools/sigh lint commit bede8d95ea4c1d7c5115f9d34bd2476e0deb7fef Author: Alex Rosengarten Date: Mon Mar 2 14:52:02 2020 -0800 Fixed build rule, simplified runtime commit 62dcc57fac16e26dd9c724d54be4a6c8ee87bc98 Author: Alex Rosengarten Date: Mon Mar 2 14:38:14 2020 -0800 Fixed test, added TODO * fixed nested tests * rewording, reorganizing * assert more ergonomic commit 298fdccfac759798efef358bf840eef8f010e03a Author: Maria Mandlis Date: Wed Mar 4 15:29:37 2020 -0800 update wasm readme (#4832) commit d15008eb184f62ec559a3f4309e0fb490694078e Author: Alex Rosengarten Date: Wed Mar 4 15:11:14 2020 -0800 fixed build commit bbfc3d2ece4f53f07c60d68efab779c2946baf2e Author: Alex Rosengarten Date: Wed Mar 4 15:04:11 2020 -0800 EntitySpec Interface can return a schema commit 3b75c3411042448f9eff919da2f6066b40e1779a Author: Alex Rosengarten Date: Wed Mar 4 14:09:36 2020 -0800 ktlint fix commit 506457e1d3a091af956ef551432352d5b23ffbc1 Author: Alex Rosengarten Date: Wed Mar 4 13:59:46 2020 -0800 addField arguments are more ergonomic commit 579b2e620c806efb083d19b38b8906631e17d604 Author: Alex Rosengarten Date: Wed Mar 4 13:45:28 2020 -0800 grooming schema registry commit 3cb8023529ef4dcd9478d8159062c17bd9c0d126 Author: Alex Rosengarten Date: Wed Mar 4 11:39:30 2020 -0800 rm unused field commit 1cff99f726f69b49a11b2939e118d446371b20a8 Author: Alex Rosengarten Date: Wed Mar 4 11:38:32 2020 -0800 rm import commit 855401ce852d46bb0287f511740d2c8b4a088906 Author: Alex Rosengarten Date: Wed Mar 4 11:31:14 2020 -0800 no difference w/ wasm golden commit 552e9ad9fc7488f37f26f9825037a3109331e36a Author: Alex Rosengarten Date: Wed Mar 4 11:26:43 2020 -0800 Added test for mapOf commit 14309076e2a5ec45eb3433f486ea84135af57322 Author: Alex Rosengarten Date: Wed Mar 4 11:07:15 2020 -0800 sigh lint commit 9c22195065e71ab0cbadbf981bbc28d9e90f47f4 Author: Alex Rosengarten Date: Wed Mar 4 11:06:50 2020 -0800 minor adjustments commit 74ae7707dcf0615f2ae212802c5cf50498cfa61f Author: Alex Rosengarten Date: Wed Mar 4 10:58:20 2020 -0800 Improved shape of registry commit f69f99b37f791c0173106aab5410b71b3157032d Author: Alex Rosengarten Date: Wed Mar 4 10:32:31 2020 -0800 excluding wasm from generation commit dcefb1658667d596a29b5002e1228366bfe9a388 Author: Alex Rosengarten Date: Wed Mar 4 10:13:46 2020 -0800 Generation adds to schema registry commit f3b9ae5a3706278818c2a712da83d74677c5512c Author: Alex Rosengarten Date: Wed Mar 4 07:57:15 2020 -0800 packaging in a schema registry commit 2ce6da20abf56c5d3d195af3bb885476898ffdb6 Author: Alex Rosengarten Date: Tue Mar 3 22:30:39 2020 -0800 First iteration of schema gen on entities --- java/arcs/core/data/SchemaRegistry.kt | 28 ++++ java/arcs/core/data/proto/recipe.proto | 1 + java/arcs/core/storage/api/BUILD | 2 +- java/arcs/core/storage/api/Entity.kt | 4 + src/tools/schema2base.ts | 20 ++- src/tools/schema2cpp.ts | 4 +- src/tools/schema2kotlin.ts | 76 +++++++++-- src/tools/storage-key-recipe-resolver.ts | 18 +-- .../tests/goldens/generated-schemas.jvm.kt | 44 +++++- src/tools/tests/schema2kotlin-test.ts | 41 ++++++ src/tools/tests/schema2wasm-test.ts | 5 +- .../tests/storage-key-recipe-resolver-test.ts | 126 ++++++++++++++++-- src/wasm/cpp/README.md | 2 +- 13 files changed, 330 insertions(+), 41 deletions(-) create mode 100644 java/arcs/core/data/SchemaRegistry.kt create mode 100644 src/tools/tests/schema2kotlin-test.ts diff --git a/java/arcs/core/data/SchemaRegistry.kt b/java/arcs/core/data/SchemaRegistry.kt new file mode 100644 index 00000000000..298f67b4d00 --- /dev/null +++ b/java/arcs/core/data/SchemaRegistry.kt @@ -0,0 +1,28 @@ +/* + * Copyright 2020 Google LLC. + * + * This code may only be used under the BSD style license found at + * http://polymer.github.io/LICENSE.txt + * + * Code distributed by Google as part of this project is also subject to an additional IP rights + * grant found at + * http://polymer.github.io/PATENTS.txt + */ +package arcs.core.data + +typealias SchemaHash = String + +/** + * A registry for generated [Schema]s. + */ +object SchemaRegistry { + private val schemas = mutableMapOf() + + /** Store a [Schema] in the registry. */ + fun register(schema: Schema) { + schemas[schema.hash] = schema + } + + /** Given a [SchemaHash], return the [Schema] for that hash, if it exists. */ + operator fun get(hash: SchemaHash) = schemas[hash] +} diff --git a/java/arcs/core/data/proto/recipe.proto b/java/arcs/core/data/proto/recipe.proto index b4ff66a211c..200b2938493 100644 --- a/java/arcs/core/data/proto/recipe.proto +++ b/java/arcs/core/data/proto/recipe.proto @@ -121,6 +121,7 @@ message TypeVariableProto { message SchemaProto { repeated string names = 1; map fields = 2; + string hash = 3; } enum OPERATOR { diff --git a/java/arcs/core/storage/api/BUILD b/java/arcs/core/storage/api/BUILD index 77b44a15923..361d7552992 100644 --- a/java/arcs/core/storage/api/BUILD +++ b/java/arcs/core/storage/api/BUILD @@ -12,7 +12,7 @@ arcs_kt_library( srcs = glob(["*.kt"]), deps = [ "//java/arcs/core/common", - "//java/arcs/core/data:rawentity", + "//java/arcs/core/data", "//java/arcs/core/data/util:data-util", "//third_party/kotlin/kotlinx_coroutines", ], diff --git a/java/arcs/core/storage/api/Entity.kt b/java/arcs/core/storage/api/Entity.kt index 5d0b4e3dac1..9f6472cfc1b 100644 --- a/java/arcs/core/storage/api/Entity.kt +++ b/java/arcs/core/storage/api/Entity.kt @@ -13,6 +13,7 @@ package arcs.core.storage.api import arcs.core.common.Referencable import arcs.core.data.RawEntity +import arcs.core.data.Schema import arcs.core.data.util.ReferencablePrimitive import kotlin.IllegalArgumentException import kotlin.reflect.KClass @@ -38,6 +39,9 @@ interface EntitySpec { * TODO: replace this with kotlinx.serialization */ fun deserialize(data: RawEntity): T + + /** Returns the corresponding [Schema] for the specified [Entity]. */ + fun schema(): Schema } /** diff --git a/src/tools/schema2base.ts b/src/tools/schema2base.ts index 7687c6824c6..94bd25aac52 100644 --- a/src/tools/schema2base.ts +++ b/src/tools/schema2base.ts @@ -15,8 +15,16 @@ import {Runtime} from '../runtime/runtime.js'; import {SchemaGraph, SchemaNode} from './schema2graph.js'; import {ParticleSpec} from '../runtime/particle-spec.js'; +export type AddFieldOptions = Readonly<{ + field: string; + typeChar: string; + isOptional?: boolean; + refClassName?: string; + isCollection?: boolean; +}>; + export interface ClassGenerator { - addField(field: string, typeChar: string, isOptional: boolean, refClassName: string|null): void; + addField(opts: AddFieldOptions): void; generate(schemaHash: string, fieldCount: number): string; } @@ -78,14 +86,20 @@ export abstract class Schema2Base { for (const [field, descriptor] of fields) { if (descriptor.kind === 'schema-primitive') { if (['Text', 'URL', 'Number', 'Boolean'].includes(descriptor.type)) { - generator.addField(field, descriptor.type[0], false, null); + generator.addField({field, typeChar: descriptor.type[0]}); } else { throw new Error(`Schema type '${descriptor.type}' for field '${field}' is not supported`); } } else if (descriptor.kind === 'schema-reference') { - generator.addField(field, 'R', false, node.refs.get(field).name); + generator.addField({field, typeChar: 'R', refClassName: node.refs.get(field).name}); } else if (descriptor.kind === 'schema-collection' && descriptor.schema.kind === 'schema-reference') { // TODO: support collections of references + } else if (descriptor.kind === 'schema-collection') { + const schema = descriptor.schema; + if (!['Text', 'URL', 'Number', 'Boolean'].includes(schema.type)) { + throw new Error(`Schema type '${schema.type}' for field '${field}' is not supported`); + } + generator.addField({field, typeChar: schema.type[0], isCollection: true}); } else { throw new Error(`Schema kind '${descriptor.kind}' for field '${field}' is not supported`); } diff --git a/src/tools/schema2cpp.ts b/src/tools/schema2cpp.ts index 1037b67948a..31853f5ef5d 100644 --- a/src/tools/schema2cpp.ts +++ b/src/tools/schema2cpp.ts @@ -7,7 +7,7 @@ * subject to an additional IP rights grant found at * http://polymer.github.io/PATENTS.txt */ -import {Schema2Base, ClassGenerator} from './schema2base.js'; +import {Schema2Base, ClassGenerator, AddFieldOptions} from './schema2base.js'; import {SchemaNode} from './schema2graph.js'; import {ParticleSpec} from '../runtime/particle-spec.js'; import {Type} from '../runtime/type.js'; @@ -113,7 +113,7 @@ class CppGenerator implements ClassGenerator { constructor(readonly node: SchemaNode, readonly namespace: string) {} - addField(field: string, typeChar: string, isOptional: boolean, refClassName: string|null) { + addField({field, typeChar, refClassName, isOptional = false, isCollection = false}: AddFieldOptions) { const fixed = fixName(field); const valid = `${field}_valid_`; let {type, defaultVal, isString} = typeMap[typeChar]; diff --git a/src/tools/schema2kotlin.ts b/src/tools/schema2kotlin.ts index b0eeb92c654..60ab7b67a12 100644 --- a/src/tools/schema2kotlin.ts +++ b/src/tools/schema2kotlin.ts @@ -7,7 +7,7 @@ * subject to an additional IP rights grant found at * http://polymer.github.io/PATENTS.txt */ -import {Schema2Base, ClassGenerator} from './schema2base.js'; +import {Schema2Base, ClassGenerator, AddFieldOptions} from './schema2base.js'; import {SchemaNode} from './schema2graph.js'; import {ParticleSpec} from '../runtime/particle-spec.js'; import minimist from 'minimist'; @@ -28,10 +28,10 @@ const keywords = [ ]; const typeMap = { - 'T': {type: 'String', decodeFn: 'decodeText()', defaultVal: `""`}, - 'U': {type: 'String', decodeFn: 'decodeText()', defaultVal: `""`}, - 'N': {type: 'Double', decodeFn: 'decodeNum()', defaultVal: '0.0'}, - 'B': {type: 'Boolean', decodeFn: 'decodeBool()', defaultVal: 'false'}, + 'T': {type: 'String', decodeFn: 'decodeText()', defaultVal: `""`, schemaType: 'FieldType.Text'}, + 'U': {type: 'String', decodeFn: 'decodeText()', defaultVal: `""`, schemaType: 'FieldType.Text'}, + 'N': {type: 'Double', decodeFn: 'decodeNum()', defaultVal: '0.0', schemaType: 'FieldType.Number'}, + 'B': {type: 'Boolean', decodeFn: 'decodeBool()', defaultVal: 'false', schemaType: 'FieldType.Boolean'}, }; export class Schema2Kotlin extends Schema2Base { @@ -54,7 +54,14 @@ package ${this.scope} // Current implementation doesn't support references or optional field detection import arcs.sdk.* -${this.opts.wasm ? 'import arcs.sdk.wasm.*' : 'import arcs.core.storage.api.toPrimitiveValue\nimport arcs.core.data.RawEntity\nimport arcs.core.data.util.toReferencable\nimport arcs.core.data.util.ReferencablePrimitive'} +${this.opts.wasm ? + `import arcs.sdk.wasm.*` : + `\ +import arcs.sdk.Entity +import arcs.core.data.* +import arcs.core.data.util.toReferencable +import arcs.core.data.util.ReferencablePrimitive +import arcs.core.storage.api.toPrimitiveValue`} `; } @@ -130,7 +137,7 @@ abstract class Abstract${particleName} : ${this.opts.wasm ? 'WasmParticleImpl' : } } -class KotlinGenerator implements ClassGenerator { +export class KotlinGenerator implements ClassGenerator { fields: string[] = []; fieldVals: string[] = []; setFields: string[] = []; @@ -144,11 +151,13 @@ class KotlinGenerator implements ClassGenerator { fieldSerializes: string[] = []; fieldDeserializes: string[] = []; fieldsForToString: string[] = []; + singletonSchemaFields: string[] = []; + collectionSchemaFields: string[] = []; constructor(readonly node: SchemaNode, private readonly opts: minimist.ParsedArgs) {} // TODO: allow optional fields in kotlin - addField(field: string, typeChar: string, isOptional: boolean, refClassName: string|null) { + addField({field, typeChar, refClassName, isOptional = false, isCollection = false}: AddFieldOptions) { // TODO: support reference types in kotlin if (typeChar === 'R') return; @@ -181,6 +190,46 @@ class KotlinGenerator implements ClassGenerator { this.fieldSerializes.push(`"${field}" to ${fixed}.toReferencable()`); this.fieldDeserializes.push(`${fixed} = data.singletons["${fixed}"].toPrimitiveValue(${type}::class, ${defaultVal})`); this.fieldsForToString.push(`${fixed} = $${fixed}`); + if (isCollection) { + this.collectionSchemaFields.push(`"${field}" to ${typeMap[typeChar].schemaType}`); + } else { + this.singletonSchemaFields.push(`"${field}" to ${typeMap[typeChar].schemaType}`); + } + } + + mapOf(items: string[]): string { + switch (items.length) { + case 0: + return `emptyMap()`; + case 1: + return `mapOf(${items[0]})`; + default: + return `\ +mapOf( +${this.leftPad(items.join(',\n'), 4)} +)`; + } + + } + + createSchema(schemaHash: string): string { + const schemaNames = this.node.schema.names.map(n => `SchemaName("${n}")`); + return `\ +Schema( + listOf(${schemaNames.join(',\n' + ' '.repeat(8))}), + SchemaFields( + singletons = ${this.leftPad(this.mapOf(this.singletonSchemaFields), 8, true)}, + collections = ${this.leftPad(this.mapOf(this.collectionSchemaFields), 8, true)} + ), + "${schemaHash}" +)`; + } + + leftPad(input: string, indent: number, skipFirst: boolean = false) { + return input + .split('\n') + .map((line: string, idx: number) => (idx === 0 && skipFirst) ? line : ' '.repeat(indent) + line) + .join('\n'); } generate(schemaHash: string, fieldCount: number): string { @@ -256,7 +305,18 @@ ${this.opts.wasm ? ` } class ${name}_Spec() : ${this.getType('EntitySpec')}<${name}> { +${this.opts.wasm ? '' : `\ + companion object { + val schema = ${this.leftPad(this.createSchema(schemaHash), 8, true)} + + init { + SchemaRegistry.register(schema) + } + } + + override fun schema() = schema +`} override fun create() = ${name}() ${!this.opts.wasm ? ` override fun deserialize(data: RawEntity): ${name} { diff --git a/src/tools/storage-key-recipe-resolver.ts b/src/tools/storage-key-recipe-resolver.ts index 7032e274102..0ee9b8957b1 100644 --- a/src/tools/storage-key-recipe-resolver.ts +++ b/src/tools/storage-key-recipe-resolver.ts @@ -32,13 +32,14 @@ export class StorageKeyRecipeResolver { /** * Produces resolved recipes with storage keys. * - * TODO(alxr): Apply to long-running recipes appropriately. + * TODO(#4818) Add passes to assign storage keys. * @throws Error if recipe fails to resolve on first or second pass. * @yields Resolved recipes with storage keys */ async resolve(): Promise { const recipes = []; for (const recipe of this.runtime.context.allRecipes) { + this.validateHandles(recipe); const arc = this.runtime.newArc(this.getArcId(recipe), ramDiskStorageKeyPrefixForTest()); const opts = {errors: new Map()}; const resolved = await this.tryResolve(recipe, arc, opts); @@ -63,7 +64,8 @@ export class StorageKeyRecipeResolver { */ async tryResolve(recipe: Recipe, arc: Arc, opts?: IsValidOptions): Promise { const normalized = recipe.clone(); - normalized.normalize(); + const successful = normalized.normalize(opts); + if (!successful) return null; if (normalized.isResolved()) return normalized; return await (new RecipeResolver(arc).resolve(recipe, opts)); @@ -97,14 +99,14 @@ export class StorageKeyRecipeResolver { } /** - * TODO(#4818) method to match `map` and `copy` fated handles with storage keys from `create` handles. + * Checks that handles are existent, disambiguous, and initiated by a long-running arc. * - * @throws when a mapped handle is associated with too many stores (ambiguous mapping). - * @throws when a mapped handle isn't associated with any store (no matches found). - * @throws when handle is mapped to a handle from an ephemeral recipe. + * @throws when a map or copy handle is associated with too many stores (ambiguous mapping). + * @throws when a map or copy handle isn't associated with any store (no matches found). + * @throws when a map or copy handle is associated with a handle from an ephemeral recipe. * @param recipe long-running or ephemeral recipe */ - matchKeysToHandles(recipe: Recipe) { + validateHandles(recipe: Recipe) { recipe.handles .filter(h => h.fate === 'map' || h.fate === 'copy') .forEach(handle => { @@ -121,8 +123,6 @@ export class StorageKeyRecipeResolver { if (!match.recipe.isLongRunning) { throw Error(`Handle ${handle.localName} mapped to ephemeral handle ${match.localName}.`); } - - handle.storageKey = match.storageKey; }); } } diff --git a/src/tools/tests/goldens/generated-schemas.jvm.kt b/src/tools/tests/goldens/generated-schemas.jvm.kt index e6c1626263f..0a36648cd14 100644 --- a/src/tools/tests/goldens/generated-schemas.jvm.kt +++ b/src/tools/tests/goldens/generated-schemas.jvm.kt @@ -9,10 +9,11 @@ package arcs.sdk // Current implementation doesn't support references or optional field detection import arcs.sdk.* -import arcs.core.storage.api.toPrimitiveValue -import arcs.core.data.RawEntity +import arcs.sdk.Entity +import arcs.core.data.* import arcs.core.data.util.toReferencable import arcs.core.data.util.ReferencablePrimitive +import arcs.core.storage.api.toPrimitiveValue class GoldInternal1() : Entity { @@ -71,6 +72,23 @@ class GoldInternal1() : Entity { class GoldInternal1_Spec() : EntitySpec { + companion object { + val schema = Schema( + listOf(), + SchemaFields( + singletons = mapOf("val" to FieldType.Text), + collections = emptyMap() + ), + "485712110d89359a3e539dac987329cd2649d889" + ) + + init { + SchemaRegistry.register(schema) + } + } + + override fun schema() = schema + override fun create() = GoldInternal1() override fun deserialize(data: RawEntity): GoldInternal1 { @@ -179,6 +197,28 @@ class Gold_Data() : Entity { class Gold_Data_Spec() : EntitySpec { + companion object { + val schema = Schema( + listOf(), + SchemaFields( + singletons = mapOf( + "num" to FieldType.Number, + "txt" to FieldType.Text, + "lnk" to FieldType.Text, + "flg" to FieldType.Boolean + ), + collections = emptyMap() + ), + "d8058d336e472da47b289eafb39733f77eadb111" + ) + + init { + SchemaRegistry.register(schema) + } + } + + override fun schema() = schema + override fun create() = Gold_Data() override fun deserialize(data: RawEntity): Gold_Data { diff --git a/src/tools/tests/schema2kotlin-test.ts b/src/tools/tests/schema2kotlin-test.ts new file mode 100644 index 00000000000..fd940158667 --- /dev/null +++ b/src/tools/tests/schema2kotlin-test.ts @@ -0,0 +1,41 @@ +/** + * @license + * Copyright (c) 2020 Google Inc. All rights reserved. + * This code may only be used under the BSD style license found at + * http://polymer.github.io/LICENSE.txt + * Code distributed by Google as part of this project is also + * subject to an additional IP rights grant found at + * http://polymer.github.io/PATENTS.txt + */ + + +import {assert} from '../../platform/chai-node.js'; +import {KotlinGenerator} from '../schema2kotlin.js'; +import {SchemaNode} from '../schema2graph.js'; +import {Schema} from '../../runtime/schema.js'; + + +describe('schema2wasm', () => { + describe('kotlin-generator', () => { + const ktGen = new KotlinGenerator(new SchemaNode(new Schema([], {}), 'dummyNode'), {arg: '', _: []}); + it('when no items are present, it creates an empty map', () => { + const actual = ktGen.mapOf([]); + + assert.strictEqual('emptyMap()', actual); + }); + it('when one item is present, it creates a single-line map', () => { + const actual = ktGen.mapOf([`"a" to "b"`]); + + assert.strictEqual('mapOf("a" to "b")', actual); + }); + it('when multiple items are present, it creates a multi-line map', () => { + const actual = ktGen.mapOf([`"a" to "b"`, `"b" to "c"`]); + + assert.strictEqual(`\ +mapOf( + "a" to "b", + "b" to "c" +)`, actual); + }); + }); +}); diff --git a/src/tools/tests/schema2wasm-test.ts b/src/tools/tests/schema2wasm-test.ts index e0dca1fde55..757bbca5df5 100644 --- a/src/tools/tests/schema2wasm-test.ts +++ b/src/tools/tests/schema2wasm-test.ts @@ -9,9 +9,8 @@ */ import {assert} from '../../platform/chai-web.js'; import {Manifest} from '../../runtime/manifest.js'; -import {Schema} from '../../runtime/schema.js'; import {Dictionary} from '../../runtime/hot.js'; -import {Schema2Base, ClassGenerator} from '../schema2base.js'; +import {Schema2Base, ClassGenerator, AddFieldOptions} from '../schema2base.js'; import {SchemaNode} from '../schema2graph.js'; import {Schema2Cpp} from '../schema2cpp.js'; import {Schema2Kotlin} from '../schema2kotlin.js'; @@ -32,7 +31,7 @@ class Schema2Mock extends Schema2Base { const collector = {count: 0, adds: []}; this.res[node.name] = collector; return { - addField(field: string, typeChar: string, isOptional: boolean, refClassName: string|null) { + addField({field, typeChar, isOptional, refClassName}: AddFieldOptions) { const refInfo = refClassName ? `<${refClassName}>` : ''; collector.adds.push(field + ':' + typeChar + refInfo + (isOptional ? '?' : '')); }, diff --git a/src/tools/tests/storage-key-recipe-resolver-test.ts b/src/tools/tests/storage-key-recipe-resolver-test.ts index 83fe5083db2..7468696d27d 100644 --- a/src/tools/tests/storage-key-recipe-resolver-test.ts +++ b/src/tools/tests/storage-key-recipe-resolver-test.ts @@ -11,10 +11,11 @@ import {Manifest} from '../../runtime/manifest.js'; import {assert} from '../../platform/chai-node.js'; import {StorageKeyRecipeResolver} from '../storage-key-recipe-resolver.js'; +import {assertThrowsAsync} from '../../testing/test-util.js'; describe('recipe2plan', () => { describe('storage-key-recipe-resolver', () => { - it('Resolves mapping a handle from a long running arc into another long running arc', async () => { + it('resolves mapping a handle from a long running arc into another long running arc', async () => { const manifest = await Manifest.parse(`\ particle Reader data: reads Thing {name: Text} @@ -28,7 +29,7 @@ describe('recipe2plan', () => { @trigger launch startup - arcId myArcId + arcId writeArcId recipe WritingRecipe thing: create persistent 'my-handle-id' Writer @@ -36,25 +37,126 @@ describe('recipe2plan', () => { @trigger launch startup - arcId otherArcId + arcId readArcId recipe ReadingRecipe data: map 'my-handle-id' Reader data: reads data`); const resolver = new StorageKeyRecipeResolver(manifest); - for (const it of (await resolver.resolve())) { + for (const it of (await resolver.resolve())) { assert.isTrue(it.isResolved()); } }); + it('fails to resolve mapping a handle from a short running arc into another short running arc', async () => { + const manifest = await Manifest.parse(`\ + particle Reader + data: reads Thing {name: Text} + + particle Writer + data: writes Thing {name: Text} + + recipe WritingRecipe + thing: create persistent 'my-handle-id' + Writer + data: writes thing + + recipe ReadingRecipe + data: map 'my-handle-id' + Reader + data: reads data`); + + const resolver = new StorageKeyRecipeResolver(manifest); + await assertThrowsAsync(async () => await resolver.resolve(), Error, 'Handle data mapped to ephemeral handle thing.'); + }); + it('fails to resolve mapping a handle from a short running arc into a long running arc', async () => { + const manifest = await Manifest.parse(`\ + particle Reader + data: reads Thing {name: Text} + + particle Writer + data: writes Thing {name: Text} + + recipe WritingRecipe + thing: create persistent 'my-handle-id' + Writer + data: writes thing + + @trigger + launch startup + arcId readArcId + recipe ReadingRecipe + data: map 'my-handle-id' + Reader + data: reads data`); + + const resolver = new StorageKeyRecipeResolver(manifest); + await assertThrowsAsync(async () => await resolver.resolve(), Error, 'Handle data mapped to ephemeral handle thing.'); + }); + it('resolves mapping a handle from a long running arc into a short running arc', async () => { + const manifest = await Manifest.parse(`\ + particle Reader + data: reads Thing {name: Text} + + particle Writer + data: writes Thing {name: Text} + + @trigger + launch startup + arcId writeArcId + recipe WritingRecipe + thing: create persistent 'my-handle-id' + Writer + data: writes thing + + recipe ReadingRecipe + data: map 'my-handle-id' + Reader + data: reads data`); + + const resolver = new StorageKeyRecipeResolver(manifest); + for (const it of await resolver.resolve()) { + assert.isTrue(it.isResolved()); + } + }); + it('Invalid Type: If Reader reads {name: Text, age: Number} it is not valid', async () => { + const manifest = await Manifest.parse(`\ + particle Reader + data: reads Thing {name: Text, age: Number} + + particle Writer + data: writes Thing {name: Text} + + @trigger + launch startup + arcId writeArcId + recipe WritingRecipe + thing: create persistent 'my-handle-id' + Writer + data: writes thing + + @trigger + launch startup + arcId readArcId + recipe ReadingRecipe + data: map 'my-handle-id' + Reader + data: reads data`); + + const resolver = new StorageKeyRecipeResolver(manifest); + // TODO: specify the correct error to be thrown + await assertThrowsAsync(resolver.resolve); + }); // TODO(alxr): Flush out outlined unit tests - it.skip('Short + Short: If WritingRecipe is short lived, it is not valid', () => {}); - it.skip('Short + Long: If WritingRecipe is short lived and Reading is long lived, it is not valid', () => {}); - it.skip('Invalid Type: If Reader reads {name: Text, age: Number} it is not valid', () => {}); - it.skip('No arc id: If arcId of WritingRecipe is not there, it is not valid', () => {}); - it.skip('No handleId: If id of handle in WritingRecipe is not provided, it is not valid', () => {}); - it.skip('Ambiguous handle: If there are 2 WritingRecipes creating the same handle, it is not valid', () => {}); - it.skip('Ambiguous handle + tag disambiguation: If there are 2 WritingRecipes creating the same handle but with different tags and mapping uses one of the tags, it is valid', () => {}); - it.skip('No Handle: If there is no writing handle, it is not valid', () => {}); + it.skip('No arc id: If arcId of WritingRecipe is not there, it is not valid', () => { + }); + it.skip('No handleId: If id of handle in WritingRecipe is not provided, it is not valid', () => { + }); + it.skip('Ambiguous handle: If there are 2 WritingRecipes creating the same handle, it is not valid', () => { + }); + it.skip('Ambiguous handle + tag disambiguation: If there are 2 WritingRecipes creating the same handle but with different tags and mapping uses one of the tags, it is valid', () => { + }); + it.skip('No Handle: If there is no writing handle, it is not valid', () => { + }); }); }); diff --git a/src/wasm/cpp/README.md b/src/wasm/cpp/README.md index 716fc935c1e..d0c17fca5c8 100644 --- a/src/wasm/cpp/README.md +++ b/src/wasm/cpp/README.md @@ -54,4 +54,4 @@ See [here](../../../particles/Native/Wasm) for a working example. # Test -`./tools/bazelisk test //src/wasm:wasm-api-test` +`./tools/bazelisk test //src/wasm/tests:wasm-api-test`