Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add blocking trigger type #4395

Merged
merged 36 commits into from
Apr 12, 2022
Merged
Show file tree
Hide file tree
Changes from 33 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
25f3d62
tinkering with implementations
colerogers Jan 31, 2022
6fe07de
Merge remote-tracking branch 'origin/master' into colerogers.auth-blo…
colerogers Feb 17, 2022
f616332
adding in cache for discovery
colerogers Feb 21, 2022
8566842
adding in skeleton for service object and identity platform api calls
colerogers Feb 25, 2022
6a4b7b9
merging master
colerogers Feb 25, 2022
43b2316
fix conflicts
colerogers Mar 23, 2022
eb016f7
adding in all the service code
colerogers Mar 29, 2022
9ce55af
formatter
colerogers Mar 29, 2022
801cb5f
rename
colerogers Mar 30, 2022
0b22d42
Merge remote-tracking branch 'origin/master' into colerogers.auth-blo…
colerogers Apr 2, 2022
94c4db4
removing the lookup before endpointToFunction & adding v2 stuff
colerogers Apr 2, 2022
b1cbf95
add tests & changelog
colerogers Apr 3, 2022
bee1b93
adding tsdoc comments
colerogers Apr 3, 2022
ed416c8
addressing pr comments
colerogers Apr 4, 2022
a298aad
fixing conflicts
colerogers Apr 4, 2022
62999db
removing the empty object from gcp/identityPlatform
colerogers Apr 5, 2022
e35f6c0
Merge remote-tracking branch 'origin/master' into colerogers.auth-blo…
colerogers Apr 5, 2022
fc47475
added typing & fixing endpointFromFunction to set securityLevel and u…
colerogers Apr 5, 2022
43516bc
removing changelog entries
colerogers Apr 5, 2022
851c3a3
cleaning up & set default endpoint options to false
colerogers Apr 5, 2022
b674320
adding in inferBlockingDetails
colerogers Apr 5, 2022
bd9a0b9
checking if config changed before calling set blocking config
colerogers Apr 5, 2022
7f4fce2
formatter
colerogers Apr 5, 2022
7b69417
cleaning up
colerogers Apr 6, 2022
320d222
Merge remote-tracking branch 'origin/master' into colerogers.auth-blo…
colerogers Apr 6, 2022
fbd7e3e
adding event type if in fab & changing api to match prod
colerogers Apr 7, 2022
4fff5c0
adding serviceForEndpoint in fab setTrigger
colerogers Apr 7, 2022
153bffb
converting to options record & yanking auth blocking events from even…
colerogers Apr 7, 2022
c671094
removing old comments
colerogers Apr 7, 2022
3c95a35
fixing delete blocking function bug
colerogers Apr 8, 2022
9faf7a8
merging master
colerogers Apr 8, 2022
77de23e
adding a promise queue for trigger registration & deletion
colerogers Apr 8, 2022
e758d2b
merge master
colerogers Apr 8, 2022
bb56d25
making auth blocking service a class & moving trigger queue into it
colerogers Apr 12, 2022
a4bfd25
remove comment
colerogers Apr 12, 2022
a2f66cd
merging master
colerogers Apr 12, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
- Fixes Firestore emulator deadlock bug. [#2452](https://github.com/firebase/firebase-tools/issues/2452)
- Fixes console error on large uploads to Storage Emulator (#4407).
- Fixes cross-platform incompatibility with Storage Emulator exports (#4411).
- Adds a blocking trigger type (#4395).
20 changes: 19 additions & 1 deletion src/deploy/functions/backend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import * as gcf from "../../gcp/cloudfunctions";
import * as gcfV2 from "../../gcp/cloudfunctionsv2";
import * as utils from "../../utils";
import * as runtimes from "./runtimes";
import * as events from "../../functions/events";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit* remove events per linter

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in #4443

import { FirebaseError } from "../../error";
import { Context } from "./args";
import { previews } from "../../previews";
Expand Down Expand Up @@ -131,6 +132,15 @@ export interface TaskQueueTriggered {
taskQueueTrigger: TaskQueueTrigger;
}

export interface BlockingTrigger {
eventType: string;
colerogers marked this conversation as resolved.
Show resolved Hide resolved
options?: Record<string, any>;
}

export interface BlockingTriggered {
blockingTrigger: BlockingTrigger;
}

/** A user-friendly string for the kind of trigger of an endpoint. */
export function endpointTriggerType(endpoint: Endpoint): string {
if (isScheduleTriggered(endpoint)) {
Expand All @@ -143,6 +153,8 @@ export function endpointTriggerType(endpoint: Endpoint): string {
return endpoint.eventTrigger.eventType;
} else if (isTaskQueueTriggered(endpoint)) {
return "taskQueue";
} else if (isBlockingTriggered(endpoint)) {
return endpoint.blockingTrigger.eventType;
} else {
throw new Error("Unexpected trigger type for endpoint " + JSON.stringify(endpoint));
}
Expand Down Expand Up @@ -227,7 +239,8 @@ export type Triggered =
| CallableTriggered
| EventTriggered
| ScheduleTriggered
| TaskQueueTriggered;
| TaskQueueTriggered
| BlockingTriggered;

/** Whether something has an HttpsTrigger */
export function isHttpsTriggered(triggered: Triggered): triggered is HttpsTriggered {
Expand All @@ -254,6 +267,11 @@ export function isTaskQueueTriggered(triggered: Triggered): triggered is TaskQue
return {}.hasOwnProperty.call(triggered, "taskQueueTrigger");
}

/** Whether something has a BlockingTrigger */
export function isBlockingTriggered(triggered: Triggered): triggered is BlockingTriggered {
return {}.hasOwnProperty.call(triggered, "blockingTrigger");
}

/**
* An endpoint that serves traffic to a stack of services.
* For now, this is always a Cloud Function. Future iterations may use complex
Expand Down
34 changes: 34 additions & 0 deletions src/deploy/functions/prepare.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import { ensureServiceAgentRoles } from "./checkIam";
import { FirebaseError } from "../../error";
import { normalizeAndValidate } from "../../functions/projectConfig";
import { previews } from "../../previews";
import { AUTH_BLOCKING_EVENTS } from "../../functions/events/v1";

function hasUserConfig(config: Record<string, unknown>): boolean {
// "firebase" key is always going to exist in runtime config.
Expand Down Expand Up @@ -204,6 +205,7 @@ export async function prepare(

await ensureServiceAgentRoles(projectNumber, wantBackend, haveBackend);
inferDetailsFromExisting(wantBackend, haveBackend, usedDotenv);
inferBlockingDetails(wantBackend);
await ensureTriggerRegions(wantBackend);

// Display a warning and prompt if any functions in the release have failurePolicies.
Expand Down Expand Up @@ -271,3 +273,35 @@ function maybeCopyTriggerRegion(wantE: backend.Endpoint, haveE: backend.Endpoint
}
wantE.eventTrigger.region = haveE.eventTrigger.region;
}

/** Figures out the blocking endpoint options by taking the OR of every trigger option and reassigning that value back to the endpoint. */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this the correct behavior? If I have a beforeUserCreated function I must have the same security options as a beforeUserLoggedIn function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's part of the trigger proposal - http://go/cf3-auth-blocking-triggers

Since these are resource level options, they must be the same. Instead of erroring if a user sets the beforeCreate & beforeSignIn options to different values, we just take the OR of them. This will need to be made very clear in our docs

export function inferBlockingDetails(want: backend.Backend): void {
const authBlockingEndpoints = backend
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit* technically blockingEndpoints 😛

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually here I do a check in filter to see if the eventType is an Auth Blocking Trigger event

.allEndpoints(want)
.filter(
(ep) =>
backend.isBlockingTriggered(ep) &&
AUTH_BLOCKING_EVENTS.includes(ep.blockingTrigger.eventType as any)
) as (backend.Endpoint & backend.BlockingTriggered)[];

if (authBlockingEndpoints.length === 0) {
return;
}

let accessToken = false;
let idToken = false;
let refreshToken = false;
for (const blockingEp of authBlockingEndpoints) {
accessToken ||= !!blockingEp.blockingTrigger.options?.accessToken;
idToken ||= !!blockingEp.blockingTrigger.options?.idToken;
refreshToken ||= !!blockingEp.blockingTrigger.options?.refreshToken;
}
for (const blockingEp of authBlockingEndpoints) {
if (!blockingEp.blockingTrigger.options) {
blockingEp.blockingTrigger.options = {};
}
blockingEp.blockingTrigger.options.accessToken = accessToken;
blockingEp.blockingTrigger.options.idToken = idToken;
blockingEp.blockingTrigger.options.refreshToken = refreshToken;
}
}
66 changes: 63 additions & 3 deletions src/deploy/functions/release/fabricator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ import * as reporter from "./reporter";
import * as run from "../../../gcp/run";
import * as scheduler from "../../../gcp/cloudscheduler";
import * as utils from "../../../utils";
import * as services from "../services";
import { AUTH_BLOCKING_EVENTS } from "../../../functions/events/v1";

// TODO: Tune this for better performance.
const gcfV1PollerOptions: Omit<poller.OperationPollerOptions, "operationResourceName"> = {
Expand Down Expand Up @@ -54,6 +56,7 @@ export interface FabricatorArgs {
const rethrowAs =
<T>(endpoint: backend.Endpoint, op: reporter.OperationType) =>
(err: unknown): T => {
logger.error((err as Error).message);
colerogers marked this conversation as resolved.
Show resolved Hide resolved
throw new reporter.DeploymentError(endpoint, op, err);
};

Expand All @@ -64,13 +67,15 @@ export class Fabricator {
sourceUrl: string | undefined;
storage: Record<string, gcfV2.StorageSource> | undefined;
appEngineLocation: string;
triggerQueue: Promise<void>;

constructor(args: FabricatorArgs) {
this.executor = args.executor;
this.functionExecutor = args.functionExecutor;
this.sourceUrl = args.sourceUrl;
this.storage = args.storage;
this.appEngineLocation = args.appEngineLocation;
this.triggerQueue = Promise.resolve();
}

async applyPlan(plan: planner.DeploymentPlan): Promise<reporter.Summary> {
Expand Down Expand Up @@ -164,7 +169,7 @@ export class Fabricator {
assertExhaustive(endpoint.platform);
}

await this.setTrigger(endpoint);
await this.setTrigger(endpoint, false);
}

async updateEndpoint(update: planner.EndpointUpdate, scraper: SourceTokenScraper): Promise<void> {
Expand All @@ -183,7 +188,7 @@ export class Fabricator {
assertExhaustive(update.endpoint.platform);
}

await this.setTrigger(update.endpoint);
await this.setTrigger(update.endpoint, true);
}

async deleteEndpoint(endpoint: backend.Endpoint): Promise<void> {
Expand Down Expand Up @@ -248,6 +253,16 @@ export class Fabricator {
})
.catch(rethrowAs(endpoint, "set invoker"));
}
} else if (
backend.isBlockingTriggered(endpoint) &&
AUTH_BLOCKING_EVENTS.includes(endpoint.blockingTrigger.eventType as any)
) {
// Auth Blocking functions should always be public
await this.executor
.run(async () => {
await gcf.setInvokerCreate(endpoint.project, backend.functionName(endpoint), ["public"]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and elsewhere, it's public because it's an auth blocking function, not because it's a blocking function. Please make the code forwards compatible to other blocking trigger types.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a check

if (backend.isBlockingTriggered(endpoint) && AUTH_BLOCKING_EVENTS.includes(endpoint.blockingTrigger.eventType)) { ... }

})
.catch(rethrowAs(endpoint, "set invoker"));
}
}

Expand Down Expand Up @@ -317,6 +332,14 @@ export class Fabricator {
})
.catch(rethrowAs(endpoint, "set invoker"));
}
} else if (
backend.isBlockingTriggered(endpoint) &&
AUTH_BLOCKING_EVENTS.includes(endpoint.blockingTrigger.eventType as any)
) {
// Auth Blocking functions should always be public
await this.executor
.run(() => run.setInvokerCreate(endpoint.project, serviceName, ["public"]))
.catch(rethrowAs(endpoint, "set invoker"));
}

const mem = endpoint.availableMemoryMb || backend.DEFAULT_MEMORY;
Expand Down Expand Up @@ -354,6 +377,11 @@ export class Fabricator {
invoker = endpoint.httpsTrigger.invoker;
} else if (backend.isTaskQueueTriggered(endpoint)) {
invoker = endpoint.taskQueueTrigger.invoker;
} else if (
backend.isBlockingTriggered(endpoint) &&
AUTH_BLOCKING_EVENTS.includes(endpoint.blockingTrigger.eventType as any)
) {
invoker = ["public"];
}
if (invoker) {
await this.executor
Expand Down Expand Up @@ -395,6 +423,11 @@ export class Fabricator {
invoker = endpoint.httpsTrigger.invoker;
} else if (backend.isTaskQueueTriggered(endpoint)) {
invoker = endpoint.taskQueueTrigger.invoker;
} else if (
backend.isBlockingTriggered(endpoint) &&
AUTH_BLOCKING_EVENTS.includes(endpoint.blockingTrigger.eventType as any)
) {
invoker = ["public"];
}
if (invoker) {
await this.executor
Expand Down Expand Up @@ -460,7 +493,7 @@ export class Fabricator {

// Set/Delete trigger is responsible for wiring up a function with any trigger not owned
// by the GCF API. This includes schedules, task queues, and blocking function triggers.
async setTrigger(endpoint: backend.Endpoint): Promise<void> {
async setTrigger(endpoint: backend.Endpoint, update: boolean): Promise<void> {
if (backend.isScheduleTriggered(endpoint)) {
if (endpoint.platform === "gcfv1") {
await this.upsertScheduleV1(endpoint);
Expand All @@ -472,6 +505,8 @@ export class Fabricator {
assertExhaustive(endpoint.platform);
} else if (backend.isTaskQueueTriggered(endpoint)) {
await this.upsertTaskQueue(endpoint);
} else if (backend.isBlockingTriggered(endpoint)) {
await this.registerBlockingTrigger(endpoint, update);
}
}

Expand All @@ -487,6 +522,8 @@ export class Fabricator {
assertExhaustive(endpoint.platform);
} else if (backend.isTaskQueueTriggered(endpoint)) {
await this.disableTaskQueue(endpoint);
} else if (backend.isBlockingTriggered(endpoint)) {
await this.unregisterBlockingTrigger(endpoint);
}
}

Expand Down Expand Up @@ -519,6 +556,18 @@ export class Fabricator {
}
}

async registerBlockingTrigger(
endpoint: backend.Endpoint & backend.BlockingTriggered,
update: boolean
): Promise<void> {
this.triggerQueue = this.triggerQueue.then(async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q: where do we await on this triggerQueue? I was expecting something to wait until the triggerQueue is cleared in the fabricator.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're checking out an old piece of code, but we're just adding the next task in the queue to the then function. When the current task is fulfilled, the next task will be executed.

await this.executor
.run(() => services.serviceForEndpoint(endpoint).registerTrigger(endpoint, update))
.catch(rethrowAs(endpoint, "register blocking trigger"));
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we make the triggerQueue be a feature of the Auth service? That way we run auth trigger updates in serial but can run other trigger updates in parallel.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to avoid defining a class for the Auth Blocking service, but parallelizing makes a lot of sense here, I'll make the change

return this.triggerQueue;
}

async deleteScheduleV1(endpoint: backend.Endpoint & backend.ScheduleTriggered): Promise<void> {
const job = scheduler.jobFromEndpoint(endpoint, this.appEngineLocation);
await this.executor
Expand Down Expand Up @@ -546,6 +595,17 @@ export class Fabricator {
.catch(rethrowAs(endpoint, "disable task queue"));
}

async unregisterBlockingTrigger(
endpoint: backend.Endpoint & backend.BlockingTriggered
): Promise<void> {
this.triggerQueue = this.triggerQueue.then(async () => {
await this.executor
.run(() => services.serviceForEndpoint(endpoint).unregisterTrigger(endpoint))
.catch(rethrowAs(endpoint, "unregister blocking trigger"));
});
return this.triggerQueue;
}

logOpStart(op: string, endpoint: backend.Endpoint): void {
const runtime = getHumanFriendlyRuntimeName(endpoint.runtime);
const label = helper.getFunctionLabel(endpoint);
Expand Down
2 changes: 2 additions & 0 deletions src/deploy/functions/release/planner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,8 @@ export function checkForIllegalUpdate(want: backend.Endpoint, have: backend.Endp
return "a scheduled";
} else if (backend.isTaskQueueTriggered(e)) {
return "a task queue";
} else if (backend.isBlockingTriggered(e)) {
return e.blockingTrigger.eventType;
}
// Unfortunately TypeScript isn't like Scala and I can't prove to it
// that all cases have been handled
Expand Down
8 changes: 7 additions & 1 deletion src/deploy/functions/release/reporter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ export type OperationType =
| "create topic"
| "delete topic"
| "set invoker"
| "set concurrency";
| "set concurrency"
| "register blocking trigger"
| "unregister blocking trigger";

/** An error with a deployment phase. */
export class DeploymentError extends Error {
Expand Down Expand Up @@ -245,5 +247,9 @@ export function triggerTag(endpoint: backend.Endpoint): string {
return `${prefix}.https`;
}

if (backend.isBlockingTriggered(endpoint)) {
return `${prefix}.blocking`;
}

return endpoint.eventTrigger.eventType;
}
12 changes: 12 additions & 0 deletions src/deploy/functions/runtimes/discovery/v1alpha1.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ export type ManifestEndpoint = backend.ServiceConfiguration &
Partial<backend.CallableTriggered> &
Partial<backend.EventTriggered> &
Partial<backend.TaskQueueTriggered> &
Partial<backend.BlockingTriggered> &
Partial<backend.ScheduleTriggered> & {
region?: string[];
entryPoint: string;
Expand Down Expand Up @@ -102,6 +103,7 @@ function parseEndpoints(
eventTrigger: "object",
scheduleTrigger: "object",
taskQueueTrigger: "object",
blockingTrigger: "object",
});
let triggerCount = 0;
if (ep.httpsTrigger) {
Expand All @@ -119,6 +121,9 @@ function parseEndpoints(
if (ep.taskQueueTrigger) {
triggerCount++;
}
if (ep.blockingTrigger) {
triggerCount++;
}
if (!triggerCount) {
throw new FirebaseError("Expected trigger in endpoint " + id);
}
Expand Down Expand Up @@ -192,6 +197,13 @@ function parseEndpoints(
});
}
triggered = { taskQueueTrigger: ep.taskQueueTrigger };
} else if (backend.isBlockingTriggered(ep)) {
requireKeys(prefix + ".blockingTrigger", ep.blockingTrigger, "eventType");
assertKeyTypes(prefix + ".blockingTrigger", ep.blockingTrigger, {
eventType: "string",
options: "object",
});
triggered = { blockingTrigger: ep.blockingTrigger };
} else {
throw new FirebaseError(
`Do not recognize trigger type for endpoint ${id}. Try upgrading ` +
Expand Down