Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds new alerting event provider #4258

Merged
merged 14 commits into from
Mar 8, 2022
Merged
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
- Fixes bug where functions' memory configurations weren't preserved in batched function deploys (#4253).
- Adds alerting event provider (#4258).
15 changes: 10 additions & 5 deletions src/deploy/functions/checkIam.ts
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ export function mergeBindings(policy: iam.Policy, allRequiredBindings: iam.Bindi
* @param have backend that we have currently deployed
*/
export async function ensureServiceAgentRoles(
projectId: string,
project: { projectId: string; projectNumber: string },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit* In theory, having just projectNumber should be sufficient since all GCP API calls accept either projectId or projectNumber. Will leave it up to you to decide whether that would be appropriate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh snap, I didn't realize treated them as interchangeable!!

want: backend.Backend,
have: backend.Backend
): Promise<void> {
Expand All @@ -164,7 +164,7 @@ export async function ensureServiceAgentRoles(
// get the full project iam policy
let policy: iam.Policy;
try {
policy = await getIamPolicy(projectId);
policy = await getIamPolicy(project.projectId);
} catch (err: any) {
utils.logLabeledBullet(
"functions",
Expand All @@ -176,15 +176,20 @@ export async function ensureServiceAgentRoles(
return;
}
// run in parallel all the missingProjectBindings jobs
const findRequiredBindings: Array<Promise<Array<iam.Binding>>> = [];
const findRequiredBindings: Array<Promise<Array<iam.Binding>> | Array<iam.Binding>> = [];
newServices.forEach((service) =>
findRequiredBindings.push(service.requiredProjectBindings!(projectId, policy))
findRequiredBindings.push(
service.requiredProjectBindings!(
{ projectId: project.projectId, projectNumber: project.projectNumber },
policy
)
)
);
const allRequiredBindings = await Promise.all(findRequiredBindings);
mergeBindings(policy, allRequiredBindings);
// set the updated policy
try {
await setIamPolicy(projectId, policy, "bindings");
await setIamPolicy(project.projectId, policy, "bindings");
} catch (err: any) {
throw new FirebaseError(
"We failed to modify the IAM policy for the project. The functions " +
Expand Down
4 changes: 3 additions & 1 deletion src/deploy/functions/prepare.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import { logger } from "../../logger";
import { ensureTriggerRegions } from "./triggerRegionHelper";
import { ensureServiceAgentRoles } from "./checkIam";
import { FirebaseError } from "../../error";
import { getProjectNumber } from "../../getProjectNumber";

function hasUserConfig(config: Record<string, unknown>): boolean {
// "firebase" key is always going to exist in runtime config.
Expand All @@ -37,6 +38,7 @@ export async function prepare(
payload: args.Payload
): Promise<void> {
const projectId = needProjectId(options);
const projectNumber = await getProjectNumber({ projectId });
colerogers marked this conversation as resolved.
Show resolved Hide resolved

const sourceDirName = options.config.get("functions.source") as string;
if (!sourceDirName) {
Expand Down Expand Up @@ -160,7 +162,7 @@ export async function prepare(
});

const haveBackend = await backend.existingBackend(context);
await ensureServiceAgentRoles(projectId, wantBackend, haveBackend);
await ensureServiceAgentRoles({ projectId, projectNumber }, wantBackend, haveBackend);
inferDetailsFromExisting(wantBackend, haveBackend, usedDotenv);
await ensureTriggerRegions(wantBackend);

Expand Down
48 changes: 48 additions & 0 deletions src/deploy/functions/services/firebaseAlerts.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import * as backend from "../backend";
import * as iam from "../../../gcp/iam";
import { getProjectNumber } from "../../../getProjectNumber";
import { FirebaseError } from "../../../error";

export const SERVICE_ACCOUNT_TOKEN_CREATOR_ROLE = "roles/iam.serviceAccountTokenCreator";

/**
* Finds the required project level IAM bindings for the Pub/Sub service agent
* If the user enabled Pub/Sub on or before April 8, 2021, then we must enable the token creator role
* @param projectId project identifier
* @param existingPolicy the project level IAM policy
*/
export function obtainFirebaseAlertsBindings(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really a Firebase Alerts feature or something that all event providers are going to need?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NVM. See below comments. I think we could have had something like:

export function obtainFirebaseAlertBindings(projectNumber) {
  iam.ensureBinding(firebaseAlertRole, firebaseAlertPermission)
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're correct, this is not a Firebase Alerts specific binding. It was yanked in #4324 and I added a noop instead

project: { projectId: string; projectNumber: string },
existingPolicy: iam.Policy
): Array<iam.Binding> {
const pubsubServiceAgent = `serviceAccount:service-${project.projectNumber}@gcp-sa-pubsub.iam.gserviceaccount.com`;
let pubsubBinding = existingPolicy.bindings.find(
(b) => b.role === SERVICE_ACCOUNT_TOKEN_CREATOR_ROLE
);
if (!pubsubBinding) {
pubsubBinding = {
role: SERVICE_ACCOUNT_TOKEN_CREATOR_ROLE,
members: [],
};
}
if (!pubsubBinding.members.find((m) => m === pubsubServiceAgent)) {
pubsubBinding.members.push(pubsubServiceAgent);
}
return [pubsubBinding];
}

/**
* Sets a Firebase Alerts event trigger's region to 'global' since the service is global
* @param endpoint the storage endpoint
* @param eventTrigger the endpoints event trigger
*/
export function ensureFirebaseAlertsTriggerRegion(
endpoint: backend.Endpoint & backend.EventTriggered
): void {
if (!endpoint.eventTrigger.region) {
endpoint.eventTrigger.region = "global";
}
if (endpoint.eventTrigger.region !== "global") {
throw new FirebaseError("A firebase alerts trigger must specify 'global' trigger location");
}
}
22 changes: 19 additions & 3 deletions src/deploy/functions/services/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,25 @@ import * as backend from "../backend";
import * as iam from "../../../gcp/iam";
import * as v2events from "../../../functions/events/v2";
import { obtainStorageBindings, ensureStorageTriggerRegion } from "./storage";
import { obtainFirebaseAlertsBindings, ensureFirebaseAlertsTriggerRegion } from "./firebaseAlerts";

const noop = (): Promise<void> => Promise.resolve();

interface Project {
projectId: string;
projectNumber: string;
}

/** A service interface for the underlying GCP event services */
export interface Service {
readonly name: string;
readonly api: string;

// dispatch functions
requiredProjectBindings: ((pId: any, p: any) => Promise<Array<iam.Binding>>) | undefined;
ensureTriggerRegion: (ep: backend.Endpoint & backend.EventTriggered) => Promise<void>;
requiredProjectBindings:
| ((project: Project, policy: iam.Policy) => Promise<Array<iam.Binding>> | Array<iam.Binding>)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this type got really complicated, what if we instead keep the original type signature and instead have obtainFirebaseAlertsBindings return a promise instead? e.g.

export function obtainFirebaseAlertsBindings(
  project: { projectId: string; projectNumber: string },
  existingPolicy: iam.Policy
): Promise<Array<iam.Binding>> {
  // ...implementation
  return Promise.resolve([pubsubBinding]);
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the latest commit

| undefined;
ensureTriggerRegion: (ep: backend.Endpoint & backend.EventTriggered) => Promise<void> | void;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same trick as I mentioned above - what if we "clean" this type up by requiring everything to return a promise - it is easy to make non-promise returning fns to return a promise instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in the latest commit

}

/** A noop service object, useful for v1 events */
Expand All @@ -30,12 +38,19 @@ export const PubSubService: Service = {
ensureTriggerRegion: noop,
};
/** A storage service object */
export const StorageService = {
export const StorageService: Service = {
name: "storage",
api: "storage.googleapis.com",
requiredProjectBindings: obtainStorageBindings,
ensureTriggerRegion: ensureStorageTriggerRegion,
};
/** A firebase alerts service object */
export const FirebaseAlertsService: Service = {
name: "firebasealerts",
api: "logging.googleapis.com",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really the API for Firebase Alerts?? How is this not a conflict with Cloud Audit Logs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was just a placeholder, I'll update it with the correct API. Additionally, this field is not currently being used in the CLI

requiredProjectBindings: obtainFirebaseAlertsBindings,
ensureTriggerRegion: ensureFirebaseAlertsTriggerRegion,
};

/** Mapping from event type string to service object */
export const EVENT_SERVICE_MAPPING: Record<v2events.Event, Service> = {
Expand All @@ -44,6 +59,7 @@ export const EVENT_SERVICE_MAPPING: Record<v2events.Event, Service> = {
"google.cloud.storage.object.v1.archived": StorageService,
"google.cloud.storage.object.v1.deleted": StorageService,
"google.cloud.storage.object.v1.metadataUpdated": StorageService,
"firebase.firebasealerts.alerts.v1.published": FirebaseAlertsService,
};

/**
Expand Down
4 changes: 2 additions & 2 deletions src/deploy/functions/services/storage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ const PUBSUB_PUBLISHER_ROLE = "roles/pubsub.publisher";
* @param existingPolicy the project level IAM policy
*/
export async function obtainStorageBindings(
projectId: string,
project: { projectId: string; projectNumber: string },
existingPolicy: iam.Policy
): Promise<Array<iam.Binding>> {
const storageResponse = await storage.getServiceAccount(projectId);
const storageResponse = await storage.getServiceAccount(project.projectId);
const storageServiceAgent = `serviceAccount:${storageResponse.email_address}`;
let pubsubBinding = existingPolicy.bindings.find((b) => b.role === PUBSUB_PUBLISHER_ROLE);
if (!pubsubBinding) {
Expand Down
2 changes: 1 addition & 1 deletion src/deploy/functions/triggerRegionHelper.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { serviceForEndpoint } from "./services";
* @param have the list of function specs we have deployed
*/
export async function ensureTriggerRegions(want: backend.Backend): Promise<void> {
const regionLookups: Array<Promise<void>> = [];
const regionLookups: Array<Promise<void> | void> = [];
colerogers marked this conversation as resolved.
Show resolved Hide resolved

for (const ep of backend.allEndpoints(want)) {
if (ep.platform === "gcfv1" || !backend.isEventTriggered(ep)) {
Expand Down
7 changes: 6 additions & 1 deletion src/functions/events/v2.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,9 @@ export const STORAGE_EVENTS = [
"google.cloud.storage.object.v1.metadataUpdated",
] as const;

export type Event = typeof PUBSUB_PUBLISH_EVENT | typeof STORAGE_EVENTS[number];
export const FIREBASE_ALERTS_PUBLISH_EVENT = "firebase.firebasealerts.alerts.v1.published";

export type Event =
| typeof PUBSUB_PUBLISH_EVENT
| typeof STORAGE_EVENTS[number]
| typeof FIREBASE_ALERTS_PUBLISH_EVENT;
13 changes: 9 additions & 4 deletions src/test/deploy/functions/checkIam.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ import * as storage from "../../../gcp/storage";
import * as rm from "../../../gcp/resourceManager";
import * as backend from "../../../deploy/functions/backend";

const project = {
projectId: "project",
projectNumber: "123456789",
};

const STORAGE_RES = {
email_address: "service-123@gs-project-accounts.iam.gserviceaccount.com",
kind: "storage#serviceAccount",
Expand Down Expand Up @@ -105,7 +110,7 @@ describe("checkIam", () => {
...SPEC,
};

await expect(checkIam.ensureServiceAgentRoles("project", backend.of(wantFn), backend.empty()))
await expect(checkIam.ensureServiceAgentRoles(project, backend.of(wantFn), backend.empty()))
.to.not.be.rejected;
expect(getIamStub).to.have.been.calledOnce;
expect(getIamStub).to.have.been.calledWith("project");
Expand Down Expand Up @@ -155,7 +160,7 @@ describe("checkIam", () => {
};

await checkIam.ensureServiceAgentRoles(
"project",
project,
backend.of(wantFn),
backend.of(v1EventFn, v2CallableFn, wantFn)
);
Expand Down Expand Up @@ -199,7 +204,7 @@ describe("checkIam", () => {
...SPEC,
};

await checkIam.ensureServiceAgentRoles("project", backend.of(wantFn), backend.of(haveFn));
await checkIam.ensureServiceAgentRoles(project, backend.of(wantFn), backend.of(haveFn));

expect(storageStub).to.not.have.been.called;
expect(getIamStub).to.not.have.been.called;
Expand Down Expand Up @@ -242,7 +247,7 @@ describe("checkIam", () => {
...SPEC,
};

await checkIam.ensureServiceAgentRoles("project", backend.of(wantFn), backend.empty());
await checkIam.ensureServiceAgentRoles(project, backend.of(wantFn), backend.empty());

expect(storageStub).to.have.been.calledOnce;
expect(getIamStub).to.have.been.calledOnce;
Expand Down
125 changes: 125 additions & 0 deletions src/test/deploy/functions/services/firebaseAlerts.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
import { expect } from "chai";
import { Endpoint } from "../../../../deploy/functions/backend";
import * as firebaseAlerts from "../../../../deploy/functions/services/firebaseAlerts";
import * as getProjectNumber from "../../../../getProjectNumber";
import * as sinon from "sinon";

const project = {
projectId: "project",
projectNumber: "123456789",
};

describe("obtainFirebaseAlertsBindings", () => {
let projectNumberStub: sinon.SinonStub;

const iamPolicy = {
etag: "etag",
version: 3,
bindings: [
{
role: "some/role",
members: ["someuser"],
},
],
};

beforeEach(() => {
projectNumberStub = sinon.stub(getProjectNumber, "getProjectNumber").resolves("123456789");
});

afterEach(() => {
sinon.verifyAndRestore();
});

it("should add the binding", () => {
const policy = { ...iamPolicy };

const bindings = firebaseAlerts.obtainFirebaseAlertsBindings(project, policy);

expect(bindings.length).to.equal(1);
expect(bindings[0]).to.deep.equal({
role: firebaseAlerts.SERVICE_ACCOUNT_TOKEN_CREATOR_ROLE,
members: ["serviceAccount:service-123456789@gcp-sa-pubsub.iam.gserviceaccount.com"],
});
});

it("should add the service agent as a member", () => {
const policy = { ...iamPolicy };
policy.bindings = [
{
role: firebaseAlerts.SERVICE_ACCOUNT_TOKEN_CREATOR_ROLE,
members: ["someuser"],
},
];

const bindings = firebaseAlerts.obtainFirebaseAlertsBindings(project, policy);

expect(bindings.length).to.equal(1);
expect(bindings[0]).to.deep.equal({
role: firebaseAlerts.SERVICE_ACCOUNT_TOKEN_CREATOR_ROLE,
members: [
"someuser",
"serviceAccount:service-123456789@gcp-sa-pubsub.iam.gserviceaccount.com",
],
});
});

it("should do nothing if we have the binding", () => {
const policy = { ...iamPolicy };
policy.bindings = [
{
role: firebaseAlerts.SERVICE_ACCOUNT_TOKEN_CREATOR_ROLE,
members: ["serviceAccount:service-123456789@gcp-sa-pubsub.iam.gserviceaccount.com"],
},
];

const bindings = firebaseAlerts.obtainFirebaseAlertsBindings(project, policy);

expect(bindings.length).to.equal(1);
expect(bindings[0]).to.deep.equal({
role: firebaseAlerts.SERVICE_ACCOUNT_TOKEN_CREATOR_ROLE,
members: ["serviceAccount:service-123456789@gcp-sa-pubsub.iam.gserviceaccount.com"],
});
});
});

describe("ensureFirebaseAlertsTriggerRegion", () => {
const endpoint: Endpoint = {
id: "endpoint",
region: "us-central1",
project: "my-project",
eventTrigger: {
retry: false,
eventType: "firebase.firebasealerts.alerts.v1.published",
eventFilters: [],
},
entryPoint: "",
platform: "gcfv2",
runtime: "nodejs16",
};
it("should set the trigger location to global", () => {
const ep = { ...endpoint };

firebaseAlerts.ensureFirebaseAlertsTriggerRegion(ep);

expect(endpoint.eventTrigger.region).to.eq("global");
});

it("should not error if the trigger location is global", () => {
const ep = { ...endpoint };
ep.eventTrigger.region = "global";

firebaseAlerts.ensureFirebaseAlertsTriggerRegion(endpoint);

expect(endpoint.eventTrigger.region).to.eq("global");
});

it("should error if the trigger location is not global", () => {
const ep = { ...endpoint };
ep.eventTrigger.region = "us-west1";

expect(() => firebaseAlerts.ensureFirebaseAlertsTriggerRegion(endpoint)).to.throw(
"A firebase alerts trigger must specify 'global' trigger location"
);
});
});