Create SQL Backup and export module

modules/backup/README.md

@@ -0,0 +1,54 @@
+# GCP CloudSQL Backup
+
+## Internal Backups
+
+Cloud SQL is providing an [internal default backup options](https://cloud.google.com/sql/docs/mysql/backup-recovery/backups#automated-backups).
+This supports standard use-cases. (One Backup ~ every 24h stored for up to 365 days).
+
+For many use cases this might be enough but for many other not:
+e.g. for SQL Server there is currently no Point-in-time Recovery (PITR) available so you might want to take more updates.
+Or your instance is not that critical and to [save costs](https://cloud.google.com/sql/docs/mysql/backup-recovery/backups#what_backups_cost) you are fine with one backup every 7 days.
+
+> **Note**: To enable or disable the internal backups use the Terraform variable `backup`
+
+For internal backups the workflow also takes care about deleting the old backups.
+
+## Exports to GCS
+
+But the backups mentioned above are bound to the SQL instance.
+This means as soon as you [delete the instance](https://cloud.google.com/sql/docs/mysql/delete-instance) you are also loosing your backups.
+To avoid this you can also export data. There is a second workflow doing exactly that.
+
+> **Note**: To enable or disable the exports to GCS Buckets use the Terraform variable `export`
+
+### Deleting old exports
+
+The export workflow does not take care about deleting old backups. Please take care about that yourself.
+You can use [Object Lifecycle Management](https://cloud.google.com/storage/docs/lifecycle) to delete old exports.
+
+## Required APIs
+
+- `workflows.googleapis.com`
+- `cloudscheduler.googleapis.com`
+
+## Monitoring
+
+Monitoring your exports/backups is not part of this module.
+Please ensure that you monitor for failed workflows to ensure you have regular backups/exports.
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_backup"></a> [backup](#input\_backup) | Weather to create internal backups with this module | `bool` | `true` | no |
+| <a name="input_backup-retention-time"></a> [backup-retention-time](#input\_backup-retention-time) | The number of Days Backups should be kept | `number` | `30` | no |
+| <a name="input_backup-schedule"></a> [backup-schedule](#input\_backup-schedule) | The cron schedule to execute the internal backup | `string` | `"45 2 * * *"` | no |
+| <a name="input_export"></a> [export](#input\_export) | Weather to create exports to GCS Buckets with this module | `bool` | `true` | no |
+| <a name="input_export-databases"></a> [export-databases](#input\_export-databases) | The list of databases that should be exported - if is an empty set all databases will be exported | `set(string)` | `[]` | no |
+| <a name="input_export-schedule"></a> [export-schedule](#input\_export-schedule) | The cron schedule to execute the export to GCS | `string` | `"15 3 * * *"` | no |
+| <a name="input_export-uri"></a> [export-uri](#input\_export-uri) | The bucket and path uri for exporting to GCS | `string` | n/a | yes |
+| <a name="input_project-id"></a> [project-id](#input\_project-id) | The project ID | `string` | n/a | yes |
+| <a name="input_region"></a> [region](#input\_region) | The region where to run the workflow | `string` | n/a | yes |
+| <a name="input_scheduler-timezone"></a> [scheduler-timezone](#input\_scheduler-timezone) | The Timezone in which the Scheduler Jobs are triggered | `string` | `"Etc/GMT"` | no |
+| <a name="input_service-account"></a> [service-account](#input\_service-account) | The service account to use for running the workflow - If empty or null a service account will be created. If you have provided a service account you need to grant the Cloud SQL Admin and the Workflows Invoker role to that | `string` | `null` | no |
+| <a name="input_sql-instance"></a> [sql-instance](#input\_sql-instance) | The name of the SQL instance to backup | `string` | n/a | yes |

modules/backup/locals.tf

@@ -0,0 +1,20 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  create-service-account = var.service-account == null || var.service-account == "" ? true : false
+  service-account        = local.create-service-account ? google_service_account.sql_backup_serviceaccount[0].email : var.service-account
+}

modules/backup/main.tf

@@ -0,0 +1,133 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+################################
+#                              #
+#   Service Account and IAM    #
+#                              #
+################################
+resource "google_service_account" "sql_backup_serviceaccount" {
+  count        = local.create-service-account ? 1 : 0
+  account_id   = substr("backup-${var.sql-instance}", 0, 28)
+  display_name = "Managed by Terraform - Service account for backup of SQL Instance ${var.sql-instance}"
+  project      = var.project-id
+}
+
+resource "google_project_iam_member" "sql_backup_serviceaccount_sql_admin" {
+  count   = local.create-service-account ? 1 : 0
+  member  = "serviceAccount:${google_service_account.sql_backup_serviceaccount[0].email}"
+  role    = "roles/cloudsql.admin"
+  project = var.project-id
+}
+
+resource "google_project_iam_member" "sql_backup_serviceaccount_workflow_invoker" {
+  count   = local.create-service-account ? 1 : 0
+  member  = "serviceAccount:${google_service_account.sql_backup_serviceaccount[0].email}"
+  role    = "roles/workflows.invoker"
+  project = var.project-id
+}
+
+data "google_sql_database_instance" "backup_instance" {
+  name    = var.sql-instance
+  project = var.project-id
+}
+
+################################
+#                              #
+#       Internal Backups       #
+#                              #
+################################
+#TODO: allow multiple backups
+resource "google_workflows_workflow" "sql_backup" {
+  count           = var.backup ? 1 : 0
+  name            = "sql-backup-${var.sql-instance}"
+  region          = var.region
+  description     = "Workflow for backing up the CloudSQL Instance "
+  project         = var.project-id
+  service_account = local.service-account
+  source_contents = templatefile("${path.module}/templates/backup.yaml.tftpl", {
+    project             = var.project-id
+    instanceName        = var.sql-instance
+    backupRetentionTime = var.backup-retention-time
+  })
+}
+
+resource "google_cloud_scheduler_job" "sql_backup" {
+  count       = var.backup ? 1 : 0
+  name        = "sql-backup-${var.sql-instance}"
+  project     = var.project-id
+  region      = var.region
+  description = "Managed by Terraform - Triggers a SQL Backup via Workflows"
+  schedule    = var.backup-schedule
+  time_zone   = var.scheduler-timezone
+
+  http_target {
+    uri         = "https://workflowexecutions.googleapis.com/v1/${google_workflows_workflow.sql_backup[0].id}/executions"
+    http_method = "POST"
+    oauth_token {
+      scope                 = "https://www.googleapis.com/auth/cloud-platform"
+      service_account_email = local.service-account
+    }
+  }
+}
+
+################################
+#                              #
+#       External Backups       #
+#                              #
+################################
+resource "google_workflows_workflow" "sql_export" {
+  count           = var.export ? 1 : 0
+  name            = "sql-export-${var.sql-instance}"
+  region          = var.region
+  description     = "Workflow for backing up the CloudSQL Instance "
+  project         = var.project-id
+  service_account = local.service-account
+  source_contents = templatefile("${path.module}/templates/export.yaml.tftpl", {
+    project             = var.project-id
+    instanceName        = var.sql-instance
+    backupRetentionTime = var.backup-retention-time
+    databases           = jsonencode(var.export-databases)
+    gcsBucket           = var.export-uri
+    dbType              = split("_", data.google_sql_database_instance.backup_instance.database_version)[0]
+  })
+}
+
+resource "google_cloud_scheduler_job" "sql_export" {
+  count       = var.export ? 1 : 0
+  name        = "sql-export-${var.sql-instance}"
+  project     = var.project-id
+  region      = var.region
+  description = "Managed by Terraform - Triggers a SQL Export via Workflows"
+  schedule    = var.export-schedule
+  time_zone   = var.scheduler-timezone
+
+  http_target {
+    uri         = "https://workflowexecutions.googleapis.com/v1/${google_workflows_workflow.sql_export[0].id}/executions"
+    http_method = "POST"
+    oauth_token {
+      scope                 = "https://www.googleapis.com/auth/cloud-platform"
+      service_account_email = local.service-account
+    }
+  }
+}
+
+resource "google_storage_bucket_iam_member" "sql_instance_account" {
+  count  = var.export ? 1 : 0
+  bucket = split("/", var.export-uri)[2] #Get the name of the bucket out of the URI
+  member = "serviceAccount:${data.google_sql_database_instance.backup_instance.service_account_email_address}"
+  role   = "roles/storage.objectCreator"
+}

modules/backup/templates/backup.yaml.tftpl

@@ -0,0 +1,50 @@
+main:
+  steps:
+  - init:
+      assign:
+        - deletedBackups: []
+        - allBackups: []
+  - create new backup:
+      call: googleapis.sqladmin.v1.backupRuns.insert
+      args:
+        project: ${project}
+        instance: ${instanceName}
+        body:
+          description: Backup triggered by the Backup Workflow
+      result: backupRun
+
+    # By calling the backups list and delete only after the new backup was created
+    # we can be sure that there is always a backup existing even if the backup run
+    # is failing
+
+  - get older backups:
+      call: googleapis.sqladmin.v1.backupRuns.list
+      args:
+        project: ${project}
+        instance: ${instanceName}
+      result: backupList
+  - delete old backups:
+      for:
+        value: backup
+        in: $${backupList.items}
+        steps:
+          - get backup endtime:
+              assign:
+                - backupEndTime: $${time.parse(backup.endTime)}
+          - delete only old backups:
+              switch:
+                - condition: $${backupEndTime < sys.now() - 60 * 60 * 24 * ${backupRetentionTime}  AND backup.type == "ON_DEMAND"  }
+                  steps:
+                    - delete:
+                        call: googleapis.sqladmin.v1beta4.backupRuns.delete
+                        args:
+                            project: ${project}
+                            instance: ${instanceName}
+                            id: $${backup.id}
+                    - add to list of deleted backups:
+                        assign:
+                          - deletedBackups: $${list.concat(deletedBackups, backup.id)}
+
+  - return:
+      return:
+        deletedBackups: $${deletedBackups}

modules/backup/templates/export.yaml.tftpl

@@ -0,0 +1,76 @@
+main:
+  steps:
+  - collectInfos:
+      assign:
+        - databases: ${databases}
+        - database_type: ${dbType}
+        - backupTime: $${string(int(sys.now()))}
+%{ if databases == "[]" && (dbType == "POSTGRES" || dbType == "SQLSERVER") }
+  - list of databases:
+      call: googleapis.sqladmin.v1.databases.list
+      args:
+        project: ${project}
+        instance: ${instanceName}
+      result: dbListResult
+  - collect DB list:
+      for:
+        value: db
+        in: $${dbListResult.items}
+        steps:
+          - iterate:
+              assign:
+                - databases: $${list.concat(databases, db.name)}
+
+%{ endif }
+
+%{ if dbType == "POSTGRES" }
+  - export databases:
+      for:
+        value: database
+        in: $${databases}
+        steps:
+          - export database:
+              call: googleapis.sqladmin.v1.instances.export
+              args:
+                project: ${project}
+                instance: ${instanceName}
+                body:
+                  exportContext:
+                    databases: [$${database}]
+                    uri: $${"${gcsBucket}/${instanceName}-" + database + "-" + backupTime + ".sql"}
+%{ endif }
+
+%{ if dbType == "SQLSERVER" }
+  - export databases:
+      for:
+        value: database
+        in: $${databases}
+        steps:
+          - exclude System DB:
+              switch:
+                - condition: $${database != "tempdb" } #tempdb has to be excluded in an export
+                  steps:
+                    - export database:
+                        call: googleapis.sqladmin.v1.instances.export
+                        args:
+                          project: ${project}
+                          instance: ${instanceName}
+                          body:
+                            exportContext:
+                              databases: [$${database}]
+                              uri: $${"${gcsBucket}/${instanceName}-" + database + "-" + backupTime + ".bak"}
+                              fileType: BAK
+%{ endif }
+
+%{ if dbType == "MYSQL" }
+  - create new export:
+      call: googleapis.sqladmin.v1.instances.export
+      args:
+        project: ${project}
+        instance: ${instanceName}
+        body:
+          exportContext:
+            databases: $${databases}
+            uri: $${"${gcsBucket}/${instanceName}-" + backupTime + ".sql"}
+%{ endif }
+