Policy Datasources for leveraging built-in services

[akhomyakov]

Hi folks,
Looking at the nsxt_ns_groups and nsxt_ns_services data resources (currently in Beta) they appear like exactly what is needed to make use of built in services and so on.
What I can't figure out is how to refer to items imported. Is there a way to output the content of what is imported?

<snip for brevity>
data "nsxt_ns_groups" "NS-GROUPS" {
}
data "nsxt_ns_services" "NS-SERVICES" {
}
resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE" {
  display_name = "SHARED-INFRASTRUCTURE"
  description  = "Service access policy shared by all tenants"
  category     = "Infrastructure"
  locked       = false
  stateful     = true
  tcp_strict   = false

  rule {
    display_name       = "ICMP"
    destination_groups = [data.nsxt_ns_groups.NS-GROUPS.items["INTDMZ"]]
    action             = "ALLOW"
    services           = [data.nsxt_ns_services.NS-SERVICES.items["ICMP-ALL"]]
    logged             = true
  }
<snip for brevity>

<snip for brevity>
╷
│ Error: Invalid index
│
│   on firewall-policy-SHARED-INFRASTRUCTURE.tf line 12, in resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE":
│   12:     destination_groups = [data.nsxt_ns_groups.NS-GROUPS.items["INTDMZ"]]
│     ├────────────────
│     │ data.nsxt_ns_groups.NS-GROUPS.items is map of string with 9 elements
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Invalid index
│
│   on firewall-policy-SHARED-INFRASTRUCTURE.tf line 14, in resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE":
│   14:     services           = [data.nsxt_ns_services.NS-SERVICES.items["ICMP-ALL"]]
│     ├────────────────
│     │ data.nsxt_ns_services.NS-SERVICES.items is map of string with 373 elements
│
│ The given key does not identify an element in this collection value.
╵
<snip for brevity>

Just using the API doesn't seem to yeild the right results

// GET https://{{nsxmanager}}/policy/api/v1/infra/domains/default/groups
{
    "results": [
        {
            "expression": [
                {
                    "ip_addresses": [
                        "10.223.35.0/24"
                    ],
                    "resource_type": "IPAddressExpression",
                    "id": "b67d26da-ae46-47b0-9ed9-b954c4447864",
                    "path": "/infra/domains/default/groups/a8839a31-dbbf-4c61-ada1-32963711ef94/ip-address-expressions/b67d26da-ae46-47b0-9ed9-b954c4447864",
                    "relative_path": "b67d26da-ae46-47b0-9ed9-b954c4447864",
                    "parent_path": "/infra/domains/default/groups/a8839a31-dbbf-4c61-ada1-32963711ef94",
                    "marked_for_delete": false,
                    "overridden": false,
                    "_protection": "NOT_PROTECTED"
                }
            ],
            "extended_expression": [],
            "reference": false,
            "resource_type": "Group",
            "id": "a8839a31-dbbf-4c61-ada1-32963711ef94",
            "display_name": "INTDMZ",
            "description": "Terraform provisioned Group",
            "tags": [],
            "path": "/infra/domains/default/groups/a8839a31-dbbf-4c61-ada1-32963711ef94",
            "relative_path": "a8839a31-dbbf-4c61-ada1-32963711ef94",
            "parent_path": "/infra/domains/default",
            "unique_id": "d214f969-60f7-44c5-acd6-8b436639f606",
            "marked_for_delete": false,
            "overridden": false,
            "_create_user": "<snip>",
            "_create_time": 1637272533808,
            "_last_modified_user": "adm_dam@athenahealth.com",
            "_last_modified_time": 1637272533809,
            "_system_owned": false,
            "_protection": "NOT_PROTECTED",
            "_revision": 0
        },
<snip for brevity>

// GET https://{{nsxmanager}}/policy/api/v1/infra/services
<snip for brevity>
        {
            "is_default": true,
            "service_entries": [
                {
                    "protocol": "ICMPv6",
                    "resource_type": "ICMPTypeServiceEntry",
                    "id": "ICMPv6-ALL",
                    "display_name": "ICMPv6-ALL",
                    "path": "/infra/services/ICMP-ALL/service-entries/ICMPv6-ALL",
                    "relative_path": "ICMPv6-ALL",
                    "parent_path": "/infra/services/ICMP-ALL",
                    "marked_for_delete": false,
                    "overridden": false,
                    "_create_user": "system",
                    "_create_time": 1555511297346,
                    "_last_modified_user": "system",
                    "_last_modified_time": 1555511297346,
                    "_system_owned": true,
                    "_protection": "NOT_PROTECTED",
                    "_revision": 0
                },
                {
                    "protocol": "ICMPv4",
                    "resource_type": "ICMPTypeServiceEntry",
                    "id": "ICMPv4-ALL",
                    "display_name": "ICMPv4-ALL",
                    "path": "/infra/services/ICMP-ALL/service-entries/ICMPv4-ALL",
                    "relative_path": "ICMPv4-ALL",
                    "parent_path": "/infra/services/ICMP-ALL",
                    "marked_for_delete": false,
                    "overridden": false,
                    "_create_user": "system",
                    "_create_time": 1555511297345,
                    "_last_modified_user": "system",
                    "_last_modified_time": 1555511297345,
                    "_system_owned": true,
                    "_protection": "NOT_PROTECTED",
                    "_revision": 0
                }
            ],
            "service_type": "NON_ETHER",
            "resource_type": "Service",
            "id": "ICMP-ALL",
            "display_name": "ICMP ALL",
            "description": "ICMP ALL",
            "path": "/infra/services/ICMP-ALL",
            "relative_path": "ICMP-ALL",
            "parent_path": "/infra",
            "marked_for_delete": false,
            "overridden": false,
            "_create_user": "system",
            "_create_time": 1555511297344,
            "_last_modified_user": "system",
            "_last_modified_time": 1620756933397,
            "_system_owned": true,
            "_protection": "NOT_PROTECTED",
            "_revision": 1
        },
<snip for brevity>

[akhomyakov]

Actually, seems like terraform's "output" resource help here

output "ns_groups" {
  value = data.nsxt_ns_groups.NS-GROUPS.items
}

output "ns_services" {
  value = data.nsxt_ns_services.NS-SERVICES.items
}

but the names don't match what is seen in the GUI or even in the API

// API output shown in the OP.
// TF output
Changes to Outputs:
  + ns_groups   = {
      + "Edge_NSGroup"                                    = "7b98e78e-79ae-40b8-aaca-7b959ee47382"
      + "ServiceInsertion_NSGroup"                        = "e2e8ec10-bdaa-4a35-99d2-0d7727380bb0"
      + "default.INTDMZ"                                  = "d214f969-60f7-44c5-acd6-8b436639f606"
      <snip for brevity>
    }
  + ns_services = {
      <snip for brevity>
      + "ICMP Destination Unreachable"                                 = "63d122cf-47d6-44e8-bb58-0453d9bb6493"
      + "ICMP Echo Reply"                                              = "c54b2d86-6327-41ff-a3fc-c67171b6ba63"
      + "ICMP Echo Request"                                            = "5531a880-61aa-42cc-ba4b-13b9ea611e2f"
      + "ICMP Redirect"                                                = "82e6a462-431b-4e2d-8e5e-492a3461913b"
      + "ICMP Router Advertisement"                                    = "2255c641-fc57-449d-b973-e2da77353fe6"
      + "ICMP Router Solicitation"                                     = "93d11f70-6a67-46b9-bcb8-4761d0ed0148"
      + "ICMP Source Quench"                                           = "4f2e4cf1-09ff-4ba2-9f50-065bbe2f6b77"
      + "ICMP Time Exceeded"                                           = "76f12d8c-e9ea-4d08-bb6c-267be759756e"
      + "ICMP-entry"                                                   = "dde6a2fc-fe52-472e-8b10-44573dcbc1bb"
      + "ICMPv4-ALL"                                                   = "82e6a462-431b-4e2d-8e5e-492a3461913c"
      + "ICMPv6-ALL"                                                   = "82e6a462-431b-4e2d-8e5e-492a3461913d"

So is ICMP-ALL object visible in API and GUI returned by TF as "ICMP-entry"? Seems odd. And conversely, "ICMP-entry" is not a valid name that can be found in the GUI

<snip for brevity>
╷
│ Error: expected rule.0.destination_groups.0 to contain a valid IP, Range, CIDR, or Group Path. Got: d214f969-60f7-44c5-acd6-8b436639f606
│
│   with nsxt_policy_security_policy.SHARED-INFRASTRUCTURE,
│   on firewall-policy-SHARED-INFRASTRUCTURE.tf line 12, in resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE":
│   12:     destination_groups = [data.nsxt_ns_groups.NS-GROUPS.items["default.INTDMZ"]]
│
╵
╷
│ Error: Invalid policy path: dde6a2fc-fe52-472e-8b10-44573dcbc1bb
│
│   with nsxt_policy_security_policy.SHARED-INFRASTRUCTURE,
│   on firewall-policy-SHARED-INFRASTRUCTURE.tf line 14, in resource "nsxt_policy_security_policy" "SHARED-INFRASTRUCTURE":
│   14:     services           = [data.nsxt_ns_services.NS-SERVICES.items["ICMP-entry"]]
│
╵
<snip for brevity>

[annakhm]

Hi @akhomyakov, unfortunately these data sources are based on old-style MP API, not policy API, so they won't do what you expect. These were added for a specific customer that cannot use policy API due to very specific feature set they are using.
Are you looking to improve performance? Otherwise you can just use a singular data sources nsxt_policy_service, nsxt_policy_group for same purpose?

[akhomyakov]

I see. This suggests that these two will never come out of beta. I am looking for a way to import all existing service objects and groups rather than import them one by one. In other words, NSXT comes with a long list of prebuilt services to be used in the security policy like ICMPv4 and so on. Is there a sane way to import them all in one go or it's a one by one type of endeavor only?

[annakhm]

For now there are no parallel data sources for policy APIs, but seems like people are asking for it. So these are on the roadmap. For now though prebuilt data sources need to be imported one by one.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[akhomyakov]

bump