AzureKeyDetector plugin (updated from AzureStorageKeyDetector)

[satonaoki]

I updated AzureStorageKeyDetector to AzureKeyDetector.

AzureKeyDetector supports various Azure services in addtion to Azure Storage (supported by AzureStorageKeyDetector).

Here are supported Azure services:

• Azure Storage
• Azure SQL Database
• Azure Database for PostgreSQL
• Azure Database for MySQL
• Azure Database for MariaDB
• Azure Cache for Redis
• Azure Cosmos DB
• Azure Synapse Analytics
• Azure Service Bus
• Azure Event Hubs
• Azure IoT Hub
• Azure Monitor
• Azure Functions
• Azure Web PubSub
• Azure SignalR Service

azure.py has RegEx expressions for various key/connection string format of various Azure services.

azure_test.py has corresponding payload/should_flag pairs for RegEx espressions/

azure_test.py runs succesfully.

$ python -m pytest tests/plugins/azure_test.py
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-0.13.1
rootdir: /mnt/c/repos/detect-secrets
collected 27 items                                                                                                                                                                                                  

tests/plugins/azure_test.py ...........................                                                                                                                                                       [100%]

================================================================================================ 27 passed in 4.85s =================================================================================================

However, when I scan azure_test.py, detect-secrets detects 6 secrets only. I think it should detec 27 secrets. I don't know the reason why 21 secrets are not detected. I'd like to find the reason in this thread and fix it (if needed).

$ python3 -m detect_secrets scan tests/plugins/azure_test.py 
{
  "version": "1.3.0",
  "plugins_used": [
    {
      "name": "ArtifactoryDetector"
    },
    {
      "name": "AWSKeyDetector"
    },
    {
      "name": "AzureKeyDetector"
    },
    {
      "name": "Base64HighEntropyString",
      "limit": 4.5
    },
    {
      "name": "BasicAuthDetector"
    },
    {
      "name": "CloudantDetector"
    },
    {
      "name": "GitHubTokenDetector"
    },
    {
      "name": "HexHighEntropyString",
      "limit": 3.0
    },
    {
      "name": "IbmCloudIamDetector"
    },
    {
      "name": "IbmCosHmacDetector"
    },
    {
      "name": "JwtTokenDetector"
    },
    {
      "name": "KeywordDetector",
      "keyword_exclude": ""
    },
    {
      "name": "MailchimpDetector"
    },
    {
      "name": "NpmDetector"
    },
    {
      "name": "PrivateKeyDetector"
    },
    {
      "name": "SendGridDetector"
    },
    {
      "name": "SlackDetector"
    },
    {
      "name": "SoftlayerDetector"
    },
    {
      "name": "SquareOAuthDetector"
    },
    {
      "name": "StripeDetector"
    },
    {
      "name": "TwilioKeyDetector"
    }
  ],
  "filters_used": [
    {
      "path": "detect_secrets.filters.allowlist.is_line_allowlisted"
    },
    {
      "path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies",
      "min_level": 2
    },
    {
      "path": "detect_secrets.filters.gibberish.should_exclude_secret",
      "limit": 3.7
    },
    {
      "path": "detect_secrets.filters.heuristic.is_indirect_reference"
    },
    {
      "path": "detect_secrets.filters.heuristic.is_likely_id_string"
    },
    {
      "path": "detect_secrets.filters.heuristic.is_lock_file"
    },
    {
      "path": "detect_secrets.filters.heuristic.is_not_alphanumeric_string"
    },
    {
      "path": "detect_secrets.filters.heuristic.is_potential_uuid"
    },
    {
      "path": "detect_secrets.filters.heuristic.is_prefixed_with_dollar_sign"
    },
    {
      "path": "detect_secrets.filters.heuristic.is_sequential_string"
    },
    {
      "path": "detect_secrets.filters.heuristic.is_swagger_file"
    },
    {
      "path": "detect_secrets.filters.heuristic.is_templated_secret"
    }
  ],
  "results": {
    "tests/plugins/azure_test.py": [
      {
        "type": "Azure keys/connection strings",
        "filename": "tests/plugins/azure_test.py",
        "hashed_secret": "43cd99b89520080a5e7c1349235765d70ef15274",
        "is_verified": false,
        "line_number": 12
      },
      {
        "type": "Azure keys/connection strings",
        "filename": "tests/plugins/azure_test.py",
        "hashed_secret": "fd4eebe5e7bdb850e8e6471a4186f506324687f0",
        "is_verified": false,
        "line_number": 28
      },
      {
        "type": "Azure keys/connection strings",
        "filename": "tests/plugins/azure_test.py",
        "hashed_secret": "b27ea2fb585a210fbb607a675b71abe228db1c7d",
        "is_verified": false,
        "line_number": 60
      },
      {
        "type": "Azure keys/connection strings",
        "filename": "tests/plugins/azure_test.py",
        "hashed_secret": "7b162df4c3e8682cd7d6e6815376f4ee6903e41a",
        "is_verified": false,
        "line_number": 68
      },
      {
        "type": "Azure keys/connection strings",
        "filename": "tests/plugins/azure_test.py",
        "hashed_secret": "8ca69981e32198203d53f5a9dfcad52aa9846381",
        "is_verified": false,
        "line_number": 72
      },
      {
        "type": "Azure keys/connection strings",
        "filename": "tests/plugins/azure_test.py",
        "hashed_secret": "e7722ae3eaa895362ea52b89e5a4d3b6d5440ce2",
        "is_verified": false,
        "line_number": 108
      }
    ]
  },
  "generated_at": "2022-08-05T03:19:07Z"
}

[lorenzodb1]

Hi @satonaoki, thank you for opening this PR. I appreciate your work on improving our Azure keys detector.

I ran our tests and it looks like they're failing, so I encourage you to take a look at those before we review your code. Let me know if you have any questions.

I just merged #598, which fixed the probable cause why tests are failing. I suggest you update your branch and we'll run tests again.

[satonaoki]

Hi @lorenzodb1

I rebased my branch from Yelp:master.

The reason of test failure is that the old plugin name AzureStorageKeyDetector is included in .secrets.baseline. I updated it to AzureKeyDetector. Now tests succeeded.

$ python -m pytest tests
...
====================================================================================== warnings summary =======================================================================================
tests/core/baseline_test.py::TestCreate::test_error_when_getting_git_tracked_files
  /mnt/c/repos/detect-secrets/detect_secrets/core/scan.py:74: UserWarning: Did not detect git repository. Try scanning all files instead.
    log.warning('Did not detect git repository. Try scanning all files instead.')

tests/core/usage/filters_usage_test.py::test_disable_filter
  /mnt/c/repos/detect-secrets/detect_secrets/core/usage/filters.py:192: UserWarning: Redundant --disable-filter "blah"
    log.warning(f'Redundant --disable-filter "{name}"')

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================== 972 passed, 1 skipped, 6 xfailed, 2 warnings in 265.23s (0:04:25) ==============================================================

[satonaoki]

@lorenzodb1 Any update/comment?

[jpdakran]

@satonaoki Hello. It seems the CI is still failing. Can you checkout the errors and fix them?

[satonaoki]

@jpdakran I added pragma allowlist comment to resolve the CI issue. Could you approve the workflow?

1 workflow awaiting approval
First-time contributors need a maintainer to approve running workflows. [Learn more.]

[satonaoki]

@jpdakran I resolved another CI issue. Could you approve the workflow?

[lorenzodb1]

@satonaoki looks like some checks are still failing. I encourage you to take a look at those.

[satonaoki]

@lorenzodb1 "W291 trailing whitespace" error fixed. Could you approve the workflow?

detect_secrets/plugins/azure.py:78:94: W291 trailing whitespace
detect_secrets/plugins/azure.py:96:79: W291 trailing whitespace
detect_secrets/plugins/azure.py:97:92: W291 trailing whitespace

[satonaoki]

Hmm. CI still has these two errors.

But it seems that azure.py has no issues for add-trailing-comma:

https://github.com/asottile/add-trailing-comma/blob/main/README.md

I will run CI in my forked repo and take a close look at it.

Add trailing commas......................................................Failed

• hook id: add-trailing-comma
• exit code: 1
• files were modified by this hook

Rewriting detect_secrets/plugins/azure.py

autopep8.................................................................Failed

• hook id: autopep8
• files were modified by this hook