skip duplicate templates

[dogancanbakir]

Proposed changes

Closes #4690.

dir:

$ tree custom-templates 
custom-templates
├── custom-temp-1.yaml
└── custom-temp-2.yaml

1 directory, 2 files

custom-temp-1.yaml:

id: custom-temp-1

info:
  name: custom-temp-1
  author: pdteam
  severity: info

http:
  - method: GET
    path:
      - "{{BaseURL}}"
    matchers:
      - type: status
        status:
          - 200

custom-temp-2.yaml:

id: custom-temp-1

info:
  name: custom-temp-1
  author: pdteam
  severity: info

http:
  - method: GET
    path:
      - "{{BaseURL}}"
    matchers:
      - type: status
        status:
          - 200

$ go run . -u hackerone.com -t custom-templates -v

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.0-dev

                projectdiscovery.io

[VER] Started metrics server at localhost:9092
[WRN] Duplicate ID 'custom-temp-1' found in '../custom-templates/custom-temp-2.yaml'; skipping.
[INF] Current nuclei version: v3.2.0-dev (development)
[INF] Current nuclei-templates version: v9.7.6 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 49
[INF] Templates loaded for current scan: 1
[WRN] Executing 1 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[VER] [custom-temp-1] Sent HTTP request to https://hackerone.com
[INF] No results found. Better luck next time!

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[Mzack9999]

If #4867 will be approved I think we can consider using the cache within catalog.ExecuterOptions since it should contain a map with O(1) access to templates (I'm not sure if we store by id or path).
Also what do you think about using some hashing algorithm or text similarity? This might help as well to spot similar existing templates in nuclei-templates repository before committing (I think we might have some duplicates already doing the same things)

[ehsandeep]

This should be also optional, as removing as default means any users with custom version of public CVE template won't be able to run public templates along with custom one, additionally we can display counter of loaded templates with duplidate template ids as indicator as default.