Package that utilizes pdm as build backend cannot be debugged with breakpoints in PyCharm

[CaptainDriftwood]

I use PyCharm as my IDE of choice, and one thing I have noticed is that packages built using the PDM backend, that are installed either editable or as a regular dependency, will not have any associated breakpoints get hit with the PyCharm debugger. Regular packages installed via PyPi work as expected (as well as std lib packages / modules). Could this be something to do with how my pyproject.toml file is structured? As an example, here is what my pyproject.toml looks like:

[project]
name = "my-package"
description = "My package"
version = "0.1.0"
authors = [{ name = "My name" }]
requires-python = ">=3.9"

[tool.pdm.dev-dependencies]
dev = [
    "boto3",
    "moto",
    "pytest>=6.2.2"
]

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

[frostming]

I use VSCode and everything works fine, it has nothing to do with the build backend.
Debugger looks for file at the sources configured so make sure the files are installed correctly to the desired location.