Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tracking issue] Add raw_training_scoresaccessor function #960

Open
10 tasks
ablaom opened this issue Aug 19, 2022 · 0 comments
Open
10 tasks

[Tracking issue] Add raw_training_scoresaccessor function #960

ablaom opened this issue Aug 19, 2022 · 0 comments

Comments

@ablaom
Copy link
Member

ablaom commented Aug 19, 2022

To address #841 (comment) I suggest we add a raw_training_scores accessor function, following the established pattern for feature_importances and training_losses.

@davnn Are you happy with the name of this function? I'm thinking here of consistency with training_losses.

Here are the steps (adapted from #747).

Step 1. Prepare a bunch of PR's to be rolled out after testing, all called scores:

  • Add reports_raw_training_scores trait to StatisticalTraits, defaulting to false, as here for feature_importances (but need to also add to list of all traits at top of file)
  • Add raw_training_scores(model, fitresult, report) stub to MLJModelInterface (in model_api.jl), as here; fallback to return nothing.
  • Overload MLJModelInterface.raw_training_scores(mach::Machine) following this pattern and re-export raw_training_scores (the reports_raw_training_scores trait will be automatically re-exported in MLJBase and MLJ - no action required)
  • In MLJ: re-export raw_training_scores

Step 2

  • @blaom to check MLJTestIntegration runs with no new surprises (and maybe adds some detector-specific testing)

Step 3

  • Merge and rollout the scores PRs and tags new releases, taking care to bump compats accordingly. Probably @ablaom to do.
  • Update MLJ model API docs

Step 4

  • Roll out implementations for packages that already report
    scores (in, eg, their report); @davnn to complete this checklist:
    • OutlierDetection.jl
    • ?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant