Add websocket API to get list of recorded entities

[bdraco]

related thread https://community.home-assistant.io/t/2023-5-lets-talk/567331/273?u=bdraco

Proposed change

frontend PR home-assistant/frontend#19621

In #90992 we now require a list of entity ids to be passed to history to avoid the cases were systems were getting overloaded because the default was to return everything and that behavior was no longer viable (see the issue queue) now that most systems have hundreds of entities.

Some users have expressed a desire to know which entities are being recorded so they can feed the list back into the history APIs to achieve what they were doing before.

This change adds a websocket API to get list of recorded entities. It may be better to implement this as a REST api since we do not generally document websocket APIs, but I did it as a websocket API since we have generally stopped adding new REST apis except for large binary data.

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of recorder can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign recorder Removes the current integration label and assignees on the pull request, add the integration domain after the command.

[alexarch21]

Very interesting ! Going to try this out asap.

[alexarch21]

Finished my testing of this patch on Core 2023.5. This works really well and behaves exactly as expected.

I really hope this will be merged into Core. Not only will it restore functionality lost with #90992 in a much more flexible way, it also provides a very low latency and resource friendly way to retrieve the recorded entity list. This feature is very useful for frontend cards that provide some kind of UI to let users select entities based on recorder inclusion, like for example alternative history panels. Another use case is the automated creation of history graphs based on user supplied patterns. As an example, it will make alexarch21/history-explorer-card#162 possible in a way that was not possible before (due to the latency of the required database fetch).

[roykrikke]

I hope this patch will be part of Core. As @alexarch21 wrote, this is really helpful!

[bdraco]

This is probably useful for populating the history panel and editor with the list of entities that are actually available in history instead of showing everything.

We should work that into the frontend to see if the api as I threw it together makes sense for that use case before moving this forward

[github-actions]

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days.
Thank you for your contributions.

[bdraco]

Since the plan is to use this on the frontend now this should be ready to go.

cc @karwosts

[karwosts]

Can you elaborate a bit how this selects entities? It's not quite working how I expected, though I might be missing something.

When trying to test no matter what I add to recorder excludes, it doesn't seem to change the value returned. If I add an entity to recorder exclude (and restart), it is still returned in this list. If I additionally purge that entity, it is still returned in this list. If I add a domain to excludes, entities from that domain are still returned in this list.

Do I need to flush some metadata, or repack the database, or anything like that to make an entity fully "nonrecorded"?

[bdraco]

You need to remove all history for the entity from the database and restart to make it go away after excluding it

[karwosts]

Ok. I guess that's a good argument for keeping the interface this way.

[karwosts]

What's somewhat interesting about this is it's finding a lot of junk that I had no idea was still there, it's returning me entities with no history records that I deleted months ago that are mentioned nowhere else.

Not sure if that's addressed fixed by your earlier bugfix, but it's interesting.

Was sort of prototyping a devtools panel that could show this information, not sure if that's too useful or not yet though.

[bdraco]

I think thats super useful. Users ask how to see that data all the time

[karwosts]

Frontend change approved to consume the API, now waiting for backend.

[github-actions]

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days.
If you are the author of this PR, please leave a comment if you want to keep it open. Also, please rebase your PR onto the latest dev branch to ensure that it's up to date with the latest changes.
Thank you for your contribution!

[karwosts]

unstale

[home-assistant]

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

[emontnemery]

LGTM, thanks @bdraco 👍

[bdraco]

Thanks