I think I need to split this into two PRs.

1. Refactor the setup (to make it easier extendible.)
2. Add the repo support

I was thinking of about the same structure of the grafana role, but keeping the possibility to install via downloaded binary. (Which devop or sysadmin ever will install software like that.)
@ishanjainn @v-zhuravlev are there any requirements of non breaking changes etc?

The more I get into the role, the more I'm questioning how that thing even works and the more confused I get.
There are so many issues in installing the agent like that.

1. It uses a different systemd unit as upstream install via a repo
2. It does a preflight check if _grafana_agent_system_dir is available?!
3. It has a loop that install/directories.yaml is executed twice
4. The download of the binary is made to the localhost (more useful on the host where it is gonna be installed, otherwise you have to upload it to the host again)
5. Configuration is only working for static agent (config is spliced into several variables)
6. The role itself is very complex
7. No tests
8. Not tested: The role doesn't seem to be idempotent it will always try to install the newest version. It always has to download the binary and overwrite it, even when the version is fixed.
9. http/grpc ports are different from upstream packages (random port: 12345)
10. Add support for ansible-playbook serial > 1 (manage multiple hosts at a time) #153
11. Implement templating of grafana_agent_provisioned_config_file #145

This list is not complete yet I guess.

All of my statement are of course rather opinionated. I'm open to discussion. You can find my progress in the PR.

Maybe we should also discuss if a support for this kind of manual install is still necessary. As it could be quite some work keep it up to date with the package install, which I guess is done by another team within grafana.

Sorry got a bit carried away today haha

really really looking forward to your final PR 🚀🚀