A reverse proxy will achieve what you’re looking for, yes. I do precisely what you’re describing. I use “local DNS” on my pihole, with CNAMEs for each service pointing to my server IP address. I’m running Caddy on the server, specifically this (because my services are running in Docker containers): https://github.com/lucaslorentz/caddy-docker-proxy
As far as a “dashboard” app, you have options there as well. I’m using Heimdall currently, but Homarr also looks pretty nice.
OP listed two solutions they were considering, one of which explicitly mentioned using a reverse proxy. The other mentioned using a “dashboard app”. Commenters have given useful advice regarding both.