Any way to have yatse be just a remote control for jellyfin?

You just need a player, Kodi can be a player, Wiki • Yatse: Kodi remote control and cast explain how to use Yatse to connect to your Jellyfin server and play on any renderer.

See also Yatse: Cast - YouTube

There’s no need to remote control to play things it’s faster to directly use Yatse.