Self-hosted messaging was solved decades ago, it's called XMPP or Jabber depending on who do you ask. There are a number of services for it, such as Prosody or ejabberd, and there are both client apps (eg.: Conversations for Android, Gajim or Dino for PC) and a number of webclients (such as converse.js which can actually be shipped via a ejabberd install).
That said you could just set up an alternative Signal client that allows for custom servers, since the centralization of Signal only really exists at the level of the client only connecting to Signal's servers.. Signal's people have pointed this out themselves, when offering bridge setups so that people could connect from Iran or other countries where those countries blocked Signal's endpoints.