Finally got around to writing the code, here's the link to the Rust playground with it. I was too lazy to do the unsafe stuff, but where I call clone in the deny and accept functions are where you'd rewrite and do unsafe magic to swap out elements without the clone overhead.
I also tried to implement a take function (like in Option) for the Allowable enum. Transferring ownership like with take would keep you in safe land and get what you want done.
Keep in mind I am not a game dev. I don't do cache optimizations. I have no idea how this code would function in a game context, but this is just my first idea for implementation of what you asked.
How is it a code smell? I'm pretty sure interior mutability is a well established design pattern. I'm not really familiar with the pattern you're describing using Ref/RefMut wrapping, could you show me what you mean?
Yeah I didn't want to pour a bunch of time into wrangling borrow checker stuff for a small code snippet.