You really need to wait until the hog has vacated – by which I mean properly, not just gone out for the night.
There is some advice here: https://www.hedgehogstreet.org/cleaning-out-boxes/
If you get another box the hog may decide to move into that one instead which will give you the chance to clean out the old one.
You say that all your returners are males, which is quite normal. The males tend to go into hibernation earlier than the females and return earlier. Apart from anything else, they don’t have any hoglet rearing duties to perform in the Autumn, as the females do. The females can sometimes return as much as 2 months later. Everyone needs to bear in mind when thinking of cleaning out boxes that a female might still be there hibernating quite a while after some other hogs have returned.