Good idea to put holes in the field fence. The hogs may even be able to get into other gardens from the field direction.
Hogs do sometimes sniff around hog houses when there is another one in there, either eating, sleeping or hibernating. Normally if a hog is hibernating, the hog house would be fairly full so another hog may not be able to get in anyway. But even after most hogs have hibernated, hoglets sometimes turn up – i.e. maybe where they came from stopped feeding, so they are exploring further afield.
Good luck with the hog tonight.