Don’t worry – you haven’t done anything wrong. Hogs will usually build a different nest for hibernating than the one they have been sleeping in over-day. So he may have gone to hibernate elsewhere. You are very lucky that a hog used your box at all so soon after you got it. Some hogs take years before they accept a hog house! They do tend to move nests during hibernation time, so you never know, your house may come in handy then. Otherwise, it will offer a hog the chance of nesting place next year, which is good.
Although you said you believed it was empty. So I suppose there is a slight chance that a hog is in there hibernating?