No you shouldn't feel guilty. Maybe if you were supposed to be at work, and you called in sick, then that's a reason to feel guilty, or if you're like me, you wouldn't give a shit either way. It was your day off man and you should enjoy it, so fuck them.