=Ok so I think I may have figured it out, but cannot test it until I get home.
I think it has to do with Mystcraft and "Instability". You get debuffs from certain Mystcraft Wolrds that have these Instabilities.
Hunger just happens to be one of these and Fatigue also.
I think they stay with you...