If I limit my FPS my frames drop even lower than 60, which I don't want. But I'll give it a shot without Optifine!
EDIT: It seems to be a random occurrence that I can't replicate, so I can't really expect anyone to solve it. When it happens, it a restart appears to fix it, so I can live with it...