Hard to say... I have a lot more stuff than that, with a bunch of automated crucibles, sieving, a mob tower running all the time with a small sorting system attached to it, a bunch of farms, lots of animals etc... No lag ^^.
Have you tried using Opis to find the cause? Did it lag right from the...