If a pack is trimmed and configured for a particular feel, adding or removing mods can and will affect the pack. For instance, dropping mods into (or removing mods from) a pack such as Regrowth will have dire consequences in regards to the atmosphere and balance of the pack.
On the other hand, adding or removing mods from Infinity (non-expert mode) happens all the time. Any large(ish) pack with no overarching theme or goal can be, and tends to be, a good starting point for people to tailor the pack to their own play preference.
The obvious drawbacks include not being able to auto update your new pack, screwing with any existing scripts (Expert Mode, for instance), and the inability to connect to servers who haven't done the exact same changes as you. If you're OK with these, there really isn't much of a problem.
If you have second thoughts, make a backup before adding your mods (always highly recommend practice anyway). Then, if down the road you decide you made a bad choice, revert to before you added and move forward from there.