I lacked alot of SP as well.. but I over hit all the mobs..
My suggestion is to take important skills first, like new summon levels, cubics, cures/buffs for summons.. leave robe/light mastery, weapon mastery for when you have spare SP. I haven't invested one point in Light armor masteries past level 3 (subclass, got it free!), since I plan to use robes all the way.
Aye, servitor recharge seems completely useless for PS, never put in SP for that either. Robe mastery is the last I get usually, if I can manage to get myself +6 BW light set I'll consider going light, else I'll stay in robes.
I usually learn servitors except Silhouette first (I never use her), then cubics, then weapon mastery, and then if I have MP spare wind shackle, servitor heal and the masteries (they usually come later in the skill level bracket).
Silhouette, Soulless, and Nightshade are absolutly nessesary. It is shadow who is basically useles. I will sometimes leave him out when I am low on SP. Also, as said before, recharge is a waste, don't buy it. Silhouette is used for PvP, she is completely nessesary unless you are some kind of carebear. Soulless is you main grinding summon, only takes 10% and can do massive damage fast. Nightshade is your party healer/mass hex+gloom summon.
And the armor masteries depend on your play style. Some people go robe for mana, I go light for the defense, which allows me to do more attacking (thus using my cubics) and less sitting on the bench, watching soulless kill mobs by himself.
I've honestly not been in a situation where I would have been wanting for the silhouette - the shadow is neccessary for me because I like grinding 'relaxedly' sometimes. He is simply infinitely more sturdy than soulless - also helps in paralyse areas, as he can keep going with ease hwen I am paralysed.
I've never found the incredibly slow and not particularly powerful nuke on the silhouette worth calling her out for.
Robes is, by the way, not for mana but for cast speed. Robes do not at all mean 'sitting on the bench', a summoner in robes is quite sturdy enough.
I tank mobs fine in robes, though I have VR and lots of buffs available.. I get low wometimes, then I just battleheal and CLD myself up if VR doesn't cut it.
This is common just try not get all the skills, oncemany of them a pretty much useless.
Things like, if you use light armor why you keep getting robe passives? Save the sp, unless that you're so undecided of what you will use.
Try stick with the basic skills, the one's you REALLY need, when the sp starts to be easy to get you start to spend it on the useless skills. But for now you can see it's not the case.