That's right. And, since their hp is already low, it won't hurt much to make it a bit lower.
After a due consideration, I went -4 CON +4 DEX, so atm my soulbreaker is at -5 CON (got -1 con due to PL set), and I don't see much difference.
Yes I get stunned by mobs a lot, but I'd expect that in any case.