Nightmare can be replaced by Demon for PvE, DC can't. For the olympiad, I'm not sure whether having extra sleep resistance wouldn't be worth it though.
Imho, for pvp as a scorc you need speed. We really only have one heavy nuke, so we need that to recharge quickly. When I had demon set on, I couldn't win in pvp cause I could never cast anything before my enemy interupted it. With karmian and homu w/acu I started to win since I could actually get spells off.
For pve hunting, sure NM will work better then DC for the simply matter of increased damage per nuke. However, as the previous poster pointed, demon is a great substitute for it. The +4 int of demon is just..... wow, especially considering the price difference between NM and demon.
PvE:
Demon > Nightmare (damagewise), so always go DC and have a Demon with you to wear if it can save you a shot.
PvP:
Hmmm, DC makes u cast very fast, but with NM you have a very good sleep resistance, that might save you against other mages.
A Sorc in NM would win against a Sorc in DC any day. Against all classes that don't have sleep/hold, DC is better.
Also consider that the NM robes give 257 MP and 35 more pdef than the Demon set. The little bit of matk the Demon set gives over the NM, imo, isn't worth it.
With full buffs (Empower, Zerk, Dance of Mystic) the difference in matk beween DC and NM is around 300. With an Acumen weapon and Acumen buff, casting speed in NM is still decent.
I don't PVP a lot and I'm mostly in parties. For me, NM is the the best choice.