If you want complex magic system where you create your own spells try Noita. It's a rogue like where you can mix and match spell components to create better magic. Nothing on the non-combat front though.

deleted by creator

Kinda like Space Engineers makes building any sort of vehicle super complicated because it's all manual voxel placement and combination. It'd be neat to build a game with a fancy magic system though I don't know what you could do with the magic system without building an also complex single player magic user simulation type thing unless it's just a sandbox type game (at least to start).

I think Dungeon Master tried something like that back in the 80's. I ended up getting the SNES version but never figured it out because I didn't have the manual.

Magicka taught me that magic isn't fun without the ability to disastrously fuck up.

I want to say one of the old Ultima games had something like this, where you had to "draw" multiple runes with your mouse to make the "sentence" that casts a spell.