Ctrl + 1 is pet actions
the rest you can modify with alt or shift on your mouse and you have tons of bindings for whatever spells you want. 1 through = on the keyboard
whenever i need to move in a direction and cast those pet actions, i just hold LMB + RMB and press them. or jump strafe in the direction. You're right though, it's not ideal. But it works.