Probably still not the best design but hopefully break fewer UX rules:
1. Use red to confirm delete rather than cancel and green to cancel
2. Show the action you're about to perform in the confirmation
3. Label confirmation button with the action rather than yes/no.