Leaving a holiday gathering is a three step process.
First, you state that you really need to get going.
Second, everyone moves to the entryway, where you visit for another 15 to 30 minutes. You repeat that you really should be on your way.
Third, everyone moves to the yard/driveway where another 15 to 30 of conversation takes place.
Only then do you get in the car and go home.
Once you understand this process, you can plan for it and time your initial "We really should be going" accordingly.