How to turn down an invitation when invitee was clearly an afterthought?

My 5 yo son has a female clmate who is having a birthday party tomorrow to which my son was not invited (not a big deal). As we were leaving school some of the kids were talking about the party and my son overheard. The girl's mom was standing right there and stammered an awkward invitation to my son and I. I told her that I didn't think it would work but had to check my calendar and get back to her. My son really is not bothered by this, and obviously we were all in an awkward situation. We are going to decline the invitation, but there is a part of me that would like to tell the mom (diplomatically, of course) that the last minute invitation was completely transparent. I did explain to my son when he came out talking about the party that people can only invite so many people, that we have fun things planned for tomorrow, etc. (I didn't want the mom to feel awkward about it). I am sort of annoyed because this girl mentioned to my son early on that he was invited to her party but then never actually got invited. Should I just leave well enough alone and say, "Thanks but no thanks" or I was thinking of something like, "We have XYZ going on and with such short notice we just can't make it- thank you for thinking of us." Thoughts?