Have I [26F] overreacted to this situation with my boyfriend [27M] and his coworker [23F]??

We've been together for nearly 5 years now and we have a 9 month old baby. Like all long term relationships we have up and downs but ultimately we love each other and we do plan to get married when we are more financially stable.
Having a baby was not planned and it has sometimes been difficult, i'm back at work now so we're both always exhausted and tired people can be irritable. We don't have sex as often as we used to but I really try to make an effort, though I still don't look like how I used to before my pregnancy.
So a couple months back this new girl started at his work and he she lives in the neighborhood so my boyfriend offered to give her rides to work. So they spend every day together in the car and honestly, I feel like she gets to spend more time with him than I do. I can see that they're friends on facebook and she tags him in things all the time. I've never actually met her but I've stalked her facebook and she's super pretty and young and definitely isn't carrying baby weight.
Yesterday I found a woman's jacket that wasn't mine in the house. I asked my Bf about it and it was her's. I got pretty mad that he invited her round without telling me and did it when I wasn't home. He looked guilty when he told me, like he purposefully did it when he knew I was away and wasn't going to tell me.
It really upset me that he was hiding it from me and he promises nothing happened and she has a boyfriend and its not like that, it still feels sneaky. I do want to trust him and i do believe that nothing so far has happened but I can tell he's attracted to her. He also said that was the only time but i'm not so sure.
So I got mad and I banned him from having her over when i'm not there or to at least tell me when she's coming. Is that unreasonable? Am I overreacting?
He thinks i'm overreacting. I know part of it my own insecurity because i'm unhappy with how I look right now and she is like how I used to look before having a baby so I could be just being oversensitive and jealous. But the though of him sneaking around with a younger, prettier version of me and deliberately hiding it from me makes me feel like ****
tldr: my bf has become friends with a his female coworker and spends a lot of time with her and invited her over when I wasn't there and was hiding it from me. Have I overreacted by banning him inviting her over when i'm not there?


Source.