My husband insists I broke a promise

TL;DR: My(34F husband(45M) of 9 years is threatening to leave me and our 3 kids unless I give him 18000$ to spend on a garage for our new house we just bought together.
Me(34F) and my husband (45M) have been married for 9 years and we've recently bought a house together. I'm from Europe, he's from Japan and We've been living in my home country for a decade. We've been renting a house for 9 years and in that time we've had three kids aged 3, 6 and 8. The house we've been renting has only 2 bedrooms so our youngest have been bedsharing with us while the oldest two shared a 8 square meter bedroom. It has come to the point where that option is becoming impossible, and we have only one toilet and the bathroom is in the cellar. I've been wanting to move ever since we knew baby #3 was on the way, the house is too inaccessible and also, the landlord has promised maintenance for years but never come through, so the balcony railing is falling apart and water is leaking into the cellar through the wall. My husband, however, initially wanted to buy that rental house, solely because of the garage(his own words), which is very well kept, as it used to be a workshop garage. I've said no because not only would we be buying ourselves a major maintenance project, we'd still be two bedrooms short.
Ever since the house next door went for sale, I've been talking about it. It's a huge house, very well kept and it has a 100 square meter garage/shed, well landscaped garden, a functional kitchen with great space, two full bathrooms one in each main floor and 5 bedrooms total! So I thought it was perfect for us. But my husband was on the fence, and wanted to wait until something better came along. It should be mentioned we live in a small town and this house is as good as it gets, both location wise and value for money-wise. And I just couldn't stand the thought of living in the rental one more year. I've felt like a prisoner in that house, because the doors are narrow, there's so little windows, it sits on a hill and it's drafty. And the room plan is awful. I basically said to my husband: "Let's buy the house, live there for 5 years then sell it and use the money to move closer to the city"(which is our ultimate goal but all we can afford is a house within this town). He agreed, but on one condition: That we get a loan that's higher than the house price and then use that money to renovate that 100 square meter garage. I agreed because I had been under the impression and advice that that was something we could do with a house loan(basically a family member did just that and renovated the house). All was well, we bought the house and signed the contract. Then we got the word from our bank that they wouldn't let us loan a single penny more than the house price and fees. So our plan and his condition fell through. It was a huge disappointment but now my husband has been blaming me, saying I've broken my promise to him and that he only agreed to buy the house because I had promised him we'd spend 18000$ on the garage. I am shocked, because I certainly haven't agreed to spending 18000 quid on a garage(that amount would be insane, we wouldn't need even close to half of that for the renovations needed). He was livid when we found out, and immediately said I had broken my promise to him. To end the argument I said I'd "fix it" by getting my stepfather who works in architecture to help calculate a budget and we'd take the finances off our income instead.
Yesterday we got the keys to the house, today we started moving our stuff. My husband greeted me with "so when are you going to fix the promise you broke?" and going off on me, he actually gave me a week to take up a loan for 18000$ for him to spend solely on that garage, or else he'd refuse to move with me and the kids and he'd just stay in the rental. Throughout these 3 months since we bought the house he hasn't packed a single thing, stated that he would pack only his clothes and all he's done this past week is move his motorcycles into the garage/shed of our new house. I've been completely alone in packing and I've had to do it while working full time and taking care of the kids. There's still a lot to pack and move. I got my family, who lives nearby, to help me today but my husband should be participating as well.
There's another factor in play here: in order to take up a house loan, my husband had to start working, as we couldn't make it on only my income. He REALLY didn't want to get a job, so I had to persuade him, write his resume, his job application and arrange for his interview. He can speak my language but relies heavily on me because I speak his language fluently. I now have to be a mediator between him and his boss(who is a little slack around union laws), and I'm also the designated parent doing literally all of the communication regarding school/kindergarten/healthcare for our kids. And now he is working, it's really rough hours, so he's hardly ever home and we haven't spent family quality time, let alone couples quality time, for over 6 months. But now we HAVE the house loan he absolutely can try and find a new job with more family friendly hours, but this he refuses. He also will hold it over my head every time I mention his work hours(we knew the working hours upon him starting the job, but we agreed that it was worth a shot and that he'd be able to change jobs after we'd bought the house). Now he's turning it around to it being me who "forced" him into taking that job and therefore I can't complain.
What hurts the most now is that he is so extremely cold and distant towards the move, and the fact that he's giving me threats of leaving me over a ****ing garage. It sounds so insane, I just need some advice. How can I fix this? I don't feel like I've misled him in any way, I also don't feel like this was a promise I specifically made to him, just that we were both led to believe that the money would be there, so of course I happily agreed to his "condition", because I want him to have a real man cave as I know it's important to him. I have been very adamant on wanting that specific house, though. But I've sure as **** put my wants and needs on hold so that he could do his thing, too. And I have not and would never hold it against HIM. It hurts me that he so easily makes this ultimatum, because that, to me feels like a betrayal. It makes me feel like he's already emotionally left this family. At the start of our marriage he did threaten to divorce me when we had our first fight, but when we talked it out I had him promise me not to throw around the divorce card during a fight. But he did that today. But would he really leave me and his three kids over a goddamn man cave? Is that a real thing?! I appreciate any advice.
Update: I got to talk with him after he got off work now, it seems he iwas set on the 18000$ mark partly because that was the amount we were led to believe we could get extra on the loan(at least what was initially discussed) but also he genuinely thought that that amount was something I had promised him we'd spend on the garage and also he thought that's the amount that would be necessary. I was able to explain that we'd need AT MOST 1500$ to do the changes he wants, based upon my extensive research into the matter, and that that money can be taken from our finances without us going bankrupt, if we do it little by little, which we would have had to do anyway. So that put the fire out somewhat but he's still not coming away from him thinking I broke a promise and he obviously sees that as a huge betrayal. I tried telling him that what I promised was that we'd renovate the garage, and that promise still stands, but the circumstances around where the finances would come from changed because of reasons out of our control. He accepted that eventually and he has backed down from the strict 18000$ limit. I did ask him if there was something specific he needed that specific amount on but that wasn't the case(I don't think he would have backed down if it was). I also did tell him that him telling me he'd leave us if I didn't give him the 18000$ was very hurtful and I've spent all day worrying if he'd meant it. He responded by saying that HE'D never specifically used the word "divorce", I was the one who said that and why would us living in two different locations mean we'd be divorcing, to which I replied "How are we going to continue living like a married couple if were neighbors? He also insisted that he money for the renovation of the garage should come from my account. I told him basically that jokes on him, we have joint finances so everything that goes from my account goes from his finances, lol. I do think we'll be seeking out couples counseling as this is not a good situation and we need to figure this out. Thank you to everyone who's shared their advice, I have no one to talk about this to, I guess I'm ashamed and everyone around us wishes and believes that we are just this successful and dynamic couple so I don't want to disrupt our whole network with this until necessary(if this ends in divorce).


Source.