So fed up but feel guilty?

Take him up on his offer to go out without him. Leave a list of chores for him to do while he takes care of the baby. He'll get a chance to find out how much hard work baby care is, and he'll gain more respect for the work you do, Then he'll stop wanting you to leave him with the baby. Then you can tell him that you're going out whether he comes along or not. Tell him if he comes along, he won't have to do any of the chores on the list and you'll take care of the baby while you two are out if he comes along.