It's too bad the ferry between Maine and New Brunswick has been discontinued That would have saved you a lot of time.
Given that your husband has a 6pm flight to catch, I'd say yes, it's too much to squeeze in unless you want only a cursory glance of places. Also, customs is hit-or-miss. Sometimes it can be sloooooow. I don't know the priorities of your interests, but I think you're misjudging some timing.
Some things depend on timing. For instance, if you want to see the Fundy tides, something I recommend, you have to time your visit accordingly. The weather can also get bad, preventing you from enjoying some of the sights until a later day. This is more relevant in Newfoundland, but the weather in the Maritimes is unpredictable in general.
Why don't you fly to Montreal or Halifax and rent a car? Since you're in NJ, I assume you're flying out of New York, Newark, or Philadelphia. All those airports have direct flights to both Montreal and Halifax.
If you fly to Montreal, which is probably a better option, I would recommend not renting a car until you're done with Montreal, as it can be difficult to find parking.
The Maine coast is also stunning, but I don't think you'll have time for that.