I've lived in Vancouver and now live in Seattle, and for me, Victoria is a great treat. The issue with traveling to Victoria is that you have to get to the ferry terminal in time to get a spot on the boat, and early morning during the week is commuter time. (If they let you reserve a spot, and you know when you want to leave, it's probably worth the money.) The ferry ride itself is 1 hour 35 minutes, and then it's another 20-30 minutes to Victoria, depending on traffic and where you're staying.
It's also possible to take a bus: it takes you to the ferry, you walk on the ferry, and then a bus picks you up at the other end and takes you from Sidney, where the ferry lands, to downtown Victoria, and you can add tours to it. Pacific Coach has buses to Victoria (and to Whistler).
How long it takes to get to Seattle from Vancouver depends on traffic and the border crossing, but there are two outlet malls on the way. Driving is about three hours with clear traffic, even with the few places that are almost always bottle-necks.
If you're going without a car, you can do a Vancouver -> Seattle (train or bus) -> Victoria (Victoria Clipper hydrofoil, leaves downtown early in the morning) -> Vancouver (bus or transit, but transit adds over an hour) triangle. To do it with a car is a long drive from Seattle to Anacortes, and a long ferry to Sidney, if it even runs in the winter.