Victoria's winter is less rainy than Vancouver's, so if it looks like you'll have a dry day, go for it. I remember ditching my coat for a couple of days at Nationals in Victoria in 2011. (The locals all had coats, but I was fine in a sweater.) Not sure if that's typical or not. Day tripping to Victoria is difficult without a car, but since you mentioned Whistler, I assume you are renting one. My husband and I are making the trip, but after Nationals is over. I don't think we will spend the money on Butchart Gardens in January, but there's a nice museum in downtown Victoria, tea at the Empress, walking around the Inner Harbor, pretty coastal drives...I think there's enough to do for a day, especially if you haven't been before.
Seattle might be the same-ish amount of travel time as Victoria, but a less relaxing trip, since traffic vs ferry. And rainier. I've done day trips there in January/February because I flew into Seattle and drove up to Vancouver for nationals/4 Continents/Olympics. The first time, the weather was nice, and we walked around by the market, but I didn't really get the best of Seattle until a few years later, when I finally got to visit in summer.