It could be due to both Yagudin's upbringing and his suppression of 'same boat' feelings, especially since he exhibits various kinds of bias, chiefly in this instance, racism and homophobia.
I linked to the interview Yagudin gave where he flat-out said he had no friends in America except for Rudy Galindo for some period in 1998 or 1999, and with that people came to the conclusion that Alexei must've been gay as well and he had to deal with that.
If it was his upbringing, I find it unlikely he would waver between addressing the rumors and saying it didn't bother him-- in his peak of his career.. to all of a sudden it (repeatedly) seeming to bother him so much.