Nationalistic bias in figure skating judging, 2018-present

[SkatingSquirrel]

A skating fan recently shared in other skating circles a statistical analysis they wrote about nationalistic bias in figure skating. The excerpt below is from the post created by the author at another skating forum:

PS. I'm not the author of this article, just merely sharing it with the approval of the author themselves.

"I have put together some quite comprehensive statistics on nationalistic bias in figure skating judging, which includes every single senior level ISU competition. Unfortunately, the write-up is a bit unwieldy for a forum post as it's quite long and contains graphs and so forth, so I will direct you to this website, where you can read the whole thing.

Here are the top line conclusions:

The database currently contains 312 judges, 177 of which I examined for nationalistic bias (the other judges didn’t have an extensive enough judging record), 92 of which showed statistically significant evidence (p<0.05) of nationalistic bias. Of those 74 showed strong (p<0.01) evidence.
Biased judging was widespread across most federations (shout out to Australia the one large-ish federation that was pretty clean), but some federations (notably Russia, France, Germany, and Italy) were particularly bad, though not always in the same way.

Considering that the judges play a quite significant role in the careers of young skaters, I thought I would share this here in order to open up a conversation. Of course, I'm sure you guys have many questions as well, so feel free to ask away."

 

[Foolhardy Ham Lint]

This is possibly not the right thread. But, political clout has and always will rule the sport.

I'm actually quite fascinated when a skater changes countries for whatever reason, and whether they are judged differently because of that.

Athletes whom at one point represented one of the power nations, and have moved on, seem to be the most affected.

Leaving almost feels like betrayal, and it can take months, maybe years, or sometimes never, to regain that respect.

 

[Tinami Amori]

Here are the top line conclusions:

Judging Bias and Figure Skating: Part One – Nationalistic Bias

As the Senior Grand Prix season is about to begin, I thought now would be an appropriate time to take a systematic look at the judging records of our judges. Despite the fact that figure skating ju…

fsjudging.wordpress.com

Biased judging was widespread across most federations (shout out to Australia the one large-ish federation that was pretty clean), but some federations (notably Russia, France, Germany, and Italy) were particularly bad, though not always in the same way.

What i find interesting is that the author of the article talks about "national bias", and then highlights and lists Russia, France, Germany, Italy first, attracting attention to these countries which indirectly suggest "worst offenders".

..... and then in his/her very own work the statistics/counting shows this...

Worst offenders (by number of cases).
Canada (23)
USA (21)
--
--
--
--
Japan (16)
Russia (14)
Germany (13)
Italy (11)
France (8)

i am not sure this "author" is qualified to talk about "national bias"...

 

[Foolhardy Ham Lint]

Australia is very good at national judging in line with international marking. That their results are fair and biased for the most part, provides its athletes with good preparation for international competition, knowing what to expect with the same performance, as well as detailing what elements need to be worked on as well.

 

[SkatingSquirrel]

Worst offenders (by number of cases).
Canada (23)
USA (21)
--
--
--
--

I believe these numbers are the total of judges tested from each country. It isn't the total of judges found to have higher bias. You can see for example for Canada there's the number (23) but there's only 9 names written. So 9 judges out of 23 tested were found to have high nationalistic bias. The table probably could have been done with a better layout.

 

[Tinami Amori]

I believe these numbers are the total of judges tested from each country. It isn't the total of judges found to have higher bias. You can see for example for Canada there's the number (23) but there's only 9 names written. So 9 judges out of 23 tested were found to have high nationalistic bias. The table probably could have been done with a better layout.

No!..... . First of all, the Author just changed the paragraph title where the list of judges by country started. It said originally "Wall of Shame" and then listed offenses. Now the author found it "inconvenient"...

Now it states "Judges with biased judging records"
...and shows the same thing: USA and Canada with highest numbers.

Canada (23)
USA (21)
--
--
--
--
Japan (16)
Russia (14)
Germany (13)
Italy (11)
France (8)

I don't know if the "author" will now change something in the text AGAIN to divert attention from his own "numbers" showing USA and Canada as "worst offenders".

 

[wickedwitch]

Very interesting. Other than the top plot, which I have some questions about, the statistics seem sound.

 

[Aerobicidal]

I believe these numbers are the total of judges tested from each country. It isn't the total of judges found to have higher bias. You can see for example for Canada there's the number (23) but there's only 9 names written. So 9 judges out of 23 tested were found to have high nationalistic bias. The table probably could have been done with a better layout.

Yes, and the author makes this abundantly clear when they write, "In parentheses next to the federation is the number of judges of that federation tested (not the total number of judges recorded, some of whom may not be tested due to insufficient data)."

The conclusion, IMO, is also quite clear:

As you can see, some federations are all over the place like Canada, with some judges falling into each of the bias categories, whereas for others (chiefly Russia), every judge has a record that favors home country skaters. However, though Russia is the most consistently biased of the federations examined, other federations have a larger percentage of judges who have records displaying a high degree of bias.

The U.S. and Canada both have a higher percentage of highly biased judges compared to Russia, but overall I think it's persuasive that (based on these statistics, which could obviously be critiqued on many levels, although probably less thoroughly than Tinami's reading comprehension) Italy, France, Germany, and Russia are more biased overall. If anything, the author could have drawn more attention to the Chinese judges for being biased.

But I'm sure the author is a rabidly anti-Russia Australian who does even follow Trusova's dog on Instagram.

 

[LeafOnTheWind]

Thanks to those who tried to explain the concept of statistics and how to read the analysis. I started a response earlier but I have to do this too often at work and I just quit typing. I sometimes just want to slam my head into a wall depending on the person sitting across from me.

 

[Seerek]

I expect to see more of these studies in the future if the "other" FSU (Fair Skating Union) is affiliated.

 

[Tinami Amori]

The U.S. and Canada both have a higher percentage of highly biased judges compared to Russia, but overall I think it's persuasive that (based on these statistics, which could obviously be critiqued on many levels, although probably less thoroughly than Tinami's reading comprehension) Italy, France, Germany, and Russia are more biased overall. If anything, the author could have drawn more attention to the Chinese judges for being biased.

See.... you can apply "brain" vs. "emotions and fantasies" when you try... and to sound intelligent... occasionally..

By the way, where is Canadian David Dore nowdays, there is not much "swindling" in the Canadian Federation lately..

 

[CaliSteve]

You just cant look at the score, you have to see their overall placement, which they did not show in the Yuzu ACI 2018 example. Plus, you need to look at their sampling of events.

 

[Fairuza]

You just cant look at the score, you have to see their overall placement, which they did not show in the Yuzu ACI 2018 example. Plus, you need to look at their sampling of events.

A quote for you in case you missed it.
“First, I compiled a database of the judging records of every senior international level judge who has judged a competition from the beginning of the 2018-2019 season onwards. Yes, every competition and every judge”
What does placement have to do with the whole thing?

 

[CaliSteve]

A quote for you in case you missed it.
“First, I compiled a database of the judging records of every senior international level judge who has judged a competition from the beginning of the 2018-2019 season onwards. Yes, every competition and every judge”
What does placement have to do with the whole thing?

A judge can be a low scorer but still place the skater 1st. Example at 2018 ACI, the US Judge gave Yuze a 157.34, the lowest score, but still had him in first place. The Canadian gave him 167.40 (the second highest) but placed him in second.

Skating Scores: Latest Figure Skating Scores, Rankings & Statistics

The best darn Figure Skating scores site on the web.

skatingscores.com

 

[Fairuza]

A judge can be a low scorer but still place the skater 1st. Example at 2018 ACI, the US Judge gave Yuze a 157.34, the lowest score, but still had him in first place. The Canadian gave him 167.40 (the second highest) but placed him in second.

Skating Scores: Latest Figure Skating Scores, Rankings & Statistics

The best darn Figure Skating scores site on the web.

skatingscores.com

And what does it change? The thing is, all this review is researching into is nationalistic bias. If you want to research into the link between placements and scores, you’re more than welcome to develop your method.

 

[love_skate2011]

as usual the big culprit wasn't mentioned
not surprise its Canada and USA

hello Chan, Osmond and soon Liu

 

[Aussie Willy]

A judge can be a low scorer but still place the skater 1st. Example at 2018 ACI, the US Judge gave Yuze a 157.34, the lowest score, but still had him in first place. The Canadian gave him 167.40 (the second highest) but placed him in second.

Skating Scores: Latest Figure Skating Scores, Rankings & Statistics

The best darn Figure Skating scores site on the web.

skatingscores.com

The whole point of the system is not to place skaters but evaluate them on their performance. So you give GOEs and components on what the skater is doing on the ice. Many times I put my marks into the system not thinking or considering what the outcome will be.

You are right, you can be scoring low but the placement shouldn't come into it. After I have been on a panel, the judges may look at the protocols and say they were the nice judge or the mean judge.

Whether there is intentional bias or it isn't intended, I think the point of the exercise is to show where biases appear. However I think when you see behaviour that compromises results, that is what you really need to be concerned about.

Having had a quick look at the article itself, my main concern is the way it is written. It starts off basically accusing judges of bias in a negative context. It would be better to explain what bias is first rather than launch into accusations. As the first comment about the article says it appears the author is not completing unbiased when it comes to their methodologies and what they are trying to seek in relation to skating judging. It also names and shames which I don't think is helpful and could put people on the defensive if what they are trying to do is be helpful rather than just being accusatory. In fact naming and shaming and putting it in writing could go against member protection, code of conduct and the rules of fair play.

I do think the intention is good and it could be used as part of judges training however it doesn't appear entirely objective in what it is trying to achieve.

Also the author while suggesting they hope to improve accountability, do not list their credentials or even their name. If they want to give credibility to this "project" they need to show these things.

 

[SkateGuard]

I expect to see more of these studies in the future if the "other" FSU (Fair Skating Union) is affiliated.

Promoted (paid) tweets and pie charts...

 

[doritos]

By the way, where is Canadian David Dore nowdays, there is not much "swindling" in the Canadian Federation lately..

David Dore passed away from cancer in 2016.

 

[gotoschool]

accidental comment

 

[Aussie Willy]

But I'm sure the author is a rabidly anti-Russia Australian who does even follow Trusova's dog on Instagram.

Nah most Australian's couldn't give a sh*t about this obsessive nutjob type stuff.

 

[Aerobicidal]

Nah most Australian's couldn't give a sh*t about this obsessive nutjob type stuff.

I was kidding (and maybe you know that, but this response makes it very unclear to me).

 

[Orm Irian]

Having had a quick look at the article itself, my main concern is the way it is written.

Yeah, it's written more like a blog post than an academic study; if it were for publication it would need to be significantly reframed as to tone, and present a bit more of a thesis (can you tell I'm an editor in my daily life?). But as a description of a data collection and analysis process targeted at the statistics lay reader, it sets out its method and methodology clearly enough and makes for quite interesting reading. I'll look forward to seeing what the updated data at the end of this season adds to the project.

 

[Aussie Willy]

I was kidding (and maybe you know that, but this response makes it very unclear to me).

I know you were

 

[CaliSteve]

The whole point of the system is not to place skaters but evaluate them on their performance. So you give GOEs and components on what the skater is doing on the ice. Many times I put my marks into the system not thinking or considering what the outcome will be.

You are right, you can be scoring low but the placement shouldn't come into it. After I have been on a panel, the judges may look at the protocols and say they were the nice judge or the mean judge.

The point I was trying to make is that you cant just look at (or highlight) the scores the judges gave to one skater and come to any sort of conclusion. You have to look at the scores given to all the competitors and evaluate those scores.

Whether there is intentional bias or it isn't intended, I think the point of the exercise is to show where biases appear. However I think when you see behaviour that compromises results, that is what you really need to be concerned about.

I agree! IMHO, before you can create any type of model to analyze scores, you first have to recognize one's own bias and various types of bias.

 

[wickedwitch]

The point I was trying to make is that you cant just look at (or highlight) the scores the judges gave to one skater and come to any sort of conclusion. You have to look at the scores given to all the competitors and evaluate those scores.

They do. If a judge is high on all skaters or low on all skater, then that's noted and taken into account.

 

[Willin]

On Reddit, the author posted in an earlier thread when they posted the raw data that Ukraine and Kazakhstan's judges were significantly biased towards Russian skaters. This is the only case in which any other country's judges were biased in a way that consistently benefited another (specific) country. Interesting considering the close ties of the federations...

On those same lines I wonder if there's data about whether or not certain judges are biased towards specific skaters from another country as opposed to that country as a whole. I know a lot of judges have a "type" of skater they like so I wonder if it's consistent regardless of country.

 

[overedge]

A couple of thoughts on the methodology:
- A judge's marks being an outlier is not evidence in and of itself of that judge being biased. The author's explanation that "outlier scores only “count against” judges if they align with expected patterns of nationalistic bias" doesn't make any sense, because it sounds like s/he is only including outlier scores when those conform to his/her theory that there is national bias.

- Calculating a judge's scores for the same skater across multiple competitions is meaningless, because a skater doesn't perform identically all the time. A judge's score for a skater might vary across time because the quality of the skater's performance varies, not because the judge is biased for or against that particular skater.

- The analysis only looks at the "top" federations, i.e. those with the most judges. IIRC some of the most blatant examples of cheating judges in recent years have been judges from smaller federations - federations that have a lot to gain (e.g. assignments to more competitions) by having their skaters do well. The analysis should include all judges if the author really wants to see whether there is national bias.

- Maybe a biased judge is going to be a biased judge no matter what country they judge for. IMO the problem is that there is biased judging at all, regardless of whether it's nationalistic bias, helping-out-a-coach-friend bias, gender bias, anti-Carmen bias, whatever. I'm not convinced that a focus on nationalistic bias alone is really getting at the problem of biased judging.

 

[Holy Headband]

Yeah, it's written more like a blog post than an academic study; if it were for publication it would need to be significantly reframed as to tone, and present a bit more of a thesis (can you tell I'm an editor in my daily life?). But as a description of a data collection and analysis process targeted at the statistics lay reader, it sets out its method and methodology clearly enough and makes for quite interesting reading. I'll look forward to seeing what the updated data at the end of this season adds to the project.

But it IS a blog post. That doesn't make it less valuable. There's no need to criticise it for not conforming to the stylistic norms of a different, unrelated genre of writing.

 

[Holy Headband]

A couple of thoughts on the methodology:
- A judge's marks being an outlier is not evidence in and of itself of that judge being biased. The author's explanation that "outlier scores only “count against” judges if they align with expected patterns of nationalistic bias" doesn't make any sense, because it sounds like s/he is only including outlier scores when those conform to his/her theory that there is national bias.

- Calculating a judge's scores for the same skater across multiple competitions is meaningless, because a skater doesn't perform identically all the time. A judge's score for a skater might vary across time because the quality of the skater's performance varies, not because the judge is biased for or against that particular skater.

- The analysis only looks at the "top" federations, i.e. those with the most judges. IIRC some of the most blatant examples of cheating judges in recent years have been judges from smaller federations - federations that have a lot to gain (e.g. assignments to more competitions) by having their skaters do well. The analysis should include all judges if the author really wants to see whether there is national bias.

Your first and second points are addressed in the analysis.

What the author says is not that they decided to disregard a judge's irregular scoring record when it doesn't affect their home federation's skaters but that a judge who scores both groups (compatriot and foreign skaters) irregularly will not be flagged as biased because the algorithm looks for differences between the two groups (in favour of the compatriot group). In other words, the author is "only including outlier scores when those conform to his/her theory that there is national bias" IN THE NATIONAL BIAS CATEGORY. When outlier scores don't conform to this theory, they go into the category of normal scores because they don’t suggest national bias.

The algorithm asks the question, is this judge biased in favour of their federation’s skaters?, and the answer can be (to oversimplify a bit) either ‘yes’ or ‘no.’ If you want to ask other questions of the data, they’re freely available.

As for the second point, scoring bias is calculated by comparing a judge's score to the mean score of the judging panel for that competition, so performance level variance between competitions doesn't matter.

The third point is actually addressed in the discussion section of the post as well, though you may not be satisfied with the author's explanation.