NEW RELEASE: Largest study ever conducted on judging and technical panel behavior

M

marc_phenologic

Member
Messages
13
(Mod note: This is a 100% free academic research project and interactive data tool created for the figure skating community. There are no paywalls, advertisements, or commercial products. Hope this is welcome here!)

Hello everyone!

Following up on my intro post from yesterday, I’m excited to share a major econometric project I’ve been working on: a comprehensive analysis and discussion of evaluation behavior in 14,382 senior international performances across the complete 2022–2026 Olympic quadrennial.

I wanted to share both the free interactive dashboard I built as well as the full academic paper so you can explore the numbers yourself:


I'd love to hear your thoughts, feedback on the methodology, and what you discover as you explore the data and results!
 
Hello everyone!

Thank you to everyone who has explored the dashboard and checked out the paper over the last couple of days.

Because the full working paper is over 70 pages and covers a massive dataset (14,382 international performances across the complete 2022–2026 quadrennial), I’d like to start an ongoing, serialized discussion in this thread breaking down specific findings—focusing on what the hard econometric data actually reveals about evaluation behavior.

Part 1: Quantifying the "Compatriot Judge Bonus" (Nationalistic Bias)​

One of the foundational questions in figure skating analytics is measuring the exact point value of having a judge from your own national federation on the panel.

To evaluate this empirically, our models isolated the scoring differential when skaters are evaluated by compatriot (home-country) judges versus the rest of the international panel across every discipline.

Here is the statistical breakdown across the entire quadrennial:

Segment_bias.png


Key Observations from the Data:

  1. Statistical Significance Across All Disciplines: The compatriot scoring premium is not a random occurrence; it is statistically significant across every single segment (with sample sizes exceeding 10,000 international evaluations per category and robust FDR-adjusted p-values).
  2. The Ice Dance Disparity: Ice Dance exhibits by far the most extreme nationalistic inflation in the sport. A compatriot bonus of +2.615 points in the Free Dance (and +2.235 in the Rhythm Dance) represents a massive effect size (Cohen's d > 0.75)—nearly double the magnitude observed in Men's, Women's, or Pairs.
  3. Segment Compounding: In every discipline, the compatriot bonus increases significantly from the Short Program to the Free Skating/Dance segment, demonstrating that higher point totals provide more mathematical room for nationalistic inflation.
Questions for Discussion:

  • For those who follow Ice Dance closely: Why do you think compatriot judges in Ice Dance are able (or willing) to award substantially higher point premiums compared to judges in the Singles disciplines?
  • In your observation, is this compatriot boost primarily channeled through subjective Program Components (PCS), or are compatriot judges also aggressively inflating Grades of Execution (GOE) on technical elements?
I look forward to hearing your thoughts, observations, and feedback!
 
marc_phenologic said:
(Mod note: This is a 100% free academic research project and interactive data tool created for the figure skating community. There are no paywalls, advertisements, or commercial products. Hope this is welcome here!)

Hello everyone!

Following up on my intro post from yesterday, I’m excited to share a major econometric project I’ve been working on: a comprehensive analysis and discussion of evaluation behavior in 14,382 senior international performances across the complete 2022–2026 Olympic quadrennial.

I wanted to share both the free interactive dashboard I built as well as the full academic paper so you can explore the numbers yourself:


I'd love to hear your thoughts, feedback on the methodology, and what you discover as you explore the data and results!
Click to expand...
Hi everyone,

Following up on the post about "Compatriot Judge Bonus" from the other day, I wanted to share the next layer of the data.

It’s one thing to look at the overall averages, but to really understand how judges behave, we have to look at the extremes. In which competition segments do the most egregious cases of nationalistic bias actually occur?

Part 2: Mapping the Extremes of Nationalistic Bias at the Segment Level​

We plotted the Nationalistic Bias Level (% of total compatriot score inflation or deflation) against the volume of home-country judgments for every competition segment from 2022-2026.

  • Red (Right): Extreme Generosity / Inflation by compatriot judges.
  • Green (Center): True Equilibrium / Objective scoring.
  • Blue (Left): Extreme Punitive / Deflation (judges actively scoring their own skaters lower than the international panel).
Here is what the scatterplot of the 2022–2026 quadrennial looks like:

Competition_bias.png


Key Observations from the Data:

1. The "Red Zone" is Dominated by Ice Dance:
Look at the top 10 most extreme cases of compatriot inflation (boosting scores by +6% to over +8%). 9 out of the top 10 events are Ice Dance segments. This mathematically confirms that when nationalistic inflation occurs, it happens at a magnitude that dwarfs the other disciplines.

2. The "Blue Zone" is Dominated by Singles:
Interestingly, the data is not strictly right-skewed, meaning that there are cases in which judges actually evaluate their home-country skaters MORE HARSHLY than the rest of the panel. The top 10 most punitive cases happened almost exclusively in Men's and Women's Singles events.

3. Equilibrium is Possible (The "Green Zone"):
It is important to note that nationalistic bias does not always occur at every competition. These are the competitions where compatriot judges scored in near-perfect equilibrium with the international panel (variance of 0.05% or less).


Looking forward to your insights!
 
Ice Dance is no surprise to me. It's always been highly political and biased.

I found it very interesting how biased some individual US judges were, but how US judges as a whole weren't among the most biased. It seems like the large number of judges from larger feds offset some of the more biased ones to regress towards the mean.

I'd be interested to see if there's a bias of former soviet republics and feds Russia took over (namely, Hungary) towards helping Russian skaters, but due to the ban we can't see it in this data set.
 
Willin said:
Ice Dance is no surprise to me. It's always been highly political and biased.

I found it very interesting how biased some individual US judges were, but how US judges as a whole weren't among the most biased. It seems like the large number of judges from larger feds offset some of the more biased ones to regress towards the mean.

I'd be interested to see if there's a bias of former soviet republics and feds Russia took over (namely, Hungary) towards helping Russian skaters, but due to the ban we can't see it in this data set.
Click to expand...
Thanks for the thoughtful reply! I will show the full results in a future post but you might be interested to know the federation showing the highest magnitude of nationalistic bias is Georgia.
 
marc_phenologic said:
(Mod note: This is a 100% free academic research project and interactive data tool created for the figure skating community. There are no paywalls, advertisements, or commercial products. Hope this is welcome here!)

Hello everyone!

Following up on my intro post from yesterday, I’m excited to share a major econometric project I’ve been working on: a comprehensive analysis and discussion of evaluation behavior in 14,382 senior international performances across the complete 2022–2026 Olympic quadrennial.

I wanted to share both the free interactive dashboard I built as well as the full academic paper so you can explore the numbers yourself:


I'd love to hear your thoughts, feedback on the methodology, and what you discover as you explore the data and results!
Click to expand...
Hi everyone,

Moving to Part 3 of the data discussion. In our nationalistic bias modeling, we tested for two different axes of bias: National Premium (inflating your own skater) and Rival Suppression (deflating the scores of direct leaderboard threats).

Here is the Federation breakdown across the 14,000+ performances:

Federation_bias1.png



Key Takeaway 1: The "Dual-Axis" Federations
Almost every country exhibits a statistically significant National Premium (Green text). But only four federations flag for extreme, statistically significant Rival Suppression (Orange text): Georgia (GEO), Hungary (HUN), China (CHN), and Azerbaijan (AZE).

These federations mathematically suppress competitors to artificially widen the point spread for their own skaters.

Key Takeaway 2: Objective Federations Do Exist
It is equally important to highlight the federations at the bottom of the table (MON, SRB, HKG, DEN, CRO, AUS). The math shows these federations exhibited no statistically significant bias in either direction. They serve as empirical proof that completely objective, neutral evaluation is entirely possible within the current IJS framework.



(As always, if you want to filter specific skaters or countries, the dashboard is at www.phenologic.net/skating-bias-details)
 
marc_phenologic said:
Key Takeaway 1: The "Dual-Axis" Federations
Almost every country exhibits a statistically significant National Premium (Green text). But only four federations flag for extreme, statistically significant Rival Suppression (Orange text): Georgia (GEO), Hungary (HUN), China (CHN), and Azerbaijan (AZE).
Click to expand...
This is my shocked face. Georgian and Azerbaijan federations are nothing more than satellites for Russia. Corrupt, corrupt, corrupt.

Fascinating analysis. Thank you.
 
marc_phenologic said:
(Mod note: This is a 100% free academic research project and interactive data tool created for the figure skating community. There are no paywalls, advertisements, or commercial products. Hope this is welcome here!)

Hello everyone!

Following up on my intro post from yesterday, I’m excited to share a major econometric project I’ve been working on: a comprehensive analysis and discussion of evaluation behavior in 14,382 senior international performances across the complete 2022–2026 Olympic quadrennial.

I wanted to share both the free interactive dashboard I built as well as the full academic paper so you can explore the numbers yourself:


I'd love to hear your thoughts, feedback on the methodology, and what you discover as you explore the data and results!
Click to expand...
Hi everyone,

Moving to the next phase of our data discussion. When fans and commentators talk about "judging bias," they almost always treat it as a single concept (usually just favoring one's own country).

However, to accurately audit the IJS across 14,000+ performances, our econometric models had to separate evaluation behavior into three distinct criteria (or orthogonal axes). We monitor every judge and federation based on these three dimensions:

  1. Baseline Leniency: Is the official naturally an "easy marker" or a "tough grader" to the entire field of competitors, regardless of nationality?

  2. Nationalistic Premium: How many extra points does the official award specifically to skaters from their own federation?

  3. Rival Suppression: Does the official actively suppress or elevate the scores of direct leaderboard threats who are competing against their own skaters?
When we map the world's federations across these three distinct criteria, very specific strategic profiles emerge.
Federation_scatter.png


(Top Chart: Leniency vs. Premium. Bottom Chart: Rival Suppression vs. Premium)

Key Observations from the Data:

  • The "Premium Only" Majority (Green): The vast majority of major federations (USA, CAN, JPN, FRA, ITA) fall into this cluster. They give a statistically significant boost to their own skaters (the Y-axis), but they remain relatively neutral when evaluating rivals.
  • The "Dual-Axis" Suppressors (Orange): Look at the bottom chart. Georgia (GEO), Hungary (HUN), and China (CHN) don't just boost their own skaters; they utilize a highly aggressive "Dual-Axis" strategy where they also systematically suppress their rivals. This mathematically maximizes the point spread for their athletes.
  • The Tough Graders (Top Chart): Interestingly, some federations like Latvia (LAT) and the Netherlands (NED) are extremely strict graders to the overall field (sitting far to the left on the Leniency axis), but they still carve out a premium for their own skaters.

 
Vagabond said:
@marc_phenologic I don't know how long you may have lurked here before posting, but leniency, especially on jump rotation and blade edges, is a fertile topic of discussion here on FSU, probably moreso than national bias and rival suppression.
Click to expand...
Thanks, you are absolutely right. If you check out the paper you will also find a full workup of bias among technical specialists, where I detect significant anomalies in their technical calls for edges and rotations. I'll post more about this in the future - it is certainly worth discussing.
 
jägerbomb said:
This is my shocked face. Georgian and Azerbaijan federations are nothing more than satellites for Russia. Corrupt, corrupt, corrupt.
Click to expand...
Whose fault is it. Your elected by you governments contributed a lot in dissolution of the USSR at your taxpayers expense. Now you have to deal with not one Soviet judge but with a bunch of them. Enjoy.
 
Hi everyone,

Moving to the next phase of our discussion on evaluation behavior. Having established that nationalistic bias is still pervasive in figure skating, we examined for the first time exactly where judges implement their bias points.
Sports analytics demonstrates that the magnitude of in-group bias is strictly governed by "verifiability", the degree to which an evaluator's decision can be objectively proven wrong. A classic example is found in Major League Baseball. Parsons et al. (2011) demonstrated that while umpires typically exhibited in-group bias when calling balls and strikes, this bias vanished entirely in stadiums equipped with electronic pitch-tracking systems. Because the pitch location was objectively verifiable, the umpires could not exhibit bias without compromising their own professional credibility.

Where do you think that figure skating judges implement their nationalistic bias?

Category_bias.png



 
BittyBug said:
The above shows why increasing the number of choreographic elements in a program, as the ISU has done for this season, makes the judging more vulnerable to corruption.
Click to expand...
I tend to agree with you. I think that having new choreographic elements will make the performances less repetitive in having new kinds of spins and lifts, etc. But these new choreo elements provide additional vectors for biased judges to influence scores.
 
jägerbomb said:
Your perpetual victimhood is impressive, my dear.

Enjoy your beautiful turquoise flag. I'll listen to your awe-inspiring anthem tonight in your honor. Cheers!
Click to expand...
Considering how Russians wear it winning everything left and right, they don't give a shit. Because only ISU old farts could come up with the idea of hiding bears behind turquoise. Cheer to Japanese fans for waving Russian flags at Kinoshita arena. Cheer to them all for not giving a Japanese flag to Sato knowing that the winner doesn't have a flag. But your loserdom keep wining.
 
Russians with their Jumping tournament proved that jumps is the most objective thing in figure skating. Jumping tournament should be an Olympic sport with a separate medal.
 
You must log in or register to reply here.

Users who are viewing this thread

Total: 1 (members: 0, guests: 1)
Back
Top