Jackie Wong Analyzes Program Component Scores

[missing]

He studied scores from the current Grand Prix series.

The overwhelming hypothesis is that program component scores are ridiculously narrow. There are five components to mark, but for what it's worth, judges basically stick to a mark and won't deviate from it all that much. For me, "all that much" always meant around one point. But in fact, it's more like half a point.

Methodology: I took a look at the four competitions in the Grand Prix series so far this season, went into each discipline, randomly drew the short program or the free skate, and took all of the judges' component scores for the 1st place, 4th place, and last place skaters to try to sample the extremes and the averages. For each judge for each skater, I took the range of component marks from highest mark to lowest mark (e.g., if a judge rated a skater 8.00, 7.50, 7.75, 8.25, 8.25, the range would be 8.25-7.50=0.75). The result was a numerical analysis of 432 data points from over 100 judges over four competitions.

 

[Spun Silver]

I wonder WHY this happens. Jackie suggests that the judges have too much to do, but I think if fans can distinguish between, say, skating skills and performance, surely judges are at least as capable of doing so, regardless of time pressure.

 

[Amantide]

What's there to analyze? We all know the PCS are rep. scores. Even the GoE's most of the times are about rep.
Don't bother too much with that, otherwise you'll go nuts.

 

[Coco]

It's important to have proof, though. Glad he did this.

 

[MAXSwagg]

I don't get this argument that Adam should've gotten higher component marks. I wouldn't give his skating skills higher than an 8.00, and transitions any higher than a 7.75 (all of these marks are considered "good" or "very good"). Even in performance and interpretation, all that goes when he is preparing for jumps, of which have little to transitions into them. Voronov at an 86? That's is outrageous.

Even more outrageous and has been for some time now is the components in the ladies. Men are getting there too, as the judges seem to think that all of the top 6 men have similar competence in all of the components which, IMO, is absolutely not true. Medvedeva and Kostner's programs, in the eyes of the judges, were OUTSTANDING and essentially PERFECT. Are you kidding me?

Judges either don't know what they're doing or don't care. Or both. They also need to be consistent in how their scoring. Why was it that Mao got absolutely no favors yet Carolina got 9+ for this skate of 1980s-level technical content (her skating skills and transitions are also not where they used to be). Why are some skaters getting favors but others not?

It's the same for GOEs also. Why should skaters with crap technique get similar max. GOEs as skaters who have incredibly pure technique?

 

[MarciajsR]

I have always thought that no judge could be accurate in their needed rapid assessment of GOE and they are penalized if their component scores are out of line with the technical levels and international reputation of a skater. We ask a poor human to evaluate too much.

 

[tony]

I don't get this argument that Adam should've gotten higher component marks. I wouldn't give his skating skills higher than an 8.00, and transitions any higher than a 7.75 (all of these marks are considered "good" or "very good"). Even in performance and interpretation, all that goes when he is preparing for jumps, of which have little to transitions into them. Voronov at an 86? That's is outrageous.

Even more outrageous and has been for some time now is the components in the ladies. Men are getting there too, as the judges seem to think that all of the top 6 men have similar competence in all of the components which, IMO, is absolutely not true. Medvedeva and Kostner's programs, in the eyes of the judges, were OUTSTANDING and essentially PERFECT. Are you kidding me?

Judges either don't know what they're doing or don't care. Or both. They also need to be consistent in how their scoring. Why was it that Mao got absolutely no favors yet Carolina got 9+ for this skate of 1980s-level technical content (her skating skills and transitions are also not where they used to be). Why are some skaters getting favors but others not?

It's the same for GOEs also. Why should skaters with crap technique get similar max. GOEs as skaters who have incredibly pure technique?

You bring up some points I agree with and some that I don't agree with at all.

As far as the case of Rippon, I'm another one that thinks that the low 8.00's average are about his max as far as where the state of his skating is. He's like Pogorilaya and several others who can be so invested in a program until the big elements come around. Then that interpretation is completely broken to focus on the element-- completely agreed there. Once it's landed, it's back to 'feeling' the music. It's distracting to me, but it's not awful. Everyone knows I think the first 2/3 of his short program is about as juvenile of a program as possible (really nothing going on no matter how you look at it), and he's capable of much more. His free skate, because of the 'pretty' music choices, helps him carry the mood more but there are some big empty spots as well.

But I'm somebody that has absolutely no problem with Kostner receiving high or the highest PCS marks. She has remarkable control over her entire body and the movements are done with such ease and little effort. She should always be getting higher PCS than Medvedeva, even if Evgenia is spot on. Carolina's last few free skates have been masterpieces to me. As far as Asada goes, she had her fair share of technique problems, stalking jumps (that preparation for the 3A... I realize it's insanely difficult and all..) but I do agree on certain occasions the judges did her no favors-- 2014 Olympics LP being the big obvious one.

 

[tony]

As far as judges having too much to do.. I agree. That and they are having to look down every few seconds to input their GOE scores and many times we see them missing little details. Rippon's closing combination spin in the short program at NHK is a great example. I guarantee those judges who put +2 had already looked down to put in their scores and missed the very obvious fall out he had.

I think the judges get an idea of how the program should go from Youtube or from watching practices or whatever, and then they probably have in their minds the PCS that they'd give if the skater does extremely well, and so on. It's hard to be watching the small intricacies (jump landings, spin rotations, etc) and then also be able to watch the transitional movements and choreography going out of all of the elements. In other words, I find it difficult to have to be so technical and get a fair 'feeling' from the overall program.

We all have known for a long time (and this is where the article is basically going) that the judges follow a pattern similar to this:

Skating Skills Score
Transition score goes down 0.50-1.00
Performance, Composition, Interpretation all go back up to +/- 0.25 of the Skating Skills score

This is the typical pattern of a normal performance. If somebody is really exceptional in one area (think Misha Ge on the composition and interpretation), we may (and do) see judges using a variety and different pattern than the above. But usually, it goes back to the same old.

 

[clairecloutier]

I wonder WHY this happens. Jackie suggests that the judges have too much to do, but I think if fans can distinguish between, say, skating skills and performance, surely judges are at least as capable of doing so, regardless of time pressure.

It's important to remember that, in the current system, the judges must issue 17 to 18 separate marks per each free skate.

17 to 18 separate scores ... that's a lot. And they don't have much time to do all this.

My personal guess is that the judges spend the majority of their time getting the GOE quality marks fairly accurate. Then they decide which general corridor a skater's PCS belongs in, and just put them there, adding or taking off .25 for various component marks so they're not all the same.

 

[Amantide]

Even more outrageous and has been for some time now is the components in the ladies. Men are getting there too, as the judges seem to think that all of the top 6 men have similar competence in all of the components which, IMO, is absolutely not true. Medvedeva and Kostner's programs, in the eyes of the judges, were OUTSTANDING and essentially PERFECT. Are you kidding me?
Judges either don't know what they're doing or don't care. Or both. They also need to be consistent in how their scoring. Why was it that Mao got absolutely no favors yet Carolina got 9+ for this skate of 1980s-level technical content (her skating skills and transitions are also not where they used to be). Why are some skaters getting favors but others not?

I agree with the rest of your comment but not on this one, although I do agree about Mao.
Carolina does deserve high PCS and should be ahead of Evgenya there. Her technical content shouldn't affect her PCS, if she is skating clean and even when not clean some of PCS still shouldn't be affected. That's the whole point of the PCS, and that's the main problem with the judges using the PCS as rep points.

 

[Spun Silver]

I wonder if technology could help the judging overload predicament, which is only going to get worse if GOE goes from a scale of 3+/- one of 5. For instance, if they used voice recognition technology to whisper their scores on the completion of each element, to eliminate the looking down and missing the action problem. (Wowza, never thought of that!) Maybe inside soundproof booths. All collapsible, lightweight and portable so they can be sent from event to event! One can dream....

 

[Foolhardy Ham Lint]

I'm with Tony Wheeler here. Judges have far to much to evaluate within a very limited time frame.

The impression I get from IJS, particularly in singles, is that a high PCS is directly tied to high TES. Both are more often than not, determined by how consistent a skater is, too.

No matter how good a skater is, they are still at the mercy of a judging panel.

 

[MAXSwagg]

You bring up some points I agree with and some that I don't agree with at all.

As far as the case of Rippon, I'm another one that thinks that the low 8.00's average are about his max as far as where the state of his skating is. He's like Pogorilaya and several others who can be so invested in a program until the big elements come around. Then that interpretation is completely broken to focus on the element-- completely agreed there. Once it's landed, it's back to 'feeling' the music. It's distracting to me, but it's not awful. Everyone knows I think the first 2/3 of his short program is about as juvenile of a program as possible (really nothing going on no matter how you look at it), and he's capable of much more. His free skate, because of the 'pretty' music choices, helps him carry the mood more but there are some big empty spots as well.

But I'm somebody that has absolutely no problem with Kostner receiving high or the highest PCS marks. She has remarkable control over her entire body and the movements are done with such ease and little effort. She should always be getting higher PCS than Medvedeva, even if Evgenia is spot on. Carolina's last few free skates have been masterpieces to me. As far as Asada goes, she had her fair share of technique problems, stalking jumps (that preparation for the 3A... I realize it's insanely difficult and all..) but I do agree on certain occasions the judges did her no favors-- 2014 Olympics LP being the big obvious one.

I don't think we disagree on anything. Carolina should be getting much higher PCS than Medvedeva, but both should be much lower for these performances. Like more on the average of 8, not 9.25. Frankly, Carolina should be getting a 7 for transitions (which is still good). Medvedeva has many transitions, but most of them have little to nothing to do with the music. Both should be getting a 7.25 max. for performance. A skate with a fall and stumbles is NOT a good performance. The PCS of both should've been in the low-mid-60s.

 

[MAXSwagg]

I wonder if technology could help the judging overload predicament, which is only going to get worse if GOE goes from a scale of 3+/- one of 5. For instance, if they used voice recognition technology to whisper their scores on the completion of each element, to eliminate the looking down and missing the action problem. (Wowza, never thought of that!) Maybe inside soundproof booths. All collapsible, lightweight and portable so they can be sent from event to event! One can dream....

They just need two panels. The judges are too focused on scrutinizing every little thing that they aren't able to evaluate the performance as a whole. Is the program cohesive? How do the transitions and body movements and expression relate to the music? Is there a "beginning, middle, and end?" Is there choreography and interpretation throughout, or does it disappear when a jump is coming? Do the program highlights match the phrasing and structure of the music that's being used? Are the skaters picking up on the subtle nuances in the music?

 

[Amantide]

Yeah, I think two panels is the best option. Still, I'm not sure it would change much, if the PCS are about rep. which I believe they are.

 

[Foolhardy Ham Lint]

I'm substituting my interest in competitive skating for competitive tickling. Seems you can make a sport out of anything these days!

https://www.youtube.com/watch?v=8XHkdgualJo

 

[MAXSwagg]

They can also institute a new rule that states you can only judge one discipline per season. If you judge the men's event at NHK, you can't judge anymore men's events for that season.

 

[Coco]

I favor paring down the GOE bullet points and letting the tech panel assess GOE.

For example, 'paired with the musical structure' could be moved to composition or P/E.

 

[MAXSwagg]

I favor paring down the GOE bullet points and letting the tech panel assess GOE.

For example, 'paired with the musical structure' could be moved to composition or P/E.

I would prefer it to be three panels:

1) Technical panel that calls elements and evaluates levels.

2) Jury A that evaluates technical elements: giving GOE, giving UR (the average of UR is used as the final reduction), etc.

3) Jury B that evaluates program components.

 

[Amantide]

They can also institute a new rule that states you can only judge one discipline per season. If you judge the men's event at NHK, you can't judge anymore men's events for that season.

That would require a lot of high level judges. I'm not sure they can afford thart.

 

[MAXSwagg]

That would require a lot of high level judges. I'm not sure they can afford thart.

They can.

 

[analia]

I've seen judges clicking buttons on their screen. They absolutely have enough time. Most of them are done way before tech panel goes through all the elements. Marking all the components takes 5 seconds. The idea that time is a factor is absolutely untrue.
But essentially the last three components are the same thing. I can't think of a skater who has great performance and choreography but does not interpret the music well. Even if it could be true it's too abstract. I will be okay with just three component marks: SS, TR, PE.

 

[Frida80]

They can also institute a new rule that states you can only judge one discipline per season. If you judge the men's event at NHK, you can't judge anymore men's events for that season.

Even with just ISU it would take a lot of judges. Seven JGP, 10 CS, 6 GP, Europeans, 4CC, Junior Worlds, Worlds, and the Olympics. At nine judges per completion, and seven for each CS, That’s 223 judges! For just men’s! I haven’t even included the tech controllers or refs. That doesn’t count the other fifty non-ISU competitions. It’s not feasible at all.

 

[MAXSwagg]

Even with just ISU it would take a lot of judges. Seven JGP, 10 CS, 6 GP, Europeans, 4CC, Junior Worlds, Worlds, and the Olympics. At nine judges per completion, and seven for each CS, That’s 223 judges! For just men’s! I haven’t even included the tech controllers or refs. That doesn’t count the other fifty non-ISU competitions. It’s not feasible at all.

Just for seniors and only for GP events and ISU championships (Europeans and 4CC can be “reused”). The smaller stuff is less of a concern...

 

[zebobes]

They can also institute a new rule that states you can only judge one discipline per season. If you judge the men's event at NHK, you can't judge anymore men's events for that season.

What problem would this rule solve? Judges gain experience by judging. If they can only judge once a season, it will be difficult for them to have enough opportunities to improve.

 

[MAXSwagg]

What problem would this rule solve? Judges gain experience by judging. If they can only judge once a season, it will be difficult for them to have enough opportunities to improve.

Clearly, experience does not matter given the marks we’re seeing. As I said, either the judges are incompetent or they don’t care. Or both.

 

[Kateri]

Clearly, experience does not matter given the marks we’re seeing. As I said, either the judges are incompetent or they don’t care. Or both.

But what problem would your suggestion solve?

 

[SkateFanBerlin]

So much number crunching. One, it attracts judges who can do it and not necessarily good judgres. Two, figuring the deductions, judges lose sight of the overall picture. Reduce/combine categories. Eliminate as many auto-deductions as possible. A judge who values crammed programs and one who appreciates flow, edges, positions should both be free to give a PCS score of 7.5.

 

[clairecloutier]

I've seen judges clicking buttons on their screen. They absolutely have enough time. Most of them are done way before tech panel goes through all the elements. Marking all the components takes 5 seconds. The idea that time is a factor is absolutely untrue.
But essentially the last three components are the same thing. I can't think of a skater who has great performance and choreography but does not interpret the music well. Even if it could be true it's too abstract. I will be okay with just three component marks: SS, TR, PE.

Interesting, but if this is the case, I question if judges are making sufficient use of replay. They have the capability to review video, I believe? For some elements, replay can be extremely helpful for fair marking.

I agree about PE/CH/IN.

 

[gkelly]

That would require a lot of high level judges. I'm not sure they can afford thart.

They can.

Only at the highest levels, e.g., senior championships and GP only. It would kill the sport financially to require even more officials at lower levels.

Worth the investment where the stakes are highest? Train the judges to do everything at once on the way up, and then ease their burden when they get to the top of the judging hierarchy?

But essentially the last three components are the same thing. I can't think of a skater who has great performance and choreography but does not interpret the music well.

The other way around is probably more likely: a skater who interprets the music well by being naturally musical, even when given poorly thought out choreography, or choreography designed primarily to allow them to complete the elements and earn points.

I'll start a thread in The Trash Can to come up with examples.

Even if it could be true it's too abstract.I will be okay with just three component marks: SS, TR, PE.

With fewer components, how should they be factored with respect to each other and to the TES?

Is there value in using separate marks that are usually very similar to each other to reflect subtle differences between skaters, or between different aspects of general "presentation/artistry" within the same performance? As well as leaving room to reflect not-so-subtle differences in the minority of performances where there is a more distinct difference?