Jackie Wong Analyzes Program Component Scores

[Skittl1321]

I've seen judges clicking buttons on their screen. They absolutely have enough time. Most of them are done way before tech panel goes through all the elements. Marking all the components takes 5 seconds. The idea that time is a factor is absolutely untrue.
But essentially the last three components are the same thing. I can't think of a skater who has great performance and choreography but does not interpret the music well. Even if it could be true it's too abstract. I will be okay with just three component marks: SS, TR, PE.

It takes 5 seconds because the scores are used by reputation. There is no actual evaluation of the bullet points that should make up the component score.

Using these scores appropriately would take way more than 5 seconds. A skater could theoretically have amazing SS and horrible PE- but when have you ever seen them more than a point or half a point different?

 

[Amantide]

The other way around is probably more likely: a skater who interprets the music well by being naturally musical, even when given poorly thought out choreography, or choreography designed primarily to allow them to complete the elements and earn points.

I'll start a thread in The Trash Can to come up with examples.

Please do. I would be interesting and very much appreciated.

 

[gkelly]

Unbalanced Programs thread. Hope to see you over there for examples and discussion

 

[Rock2]

What worries me the most over the short term is that a narrowing range of PCS scores from each judge makes me feel like they have made up their mind on where that skater slots in for this Olympic year. The wider the range of PCS marks the more math it takes to slot a skater based on PCS at least. So I wonder if judges don't want to throw off their intentions by scrambling up the marks too much.

It's that and the issue that outlier marks tend to be questioned, so they don't want to be noticed too much. That has always been an issue under 6.0. Remember Vanessa Riley's reputation? She judged what she really believed she saw vs laying out ordinals to be consistent with what the others would likely do. And that made her controversial, although many of us respected her for it.

I find GoE is very much going in that direction as well. Judges using whatever subjective tools are at their disposal to slide the designated athlete as close as possible into the position they want them. Arrrgh.

This dynamic makes me dread ISU moves to make PCS and GoE more influential in the scoring going forward. We're effectively going back to 6.0 IMHO.

 

[Rock2]

To come up with solutions we have to consider what works for the integrity of skating along with what can possibly be implemented by the ISU given their constraints. Ideally we'd want separate judges for GOE and for each component so they can focus. Too many judges. Not possible.

I have often said that PCS should be compressed into only 3 components since judges can't effectively monitor and think through all five while also marking elements. I'd group them this way:

1. SS
2. TR + CH
3. PE + IN

Whether they are combined or left as is, I would consider having each judge mark GoE plus only one of the three groupings. Yes, you only get 3 marks from judges on each component but the idea is that those marks are better thought out. Also, those judges don't mark other components so they have limited ability to sway entire PCS.

Can consider 12 judges = 4 marks for each component grouping since at big events you have more judges present anyway and you select which marks count just prior to event.

If you compress PCS into 3 marks you can still raise the factor to make the marks important in totality. I would just do it to simplify the judging. I mean why separate choreo/moves that link elements vs those that don't? Makes no sense. Simplify it.
I similarly think many of the notions behind PE (e.g. clear intention of moves) can easily be considered in the context of IN (clear intention of moves that appropriately highlight the music). I think many of the thoughts can be connected pretty seamlessly to easily combine the two components.

Anyway, that's the direction I'd go in to make PCS judging more authentic.

 

[Rock2]

This thread is about PCS but GoE is going to need similar work. I have thought this through less but let me share the general direction my mind is going in.

For TES we have callers (two people, a tech specialist and assistant) who have big influence on base value with authority to call URs and levels. I'd like similar authority be given to two people on GoE. Make even wider bands on GoE although each GoE point would have less impact on overall element score since. you might be able to get like a +7 or something.

Anyway, don't get hung up on the pt system. The idea is that the 'callers' would identify what GoE band the element operates within and the judges would assign a mark within that band. In order to be judged in the top GoE band there would be certain mandatories (e.g. a jump needs speed of approach, ice coverage, height and clear run out); without ALL of these things you can't be elite level. Spin must be centered and average a certain number of rotations per second, which is timed. Footwork needs deep and clear edges with the same scrutiny as dance to be top tier.

I'm making stuff up but the idea is to bring more clarity and focus to the idea of what gets top GoE levels. Make it harder to get those grades because right now some pretty unimpressive stuff is getting +3.

More thinking to come on this...but that's where my head is going for now.

 

[Seerek]

My theory is that judges are using the skating skills mark as their "anchor" (almost ordinal-like) component and "wrap" the other 4 component scores in and around the skating skills mark on a relative scale. That's the reason why you almost never see -1/+1 differentials.

 

[SkateFanBerlin]

^^Is Rock2 a Shockwave alias?

 

[peibeck]

I think the issue really is more that judges are expected to put skaters in a corridor, rather than actually be able to give the marks they would like to give. For this, I blame the ISU, not the judges.

It's easy enough for us "armchair" judges to see Skater X has fewer/easier transitions than Skater Y, or how well balanced a program is in the different location of elements on the ice surface or if a skater is utilizing the whole ice surface effectively. But the fear judges are put into, that they must place skaters within a reasonable (what ever that means) corridor with the other judges or face potential reprimand or get fewer assignments, is only going to lend itself to the reputation judging we are seeing.

That's my two cents.

 

[clairecloutier]

I have often said that PCS should be compressed into only 3 components since judges can't effectively monitor and think through all five while also marking elements. I'd group them this way:

1. SS
2. TR + CH
3. PE + IN

I've been coming to a similar conclusion myself, recently--with a couple exceptions.

Here's what I would explore if I were in the ISU:

1. Set SS as its own separate scoring category. Skaters would get TES, SS, and PCS marks. The split could be set/factored to be roughly equal--33.3/33.3/33.3. Or it could, perhaps, be 50/25/25 (more like current setup), or even 50/20/30. The ISU would need to use mathematical models to see what works best.

Reasoning-- Skating skills are not artistry. They are one of the most important factors in judging a skater's overall skill, but they are not artistry. You can have great skating skills but not great artistry (e.g., Kaori Sakamoto). Ultimately, skating skills are more technique-related than artistry-related. They could arguably be rolled under TES, as more of a technical skill. But that could be problematic in the overall scoring scheme, giving too much weight to the total "technical" category.

2. Eliminate TR component.

Reasoning-- If they want to increase artistry, the skaters need more freedom within the programs for movement that actually works with the music & isn't there simply to count as difficult transition #15 or #32. Just get rid of it. Unusual difficulty in choreography can be rewarded in new PCS category (if executed well).

3. Combine PE, CH, and IN into 1 PCS mark. Could change name to "Artistry" or whatever works.

Reasoning-- The combined PCS mark would represent the judge's perception of the program artistically. Trying to mark separate artistic components has proven unrealistic. Just go with one overall mark.

For TES we have callers (two people, a tech specialist and assistant) who have big influence on base value with authority to call URs and levels. I'd like similar authority be given to two people on GoE. Make even wider bands on GoE although each GoE point would have less impact on overall element score since. you might be able to get like a +7 or something.

Anyway, don't get hung up on the pt system. The idea is that the 'callers' would identify what GoE band the element operates within and the judges would assign a mark within that band. In order to be judged in the top GoE band there would be certain mandatories (e.g. a jump needs speed of approach, ice coverage, height and clear run out); without ALL of these things you can't be elite level.

I think I get the intent here ... but I would oppose this idea because it adds yet more complexity to the system. And therefore, also more room for mistakes/political judging/manipulation.

The system needs to be simpler, more common-sense, more easy for everyone to understand.

 

[Spun Silver]

@clairecloutier, would you propose giving equal weight to your three categories or what?

 

[clairecloutier]

@clairecloutier, would you propose giving equal weight to your three categories or what?

Honestly, I think it would need to be studied and modeled mathematically to come up with the right split.

My initial inclination would be toward: 50% TES / 20% SS / 30% PCS.

 

[Spun Silver]

Maybe Jackie Wong could test that out for you.

 

[AxelAnnie]

I don't think the new judging is any better than the old. In fact, I think it is worse because to rack up points, skaters have to squish so much into a few minutes it is ruining a lot of what is gorgeous about FS....to say nothing about injuring skaters. Hanyu and likely Med.

There are 9 judges there, right? Why cant three judge each of the three segments, and average the score. In fact.....during the time they wait for the marks, the judges could practically watch the whole thing again! Also, the judges could specialize in each of the three areas.

 

[kwanfan1818]

There's a different kind of attention for marking individual elements in depth and trying to evaluate a series of qualities over the duration of a program.

 

[VGThuy]

I would support any system where an element can get graded on GOE and levels on their own terms where we can see a high GOE mark on a lower level element that can actually score higher than a higher level element that isn’t executed well. Theoretically, we have that already but I don’t really see it in practice for like the 12 years IJS has been around.

I would also support that system if the element causes some of the more artistic minded PCS categories marks to go down if it seems to not fit the program/music apart from how good that element is out of context of the program. I also wish GOE didn’t have a bullet point checklist that serves as near mandatory minimums if a skater checks like 3-4 boxes. I rather it just be a list of things to look out for (like suggestions) and let the judges make their own discretionary decision. Sure that’s more subjective and may give the judges way too much power but I think that might get rid of the more robotic style of skating where everyone is trying to check off the same boxes. Of course, that may make the actual skaters and coaches angry because their scores are less in their hands.

 

[tony]

There's a different kind of attention for marking individual elements in depth and trying to evaluate a series of qualities over the duration of a program.

That's how I feel. It's hard to keep shifting between looking at the small details of every single element and then having to look at the program and how it works as a whole.

But, I do agree that a shifting of the program components is needed and I am in total agreement that skating skills needs to have more weight than it does.

Transitions are already reflected, in part, in the GOE's of the elements. Skaters are fulfilling the checkboxes for higher GOE by including steps into and out of jumps, and using unusual entries into spins, etc. Where does the transition stop and the element criteria start? I think the judges are rewarding both categories (GOE and transition mark) for the majority of the same things.

So, either get rid of transitions completely, reward the elements as they are now, or throw transitions into a sub-category of the composition of the program. Performance and interpretation have a lot of similar wording in the criteria and there is no reason IMO to double reward (or punish) skaters because basically a slight modification in wording between the two.

I wouldn't mind seeing a 50 / 20 / 30 split either, and just separate it into TES, Skating Skills, and Presentation. But in order to have that actual 50% come from TES, an absolute scale would have to be used for the technical score, and a lot of people bit my head off the last time I suggested it

 

[analia]

I don't actually think judges treat TR mark the same way people think it means. Transitions in and out of elements reflect on GOE, not necessarily on TR. If all the leg bending and steps out of jumps count as TR then Eteri's juniors like Tarakanova with her extreme busyness should be getting a 10. That would be out of corridor judging.
And judges do have time to watch replays. The system they use replays each element. Normally after putting in PCS marks the judges will rewatch elements marked by the tech panel (I think US broadcast shows these marks.) All in all time is a non-factor. Some other issues are there. For example Mai Mihara's SS is better than most, but she is really small in person and it does affect perception. I have no doubt Carolina's figure helps her getting higher marks. To ask judges take physical qualities out from assessment is unrealistic.

 

[Rock2]

I've been coming to a similar conclusion myself, recently--with a couple exceptions.

Here's what I would explore if I were in the ISU:

1. Set SS as its own separate scoring category. Skaters would get TES, SS, and PCS marks. The split could be set/factored to be roughly equal--33.3/33.3/33.3. Or it could, perhaps, be 50/25/25 (more like current setup), or even 50/20/30. The ISU would need to use mathematical models to see what works best.

Reasoning-- Skating skills are not artistry. They are one of the most important factors in judging a skater's overall skill, but they are not artistry. You can have great skating skills but not great artistry (e.g., Kaori Sakamoto). Ultimately, skating skills are more technique-related than artistry-related. They could arguably be rolled under TES, as more of a technical skill. But that could be problematic in the overall scoring scheme, giving too much weight to the total "technical" category.

2. Eliminate TR component.

Reasoning-- If they want to increase artistry, the skaters need more freedom within the programs for movement that actually works with the music & isn't there simply to count as difficult transition #15 or #32. Just get rid of it. Unusual difficulty in choreography can be rewarded in new PCS category (if executed well).

3. Combine PE, CH, and IN into 1 PCS mark. Could change name to "Artistry" or whatever works.

Reasoning-- The combined PCS mark would represent the judge's perception of the program artistically. Trying to mark separate artistic components has proven unrealistic. Just go with one overall mark.




I think I get the intent here ... but I would oppose this idea because it adds yet more complexity to the system. And therefore, also more room for mistakes/political judging/manipulation.

The system needs to be simpler, more common-sense, more easy for everyone to understand.

Many interesting points here.
You step on a big issue with your sensible commentary on components vs artistry.
That's to me more of a marketing problem, not a scoring application program. Those within the sport totally get it and I'm not convinced there is confusion here among officials causing a stray on scoring.

The issue is the public doesn't get it. They can't rationalize the marking because they look at all performances through the lens of an artistic performance. Other ways to fix that but your idea is worth chewing on.

As for 'make it simple' I mostly agree, other than simplicity breeds subjectivity. It's why the tax act isn't 'simple'. Allows for too many loopholes.

The sum-total of my ideas actually bring simplicity and focus. I would disagree that my GoE thoughts are neutral in terms of driving simplicity or complexity. Just altering the way GoE is managed. Put the hard work in the hands of fewer people so the broader judging panel can manage the broad scope of their job better.

Let's keep going.

 

[Rock2]

I don't actually think judges treat TR mark the same way people think it means.

The confusion is fairly widespread within the sport I think. I remember not long ago Alexei Urmanov spoke of the 'mythical transitions' I think he called it...

 

[Rock2]

My theory is that judges are using the skating skills mark as their "anchor" (almost ordinal-like) component and "wrap" the other 4 component scores in and around the skating skills mark on a relative scale. That's the reason why you almost never see -1/+1 differentials.

A USFSA judge said as much to me in mid 2000s. They tested marking in smaller comps where the order of input of the PCS was switched. She said marks came out very different with SS not first....but didn't say what was to come of it. Clearly, nothing

 

[tony]

I don't actually think judges treat TR mark the same way people think it means. Transitions in and out of elements reflect on GOE, not necessarily on TR. If all the leg bending and steps out of jumps count as TR then Eteri's juniors like Tarakanova with her extreme busyness should be getting a 10. That would be out of corridor judging.

But the problem is that most skaters like to cram many of the elements together and link these elements such as doing some edge-work or steps out of a jump and directly into a spin-- getting high GOE's AND probably being rewarded with high transitions. Or the skaters that cram many jump elements together during the half-way point for the bonus. So transitions really become trivial and the way everything 'flows' together should just be considered part of the overall composition of the program, or in the idea several of us seem to agree upon-- the presentation.

 

[tony]

Actually, it's pretty laughable that the same amount of weight is put into the transitions as the skating skills in a short program where there is very limited time to do anything aside from the required elements. The long program isn't much better, but I really think they just need to go away.

 

[bardtoob]

To remove bias, I say the ISU should use a machine learning algorithm to call technical elements and judge skating, given that the ISU is already eyeballing angles in technical elements and is using a linear equation to score skating

 

[SkateFanBerlin]

I also wish GOE didn’t have a bullet point checklist that serves as near mandatory minimums if a skater checks like 3-4 boxes. I rather it just be a list of things to look out for (like suggestions) and let the judges make their own discretionary decision. Sure that’s more subjective and may give the judges way too much power but I think that might get rid of the more robotic style of skating where everyone is trying to check off the same boxes. Of course, that may make the actual skaters and coaches angry because their scores are less in their hands.

I think this bullet list/mandatory deduction idea sometimes causes the wrong skater to win. All the boxes are checked but the skating ugly. (Partly the reason juniorish skaters are running off with medals. Boxes checked; they must get a medal)

 

[tony]

I think this bullet list/mandatory deduction idea sometimes causes the wrong skater to win. All the boxes are checked but the skating ugly. (Partly the reason juniorish skaters are running off with medals. Boxes checked; they must get a medal)

Some of the checkboxes for the elements are even super subjective. One of the features is 'element matched to the musical structure', which to me would be double rewarding skaters for good performance/composition/interpretation (whichever or all of the three) while giving them a potential boost in GOE. And skating to slow music? I mean any judge could argue it matches the music then

 

[shine]

I've been coming to a similar conclusion myself, recently--with a couple exceptions.

Here's what I would explore if I were in the ISU:

1. Set SS as its own separate scoring category. Skaters would get TES, SS, and PCS marks. The split could be set/factored to be roughly equal--33.3/33.3/33.3. Or it could, perhaps, be 50/25/25 (more like current setup), or even 50/20/30. The ISU would need to use mathematical models to see what works best.

Reasoning-- Skating skills are not artistry. They are one of the most important factors in judging a skater's overall skill, but they are not artistry. You can have great skating skills but not great artistry (e.g., Kaori Sakamoto). Ultimately, skating skills are more technique-related than artistry-related. They could arguably be rolled under TES, as more of a technical skill. But that could be problematic in the overall scoring scheme, giving too much weight to the total "technical" category.

2. Eliminate TR component.

Reasoning-- If they want to increase artistry, the skaters need more freedom within the programs for movement that actually works with the music & isn't there simply to count as difficult transition #15 or #32. Just get rid of it. Unusual difficulty in choreography can be rewarded in new PCS category (if executed well).

3. Combine PE, CH, and IN into 1 PCS mark. Could change name to "Artistry" or whatever works.

Reasoning-- The combined PCS mark would represent the judge's perception of the program artistically. Trying to mark separate artistic components has proven unrealistic. Just go with one overall mark.




I think I get the intent here ... but I would oppose this idea because it adds yet more complexity to the system. And therefore, also more room for mistakes/political judging/manipulation.

The system needs to be simpler, more common-sense, more easy for everyone to understand.

I totally agree with 1 and 2. But I think we still need at least 2 separate cateogories for your 3rd point: Choreography and Performance. There are countless examples where good performance quality is carried out in empty programs. And a jam packed or well laid out program can also have subpar execution. This brings us to [TES, SS, CH+PE].

 

[Rock2]

But the problem is that most skaters like to cram many of the elements together and link these elements such as doing some edge-work or steps out of a jump and directly into a spin-- getting high GOE's AND probably being rewarded with high transitions. Or the skaters that cram many jump elements together during the half-way point for the bonus. So transitions really become trivial and the way everything 'flows' together should just be considered part of the overall composition of the program, or in the idea several of us seem to agree upon-- the presentation.

Yup that's pretty much it. I think there should be some objective cut off point where moves immediately preceding and following an element such as a jump are part of the GOE or in some case levels, such as an allusion into a spin. And then everything else is the TR/CH/etc mishmash of non-element content of the program.

 

[Rock2]

To remove bias, I say the ISU should use a machine learning algorithm to call technical elements and judge skating, given that the ISU is already eyeballing angles in technical elements and using a linear equation to score skating

Yes, needs to go there. There was that retired judge (Austrian maybe?) who like ten years ago talked about outfitting boots with some measuring device to measure jump trajectory, height etc. Since then, Craig Buntin launched a company that I believe uses even a single camera angle to take on-court/on-ice measurements. Talked about the application to skating but it's an ISU issue not SC issue. So he took it to bigger sports and from what I know his company is doing really well.

Net net need some nerds to infiltrate ISU and add some value to the judging process!

 

[gkelly]

I would support any system where an element can get graded on GOE and levels on their own terms where we can see a high GOE mark on a lower level element that can actually score higher than a higher level element that isn’t executed well. Theoretically, we have that already but I don’t really see it in practice for like the 12 years IJS has been around.

Two separate issues there:

1) Is the Scale of Values arranged so that increasing the GOE of an element is worth more points than doing a higher level element?

2) Are judges willing to give higher GOEs to lower value elements?


The rewards have to be built into the SoV in order for skaters to focus their training on increasing quality before adding difficulty.


If most of the strong skaters (strong spinners, etc.) are doing the higher level elements because they know they'll earn more points, then we'll only see lower value elements from skaters who can't do the harder ones at all successfully. Or from stronger skaters making mistakes. Thus not too many examples of simple elements worth high GOE.


I think that's one purpose the Choreo Sequence was intended to fulfill, but too many skaters seem to use it as a throwaway rest period. Those who do chase the higher GOEs tend to include more content as well.

I also wish GOE didn’t have a bullet point checklist that serves as near mandatory minimums if a skater checks like 3-4 boxes. I rather it just be a list of things to look out for (like suggestions) and let the judges make their own discretionary decision.

In practice, that is pretty much the case. There are no actual boxes for judges to check, just guidelines. And each judge has their own cutoff for deciding when an aspect of an element is good enough to qualify as "good" for a qualitative bullet point, or how much to reduce for errors with a range of specified reductions, and also for how to handle elements with odd numbers of positive bullet points or which errors might cancel out all positive qualities vs. just being subtracted from a positive starting point.


Some fans want more of a check-box approach so that the thought processes could be captured in the computer and conveyed to audiences. And here you want a more holistic assessment of positives and negatives. Can't please everybody.

Transitions are already reflected, in part, in the GOE's of the elements. Skaters are fulfilling the checkboxes for higher GOE by including steps into and out of jumps, and using unusual entries into spins, etc. Where does the transition stop and the element criteria start? I think the judges are rewarding both categories (GOE and transition mark) for the majority of the same things.

Sometimes, yes. But some things that go into the Transitions score cannot be captured by GOEs at all. E.g., skater P does a beautiful spread eagle directly up into a 3A. That can be rewarded in both TR and GOE. Skater Q does a beautiful spread eagle, then a couple of crossovers, then sits on a back outside edge for a couple of seconds before stepping up into a 3A. That spread eagle should get some credit for the variety and quality aspects of the TR score, but it shouldn't add anything to any GOE because it wasn't connected to any element.


Skater R could do a beautiful spread eagle into a 3A< with a fall. Credit in TR for variety and quality of the spread eagle and for connecting it directly to an element, but after reducing for both the underrotation and the fall the GOE of the element is going to be -3 regardless of the entrance.


So to me it does make sense to reward Skater P twice for that transition, but Q and R only once.

The issue is the public doesn't get it. They can't rationalize the marketing because they look at all performances through the lens of an artistic performance.

Well, there are different segments of the public. Some casual viewers are all about the artistic value. Others are all about the jump content attempted or landed successfully (but not really about quality issues that aren't visible to the untutored eye).


Again, if for marketing reasons the artistry gets played up more, to the point that skaters stop risking high-value jumps, then sports fans who are already dubious about whether skating is a sport or belongs in the Olympics will lose what interest they have. On the other hand, if artistry gets played down and technical content played up to the point that skaters don't bother much with trying to be artistic (or those who do don't win and rarely get on TV), then the arts fans will lose interest.


Both camps of casual viewers probably want falls and other obvious errors to have more of an impact on final results. But if remaining upright is the most important determinant of the results, then skaters will scale back on attempting the risk elements. And skaters themselves, and the fans who have made the effort to educate themselves about technique would not like seeing weaker skaters with cautious performances win just by avoiding falls.


Can't please everybody all of the time. Can we balance the rewards that appeal to different constituencies so everyone's favorite types of skaters win sometimes?

Actually, it's pretty laughable that the same amount of weight is put into the transitions as the skating skills in a short program where there is very limited time to do anything aside from the required elements. The long program isn't much better, but I really think they just need to go away.

The factors could change, so that Skating Skills could be worth a lot more than Transitions despite being graded on the same 0-10 scale.

Also, some skills do factor into both components in different ways. E.g., a skater does one jump at the west end of the ice then skates to the east end for another jump. The skater who uses only crossovers and mohawks to get from one end to the other is not demonstrating the same level of one-foot skating or multidirectional skating (SS component criteria) as the skater who uses counters and choctaws and power pulls to get there. They're also probably not demonstrating the same difficulty or variety of transitional moves (TR component).

But if the skater with the weaker skating skating skills also uses a lot of unlisted jumps and spirals and lunges and stars into spins and upper body movement, but the stronger skater uses only difficult turns as their only type of transition, then the former would deserve credit for more varied transitions. (Not necessarily for quality. But maybe those split and spirals are beautiful.)