The Disconnect Between Technical Panels and Judges

tony

Throwing the (rule)book at them
Messages
17,679
This stems directly from the Hubbell/Donohue incident at Four Continents, but also brings up the bigger issue that we've been seeing this season, particularly in ice dance, where the GOE's don't always seem to be in line with the levels called.

Long story short (and it's been covered wonderfully in-depth in another thread here in the Trash Can by @thvudragon)- Hubbell and Donohue's opening stationary lift failed to stop traveling and per ISU Communication 2164 - 1.4.1.7 if it does not stop traveling by the completion of the first rotation / NO features will count until the couple rotates on the spot. This explains why the element was called base level.

However, the GOEs for this element were 3 +3's, 2 +4's, and 4 perfect +5's even though the lift was messy throughout.

Many people aren't aware or forget that judging panels are NOT seeing the levels that are being called for any elements. The ISU makes the judges aware that specific elements have been flagged for review (they will light up on the screen in a different color), but they have no further information as to what the decision is from the technical panel once said elements are actually reviewed. They don't have that little TES tracker in the corner of their judging screens.

When it comes to singles and pairs skating, things like the <, <<, !, and e calls ARE shown to the judges once the elements are reviewed. This obviously comes into play because judges that were giving high positive GOE's need to go back in and modify their scores with these calls.

However, with ice dance (and with the non-jump elements in singles and pairs) there is no note of the level drop, and it makes sense. Seeing all of the called levels would give the crafty judges an idea of where the base value stands for a particular performance, and it would also give them a quick way of comparing skaters to each other in terms of where the PCS likely needs to go. I'm quite sure the phenomenon of judges marking skaters with called level 1's and 2's with lower GOE would also take place, and skaters with level 4 would likely receive the highest GOE's by default as well (ie. 'They got the level, so it must've been great' mentality). Opening up all of that specific information to panels could influence many of them to mark a certain way, and I don't think that's the answer to the problem.

In the case of Hubbell and Donohue, I'm guessing that since they are a top team - 2nd in the World and GPF winners, the judges wanted to give them the benefit of the doubt on the element and assume that even with the traveling (if they even noticed), nothing would be called down. Or maybe they did notice something wrong but they didn't want to be the random -1 or 0 on the panel and stray outside that 'corridor' we hear of so often. We all know that it's much easier for the top skaters to get the highest GOE's while mid-to-low ranked skaters may be really proficient in one element or two but pulling off the highest GOEs typically doesn't happen.

So, coming out of my rambling- I'm actually curious to hear what you think needs to happen as far as getting the technical panels and judges more in line with each other and having the GOEs make more sense in these situations. Do the judges need more information, even if their jobs are solely to count the positive and negative features of the element they see to come up with a GOE number? Why do you think EVERY judge overlooked the loss of balance, not centered/on the spot, awkward position on H/D's lift? Were they really afraid of being the only to do so?

Also, I mentioned in a video earlier in the year that I think certain GOEs should just be voided out if the skater is not capable of pulling that number based on the way the element was performed. Same with PCS. The ISU is going with the (questionable) rule this season about the maximum PCS a skater/team can receive on each of the five components with a major error/fall in a program. If the skater falls once, wouldn't it make more sense to just void out the 9+ scores that are not allowed at that point rather than giving the judge the opportunity to bend the rules?
 

Tak

Well-Known Member
Messages
3,219
I've said this several times this year, but I think they need to link the GOE to the levels. So levels 1 and 2 would be eligible for GOE of 1-3, levels 3 and 4 would be eligible for GOE of 1-5. That would reward teams with greater technical proficiency and give an incentive for teams not to settle for easier levels, but to go for the more difficult ones.

But there is a wider problem here, which honestly has been going on almost since the beg of COP marking. Judges are way too generous with the GOE they hand out. It's not just this year that badly executed elements have gotten positive GOEs. I've heard all kinds of excuses for it, none of which justify the marks judges have handed out to me. I think they need to re-educate the judges on what the GOE is for and how it should be used, and then follow up and make sure judges who continue to give out excessive GOE are dinged for it.
 

millyskate

Well-Known Member
Messages
16,732
Mishin, always insightful, said when these increased GOE ranges appeared that it was a mistake because it would just give power to the judges to increase the space in scoring between the top-rated skaters and the others.
It's done exactly that.
 

mackiecat

Well-Known Member
Messages
1,774
Sorry @Tak . I disagree. The judges look at the overall quality. Just because a spin is level 4 does not mean it has great quality and the reverse is true a base level layback could be gorgeous and deserve a +5.
The Tech panel looks at the micro picture of the program and the judges the macro. In the case of H/D lift

Look at the ISU handbook for judges. https://www.isu.org/figure-skating/...ok-for-referees-and-judges-2018-19-final/file
It doesn’t go into the definitions of the lifts. The calling of the lifts is for the tech panel. When you look at the goe guidelines, there is nothing about travelling. They could have done a reduction for loss of control but did it really look like that?
 

Tak

Well-Known Member
Messages
3,219
There's nothing wrong with giving a level 4 element zero or 1 for GOE - or even negative GOE if it's done badly. That's the point. But a badly done element IMO should NOT be given positive GOE.

If a level 1 element is extremely well done, the judge can give it a 3 GOE. I don't see anything wrong with that. But it's much easier to do a level 1 element well, because it's not as difficult. It really doesn't deserve the same GOE as a well done level 4 element.

I don't want to see teams taking the easy way out. Ice Dance is already bashed enough for not being a "real" sport. You need to have some incentive for teams to go for the difficulty, that's what makes it a sport. Otherwise it becomes a performance art, like ballet.
 
Last edited:

starrynight

Well-Known Member
Messages
3,234
Cross posting from the PBP thread:

H/D s stationary lift has been travelling all season to some degree (it was travelling in both directions at the GPF). I think it’s because they go into it with a tonne of sideways momentum. The travelling sort of helps Zach put in those fast rotations with Madison on his shoulders. I do wonder if he stayed totally put on the spot if they would be able to rotate like they do.

So I do wonder if it’s not particularly fair to allow the high GOE if aspects of the execution are actually aided by the non compliance with the element requirements.

Sort of how like a lot of the really really fast spins I've seen some singles skaters post for fun on Instagram have travelled a lot because it's harder to spin like that and keep it centred and stationary. Great fun to look at on insta - but not compliant with the competitive requirements.
 
Last edited:

snoopy

Well-Known Member
Messages
12,274
However, with ice dance (and with the non-jump elements in singles and pairs) there is no note of the level drop, and it makes sense. Seeing all of the called levels would give the crafty judges an idea of where the base value stands for a particular performance, and it would also give them a quick way of comparing skaters to each other in terms of where the PCS likely needs to go. I'm quite sure the phenomenon of judges marking skaters with called level 1's and 2's with lower GOE would also take place, and skaters with level 4 would likely receive the highest GOE's by default as well (ie. 'They got the level, so it must've been great' mentality).

This contradicts the rest of your post. IMO goe should not be based on levels. If the judges aren’t judging properly in the quality of the element, fix that particular problem rather than create a potential new one.
 

tony

Throwing the (rule)book at them
Messages
17,679


This contradicts the rest of your post. IMO goe should not be based on levels. If the judges aren’t judging properly in the quality of the element, fix that particular problem rather than create a potential new one.

I said there’s no note of the level drop, and it *makes sense* because if it did, we would have the potential problems I listed.

So somehow there needs to be a better way of measuring the GOE, but straight-up showing all of the levels to the judges wouldn’t be my answer. That leads to the whole point of my post/this discussion- how it could be better balanced between the two.
 

gkelly

Well-Known Member
Messages
16,441
Often the loss of levels is based on a technicality that really doesn't affect the quality of the element as such.

Quality is a continuous variable. Each judge needs to decide for herself whether a given element is executed reaches their mental threshold to qualify as "good" (or very good in some cases) enough to earn a GOE bullet or not, but there aren't and probably never will be strict definitions of "good" for each aspect of each possible element and feature such that all judges would come to the exact same decision on every element.

I don't think that will ever be a goal of the judging guidelines. Each judge applies her own judgment within the guidelines that allow for personal judgment, and then the scoring takes an average of the various scores from the whole panel.

If an aspect of quality is borderline as to whether some judges consider it good and others just OK/above average, then there can be differences in scores awarded and the final average will end up somewhere in between. The scoring system for quality is designed to allow for averaging different judgments of where the execution falls on the continuous scale.

Levels, on the other hand, are discrete variables. Yes or no questions: Did the skater/team achieve this or not? The ideal is that all tech specialists and controllers would indeed agree, given a perfect view of what was achieved.

So there are, and need to be, clear rules on when to answer Yes and when to answer No. Where tech panels are inconsistent on how to make these determinations, new guidelines are (or, in some cases, still need to be) added to get all tech panel members on the same page.

When there's a new tech panel guideline on exactly where to draw the line between Yes and No, the same element that earned the level last year might not earn it this year (or vice versa).

And of course, if the list of what does or doesn't count as a feature at all should change, then the level calls would change.

You could see this with the same element viewed retrospectively on video.
Different calls based on last year's vs. this year's calling rules don't mean the quality changed as far as the judges are concerned.

Similarly, for something like the "8 revolutions in position" feature on spins the difference between 7.5 vs. 8 revolutions in a valid camel or sit position can make a clear difference in level, but as far as the judges are concerned there's not much difference between 7.5 and 8. Especially if balanced by extra revolutions in a different position or by some revolutions in a beautiful but nondifficult, "nonbasic" position.

And any spin or lift feature that isn't granted because it's allowed only once per program and the skater/team already got credit for it in another element earlier in the program doesn't change the quality of the execution but does change the level call.

So it doesn't make sense to me to assume that lower level always equals lower overall quality even if judges who are also trained controllers or TSs can easily recognize which features the skaters were aiming for.

Quite aside from the fact that intentionally lower levels, or lower levels because the skater and coach hadn't read the latest level rules carefully enough to make wise choices about which features to attempt in which elements, also have no direct relation to the quality of the execution.
 
Last edited:

Dobre

Well-Known Member
Messages
16,961
I don't see any reason to believe that the majority of "disconnect" this season is happening due to technicalities. It is happening frequently on 1. the pattern, 2. steps, 3. twizzles. 4. elements that teams have lost levels on more than once.
 

mackiecat

Well-Known Member
Messages
1,774
Interesting enough for singles and pairs the goe goes up/down based on the level. For dance the goe bump is only different for base. I wonder why?
In pair a group 5 toe lift with a +5 goe but is base gets a bump up of 2.3 if the same lift was level 4 the bump up is 3.1. Dance lifts are not like this
 

gkelly

Well-Known Member
Messages
16,441
I don't see any reason to believe that the majority of "disconnect" this season is happening due to technicalities. It is happening frequently on 1. the pattern, 2. steps, 3. twizzles. 4. elements that teams have lost levels on more than once.

Are we talking only about elite dance teams (who are usually aiming for as many level 4s as possible)?

Or about all disciplines at all skill levels?

If the rule would be to show the levels to the judges so they can take the levels called into account when awarding GOEs, then the rules would have to be the same for all competitions at the same level in the same discipline.

If the consensus is that it would be useful for all senior dance events, then the same procedure would need to apply to Lake Placid or Pavel Roman Memorial, etc., as well as to ISU championships.

Maybe different rules would be used for (all) junior or novice events, or for other disciplines.

Level calling and GOEs do seem to work differently in dance than in the freestyle disciplines already.
 

Dobre

Well-Known Member
Messages
16,961
Just to put this discussion into a wider context, instead of highlighting only one element from one team in one competition, this disconnect has been present in dance throughout the season. Below is a much more comprehensive look at examples of the disconnect we have been seeing. It is important to recognize the following:

1. The examples of disconnect below all belonged to fairly well recognized teams. This is because the majority of dance teams in the world do not earn +3s or above from the majority of judges on ice dance panels.

2. Disconnect undoubtedly existed prior to the current system but the impact of that disconnect has a much greater impact on dance results this year versus in the past because . . .

3. There are now more points earned for GOE.

4. There are now fewer points lost and/or gained for dropped or earned levels.

5. While the examples below all feature level 2 & below elements, the reverse form of disconnect also exists. Teams earning high levels & not receiving similarly high GOE relative to more established teams earning lower levels. This is harder to analyze as all dance teams aim for level 4s, but if I have time later, I will see if I can do this a bit with the pattern this season, where the majority of levels for even the best established teams have been relatively low.

6. I would argue that the scoring system in dance this season is "out of balance."




Examples of Level vs. GOE Disconnect in Challengers, GP Events, and Major Internationals This Season:

Lombardia
Hurtado & Khaliavin had a level 2 step. +3s from the majority of the panel.

U.S. Classic
Hubbell & Donohue had level 2 on the Tango Romantica. All +3s.
Hubbell & Donohue had level 2 on their diagonal step. They had +3s & +4s.

Autumn Classic
Weaver & Poje had level 2 on the Tango Romantica. +3s & above from the majority of the panel.
Weaver & Poje had level 2 on the diagonal steps. +3s & 4s from the majority of the panel.

Nebelhorn
Gilles & Poirier had a level 2 on the Tango Romantica. +3s from the majority of the panel.

Ondrej Nepela
Sinitsina & Katsalapov had a level 2 midline step. +3s & +4s from the majority of the panel.
Sinitsina & Katsalapov had a level 2 on the first step in their FD. All +3s.
Sinitsina & Katsalapov both had level 2s on their one-foot step. All +3s & +4s.
Popova & Mozgov had level 2 on their combination spin. +3s & above from the majority of the panel.

Finlandia
Stepanova & Bukin had level 1 twizzles. They had +4s & a +3 & +5.

Skate America
Hubbell & Donohue had level on their Tango Romantica. +3s & +4s from the majority of the panel.
Guignard & Fabbri had level 2 on their Diagonal Step. +3s from the majority of the panel.
Zach received level 2 on his one-foot step sequence. H&D had all +3s and above.
Marco had level 2 on his one-foot step sequence. +3s & +4s from the majority of the panel.
Lorraine had level 2 on her one-foot step. +3s from the majority of the panel.
Jonathan had level 2 on his one-foot step. +3s and a +4 from the majority of the panel.

Skate Canada
Hubbell & Donohue had level 1 on their Tango Romantica. They received +4s & +3s.
Sinitsina & Katsalapov had a level 2 rotational lift. They received +3s & +4s from all but two members of the panel.
Sinitsina & Katsalapov had a level 2 midline step. Also +3s & +4s.
Smart & Diaz had a level 1 on their Tango Romantica. They received +3s from the majority of the panel.
Lauriault & Le Gac had a level 2 on their diagonal steps. They received +3s from the majority of the panel.
Sinitsina & Katsalapov receive level 2s on both step sequences in the FD. They received +3s from the majority of the panel both times.
Hubbell & Donohue had level 2s on their one-foot steps. +3s & above from the majority of the panel.
Piper had a level 2 on the one-foot steps. +3s & 4s from the majority of the panel.
Lauriault & Le Gac had a level 2 on their first step sequence. +3s from the majority of the panel.

NHK Trophy
Hawayek & Baker had a level 1 on their Tango Romantica. +3s from all but one member of the panel.
Hawayek & Baker had a level 2 on their RD steps. +3s & +4s from most of the panel.

Helsinki GP
McNamara & Carpenter had a level 2 diagonal step. +3s from the majority of the panel.

Rostelecom
Stepanova & Bukin had a level 2 midline step. All +3s & +4s.

Inge Solar
Charlene had a level 2 one-foot step. +3s & +4s from the majority of the panel.
Lauriault & Le Gac had level 1 and level 2 on their one-foot step. +3s from the majority of the panel.

Internationaux de France
Papadakis & Cizeron had level 1 on their Tango Romantica. +3s & +4s from the majority of the panel.
Gilles & Poirier both had level 2 twizzles. +3s & +4s from the majority of the panel.
Hawayek & Baker had a level 2 midline step. +3s from the majority of the panel.
Sinitsina & Katsalapov both had level 2 one-foot steps. +3s & +4s from most of the panel.
Gilles & Poirier both had level 2 one-foot steps. +3s from the majority of the panel.

Grand Prix Final
Hubbell & Donohue had level 1 on the first part of their Tango Romantica. All +3s and +4s.
Hubbell & Donohue had level 2 on the second part of the Tango Romantica. +3s from the majority of the panel.
Sinitsina & Katsalapov had level 2 on the rotational lift. +3s & +4s from the majority of the panel.
Hubbell & Donohue had a level 2 spin. All +3s & +4s.
Nikita had level 2 one-foot footwork. +3s, +4s, and a +5 from the majority of the panel.
Charlene had level 2 on one-foot footwork. +3s & +4s from the majority of the panel.

Europeans
Papadakis & Cizeron had a level 2 on the Tango Romantica. All +3s & +4s.
Papadakis & Cizeron had a level 2 Midline Step. All +4s and +5s.
Stepanova & Bukin had a level 2 Tango Romantica. +3s from the majority of the panel.
Fear & Gibson had a level 2 step in the RD. +3s and a +4 from the majority of the panel.

4CCs
Hubbell & Donohue had a level 2 on the Tango Romantica. All +3s & +4s.
Hubbell & Donohue had a level 2 Midline Step. +4s & +5s from the majority of the panel.
Hawayek & Baker had a level 2 Tango Romantica. +3s from the majority of the panel.
Hawayek & Baker had a level 2 Midline Step. +3s & +4s from the majority of the panel.
Hubbell & Donohue had a level 2 Spin. +4s & +5s from the majority of the panel.
Hubbell & Donohue had the level B lift. All +3s, 4s, +5s.
 
Last edited:

DreamSkates

Well-Known Member
Messages
3,364
Would this related to Zhou's LP technical score being 102 at the end of his program, but 88 when his score was announced? I read there was one UR, but how did that result in such a drop in points?
 

VGThuy

Well-Known Member
Messages
41,020
There were three URs. The reason why TES dropped so much because jumps are called as they are intended no matter if they are UR until the callers review and put the carrot next to it. So initially, his jumps were called clean and the judges gave him positive GOEs for them and then once the judges get notice that three of his jumps were UR, they changed their GOE to negative ones. If he had fallen/stumbled/fell out on a clearly UR jump, TES still would have dropped but it wouldn't have been such a dramatic drop as the GOE may not have changed as much as he would have received negative GOE on it initially for those mistakes. That's why it seems Vincent gets big drops because often times he lands on one foot and thus gets positive GOEs initially until the judges are told the jumps are UR.
 

Dobre

Well-Known Member
Messages
16,961
Would this related to Zhou's LP technical score being 102 at the end of his program, but 88 when his score was announced? I read there was one UR, but how did that result in such a drop in points?

Only that it shows how different the GOE impact is in singles or pairs when judges are informed that a jump is underrotated. So Vincent not only loses part of the base value of the jump, but the judges are made aware of the call and their GOE is expected to reflect the error in the jump's execution. Not only does Vincent have zero +3s, +4s, or +5s for the URed jumps in his free skate on his protocol; but he also has negative GOE from many judges on those elements. So, for example, he loses 4.34 points on his quad lutz, triple toe alone. Because he performs the second jump in the combination with an error during the free skate.

Theoretically, one would think one would find examples of disconnect during the men's event at 4CCs on spin levels or footwork sequences.

But intriguingly, I found none. As far as I can tell after skimming the protocols for the SP and FP, none of the 4CC men with a level 2 or below spin or footwork sequence received +3s or above from the majority of the panel.

(Footwork & spin levels were in general quite high at 4CCs. Think I will go look at the Europeans men's protocol also).
 

VGThuy

Well-Known Member
Messages
41,020
(Footwork & spin levels were in general quite high at 4CCs. Think I will go look at the Europeans men's protocol also).

You can say that again. 17 men got a level 4 footwork call in the SP. It was not the same story in the LP, however.
 

Dobre

Well-Known Member
Messages
16,961
I find no examples of disconnect similar to my dance examples above for the spins or footwork during the men's event at Europeans in the FP. I find the following examples during the SP:

Samarin had a level 2 footwork sequence. He earned +3s from the majority of the panel.
Samohin had a level 2 spin. He earned +3s and +4s from the majority of the panel.
Selevko had a level 2 step sequence. He earned +3s from the majority of the panel.

I've gotta say that I don't think it would hurt the sport to ding those guys on those particular elements more harshly on GOE either.
 

AxelAnnie

Like a small boat on the ocean...
Messages
14,463
Mishin, always insightful, said when these increased GOE ranges appeared that it was a mistake because it would just give power to the judges to increase the space in scoring between the top-rated skaters and the others.
It's done exactly that.
The GOE is being used as a placeholder. The GOE in my opinion needs to be tied to the tech score. An element that is under rotated needs a GOE deduction..etc.
 

AxelAnnie

Like a small boat on the ocean...
Messages
14,463
I find no examples of disconnect similar to my dance examples above for the spins or footwork during the men's event at Europeans in the FP. I find the following examples during the SP:

Samarin had a level 2 footwork sequence. He earned +3s from the majority of the panel.
Samohin had a level 2 spin. He earned +3s and +4s from the majority of the panel.
Selevko had a level 2 step sequence. He earned +3s from the majority of the panel.

I've gotta say that I don't think it would hurt the sport to ding those guys on those particular elements more harshly on GOE either.
And that is why figure skating has such narrow appeal...or rather one reason. People can't understand why people win with a fall and lose when clean. Momo
 

Dobre

Well-Known Member
Messages
16,961
So, coming out of my rambling- I'm actually curious to hear what you think needs to happen as far as getting the technical panels and judges more in line with each other and having the GOEs make more sense in these situations.

So honestly, Tony, I think this is a really significant topic. And there are loads of possible ways to deal with this that have been shared in threads this season. I feel like the best way to answer your question would be to go back, quote them all from their various locations, and insert them into this thread. (Which would maybe bring some of the expert voices in here and save those of us who have been talking about it all season from needing to repeat what we have previously said). It is a legitimate, quality topic and it should be discussed. And I'm sure many of the suggestions we have heard come from people with more insight than me. It would be nice to have all of those suggestions here so we could hash out the nuances of each one and/or make comparisons.

But this would mean going through all the dance play-by-play threads from--gosh--certainly the GP, and to be honest, possibly even as far back as the Quebec Summer Competition this year through 4CCs. And through the Dance Hall Thread, where we have been discussing it all season. But that thread is now 110 pages long. (And I'm perfectly OK with that because it's a really good microcosm of how the season has gone & makes it fairly easy to find a great deal of the dance-related discussion. This topic, though, isn't an easy one to pick out of the thread because it runs all the way through the discussion for this season). I just do not know that I could possibly find the time do all of the reading it would take to find these relevant suggestions. (Maybe. One never knows).

But if you are really curious about what suggestions a wider variety of dance fans have, I can recommend a lovely meandering read through those 110 pages, etc.
 

RoyThree

Active Member
Messages
84
I think I read this somewhere, but I just need some clarification. Are the singles/pairs judges informed when a jump is under-rotated or has an edge call (and have to adjust their GOE to reflect that), but are not informed on the elements that have a level?

I agree with many on here that in Ice Dance, the levels and GOE should be connected in some way. I have no idea how, as I'm just a layperson who watches and reads, but those with more knowledge should be able to figure it out. It makes little sense to me that a GOE can offset a low level. Otherwise (to me at least), the GOE tends to appear more like a reputation score instead of what just occurred on the ice.
 
D

Deleted member 19433

Guest
I think I read this somewhere, but I just need some clarification. Are the singles/pairs judges informed when a jump is under-rotated or has an edge call (and have to adjust their GOE to reflect that), but are not informed on the elements that have a level?

Yes, that is correct.
 

Users who are viewing this thread

Top
Do Not Sell My Personal Information