Opining on IJS scoring system: 5 calls to action by Jackie Wong - agree/disagree/your suggestions?

[caseyedwards]

The only true solution to all problems is admit different people can have different opinions and go back to 6.0. Even on technical elements on replay people can disagree on edges and falls even!

 

[chapis]

I really like most of his ideas.

And, I had no idea the judges had only 1 replay camera. Here I was thinking they were blatantly disregarding what we at home were shown in favour of their own agenda (lol, still prob the case sometimes), but now I can see how they can get it wrong!! Don't other sports already use pro tv camera angles? For big events, it only makes sense. In the ladies event where they havent reached into quad territory, checking edges is a big deal....

Judges really should learn to separate PCS scores...

Me neither. I remember someone saying that judges have special cameras, that we should not complain about scores because they can see everything very well with their cameras. Come on, only 1 cheap camera and located in the same place where they are seated .

 

[Roux]

Jackie for (ISU) president! I don't think he wants that responsibility, though...

I have wondered this for quite a few years now but I really have to wonder why the technology doesn't exist (or the investment in the technological development hasn't been made) to determine from a chip in either the blade or the boot, the # of rotations and edge takeoff. It would certainly take a lot of the human error out of the equation and likely eliminate the need for some of these lengthy reviews.

I'm not an expert, but I would guess that such technology would be prohibitively expensive, even if the design already existed. I also can't imagine how the chip could sense when the skate left/returned to the ice. Then it would have to be durable, reliable, attach to the skate without adversely affecting its structural integrity or balance...

But innovation does come from questions like that! Maybe someday!

How much would Poeta score today? I think once that question is answered, PCS can just be calibrated accordingly.

Oh, there's an interesting idea... Current judges could rescore an old program periodically. Of course, they would probably be affected by preconceived notions and possibly by changes in style/fashion. And then there's the difference of watching in person vs. video... Ah, but still, that appeals to me.

 

[alchemy void]

1) separate judging panel for PCS
The program components assess different aspects of skating, but generally skaters get the same or similar PCS marks across the board even if they have different PC strengths and weaknesses. This change would not have to be a huge increase in cost: just switch from 9 judges to 10 and have 5 judge TES and 5 judge PCS.

I *really* like this idea. Previously, I’ve dismissed separating the panels because of the unrealistic cost associated but I think 5 judges on each would be sufficient.

PCS categories and criteria still need to be simplified and clarified, though. Ideally, if 5 judges are only focusing on PCS, they should be able to accurately judge each component and vary the scores. I have my doubts this is possible with the current criteria.

Totally against making up some more complex rules for combinations. The goal is to make the system more clear, not more convoluted!

I agree with you. Especially in regards to PCS. TES needs a few tweaks: quad values, expand the GOEs, reduce the tano and backloading bonuses. But it isn’t broken at all.

Honestly at this point, I almost think PCS could be reverted back to just one overall score from each judge that is relative to the rest of the competition, just like the old presentation mark. Should make it a lot easier to judge, and perhaps allow the judges to judge the “big picture” for once.

Hell, the announcer could even read each judges’ PCS score (and displayed on screen by country) just like in the 6.0 days. I miss being able to instantly bitch about a specific judges’ score in the K&C, instead of waiting for the protocols.

It's actually been longer since IJS was first implemented (fall 2003 at Nebelhorn) than it was between the abolition of figures in international competition (1990) and the introduction of IJS.

That is insane. Wow.

 

[MAXSwagg]

You can’t compare PCS scores ten years ago. Component scores have become unmoored and have drifted. Before circa 2012, for example, Mao and Yuna averaged in the high 7s for components. 62 was a fantastic PCS for a free skate. Now it’s borderline dismal and it shouldn’t be, especially since 7s still count as “good.”

I completely agree with Sandra that the issue is the system doesn’t address the overall picture, the program from a holistic standpoint. It just dissects it (which is not a bad thing) and the judges give big scores when if you were to sit back and just watch the whole program the marks in most cases should be lower.

 

[Vagabond]

The replay camera problem is ridiculous. Can't the ISU use the TV camera angles? For some US Sports (College Football/Basketball at least) the referees have access to their own cameras as well as all of the TV camera angles of every play. That gives them an additional 3-5 angles and zoom options for each play (element in skating). What's even more fun is that for some games, because the replay footage is from the network, you can see the exact clips and speeds that the referees are requesting for review as they're reviewing it. So you can see it from the angle(s) the referee is considering at the same time as the referee. Imagine if we had that in skating - I bet it would explain a lot about some calls.

Me neither. I remember someone saying that judges have special cameras, that we should not complain about scores because they can see everything very well with their cameras. Come on, only 1 cheap camera and located in the same place where they are seated .

Basketball has a referee and one or two umpires, who move up and down the court. American football has six or seven officials (seven at the professional level and in Division I) moving about the field and sidelines. Cameras in these sports are stationed in different positions on the field to reflect what the officials can or should be able to see.

A figure skating judging panel stays in a single position throughout a program. Skaters construct programs so that they are directed to the judging panel, with many important elements placed in order to be in the panel's close sight. If judging cameras were placed in other locations, skaters would have to construct their programs differently, and the panel would probably have to be broken up and seated in different places around the rink.

Is that really what you want?

 

[chapis]

If judging cameras were placed in other locations, skaters would have to construct their programs differently, and the panel would probably have to be broken up and seated in different places around the rink.

Is that really what you want?

Yes.

 

[Vagabond]

Good luck with that!

 

[alchemy void]

The camera angle thing is dumb, the tech panel should have access to at least what we have access to on the replays, but I really think it is (by far) the least pressing concern. Fix the PCS first, please.

 

[overedge]

The only true solution to all problems is admit different people can have different opinions and go back to 6.0. Even on technical elements on replay people can disagree on edges and falls even!

Because judging panels always agreed on everything when 6.0 was used Do you have anything meaningful to contribute to the discussion, or do you just like hearing yourself talk?

 

[Susan M]

1. More (and better) replay cameras

Not so sure. IMO they need less reliance on replay not more. It has created a problem of real disconnect between what we see in real time and the scores. There has also developed a real issue of some skaters getting a pass, even for dodgy landings, while others have 4 or 5 landings scrutinized.

2. Properly recalibrating elements and components

I agree in general that figure skating is driven too much by planned technical content, but the current program components are so poorly defined and scored anyway, I can't get too excited over increasing their relative weight. I think the real solution is to dramatically increase the reductions from negative GOE, so that a fall on a jump does not still yield points for the skater. They should go back to the notion that a failed element gets no credit. Points just for trying is a sweet concept for age 5 but should have no place in Olympic level events.

3. Penalties in bonus scoring

I do agree with this. There are too many errors listed as requiring a reduction but still can result in positive GOE. IMO all the listed errors should require a GOE of zero or lower.

4. A review of how (and how often) bullets count

ITA. Each variation should credit for GOE only once. They do a bit of this already, so it would be consistent to apply this concept throughout.

5. An all-out effort in behavior change

Not sure about cause and effect, but I agree that judges are not using components effectively. There needs to be a complete overhaul of the PCS scoring guidance and standards to produce a lot more spread between the best and the rest.

 

[gkelly]

Me neither. I remember someone saying that judges have special cameras, that we should not complain about scores because they can see everything very well with their cameras. Come on, only 1 cheap camera and located in the same place where they are seated .

Not "special" cameras. But different cameras than are used by the broadcasts.

The point being that just because a jump looks clean (or not clean) on the broadcast video doesn't mean the tech panel saw it the same way. So just because the call doesn't match what you saw on the broadcast video doesn't mean that the tech panel must have been incompetent or corrupt. They probably called exactly what they should have called based on what they could see..

Sometimes the official camera angle may be closer to the "truth" and sometimes the broadcast camera may be closer. But they only way to know would be if we were omniscient . . . or at least if we had multiple camera angles to compare.

Ideally, with the technology currently available, it would be more accurate if the tech panel could access 4 or 5 cameras, or at least 2.

But then you get into questions of 1) financial cost, 2) time cost, 3) logistics, and 4) security.

Probably the best compromise to deal with 1) and 2) would be to use exactly two cameras on opposite sides of the rink, both required to keep the skaters whole body and blades in frame throughout the program. And/or maybe an additional camera that just does closeups on the blades.

No facial closeups or artsy longshots or overhead shots from TV directors being creative. No cuts or dissolves in the middle of programs. Broadcasts can use those techniques, which will influence home viewers' experience of the performance, but the judges and technical panel would only ever see the more objective sports camera angles, and only go to the second or third angle if the first angle is ambiguous.

And maybe the default first camera angle tech panels would look at when they start a review would be from a different angle than what they saw live with their naked eyes. A full-body camera on the opposite side of the rink and a blade close-up camera stationed near the tech panel might be a good combination.

Also if those feeds are available to the broadcasters to edit in with their own camera angles, and for replays on the jumbotron after the performance, then fans could see the same things the tech panel sees when reviewing.

A figure skating judging panel stays in a single position throughout a program. Skaters construct programs so that they are directed to the judging panel, with many important elements placed in order to be in the panel's close sight. If judging cameras were placed in other locations, skaters would have to construct their programs differently, and the panel would probably have to be broken up and seated in different places around the rink.

As is, skaters often place jumps they don't want to be scrutinized too closely in the corners along the officials' side of the rink. So there would be less benefit of the doubt afforded with more cameras.

I don't know that the tech panel or the judging panel would need to be spread out around the rink. There would be some advantages to doing that in terms of getting more perspectives on the performances.

3) Logistically it would be challenging for referees to communicate with all judges and with the technical panel if they're all in different locations, especially if the systems go down mid-competition for any reason. A real problem if the computers stop working, or headsets if that's how they're talking to each other about the details of moving the event along. Also an issue if, say, a skater has to stop in the middle of the program for any reason and then resume.

4) How should camera feeds be sent to the tech panel and judges' monitors from the far side of the rink? And data input from the tech panel and each judge to the accounting computer if they're all spread out across the building? Lots of really long cables that would need to be secured? Or wirelessly, with risk of hacking or inadvertent interference? Would more seats need to be unavailable for spectators to avoid them looking over officials' shoulders at the screens in various locations?

The answers to these questions would likely be different for large arenas vs. local rinks with riser seating for a thousand or two spectators vs. small local rinks used for lower-level qualifying and nonqualifying competitions. Probably multiple cameras are needed only at large arena events.

I'm not an expert, but I would guess that such technology would be prohibitively expensive, even if the design already existed. I also can't imagine how the chip could sense when the skate left/returned to the ice. Then it would have to be durable, reliable, attach to the skate without adversely affecting its structural integrity or balance...


But innovation does come from questions like that! Maybe someday!

I think something like that will happen eventually. It’s probably possible with today’s level of technology, but someone would have to actually invent the sensors and teach them to recognize all the different details needed to score the technique accurately, which would not be a trivial task. If people start working on it now and using partial versions for test events or training purposes, maybe it would be ready for adoption into official scoring in a decade or two.

 

[Roux]

Here's an idea I've been sitting on for a bit:

Give the judges rubrics to help differentiate the full range of component scores. I don't mean by adding more quantitative restrictions (this transition is worth more than that one), but really just a visual aid, or a tool for thinking through the scores. Here is a chart from USFSA that lays out the criteria and then, separately, identifies 9-10 as "outstanding," 8 as "very good," 7 as "good," etc. But they need to be combined!

I'm picturing rubrics that teachers would use usually for grading essays in gradeschool. Here's an example if you don't know what I'm talking about. You can have the same criteria described for different scoring ranges. So instead of saying that skating skills include deep edges, balance, flow, etc. and then asking if skating skills are outstanding, very good, etc.:

Outstanding skating skills

• Deep edges, steps and turns throughout the program.
• Perfect precision and balance.
• Continuous flow and glide throughout.

Very Good

• Deep edges, steps and turns throughout most of the program.
• Precision and balance with only one or two minor bobbles.
• Flow and glide throughout.

Good

• Moderately deep edges, steps and turns throughout.
• Precision and balance with no major bobbles.
• Frequent flow and glide.

Etc, etc, etc. Something in that vein. You can also highlight bullets from different levels (outstanding flow, very good balance, good edges) and approximate an average visually. It might take a bit longer, but I think not much, with familiarity.

Okay, thanks for bearing with me. Now, who hates it?

 

[gkelly]

Something like that does exist for ice dance:
http://www.usfigureskating.org/content/Marking Guide for PCS for SD and FD-2017-18.pdf

(See starting on p. 23, pages 4-7 of this document; I'd have to search further to find the original document this is an excerpt from.)

Should singles and pairs get something similar?

 

[Willin]

@Roux ITA. There needs to be widely published bullet points or standards for each level of PCS. Apparently they exist, but I can't find them. How are we supposed to know what if the judges are wrong to award PCS if we have no concept of what each point value means?
@gkelly Yes. I think Pairs and Singles need something similar. A lot of the PCS in those disciplines make no sense, and I think having a way to evaluate judges and punish them for rewarding marks that make no sense would help reign in judges. Or perhaps because the judges in those disciplines have no easily accessible standard to go off of, they don't even have a clear idea of what should constitute each mark.

The problem with the replay as described above is that it has become so bad in college football that the games take forever. Allowing unlimited replay only means that we will be there for 10 minutes after every four minute skate.

And as has been proven on here if you replay something long enough and slow enough you can find something wrong with every skater. So replay could easily be used to ding skaters judges don’t like while still rewarding skaters they do

I don't necessarily think there should be unlimited replay, just that they should use better camera angles. The image Jackie provided of where the camera is vs. the typical Lz corner shows how this can be an issue. One camera sitting right next to the judges is not enough to accurately evaluate a UR. I was thinking it should be more along the lines of having the judges select an angle or two to review from the HD footage the TV station provides. As you see during broadcast replays, there's plenty of time to show a couple of different angles of possible URs and still not increase the review time.
Perhaps they could implement a rule that only elements marked for review during the skate can be reviewed?

Basketball has a referee and one or two umpires, who move up and down the court. American football has six or seven officials (seven at the professional level and in Division I) moving about the field and sidelines. Cameras in these sports are stationed in different positions on the field to reflect what the officials can or should be able to see.

A figure skating judging panel stays in a single position throughout a program. Skaters construct programs so that they are directed to the judging panel, with many important elements placed in order to be in the panel's close sight. If judging cameras were placed in other locations, skaters would have to construct their programs differently, and the panel would probably have to be broken up and seated in different places around the rink.

Is that really what you want?

Not true - in fact, some sports like Football and Soccer are trying to find ways to get angles that the refs see. If you watch any game you can see that a lot of shots are overhead or far away from where the officials are - unless they can fly or jump up to the rafters at a moments' notice? I remember there was an idea of using gopros at one point to fix this problem. Instead, they're experimenting with other options. Soccer is implementing goal spotting technology. Football is now using cameras in the pylons on the goal line in some larger broadcasts. At least one network is trying to put cameras in the down markers. Ground level cameras are rarely used outside of replays and non-playing time footage because using the ground level cameras (which don't have a wide view of the whole court/field) may miss out on important parts of each play in real time.

Is that really a bad thing? It seems to cut down on innovation and interesting jump/spin entrances with all the skaters placing elements in the same place as each other. I would love to see skaters mix up where they place their jumps. Not only that, but if there were multiple angles, there would be no reason to place the judging panel around the rink - if they wanted to see the element better, they could just review a better angle.

Also if those feeds are available to the broadcasters to edit in with their own camera angles, and for replays on the jumbotron after the performance, then fans could see the same things the tech panel sees when reviewing.

The answers to these questions would likely be different for large arenas vs. local rinks with riser seating for a thousand or two spectators vs. small local rinks used for lower-level qualifying and nonqualifying competitions. Probably multiple cameras are needed only at large arena events.

I was suggesting the TV cameras because it would be cheaper for competitions to use the TV feed (particularly if they want TV footage, but I agree that shots of the skaters face are not needed. Also, for the reason you listed - I want the angle the judges see to be available to the fans. I feel it would make judging more transparent.

I agree that multiple camera angles should only be used at large national/international competitions as it's far too burdensome for smaller competitions.

 

[caseyedwards]

Because judging panels always agreed on everything when 6.0 was used Do you have anything meaningful to contribute to the discussion, or do you just like hearing yourself talk?

The whole point was that there was disagreement and everyone acknowledged that all the judges had different standards and methods and this whole thing of ijs in trying to say things can be determined objectively is false! You have a system of scoring that makes no sense!

This is what I wrote!
“The only true solution to all problems is admit different people can have different opinions and go back to 6.0. Even on technical elements on replay people can disagree on edges and falls even”

It was all about embracing the disagreements! Not trying to pretend everything is objective. The technical panels can disagree and tie breakers are built into the technical panel and every judge can read PCs bullets differently so you can’t get objective scores ever in any way whatsoever.

 

[overedge]

@caseyedwards why don't you go take a look at how easy it was to hide politicking under 6.0, and how little feedback it gave to the skaters other than they were relatively better or worse than other skaters.

And yes, there are things in skating than can be objectively compared, such as the number of revolutions in spins and jumps, the tempo of dance programs, and the number or kind of elements in a program.

IJS is a long long way from being perfect but it's better than 6.0 in several significant ways.

 

[Coco]

The underlying problem with the PCS criteria is that each component can either be used to grade a specific performance or to grade the program. The system is designed to allow judges to hold up skaters who perform poorly, imo.

There is something to be said for attempting difficult choreography, but it shouldn't be used to boost ALL 5 of the component marks.

So they need to address the built in 'fudgability' aspect of components. The corridor is part of this, so there is really no point to fixing anything until they get rid of the corridor, imo.

 

[gkelly]

Oh, there's an interesting idea... Current judges could rescore an old program periodically. Of course, they would probably be affected by preconceived notions and possibly by changes in style/fashion. And then there's the difference of watching in person vs. video... Ah, but still, that appeals to me.

For various reasons we can’t ever say definitively that the official scoring was wrong and the rescoring is more correct, or vice versa.

Live vs. video is definitely an issue, especially for Skating Skills, and some aspects of Composition.

As an exercise, it might be interesting to have some experienced judges watch a bunch of performances on video from different eras all at once and apply the same standards across the eras. How do different emphases of technical skills and styles of performance and program construction hold up in comparison to programs from other eras with other emphases?


I like to score pre-IJS programs, or comparable programs from different eras, by the latest rules just for fun.

Makes for good Trash Can threads in the off season.

I *really* like this idea.

Previously, I’ve dismissed separating the panels because of the unrealistic cost associated but I think 5 judges on each would be sufficient.

I think that could work for major events. It would be more of a hardship for lesser events that only use 5-7 judges now. As long as the skaters don’t need to change what they do, the judges could adapt.

The one time about a decade ago that there was a test use of separate GOE and PCS panels at Nebelhorn, the feedback was that the GOE-only judges were bored to tears.

I’d let them judge Skating Skills as well, to give them something to focus on between elements and also to separate the scoring of other components from being driven by Skating Skills, as often seems to be the case.

PCS categories and criteria still need to be simplified and clarified, though.

The ISU tried to simplify last year by removing or combining several of the bullet points under Performance and Composition (and changing the names of those components), and changing “Intricacy” to “Continuity of movements from one element to the other” under Transitions.

Do you think they need to go further and lose even more criteria? And/or combine some whole components, as has been periodically suggested?

If we want more difference between components and more agreement between judges, it might make more sense to separate the criteria even more so judges could score each criterion separately. E.g., if judges gave one score for “Movement and steps in time to the music (Timing)” and a separate score for “Expression of the music’s character/feeling and rhythm,” then you might see the skater who skates lyrically and emotionally to lyrical emotional music but shows no recognition of the downbeats to score low on the former and high on the latter, and vice versa for a skater who hits every beat robotically with a blank face and stiff upper body, instead of both splitting the difference and ending up with similar scores in between.

Honestly at this point, I almost think PCS could be reverted back to just one overall score from each judge that is relative to the rest of the competition, just like the old presentation mark. Should make it a lot easier to judge, and perhaps allow the judges to judge the “big picture” for once.

I completely agree with Sandra that the issue is the system doesn’t address the overall picture, the program from a holistic standpoint. It just dissects it (which is not a bad thing) and the judges give big scores when if you were to sit back and just watch the whole program the marks in most cases should be lower.

I think there would be value to having a score that reflects the big picture, but I don’t think it would be appropriate to lose all the analysis and combine everything into a single number, which would make the scores more subjective and judges even less accountable.

The underlying problem with the PCS criteria is that each component can either be used to grade a specific performance or to grade the program. The system is designed to allow judges to hold up skaters who perform poorly, imo.

There is something to be said for attempting difficult choreography, but it shouldn't be used to boost ALL 5 of the component marks.

Well, only if what is attempted is difficult areas related to all the components, and executed at least somewhat successfully.

How about if components were redefined and factored as

1) Skating skills 40%
2) Program construction 20%
3) Execution 40%

 

[Roux]

The one time about a decade ago that there was a test use of separate GOE and PCS panels at Nebelhorn, the feedback was that the GOE-only judges were bored to tears.

I'd heard about the test but not that feedback! That's enlightening

I’d let them judge Skating Skills as well, to give them something to focus on between elements and also to separate the scoring of other components from being driven by Skating Skills, as often seems to be the case.

If they could do Transitions as well, it might be easier to find judges for the other three—wouldn't require much if any technical/practical knowledge. And, as I think I've said before, that division in itself might promote better judging by (1) allowing judges to really see the performance as a whole instead of interrupting themselves to look at technical things, and (2) allowing judges who are best suited to one or the other to focus on what they do best.

 

[Susan M]

If I was making changes, these would be the top of my list

2) change the formula for base value of combination jumps
A 3A-3T and solo 3F should not be equal in worth to a 3F-3T and solo 3A. The first is infinitely harder than the second. Make each combination have its own value commiserate to its difficulty.

I think the scoring of jump combinations and sequences is a real weak point in this system. IMO first jump's base value should get a 1.1 multiplier and they should get rid of the reduction for the sequence. Another problem is that the skater can do a fine first jump, but if the skater falls on the second jump, the GOE for the whole combo goes way down. I wish they could find a way to give full credit for the good jump.

[/QUOTE]3) limit the number of jumps that can receive bonus in the second half of the FS to four jumping passes. Get rid of the bonus altogether in the SP.[/QUOTE]

Agree with both these ideas, except I would change second half to final third and limit it to two or three.

 

[gkelly]

I think the scoring of jump combinations and sequences is a real weak point in this system. IMO first jump's base value should get a 1.1 multiplier and they should get rid of the reduction for the sequence.

I'd rather see the second jump's base value get the bonus multiplier, to reflect that it's harder to put a jump after another jump than on its own. Or possibly a small multiplier for the first jump in a combo (and the last jump in a sequence, or at least no reduction), and a larger multiplier for the last jump in a true combo.

That would also encourage skaters to develop skills such as one-foot axel into triple (or quad) salchow, which should be but currently is not worth more than the salchow followed by a double toe.

Or triple toe+quad toe, which should be but isn't worth more than quad-triple.

Another problem is that the skater can do a fine first jump, but if the skater falls on the second jump, the GOE for the whole combo goes way down. I wish they could find a way to give full credit for the good jump.

If the first jump is good, it can earn some positive GOE bullets, so if the only reduction is for the fall on the second jump the final GOE can be -2, or theoretically but less likely -1 or even 0.

And they do get base value for the second jump, so it's not a total loss.

Maybe with the +5 to -5 GOE range we'll see a lot of combinations with good jumps followed by rotated but fallen jumps earning -3 or -2 or higher rather than losing the full -5. We still need to know the plan for exactly how that changes is going to work. (And whether it will in fact be adopted.)

 

[MAXSwagg]

How can GOE judges be anymore bored than PCS judges, and any more bored of they’re judging both? Also, if they’re bored, maybe they should not judge. That sounds like a piss-poor feedback to me.

 

[gkelly]

How can GOE judges be anymore bored than PCS judges, and any more bored of they’re judging both? Also, if they’re bored, maybe they should not judge. That sounds like a piss-poor feedback to me.

Well, I was paraphrasing. I don't remember exactly what the verbatim feedback was.

But my understanding is that judges in general take pride in their ability to balance detailed technical observations with big-picture assessments of each program as a whole, and to be familiar enough with the GOE rules that they can decide and input their scores for each element within a second or two of it finishing, if not before. For most elements, it becomes almost a rote exercise: that's an obvious +2; that's an obvious -2.

So once an element is over and the skater is moving on, whether in complex artistic ways or telegraphing, to the next place on the rink where s/he's going to do the next element, the judges have nothing to think about until the skater actually gets there and does it.

They signed up for a (volunteer) job that challenged their ability to think about fine details and big pictures at the same time, to analyze and to integrate simultaneously. With the change from 6.0 to IJS, part of the expertise (determining how difficult each move was in relation to other moves, and how to balance the difficulty with the quality) was taken out of their hands, but the new challenge was documenting their decisions about the quality of each element and about five separate groups of big-picture criteria at the same time.

And then with the division of labor in which all they needed to do was award GOEs, all of the integrative mental tasks were gone and all that was left was, to many of them, the least interesting task.


Also, I'm just extrapolating here, but I find in my own experience that a complex task that keeps on my toes mentally for long stretches produces better results than long periods of down time with intermittent moments of focused attention.

 

[binbinwinwin]

I disagree with his third point, it just seems to punish them more on top of things, a jump done in the second half is a jump done in the second half. I'm very annoyed with how the 2nd half bonus is being used right now and I think they should switch it by the order of the elements instead of time but that's a long story for another day. I also want to take away the compounding deduction for falls, it's not like skaters are falling on purpose, if they fall once they might have hurt themselves which increases their chances of falling later. Why kick them while they're down? Just make it a flat -2.00 deduction for falls.

I think his 5th point is very pragmatic but I remember an old interview from Scott Williams where he said that the scoring system is not the problem, but the judge's behaviour and I agree. We could wish but I think even with additional training and everything, there will still be fudging of things.

 

[MsZem]

I also want to take away the compounding deduction for falls, it's not like skaters are falling on purpose, if they fall once they might have hurt themselves which increases their chances of falling later. Why kick them while they're down? Just make it a flat -2.00 deduction for falls.

Compounding deductions should discourage skaters from attempting jumps that they are unlikely to land, which is a good idea. It also recognizes that multiple falls are disruptive to a program.

 

[BittyBug]

They signed up for a (volunteer) job that challenged their ability to think about fine details and big pictures at the same time, to analyze and to integrate simultaneously.

Except the PCS demonstrate that they are not successfully doing that. If they were, we would see more variation in scores across the components for most skaters. Instead, the range of marks is generally at most 1. Why bother having five different components if they are all going to be marked relatively equally?

 

[binbinwinwin]

Compounding deductions should discourage skaters from attempting jumps that they are unlikely to land, which is a good idea. It also recognizes that multiple falls are disruptive to a program.

But the current system rewards botched, rotated jumps more than popped jumps. Pop a lutz into a single, you get 0.6 BV, double is 2.1 BV. A fallen, round 3Lz fall is worth 2.90 after deductions (without 2nd half bonus), if it's their 3rd/4th fall it'll still be worth 1.90 so they may still take the risk. Maybe that's something else we need to look into in IJS scoring.

 

[gkelly]

Sorry for all the long posts – these topics are fascinating to me.

Except the PCS demonstrate that they are not successfully doing that. If they were, we would see more variation in scores across the components for most skaters. Instead, the range of marks is generally at most 1.

If some judges were scoring only the 5 components and not also GOEs, I think we would see more variation in scores across the components, but I don't think we would see much more than 1 point difference from the panel average for most skaters.

If there were only two (e.g., Technical Components and Artistic Components) or three components (e.g., Skating Skills, Program Construction, and Execution) with a larger number of disparate criteria within each, even less difference, because for skaters who are strong in some technical (or artistic) areas and weaker in others the strengths and weaknesses would often cancel each other out.

So if wider ranges within each skater’s components is a goal in itself, I would say the ways to achieve that would be

1) Separate the panel and let the GOE judges also judge Skating Skills (or Skating Skills + Transitions) so the artistic component judges would not be influenced by their own scores for the technical components. Of course if they’re knowledgeable skating judges, they would be influenced by the quality and difficulty of the skating that they recognize even if they’re not scoring it, but they could focus on making clear distinctions among the components they are responsible for. And their average assessments of the SS might be different than the technical judging panel’s (although probably not in a systematic way). So just by using different groups of individuals you’d end up with different ranges.

2) Separate the component scores even more, into more than 5 components. E.g., for Transitions, maybe one score for Continuity of Movements and for Quality, and a separate score for Variety and Difficulty. Then the simple but well-executed performance could score high on the former and low on the latter, and an overly ambitious complex program that the skater could score low and high respectively.


If we’re separating the panels anyway, other ways to artificially separate the scores from the different panels would be to have one group watch from close up and the other from far away, or one group watch live and the other on video, or have one group highly trained in evaluating skating with no outside knowledge about performance required or encouraged, and the other group highly knowledgeable about the arts but no knowledge about skating technique or rules or standards wanted.


Or just revise the way judges’ performance is evaluated to reward wide ranges and penalize narrow ones instead of encouraging them to fit within a corridor.


Or redefine the 0 to 10 scale for each competition level so that what’s now considered a 5 would be a 1 in senior competition, 2 in juniors, and so on down.


I don’t think we really want to force wide ranges just for the sake of wide ranges.

Why bother having five different components if they are all going to be marked relatively equally?

Even if the differences between components tend to be small for most skaters, I do think there is more value to separating them than just combining them all into a single score.


1) It allows judges to make fine distinctions between skaters who are pretty close. An individual judge might think that two skaters were both Very Good and very very close to each other overall. They both deserve average PCS around 8. But Skater A was just a bit more expressive musically than Skater B, and B had a notably more complex program than A in terms of deeper edges, more difficult turns, and more changes of direction. Is it more helpful to score them as, say, 8.25 8.0 8.0 8.0 7.75 vs. 8.0 7.5 8.0 8.0 8.25, or 8.0 vs. 7.9 (or both straight 8.0)?

In most cases under the current breakdown, individual judges have slightly larger differences in the totals between even close skaters, although the averages can take the PCS differences to less than 0.25.

2) For any individual skater, those 0.25 and 0.5 differences from each judge will tell them which areas that judge thought they were slightly stronger or weaker at. If different judges on the panel disagree about what the strengths were or where to reward them, the skater will get some mixed messages that aren’t very helpful. But most often there will be a small but clear trend that the panel as a whole thought the skater was strongest on Interpretation and weakest on Transitions, or much stronger on Skating Skills than everything else, etc.

3) For skaters whose skill sets are significantly unbalanced, at least on a given day, using separate scores with differences of 1.0 or more between strongest and weakest component can send a clear message that a single combined component score would not.

I do think there is a lot of room for improvement in the scoring of the program components. Separating the criteria differently might help. Separating the panels so that different judges score different components might help.

Developing some clear standards about what to look for in terms of musical interpretation, phrasing, choreographic purpose, and all the other artistic criteria would help. So would shared benchmarks about performances that qualify as Poor, Weak, Fair, Average, Above Average, Good, Very Good, or Outstanding, or among the all-time best ever, in each of the component areas would help. Obviously for the 9 and 10 examples the performance would have to be exceptionally strong in all criteria for a given component. But for benchmarks at 8 and below, it would be good to have several performances with different strengths and weaknesses that all average out to the same general range.

If all judges start with the same idea that “This is what a 7, 8, and 9 in Composition looks like” and “This is what a 7, 8, and 9 in Interpretation looks like,” they might be better positioned to say “That was a 9 in Interpretation but somewhere between 7 and 8 in Composition.”

More discussion here:
https://www.fsuniverse.net/forum/threads/unbalanced-performances.102300

 

[Rock2]

At the very top of the pyramid lies a key question:
Should an Olympic sport (and its method of determining winners) evolve to challenge the athleticism of the participants or evolve to grow audiences? Easy to say both, but when it comes down to it, what is the primary guiding force?

This is the conundrum that lies at the core of the debate. Whether we realize it or not, I would offer a majority of casual and serious fans alike favor fan-friendly changes. This is in part why so many are frustrated by IJS. It's redefining 'best/better' in ways that are eating away at the enjoyment of much of the audience.

I truly feel that until we understand and align on the guiding force as proposed above we will continue to have major divides and contentious debates. And further, we won't have the necessary context to effect rule changes.

Note, this is about OLYMPIC sports. And the Olympic slogan is 'faster, higher, stronger', if memory serves.