judging system

[vexlak]

Looking at the competitions from last month or so, I cannot help to question the system in use...
Without naming the competitors, the judges are really all over the place. Today, foe example, clearly a 2A with bad landing gets -1 (correct in my view) to +3 (!)...
I hate to call the judges incapable, but are we in need of another judging system overhaul? Or will we continue to accept by name or history points earning judging. This is not helping the sport.
I am sure that I am not the only one noticing it.

[Vash01]

Shouldn't this be in the Trash can?

I know we have had those discussions there in the past, and I wouldn't mind discussing it again. I never fully bought into the COP. It did not fix the real issues; the judges are still scoring their favorites high (case in point- pairs competition). We don't need another judging system; we just need more consistent judging across the board. If I had my choice, I would revamp the current system to bring back the musical interpretation which was so important in the 6.0 system, and it will not impact the current system adversely.

[Sugar]

Anonymous judging in CoP is just begging for scandals/conspiracies, at least 6.0 judging was not anonymous.

I don't know why judging in Cop must be secret?

And how that is an improvement?

Transparency is good not bad in judging, or am I wrong?

[hanca]

I think a lot would be fixed if the judging was not anonymous. Also, if the judges who are too far off had to explain why. Like you said, if one element is somewhere between GOE -1 to +3, then either the judge having +3 or -1 is wrong. If they were accountable for the judging mistakes, making they would judge more accurately. There are concrete criteria which cause an element to increase or decrease in value of GOE, so it shouldn't leave such a big space for interpretation. Difference of 4 levels GOE (from -1 to +3) shouldn't be happening. I could understand difference of 2 levels, but 4 is too far off.

Also, I wouldn't blame the judging system. It's the judges that are not using it correctly, and lack of consequences if the judges make mistakes. It is not fault of the system.

[briancoogaert]

And there is also another problem, IMO. I have the feeling that a great double axel by a top class skater has +2 and a great double axel by a less famous skater has +0.5 !

[Iceman]

I have reached the point I think it best not to look at the protocols. In her short in Germany, for her 2a, Yu-na received the following goe's: -1, 1, 2, 2, 2, 3, a range that hardly makes sense. How could that jump have been seen so differently by the judges?

[Really]

Because they are human, not machines, with all the differences in perceptions and inherent biases about which parts of an element carries more impact on the GOE.

Oh, and I thought musical interpretation was accounted for in the component score?

[gkelly]

Anonymity is a completely separate issue from which judging system is in use, as I mentioned in a similar thread a few days ago. The confusing part is that the anonymity was introduced (under 6.0) just a year or two before the new judging system started being used, so it's easy to believe they're inextricably related. Not so.

Let's assume that there is no anonymity and judges' names were always attached to scores. If so,

*Is it easier for judges to reflect their honest evaluations of the performances if they give a couple of holistic marks about the program as a whole or if they give separate marks for each element and for several different aspects of the program as a whole?

*Is it more informative to skaters and to interested fans to see scores for a couple of holistic marks about the program as a whole or separate marks for each element and for several different aspects of the program as a whole?

*If the skating community as a whole believes that the system is currently rewarding some unimportant qualities too highly and some important ones not enough, would it be easier or more effective to change the direction of the reward system by giving general instructions about how to use the couple of marks available, or by revising the Scale of Values, the factors for various scores, the program requirements, and/or the judges' or technical panels' instructions to enforce more value for whatever had been considered undervalued or vice versa.

*Assuming that different members of the skating community, including different judges, have different personal values about what should be rewarded, is it better to have them all give scores in the same system that weights the various aspects consistently or to allow each judge to determine their own rankings according to their own conscience and beliefs?

*If some judges are less knowledgeable than average, or more easily unconsciously swayed by personal preferences, but want to judge as honestly and accurately as possible, which method of scoring makes it easier to identify areas of judging weakness for improvement?

*If some judges are inclined to ignore their consciences or what they see on the ice to try to manipulate results by scoring favored skaters higher and rivals lower than the judge thinks they deserve, what method of scoring and reporting scores makes it easier for them to achieve their dishonest goals? What method makes it easier for officials to catch the cheaters?


Which of these questions would be interesting to discuss here in depth with examples (real-life or hypothetical as appropriate)?

[aftershocks]

Jeez, gkelly, they should hire you to fix IJS/ CoP scoring mess, I suppose. Or, at the least serve as PR representative, or moderator of a IJS/ CoP panel discussion.

The problem, in my humble opinion is that the above types of questions you are asking are simply too late, eh? These or similar types of questions should have been asked by TPTB before enforcing drastic changes so precipitously and then tweaking and modifying and re-changing rules and guidelines season after season.

Ultimately, the new system has as many or more systemic problems as 6.0. Like many have said, I don't think it is a matter of 6.0 being better, but just that they rushed the new system in too quickly without adequate contemplation, planning, testing, and input from all parties, including federations, coaches, former skaters, etc. My feeling is that it might possibly have worked better had they not thrown the baby (6.0 brand) out with the bath water, but incorporated it into a better thought out IJS/ CoP. Maybe I'm just a CoP idiot or a math retard (like Chan was practically accused of being after his GPF fp problems with combinations ), but I simply don't understand where the ceiling is on the CoP numbers game. It's exhausting and sorely dissatisfying to watch comps like the one that took place in Sochi.

The overly high scoring in general simply did not reflect much of what was witnessed on the ice, IMHO. I'm wondering if ISU judges at Worlds will bring things back down a bit re scoring. Again, that's part of the problem tho': the unevenness in scoring from event to event.

Maybe at GPF the skaters were burnt out a bit from the GP season and many did not have their mojo working (aside from the few who performed really well in both short and free portions). Beyond the less than stellar performances of many skaters, I think there is something a bit more problematic going on under the surface that feels dissatisfying but difficult to articulate. For me, I think my dissatisfaction stems from issues with the scoring and the overemphasis on quads, but also extends way beyond those issues to some fundamental things about this sport and the way it is run and reported on/ not reported on.

Guess I should take the rest of my whining and dissatisfaction to Trash Can, or not.

[vexlak]

...

Guess I should take the rest of my whining and dissatisfaction to Trash Can, or not.

Well said. Fully agree.

IMHO this should not be in the Trash Can.
This is about skating debate and it is directly affecting all of us here and those on ice.
I will respect the moderators decision however what ever that may be.

[aftershocks]

^ Debate away! Debating is a good thing.

Would be nice if fan debates could end up providing enough great fodder for actual viable suggestions that figure skating PTB would actually consider implementing.

[Iceman]

Because they are human, not machines, with all the differences in perceptions and inherent biases about which parts of an element carries more impact on the GOE.

Oh, and I thought musical interpretation was accounted for in the component score?

If I am wrong I am sure someone will correct me (and I would welcome that), but the "which parts of an element carries more impact on the goe" are spelled out in the judging criteria and are not left up to the discretion of individual judges. IMO there is no way two qualified and well trained judges can come up with a +3 and and -1 for the same jump. This is not a matter of being human.

[gkelly]

http://www.isu.org/vsite/vnavsite/pa...v-list,00.html

The current rules for positive and negative GOEs are in ISU Communication 1724, pages 10-13.

There are some rules for certain errors that must result in negative final GOE. But beyond I don't see any rules establishing which errors or positive bullet points should have more weight than others.

-1 to +3 does seem like an exceptionally wide range -- perhaps someone made an actual error in scoring that element. But in a lot of elements there is definitely room for disagreement, if two people don't see exactly the same thing or if they weight the different aspects differently in their minds.

[Iceman]

Others much more knowledgeable than I (and with whom I agree) have suggested that there is no way a judge can judge all the things he is required to judge, especially in the short time he or she is given. They would , at least, need to watch the taped performance a number of times. A possible improvement, although probably not a complete solution, would be to have two judging panels, one to judge tes, the other pcs. In addition, when there is a goe spread such as has been discussed in this thread, the referee would step in for immediate rectification. This would be akin to replays in tennis and football. FS is often its own worst enemy.

What the judges are asked to do is imo a severe form of multitasking and we know what recent research has shown about that.
Psychiatrist Edward M. Hallowell has gone so far as to describe multitasking as a “mythical activity in which people believe they can perform two or more tasks simultaneously as effectively as one.”

[gkelly]

Others much more knowledgeable than I (and with whom I agree) have suggested that there is no way a judge can judge all the things he is required to judge, especially in the short time he or she is given. They would , at least, need to watch the taped performance a number of times. A possible improvement, although probably not a complete solution, would be to have two judging panels, one to judge tes, the other pcs.

The parts that are most complicated to judge tend to be the PCS more than the GOEs. So separating the judging panels that way might help the PCS but would not have much effect on the GOE judging.

My understanding was when they tried a split panel that way at Nebelhorn a number of years ago, the judges who were doing only GOEs were really bored.

So maybe a better way to split it -- at important competitions that can afford to bring in larger panels -- would be to have a panel of "technical judges" doing the GOEs and the Skating Skills and Transitions, and a panel of "artistic judges" with extra training in aesthetic theory doing Performance/Execution, Choreography, and Interpretation.

In addition, when there is a goe spread such as has been discussed in this thread, the referee would step in for immediate rectification.

How can the referee step in immediately? The skater is still skating the program and the judges are still watching the next things that the skater is doing. (Also the referee doesn't see the judges' marks until after the end of the program.)

What could happen would be that as soon as the judges send their marks, the computer could flag any wide ranges or outliers for the referee and the referee could ask the judges to check those scores before finalizing the scores. This will delay the announcement for the skaters and audience, but it would be a good time to catch data input errors.

[Belinda]

I have a question regarding the protocol.
Where it says "The Judges Panel (in random order)", does one column of marks represent marks from ONE and the same judge from the first GOE to the last PCS? Or are the marks randomised for each row so that each column of marks is a mixture of marks given by different judges?

[gkelly]

At senior international competitions, I believe what they now do is keep all the marks for each judge in the same column for each skater, but the judges' columns are in random order and the order changes with each skater.

For junior and domestic competitions that don't use anonymous judging, the columns are in the same order as the list of judges on the cover sheet and stays the same for all skaters.

[dinakt]

The parts that are most complicated to judge tend to be the PCS more than the GOEs. So separating the judging panels that way might help the PCS but would not have much effect on the GOE judging.

My understanding was when they tried a split panel that way at Nebelhorn a number of years ago, the judges who were doing only GOEs were really bored.

So maybe a better way to split it -- at important competitions that can afford to bring in larger panels -- would be to have a panel of "technical judges" doing the GOEs and the Skating Skills and Transitions, and a panel of "artistic judges" with extra training in aesthetic theory doing Performance/Execution, Choreography, and Interpretation.

Something like that, yes. Because SS and TR belong to technique. But P/E, IN and CH is really another field.

For myself: I prefer the current system for TES. I think in a sport counting technical points makes great sense, and it makes many things clearer to both athletes and fans. GOE is a bit of a mess currently and either needs to be revised ot implemented more strictly.
SS and TR are in the grey zone between Technique and Artistry, as SS and TR enable artistry, but also are judged in their own right as technical elements.
The rest of PCS is where I have the most problems, simply because I am professionally most qualified to judge them. Even with a panel of "art" professionals you'll get wide difference of opinion not only because of different tastes, but because different artistic professions put different emphasis on various aspects of skating. I ( a musician) was watching skating with my SIL ( a modern dancer) and it was rather striking that we put value and attention on different things, all having to do with those remaining PCS.

PCS also does not cover well a few important things which would do better if they were spelled out. Though musicality is hiding somewhere behind Interpretation, it is really NOT the same thing; and musicality historically was very important in skating. Same for such things as unison, point, esthetic beauty and such elusive things as chemistry for Pairs and Dance. As long as they are acknowledged somewhat covertly in descriptions of PCS rather than spelled out as inportant things, they are going to be lacking.

It is incredibly tricky to find a balance between sport and art, and art, rightly ( as much as I hate to admit it) comes second to sport. But the PCS as they stand now are muddy and encourage reputation judging. Splitting panels would help.

[Belinda]

At senior international competitions, I believe what they now do is keep all the marks for each judge in the same column for each skater, but the judges' columns are in random order and the order changes with each skater.

For junior and domestic competitions that don't use anonymous judging, the columns are in the same order as the list of judges on the cover sheet and stays the same for all skaters.

Thank you for your response. Much appreciated.

[gkelly]

The rest of PCS is where I have the most problems, simply because I am professionally most qualified to judge them. Even with a panel of "art" professionals you'll get wide difference of opinion not only because of different tastes, but because different artistic professions put different emphasis on various aspects of skating. I ( a musician) was watching skating with my SIL ( a modern dancer) and it was rather striking that we put value and attention on different things, all having to do with those remaining PCS.

PCS also does not cover well a few important things which would do better if they were spelled out. Though musicality is hiding somewhere behind Interpretation, it is really NOT the same thing; and musicality historically was very important in skating. Same for such things as unison, point, esthetic beauty and such elusive things as chemistry for Pairs and Dance. As long as they are acknowledged somewhat covertly in descriptions of PCS rather than spelled out as inportant things, they are going to be lacking.

So what would be good revisions of the PCS criteria to help judges do a better job of judging things that the sport has historically valued (and encourage skaters to do a better job of incorporating them in their programs)?

Beyond a page in a rulebook, what would be good training for judges who do not have an arts background to learn how to judge these things?

Would there be any value in training visual and performing artists, or art critics, to judge skating, assuming any would be willing to volunteer their time to learn the sport and then travel around judging?