PDA

View Full Version : Interesting aticle: Natural Bias in sports judging



Pages : [1] 2 3

maatTheViking
11-04-2010, 06:17 PM
So, my main sport is dressage, and one of the main (independent!) news sites had a very interesting article posted about Natural Bias in sports judging, and how judging complex movements (such as figure skating, gymnastics or dressage) is simply too hard for the brain without using any kind of prior bias or knowledge.

The guest article is a written by a researcher in sports psychology, Inga Wolframm, and I thought it was very enlightening, and applicable to Figure Skating as well.

Enjoy!

http://www.eurodressage.com/equestrian/2010/11/04/natural-bias-hidden-controversy-judging-sports (http://www.eurodressage.com/equestrian/2010/11/04/natural-bias-hidden-controversy-judging-sports)

Reading the article, do you think figure skating judging could be made more simple? How? I don't have a firm enough grasp on all the teknik stuff to have any ideas.

I also wondered if the judges can see each other marks/screens sitting that close in competition?

mag
11-04-2010, 06:45 PM
That is a fascinating article. Thanks for posting.

joeperryfan
11-04-2010, 06:52 PM
Thanks for posting, this article is quite interesting and I'll make sure to comment once I've read it thoroughly. :)

gkelly
11-04-2010, 07:01 PM
Reading the article, do you think figure skating judging could be made more simple? How? I don't have a firm enough grasp on all the teknik stuff to have any ideas.

The author writes:


Ultimately this means that judging tasks should be made less difficult and complex. Judges need to be able to rely on systematic judgements based on objective and transparent evaluation criteria. To that end very clear guidelines should be provided highlighting, for example, which type of error deserves which kind of penalty.

In skating, I think that equates to breaking down the technical scoring into scoring each technical element separately, and getting different panels to determine what was performed and how well.

Looking at all the numbers generated by that process is more complicated than just looking at one Technical Merit mark, but the process of coming up with each of those numbers is a lot simpler than trying to keep all the elements in mind to assign a single mark, and it's less subject to skate order effects.

The PCS scoring is still holistic and therefore still subjective and still subject to all these effects.


I also wondered if the judges can see each other marks/screens sitting that close in competition?

There is such a thing as a filter that can be put over monitors to make them unreadable from anything but a straight-on view, but I don't know if it's common practice to use them on the judges' monitors.
http://www.tech-faq.com/computer-privacy-screen.html

dinakt
11-04-2010, 08:45 PM
Very interesting article and very relevant.

overedge
11-04-2010, 10:14 PM
Great article. Thanks for posting the link.


I also wondered if the judges can see each other marks/screens sitting that close in competition?

I would guess not - they look like small screens with a lot of information on them - e.g. video replay, which has to be large-ish so the details are visible - and the judges are sitting with some space between each seat. So unless they are entering the marks in 48-point font or some such, I think it would be pretty difficult to read a screen that was not your own.

krenseby
11-04-2010, 10:53 PM
So, my main sport is dressage, and one of the main (independent!) news sites had a very interesting article posted about Natural Bias in sports judging, and how judging complex movements (such as figure skating, gymnastics or dressage) is simply too hard for the brain without using any kind of prior bias or knowledge.

The guest article is a written by a researcher in sports psychology, Inga Wolframm, and I thought it was very enlightening, and applicable to Figure Skating as well.

Enjoy!

http://www.eurodressage.com/equestrian/2010/11/04/natural-bias-hidden-controversy-judging-sports (http://www.eurodressage.com/equestrian/2010/11/04/natural-bias-hidden-controversy-judging-sports)

Reading the article, do you think figure skating judging could be made more simple? How? I don't have a firm enough grasp on all the teknik stuff to have any ideas.

I also wondered if the judges can see each other marks/screens sitting that close in competition?

This is the point that I found the most interesting: "Furthermore, in aesthetic sports... different movements are extremely complex, consisting of a number of technical and artistic elements that all need to be considered at once. However, research has shown that the processing of such complex information simply exceeds human capabilities. In order to be still be able to provide relevant scores within the given timeframe, judges fall back on schemas.. or “short-cuts” .. based on a number of different information sources, such as the athlete’s reputation, their previous performances, which team they belong to etc., [these] .. help judges come up with judgement decisions that, in their mind, approximate actual performances."

This basically means that skaters are actually graded based on previous performances rather than on the current one, because all the movements in the program are too complicated to process and judges automatically fall back on an impression of the skater's previous performances.

Blair
11-04-2010, 11:19 PM
This offers a great explanation for how PCS have manifested themselves over the last 7 years since the IJS was introduced.

Really interesting article! Thanks for posting.

Ziggy
11-05-2010, 12:06 AM
The PCS scoring is still holistic and therefore still subjective and still subject to all these effects.

There is no exact criteria and no methodology for coming up with the score.

When with Monika we tried to do an experiment with judging PCS at 2010 Worlds, we had to come up with our own methodology, based on the very vague guidelines.

The other thing which really bugs me is judges only noticing the first thing and failing to notice anything that came afterwards. It's very well studied and described in Social Psychology but I can't remember what this effect was called now.

So in effect - somebody puts a hand down and then put the foot down. Judges give -1 GOE for that element.

When you look at the deduction sheet, they should have deducted -1 for the hand down and -2 for the foot down, which in combination gives you a -3 GOE deduction.

alilou
11-05-2010, 02:21 AM
Very interesting article. I'm glad I read it. I don't have any brilliant insight or suggestions, but just want to say that it kind of helps me relax about it all, like I can finally exhale about the judging because it's just the way human beings are, and it doesn't matter what system is used these same "schemas" will still apply because it's a judged sport. Still, I do think a separate panel for TES and PCS would really help but I doubt that's ever going to happen.

Ziggy
11-05-2010, 02:25 AM
Still, I do think a separate panel for TES and PCS would really help but I doubt that's ever going to happen.

It would help but with ISU keeping making all the cutbacks... No chance, yep.

dinakt
11-05-2010, 03:17 AM
It would help but with ISU keeping making all the cutbacks... No chance, yep.

Probably no chance, but that's my very strong wish, as well; separate the panels so people have specific limited tasks and actually can pay attention to technique and to performance/ choreography/ artistry- separately.

BreakfastClub
11-05-2010, 05:18 AM
Reading the article, do you think figure skating judging could be made more simple? How?

Go back to 6.0 and crack down on the cheating.

I'm really not trying to be a jerk saying that. 6.0 was a very simple system - rank the skaters. Period. Cognitive science research has proven over and over that the human mind is much more effective at comparing things to each other (6.0) rather than against an arbitrary standard (COP).

Sure it was easy to get a bloc together and that led to controversial 4/5 and 6/3 splits on the medal stand at the elite level, that were debatable for reasons of preference, politics or reputation.

They had to toss out the toe tappers, the Marie-Reines, and the Alla Shekhovtsovas, but 6.0 was simpler and generally led to more logical results than the craziness of COP. Now judges need to assign 7-12+ or more GOE marks against arbitrarily assigned pages of standards/criteria, then assign five more overall PCS marks based on even more arbitrary criteria they need to memorize.

And they need to do this all while trying to guess and stay "in the corridor" (the ultimate piece of BS) based on a skater's reputation.

Then add in the fact that the base value for each technical element is arbitrary (yank your blade over your head and get more points, wheeee!!!) quads are now suddenly worth more this year, wheeee!!!), and the fact that there's a powerful caller out there splitting hairs to assign a level, a downgrade, etc....

Ah, 6.0, where did you go?

Great article. And I love dressage. Thanks for posting!

aftershocks
11-05-2010, 06:29 AM
Thanks for posting that article ... fascinating read. I too love dressage and anything to do with horses and horseback riding. [sidenote: Johnny attributes his erect posture on jump landings to his equestrian skills]

I think what the writer said about judges relying on politics and athletes' reputation to help decide their scoring is pretty much the main modus operandi in figure skating judging. Let's not forget too that the Code of Points was essentially rushed into being mainly to protect the judges rather than helping to fairly judge the skaters. Whatever benefits may accrue as the system continues to develop, IMO, are tarnished by anonymous judging and the way CoP was rushed and forced into existence.

So true re skaters being judged by previous performances -- case in point, Jeremy Abbott at Worlds 2010 (judges apparently couldn't forget images of Abbott falling and stumbling through his Olympic short progam -- one of the two best sp of last season, the other of course belonging to Daisuke). Abbott skated his sp beautifully at Worlds, but was marked lower than he deserved.

There are rare occasions (Michelle Kwan many times, Brian Boitano at '88 Olympics, Rudy Galindo at 1996 Nationals and Worlds) where the judges had in mind to score differently, (i.e., politically, and based on things other than the skaters' performances), but couldn't in light of magical, bring down the house performances.

Generally, I think figure skating is even more difficult to judge based on a set of criteria than other sports such as Gymnastics and Diving, because figure skating is sport and performance art, while I think gymnastics and diving have important aesthetic aspects, the performance aspect does not play as significant a role as it does in figure skating. For me taking a skater's performance apart to score on specific elements, without also truly looking at the whole and judging the whole without political bias and manipulation of PCS, is largely what sucks about current system .. along with the anonymous judging.

millyskate
11-05-2010, 08:26 AM
The other thing which really bugs me is judges only noticing the first thing and failing to notice anything that came afterwards. It's very well studied and described in Social Psychology but I can't remember what this effect was called now.

When you look at the deduction sheet, they should have deducted -1 for the hand down and -2 for the foot down, which in combination gives you a -3 GOE deduction.

This can go further... When I was on a few piano panels, people would often get completely obsessed with one detail. Something they'd noticed at the start, and then failed to pay attention to any of the rest.
It was generally the all-rounders that suffered.

Starting off strong and collapsing at the end, or collapsing at the start and pulling it together at the end was often forgivable, but encountering a few problems interspersed throughout was generally the kiss of death.

Being small and cute was a MASSIVE bonus. Any child tall for their age or slightly overweight was doomed unless they were outstanding, they rarely got more than a pass. I used to take :EVILLE: pleasure in looking up all the birth dates and pointing out the "small cute one" was the oldest of the pack.