Rossano: Current Replay Systems Not Up To Task of Insuring Accurate Calls

[Sylvia]

As Many as Twenty Percent of Technical Panel Jump Reviews May Be Wrong
by George S. Rossano
(17 October 2019) Twenty percent is only an estimate. It could be more, it might be less, but it is surely a significant number. This is the sad fact of competition scoring. Since IJS development began over 16 years ago in 2013, no one has ever conducted a rigorous, unbiased, quantitative study of the accuracy of Technical Panel calls. Not once. Not during development and not since. To an engineer, IJS Technical Panel scoring is mostly a system of educated estimates, and not a proven absolute measurement system, with a quantitative validation, that it was meant to be.

Full article is here: http://www.iceskatingintnl.com/archive/commentaries/ReplaySystem.htm

 

[mackiecat]

And how much is that going to cost?

 

[Spun Silver]

A very interesting and rewarding essay (no pun intended)! It's a bit surprising after all the detail on the current system's grave flaws, a solution could be fairly easily achieved - a conclusion that isn't explained. I hope that's true and that we'll get an improved camera and analytical system soon. It sounds like there could be some surprises when that happens.

 

[Willin]

I mean if I'm being honest I don't think a lot of this is the replay system as much as it is the tech panels - as the article suggests the system is not objective but rather estimation. The tech panels are pretty inconsistent and missing a lot of calls obvious in real time. Some callers are known to be "harsh" like Shin Amano and some are known to let skaters get away with pretty much everything. So while all this tech stuff the article suggests as solutions may be a good solution, perhaps a better short term fix would just be to get all the tech panels on the same page in the first place?

A lot of fans (and this article) only talk about calls on jumps (<, !, e) because that's what fans know best and the stuff that's easiest to call. What I have a lot of annoyance about is consistency in step sequences and spins. Some callers are more strict about spin rotation than others. I think that could be more easily measured than jump things or might be a better place to start counting rotation as there's no small adjustment of feet to measure as well - the whole body can be used as a tool to measure rotation.
And forget about calls on step sequences - the Eteri girls are the best example since they are the most egregiously rewarded for turns on incorrect or flat edges, but you could count most singles skaters (and pairs are somehow worse) as having sketchy edges on turns that should lead to a lot more downgrading than they're getting. If they had an ice dance caller judge singles and pairs step sequences there'd be a whole lotta level 1s, 2s, and 3s and not a whole lotta 4. Every season would become the blood bath that was 2009-2010 where only ONE male skater got a L4 step prior to the Olympics - and at the Olympics I think only 5 or 6 L4 steps were called between the men's SP and FS. TBH I'd enjoy that a lot more than what we have right now. Back then it was a sign that they wanted only complete skaters to win; now with them ignoring proper calling it seems to be another way judges are endorsing skaters who can jump but not do much else.

 

[clairecloutier]

Interesting article! Very technical. The author is a photographer at ISU events, so no doubt his technical knowledge regarding cameras, etc., is sound.

I doubt more technology will be the answer here. It would be expensive to implement and, as the author notes, there’s no indication the ISU is making this a project priority. They prefer to spend their time, apparently, developing elaborate proposals for a new “artistic” event—even as the existing sport becomes ever more technical, with results largely determined by quads, yet with the whole technical side of the judging system resting on a shaky foundation, as this article argues. The cognitive dissonance is .

The article just reinforces my feeling that UR calls should have less impact in the scoring. they still haven’t found a way to make the calls fairly and consistently and in a way that’s not controversial. And as much as some will argue the importance of pristine jump technique, in the absence of a fair and correct way to judge URs, I think they carry too much weight in the scoring.

I do find it interesting that the author argues that the human eye can’t effectively determine URs in real time. I know that’s true for me at least—I can only tell in slow motion replay, and even then, it can be quite difficult sometimes, depending on the angle.

 

[Theoreticalgirl]

@clairecloutier re: URs - they should revert back to the "1/4 and under" rule that was in effect until 2 seasons ago. It's a small detail but enough that it's changed calling quite a bit.

 

[antmanb]

The article just reinforces my feeling that UR calls should have less impact in the scoring. they still haven’t found a way to make the calls fairly and consistently and in a way that’s not controversial. And as much as some will argue the importance of pristine jump technique, in the absence of a fair and correct way to judge URs, I think they carry too much weight in the scoring.

I think the issue is that the scoring system is trying to be as comprehensive as possible and it does come down to a question of what the scoring systems values. Under-rotations became a big issue because people (rightly IMO) wanted to see a penalty. The question that always needs to be asked is - are the penalties correct. I appreciate that the system tries to differentiate between the different GOE errors and the rotation on the jumps. I think the theory is sound with the beyond a quarter but less than half (<) and landing forwards (<<).

I do find it interesting that the author argues that the human eye can’t effectively determine URs in real time. I know that’s true for me at least—I can only tell in slow motion replay, and even then, it can be quite difficult sometimes, depending on the angle.

I think the << are easy to spot in real time, but the < can be trickier though I’ve started to get a feeling for slightly stuttered/hooked landings on those. This GP season I’ve been watching unspoiled (usually I’ve looked over the results and protocols before watching the competition), and was pleasantly surprised to see that the calls I was making in my head were more or less reflected in the protocols. I thought Zagitova might have been given a pass but she wasn’t.

 

[gkelly]

One low-tech possibility would be to stop showing the tech panel rotation and edge calls to the judges.

That way, if the tech panel sees a problem and none of the judges do, the skater would get penalized in base value but not in GOE.

Or if the tech panel doesn't see a problem but some judges do, the penalty would show up in GOE only.

For really egregious errors, everyone would penalize.

For really pristine jumps, full base value and all positive GOEs as reward.

However, some jump calls especially +COMBO in short programs do have to be shown to judges so they know which rules to apply to which elements.

 

[Foolhardy Ham Lint]

Wow. All of this new technology, and marking is still going to pot. Maybe the judges should sit on the ice like they used to years ago.

My eyesight is pretty good. I've seen calls go awry ten to twenty rows away from the ice.

I'd be happy to volunteer my services and smack any technical callers on the back of the head I think are wrong. That should get them up to speed in no time.

Talk to the hand!

 

[greenapple]

George Rossano is a scientist by profession and has a brilliant analytical mind. He also judges at lower level competitions and has studied the IJS since its inception.

 

[VGThuy]

He very articulately stated what I’ve been trying to express in the past countless seasons. So did @clairecloutier! A lot of > talk among the fans has also been a way for fans to attack skaters they don’t like either by using calls to validate their dislike or to complain about a skater not getting a call thus giving them another opportunity to bash them again.

 

[MacMadame]

His paper instantly reminded me of our discussion in Trusova's thread there people insisted that the camera showed that her quads aren't real while others of us pointed out the limitations of the cameras and that based on the angle shown, you really couldn't tell.

 

[VGThuy]

His paper instantly reminded me of our discussion in Trusova's thread there people insisted that the camera showed that her quads aren't real while others of us pointed out the limitations of the cameras and that based on the angle shown, you really couldn't tell.

That’s how I often feel when people argue about >. Like I truly don’t believe we truly see when the blade actually hits the ice when people argue it did to make their > claims and use camera angles and slo-Mo and what not. The angle, the white sheet of ice, and us not having a super close up to see exactly when the blade hit the ice is why I feel that way. I mean it may be a centimeter more or less still off the ice when people who are very sure argue the blade hit the ice. It’s like when people miss typos in a first or second draft. Their minds automatically correct the mistakes when they read it over, and if people want to see a blade hitting the ice on that angle for sure then they will see it.

 

[kwanfan1818]

Craig Buntin had been involved in a venture to create software that could make those determinations and calculations.

 

[Miki89]

The article just reinforces my feeling that UR calls should have less impact in the scoring. they still haven’t found a way to make the calls fairly and consistently and in a way that’s not controversial. And as much as some will argue the importance of pristine jump technique, in the absence of a fair and correct way to judge URs, I think they carry too much weight in the scoring.

I agree. My problem with the system's answer to jump technique issues is that it was always penalty-based when it should be solution-based. There should be more of an effort to help coaches and skaters adopt correct jump techniques. Yet, the work is mostly left to individual coaching teams as it has always been. And it is obvious not all coaches are great at teaching jump technique. Therefore, even more than a decade after IJS start enforcing these rules, URs are still a prevalent issue. Skaters are getting called more than ever but I don't really see a significant improvement in the actual jumping technique among the skaters. Not to mention that the excessive emphasis on jumps have been a detriment to everything else that was great about skating. I rather see more of an focus on accurately scoring skating skills.

 

[Rock2]

Craig Buntin had been involved in a venture to create software that could make those determinations and calculations.

Yes, using single camera IIRC. He started first with skating but got nowhere so I think he's making millions working with other sports that have money, along with an inate desire to make correct decisions.

On that last point, while I realize this is at least partially a motherhood fliippant statement, I do believe fundamentally the ISU leaders prefer to let judgment drive as much of the results as possible. They seem to love having control over the results and futures of their members and don't seem to want to let go of that. They also have no desire to police themselves like most credible sports do. Bad officiating does no go unchecked by most major sports, where it's not all that hard to get suspended, lose credentials to officiate internationally, or even lose all priviledges in the most egregious circumstances.

IJS did not come about as a result of Speedy's desire to legitimize the sport. He brought it in kicking and screaming to save the sport in the eyes of the IOC. ISU also doesn't have much $$ so I don't foresee any technology upgrades anytime soon. I'd be surprised if Craig had any meaningful conversation with a big official in skating over the last few years.

 

[Theoreticalgirl]

Just read through that article, very quick thoughts: This article is technical horseshit. I've read better arguments from my first year broadcast students.

Also, don't tell me "up to 20%" of calls may be wrong if you have NO STUDY DATA INCLUDED.

[ETA: I will actually respond to some of the points in this because I feel it is important, but today is not that day!]

 

[Theoreticalgirl]

FWIW, the author designed a replay system that can be used at non-NQS non-qual USFS events. Not that I discredit white papers (lotsa good can be in them! just not this one!), but the author has a vested interest in developing widespread adoption of his equipment.

 

[gkelly]

I agree. My problem with the system's answer to jump technique issues is that it was always penalty-based when it should be solution-based. There should be more of an effort to help coaches and skaters adopt correct jump techniques. Yet, the work is mostly left to individual coaching teams as it has always been. And it is obvious not all coaches are great at teaching jump technique. Therefore, even more than a decade after IJS start enforcing these rules, URs are still a prevalent issue. Skaters are getting called more than ever but I don't really see a significant improvement in the actual jumping technique among the skaters.

Or, more skaters are trying harder jump elements, so there are more calls, but the number of correctly executed elements remains about the same or maybe a bit higher.

To the extent that the jump attempts are approaching the limits of what is humanly possible with current equipment, merely pointing out that not all aspects of technique are perfect and that takeoff edges or rotation may be lacking, and penalizing those imperfections in the scoring, will not automatically translate into skaters and coaches being able to achieve those jumps with perfect technique.

But if there are scoring rewards for attempting the elements with less than perfect technique, more skaters will attempt them.

Improvements in coaching techniques may result in more skaters attempting quads, or triple axels or triple-triple combinations, who wouldn't have been close enough try in competition 10-15 years ago.

Not to mention that the excessive emphasis on jumps have been a detriment to everything else that was great about skating. I rather see more of an focus on accurately scoring skating skills.

Agreed. Although I'm not sure "accurate" is the best word to use for scoring skating skills, which rely on a variety of skills and qualities some of which could be objectively and thus theoretically accurately measured, and others of which are qualitative and can only be evaluated.

How would skating skills be better evaluated? And should technology be used to measure those aspects that are measurable?

 

[thvu]

Just read through that article, very quick thoughts: This article is technical horseshit. I've read better arguments from my first year broadcast students.

Also, don't tell me "up to 20%" of calls may be wrong if you have NO STUDY DATA INCLUDED.

THIS! SO MUCH THIS! Regardless of how respected the author is, the article itself is very deficient. Once he starts talking about "3D Space" I literally laughed out loud while my students are taking a test. Even sports that are highly funded don't have the level of technology he's implying is needed.

 

[Theoreticalgirl]

THIS! SO MUCH THIS! Regardless of how respected the author is, the article itself is very deficient. Once he starts talking about "3D Space" I literally laughed out loud while my students are taking a test. Even sports that are highly funded don't have the level of technology he's implying is needed.

I read that and started preparing a literature review in my head around theories of perspective in photography. I have a lot of feelings about "3D space," especially when it's being used/upheld as an objective truth.

 

[MacMadame]

I agree about the 20% comment. And also in there, he said something to the effect that a system that resulted in any bad calls at all was not acceptable. There is no system and no amount of technology that will prevent 100% of bad calls. The goal should not be perfection but to eliminate human error to the degree that is possible.

However, as a photographer and a videographer, I think he is dead on when the talks about the limitations of relying on one camera with one angle for these sorts of calls. And also when he talks about what you can and cannot tell from video replay.

 

[Foolhardy Ham Lint]

George Rossano is a scientist by profession and has a brilliant analytical mind. He also judges at lower level competitions and has studied the IJS since its inception.

Maybe the ISU should finance a movie about George, in appreciation of his good work. I've already thought of the title, too:

Hidden (Compulsory) Figures.

 

[Willin]

Agreed. Although I'm not sure "accurate" is the best word to use for scoring skating skills, which rely on a variety of skills and qualities some of which could be objectively and thus theoretically accurately measured, and others of which are qualitative and can only be evaluated.

How would skating skills be better evaluated? And should technology be used to measure those aspects that are measurable?

I think it world be hard to judge skating skills objectively, but it wouldn't be hard to judge them better than what's being done now. What's being done now in ladies is Eteri girls get 8+ if clean, everybody else gets 8s if they're clean and/or a podium contender, and everyone else gets 6s or 7s. That's not actual judging. That's straight up laziness and corruption. As I mentioned in my post and many a skating person that knows skating skills has pointed out, Eteri skaters have awful skating skills and get rewarded way above their ability - which is why I bring them up - they're only the most blatant example of a larger problem affecting skaters from all countries: jumps being confused with skating skills.

Things we see bad among Eteri girls (for an example, but you can look at less blatant examples as well) that can be objectively seen as skating skills or lack thereof:

• Posture - If you look at ice dancers and synchro skaters, they maintain good posture on crossovers and gain a lot of speed real fast, The best teams in both disciplines are full speed in 1-2 crossovers. So why does almost every female and pair's skater hunch over to help gain speed? (Also why isn't it as big of an issue in men's?) And seriously this is my pet peeve #1: if bad posture is ugly on the ice and good posture is such a basic foundational skating skill we start teaching it in Tots/Snowplow sam, why is it entirely ignored and even rewarded by judges?
• Lean of the skate/leg during edges - an easily visible problem to the trained eye, and something that can be confirmed by computer measurement
• Traced patterns of edges (or hearing the rip of edges) - can be confirmed from overhead or higher up views and by the microphones at ice level; this can show quality of edge or if the skater is a bit wobbly on an edge
• Looking for those flipped/incorrect turns - for some turns you can more easily see mistakes and flipped edges from the tracing on the ice, but for most you can see it in real time; if synchro callers can see flipped edges of every skater in a formation (to the point that they downgrade step sequences more often than our singles callers, even on top teams where everyone has good-looking turns at face value) and dance callers/judges can see a split second of an incorrect edge surely singles callers and judges can do this too
• Ability to gain speed AND maintain speed - you can have poor skating skills and gain a lot of speed, but maintaining it is another matter entirely; in order to maintain it you need to have very good skating skills as far as knee bend and timing of knee bend, maintaining edges (no skidding, wobbling, etc.), avoiding toe picks, staying in one spot of your blade (and making fine adjustments as needed); This issue is one of the reasons why Chock and Bates are said to have poor skating skills as opposed to dancers like the Shibutanis or Hubbel/Donahue - they very obviously lose a lot of speed through their elements due to minute errors in hitting toe picks or wobbling a bit on the blade; This can objectively be seen in real time (at least in the arena) and objectively measured by measuring the skater's speed across the ice.

I'm sure there are a few others, but that's what I can think of for now.

This also plays heavily into what I said in my previous post about step sequence judging/calling becoming basically nonexistent in recent years. Step sequences are skating skills on display, and I'm not seeing a whole lotta bad skating skills being knocked down appropriately.

 

[Tinami Amori]

I think some kind of "technology/sensors" will be the "solution" eventually... but not what is stated in the article.

 

[gkelly]

I think it world be hard to judge skating skills objectively, but it wouldn't be hard to judge them better than what's being done now.

Agreed.

Posture - If you look at ice dancers and synchro skaters, they maintain good posture on crossovers and gain a lot of speed real fast, The best teams in both disciplines are full speed in 1-2 crossovers. So why does almost every female and pair's skater hunch over to help gain speed? (Also why isn't it as big of an issue in men's?) And seriously this is my pet peeve #1: if bad posture is ugly on the ice and good posture is such a basic foundational skating skill we start teaching it in Tots/Snowplow sam, why is it entirely ignored and even rewarded by judges?

For better or for worse, the IJS definitions of the Skating Skills component make no mention of posture:
Use of deep edges, steps and turns
Balance, rhythmic knee action and precision of foot placement
Flow and glide
Varies use of power, speed and acceleration
Use of multi directional skating
Use of one foot skating

The only mention of "Carriage" falls under the Performance component.

I don't know that this isn't an issue in men's skating as well. However, it may be that because men have larger muscles they are able to generate more absolute speed with less effort, which could translate to less need to hunch over. However, tall male pair partners may need to get down closer to their smaller partners.

But in order to get judges to penalize it under Skating Skills, the ISU would need to add something about carriage or posture to the Skating Skills criteria.

Also -- and this is something I've noticed since going back in the 1990s to watch videos of 1970s skating -- it seems that the kind of still upright posture that figures encouraged seemed to produce a disconnect between what the upper and lower body were doing during freeskating, and that skaters such as David Santee who had what was probably considered weaker posture in the 70s looked more modern in their stroking style to 1990s (or 2010s?) eyes.

 

[kwanfan1818]

IJS did not come about as a result of Speedy's desire to legitimize the sport. He brought it in kicking and screaming to save the sport in the eyes of the IOC.

From where do you conclude this? At the time there were reports that he (and some other ISU figures) had this in his back pocket and shoved it through when the IOC gave him that excuse.

 

[Rock2]

From where do you conclude this? At the time there were reports that he (and some other ISU figures) had this in his back pocket and shoved it through when the IOC gave him that excuse.

I don't think the timeline is quite that.

When the scandal happened there were a number of articles such as this quoting Rogge, demanding the ISU clean up the sport or it would risk losing Olympic status. Many of the articles pointed at Ice Dance as the first likely to go.

My understanding is that Ted Barton was part of the team that wrote the program for IJS and he wasn't working on this prior to Salt Lake. This ended up being a rushed initiative to get in place and sort of 'working' by Torino, because Speedy was afraid of Rogge and another scandal. Sonia Bianchetti lays this out in an interview here.

Now, what Speedy did like about the new system was the anonymity of judges, so he did rush the new system through because it would mask block judging. He ended up supporting what was created because it would do just enough to save face for him in the eyes of the IOC.

There is no indication this system was ready before Salt Lake City happened. I mean this sport took decades to approve vocals in music. I have a hard time believing they would move to revolutionize the scoring system with no crisis to inspire the work and expense.

 

[kwanfan1818]

have a hard time believing they would move to revolutionize the scoring system with no crisis to inspire the work and expense.

How quickly they implemented it as and how they bypassed the procedure to involve the Congress to amend the Constitution and General Regs were certainly caused by the judging scandal at Salt Lake City. Whether Cinquanta already had COP in mind before SLC is what is in dispute. At the time, there were strong statements that he did, indeed, have this in mind before SLC, and simply used the excuse of SLC to ram it through. And having the idea in his back pocket before SLC doesn't mean he had it ready to launch. That took work.

 

[MAXSwagg]

It’s definitely more than 20%, imo. Much more.