Order in the court

By Paul Connolly

It’s five games all in the third set of a women’s tennis major final and a second serve at deuce is called long. The courtside crowd utters a collective “oooooooooh”. The server gestures to the umpire, and an electronic review process begins.

Within seconds, a 3D graphic of the ball in flight appears on stadium – and television – screens and the stadium resounds with a slow handclap, which builds to a “waaaaaah” crescendo as the ball’s elliptical-shaped landing point is revealed to be… just touching the service line. Advantage server.

The ball-tracking technology that made the call – and has come to play an increasingly influential role in officiating a number of sports, not least tennis – is called Hawk-Eye.

Currently a product of Sony-owned company Hawk-Eye Innovations, it was developed in 1999 by English amateur cricketer Paul Hawkins. A few years earlier – as a PhD student studying artificial intelligence – Hawkins was so chagrined by an LBW decision that went against him that he began thinking of technological solutions to bad umpiring.

By the early 2000s, cricket and tennis broadcasters were using his system as an illustrative tool. In a 2012 interview, Hawkins said the catalyst behind Hawk-Eye’s adoption as an official adjudication tool in tennis was the 2004 US Open quarter-final between Serena Williams and Jennifer Capriati. Williams lost in three sets and was, according to then-illustrative-only Hawk-Eye, on the wrong end of some contentious line calls. “Hawk-Eye … please”, commentator John McEnroe said at one point, advocating for the tech’s usage as an officiating aid. “This is getting ridiculous.”

By the end of 2005, after working with Hawk-Eye to test and further develop the system, the International Tennis Federation (ITF) – which runs the Grand Slam tournaments –adopted the technology. In 2006 it was used at 10 events, including the US Open. Today it’s used at more than 80 events, including three of the Grand Slams. Under current rules, players get three unsuccessful challenges per set, so they have to be judicious when calling on Hawk-Eye to settle the matter.

So how does Hawk-Eye work?

Used for tennis, Hawk-Eye relies on the information provided by 10 high-speed video cameras – five trained on one side of the court and five on the other. These cameras don’t, as you might expect, move in order to track the ball in flight. Considering the average speed of a first serve on the men’s professional tour is around 180 kilometres per hour (or 50 metres per second), that would be difficult to do over the 23.77-metre length of a tennis court.

Rather, the cameras are in fixed, precisely located positions, which differ from court to court due to differences in stadium design. Before a tournament, Hawk-Eye staff set up the cameras in their various positions. Some will be at relatively high and distant elevations (such as on the underside of a stadium roof, up to 20 metres high), others might be secured to the lower tier of a grandstand, or even – on an outside court – to poles little higher than an umpire’s chair.

The more elevated cameras capture something like a bird’s eye view of the ball – approximating, if you like, the x- and y-axes on a grid – while cameras at lower levels supply information on the height of the ball above the playing surface, or the z-axis.

After installation, the cameras are calibrated to the court – an hours-long process.

While the dimensions of all pro-standard tennis courts are ostensibly the same, there are tiny variations that must be accounted for, for instance in such things as line painting and surface flatness. For these, Hawk-Eye technicians measure all court lines and use a laser to determine if and where the court has undulations – and they’re just two of the many things measured for input to the system in order for it to be consistently accurate.

Next, Hawk-Eye techs distribute more than 70 tennis balls all over the court in precisely measured locations. Using the balls as markers, the techs ensure that each camera works in concert with the others so that their combined views, when stitched together, cover the entire playing area (including the spaces outside the court lines).

Once the calibration procedures are completed, both Hawk-Eye staff and the ITF rigorously test and verify the system – the ITF uses high-speed (2000 frames per second) cameras mounted one centimetre above the court surface to track balls fired from an air-cannon onto the court. Once the ITF has reviewed their high-speed camera footage and is satisfied that the system is tracking accurately, it’s passed for use.

A ball in play passes through multiple cameras’ fields of vision and at least three of them, at any one time, capture the passage of the ball through the air (some cameras may be obscured by a player running for, or making, a shot).

From each individual frame in every video feed, Hawk-Eye’s computers identify the centre of the ball among the pixels in the image. Given the ball is captured by a number of cameras covering that particular area of court, its position in space can be pinpointed using triangulation.

Because the process is repeated for every frame, Hawk-Eye can capture and re-create the trajectory of the ball, literally by joining the dots (the triangulated position of the ball at any given time) to make a smooth curve. By extrapolating this curve, the ball’s landing point can then be projected. When a line call is reviewed, television viewers, spectators, players and officials see this curve as a virtual reality graphic laid over a 3D rendering of the court.

According to Machar Reid, head of innovation at Tennis Australia, who works with Hawk-Eye technicians at the Australian Open, 10 cameras is about the right number to strike a balance between accuracy and cost. Reid doesn’t elaborate on the price tag but it’s been estimated that Hawk-Eye costs from US$60,000 to $100,000 – AU$78,000 to $129,000 – per court.

“You get to the point of diminishing return,” says Reid. “Beyond three or four cameras, depending on the capture volume, the benefit of having additional cameras begins to reduce.” Perhaps not surprisingly, Hawk-Eye isn’t infallible. A ball travelling at 160km/h would move 44 centimetres between the frames taken by a camera operating at 100 fps. Hawk-Eye’s website says its “ultra-motion cameras” operate at 340 fps – but even using those, our 160km/h ball would travel 12.3 centimetres between frames.

Pixel resolution might also affect the measured accuracy of the ball’s position during flight. Hawk-Eye’s self-confessed average margin for error is 2.2 millimetres – shorter than the length of the fuzz on a tennis ball.

But the result of all this is that some estimation ofthe ball’s trajectory must be made in the moments before it bounces, and that the mark the ball makes on landing – its “footprint” – seen on Hawk-Eye recreations may not be the actual footprint the ball made.

“You have to infer [the ball’s footprint],” Reid says. “Hawk-Eye talks about millimetres of error, and that error will be borne out of that [footprint] estimation.”

It’s understood that Hawk-Eye uses a series of algorithms – based on the playing surface and the speed and trajectory of the ball before it lands – to calculate the crucial ball mark. The predicted compression of the ball is also taken into account.

A lob, for example, will drop at a steeper trajectory, leaving a wider and more rounded mark than the more elongated skid marks left by flat, hard strokes or slices, which come in at more oblique trajectories.

While the margin for error isn’t acknowledged in Hawk-Eye’s graphics, its estimations are thought to be notably more accurate than those made by the human eye.

When we watch tennis, we add more subjective decision-making inputs to our sensory data – such as considering how well a stroke was played – to identify where a ball has landed.

Reid acknowledges that triangulation to track objects in space has been utilised in many fields long before Hawk-Eye. “But Hawk-Eye has just done that in a far more accessible way than anyone before them,” he says. “And they were clever enough to try and tackle real problems in sport – ones anchored in officiating”

There were early teething problems, such as at the 2007 Wimbledon men’s final when, locked in an epic five-set struggle against his great rival Rafael Nadal, the normally unflappable Roger Federer flapped when convinced that balls called in were not – only to discover that, well: “computer says no”.

But as officiating tools, electronic line judges – whether Hawk-Eye or its newer competitors such as Foxtenn (which uses slow-motion video replay from 40 cameras with speeds of up to 2500 fps) – seem set to stay. Perhaps the best reason is that the players like it and accept it.

“I think they appreciate [Hawk-Eye] holds an advantage over the human eye, and it’s been around long enough that the system is reliable, and trust has been built through that,” says Reid. “There’s also uniformity in knowing the technology is not biased against you. It’s the same for everyone.”

Paul Connolly is a Melbourne-based journalist and writer. This is an excerpt from his article in issue 85 ‘Packing for Planet Next‘ of Cosmos magazine. You can subscribe to Cosmos here.