Crash: how computers are setting us up for disaster
by Tim Harford at theguardian.com
The paradox of automation, then, has three strands to it. First, automatic systems accommodate incompetence by being easy to operate and by automatically correcting mistakes. Because of this, an inexpert operator can function for a long time before his lack of skill becomes apparent his incompetence is a hidden weakness that can persist almost indefinitely. Second, even if operators are expert, automatic systems erode their skills by removing the need for practice. Third, automatic systems tend to fail either in unusual situations or in ways that produce unusual situations, requiring a particularly skilful response. A more capable and reliable automatic system makes the situation worse.
Back in the late 1970s I was responsible for supporting an then state-of-the-art communications system (think airline reservation system) with the latest hardware and the most elegant coding. However, at that time the console for the system was a Model 33 Teletypewriter (unless, as we occasionally were forced to do, you input instructions through the front-panel toggle switches).