Edmund Burke and George Santayana both remarked that those who do not know history are doomed to repeat it. Judging by the rash of forced departures among university and college presidents in the US, however, even those who know history seem doomed to repeat it.

There are more than 4,000 higher education institutions in the US. They are headed by well-educated, well-trained and well-respected individuals, and these leaders serve their schools, on average, for eight years. Almost all do a fine job day to day, working collaboratively with academic faculty, boards of overseers, fellow staff members and community leaders to make sure their institutions perform two major functions: educate young adults and undertake serious research in all academic disciplines.

Of course, the job of college president is not for the faint of heart or those lacking physical and mental stamina: its pace is relentless and the pressures are considerable. In the past 20 years, the American press has sought to cover university people and events with a beady-eyed intensity previously reserved for those in elected political office. Colleges now come under routine media scrutiny and the presidents, as the most visible public figure on campus, are under a public relations microscope.

Every year, about 50 heads of US higher education institutions lose their jobs as a result of poor performance or scandal. This is too many. The visibility of the dismissals taints the entire educational enterprise. As tuition prices upwardly spiral, leaders’ salaries climb and public attention grows ever more intense, unplanned presidential departures also raise questions about the judgement of those who selected them in the first place.

Most non-voluntary departures are shrouded in confidentiality agreements. Whatever lessons might be learned are buried: there’s a lot of gossip and guesswork on campus and in the press, but the official silence deafens the opportunity to learn from the experience of others. So we set out to research this subject in the hope that our conclusions might help improve the probability of appointing and keeping presidents of academic institutions who serve with competence and integrity. Through conversations with reliable (although sometimes anonymous) sources, we have been able to provide accurate descriptions and analysis of the derailments without betraying confidences.

The literature on derailment in commerce, industry, government and the military is depressingly similar – that is, the causes are well known – and we have found that the reasons are remarkably close to those that end in termination in higher education. From talks with leaders of higher education in other countries, it seems their experience is similar. Some presidents get the boot because of impropriety; some because of poor judgement; others because they become arrogant and self-aggrandising, or because they seem unconscious of the fact that every utterance, indeed every nod or wink, will be scrutinised and broadcast; and some because they appear to have forgotten that every decision creates one winner who may be an ingrate and a hoard of losers who can feel personally affronted.

Others fall victim to dysfunctional boards: those dominated by a cabal; those that cannot distinguish between strategy and tactics; andthose that remain ignorant of the complexities of institutions of higher education. In the extreme, there are even people who seek board positions to enrich themselves and their cronies by directing lucrative contracts their way and demanding that the children of their pals be admitted to the institution regardless of their academic standing or chances of success.

In most cases, it is not a single incident that causes the undoing of a president: it usually takes more than one failure, lapse or incident to lead to deposition.

For the most part, presidents, chief executives and board members are able people who have earned their stripes and paid their dues. So why do they repeat behaviours that proved fatal to those who came before them? Why do some college presidents (and politicians, company chieftains and military generals, for that matter) cheat on their spouses, pad the payroll with relatives, demand kickbacks for steering lucrative contracts to friends, accept or demand personal services from contractors, browbeat their subordinates and flaunt their misbehaviour? Why do they turn a blind eye to ethical breaches committed by those under their command? Perhaps they believe they are exceptions because they are exceptional – bright, talented high achievers. Perhaps they believe that they are too smart to be found out, but they aren’t, as case after case proves. The overzealous will get away with their venal and mortal sins for a while, but not for long. The world is watching: scholars, staff, alumni, students, parents, boards, political and community leaders – all eyes are too sharp, the media too eager for scandal. The leaders cannot escape scrutiny and ultimately fall from grace. They know history but are doomed to repeat it.

In 2007, William Frawley, at that time the successful president of the University of Mary Washington in Virginia, was charged with drunk driving on two occasions over a brief period of time. Frawley’s story, an example of personal frailty leading to derailment, was splashed across the news; he was asked to leave campus (immediately) and not return.

Most non-voluntary departures are shrouded in confidentiality agreements. Whatever lessons might be learned are buried

It is not uncommon for presidents who are dismissed to take a short sabbatical and return to campus as a member of faculty, often excelling in the classroom in ways they did not in the presidential office. Although Frawley held tenure, he was not offered this opportunity.

We interviewed him about the circumstances of his fall from grace and his efforts to heal and reinvent himself. Now working in education in Saudi Arabia, he argued that the job of president had turned him “into a kind of anti-intellectual” and that “maybe to the surprise of many”, he regretted almost nothing but the pain he had caused people.

“As is true for most people, college presidents’ behavior doesn’t always match our ideals,” he wrote in an article for The Washington Post at the time.

On the other hand, the goals set for university leaders can be too ambitious. In US higher education, boards select presidents (it is one of their primary functions). They rely on search committees, professional search firms, interviews and formal background checks to line up the right candidate. Hours are spent crafting job announcements and mission statements. The former always list criteria that would qualify applicants for sainthood, not merely for a university presidency; the latter proclaim the institution’s desire to become the Harvard University of whatever niche it occupies – the Harvard of the Midwest, the Harvard of community colleges. None seem to be content with being the best they can be given their locations, histories, finances, local support, facilities, faculties and competition.

Of course, their reach should exceed their grasp and it may be OK for every 90lb weakling to dream of being Charles Atlas, but at some point failure to achieve such inflated goals will be evident and perhaps embarrassing, wasting valuable resources, and the president will be blamed for being a mere mortal. Too lofty an aspiration sinks to the ground – it is no longer inspirational but self-mocking. The president may not have been impeccable, leaving him vulnerable to being pecked at. Vultures circle on campus as quickly as they do on the prairie; the press have an acute sense of smell for a wounded warrior.

The board won’t be content with hiring the best available candidate: each seeks the best candidate full stop – a being not quite as rare as a unicorn, but almost. She should sing like Maria Callas, run like Sir Sebastian Coe, imagine like Albert Einstein, innovate like Steve Jobs, raise money and invest as successfully as Warren Buffett.

Regrettably, some hire not the candidate but the lustre of where she or he was previously employed. In many public institutions, pressure from elected officials in state and local government may determine the selection. Well-crafted presidential contracts with clear performance expectations in line with the institution’s capabilities are essential but all too rare. In general, presidential searches are too often a matter of luck rather than the outcome of sound processes.

Derailed presidencies matter for many reasons, not just for the individuals involved and the reputation of the sector. They can be an expensive business, too, as multiple costs are incurred: severance payments are usually negotiated; legal bills have to be paid. Meanwhile, the reputation of the institution suffers: faculty, students, prospective students and their parents, alumni, staff, vendors, donors, foundations and grant-making agencies, quality accreditors and bond rating agencies all grow anxious about the institution’s future. The snail’s pace of change typical of higher education decelerates further.

And the derailments continue. Very recently, an incident at the distinguished University of Virginia (founded and designed by Thomas Jefferson) gave the public the opportunity to watch the ill-advised behaviour of a board of visitors collide with a sitting president, Teresa Sullivan. In June 2012, she was capriciously ousted in a nanosecond, only to be re-engaged after a forceful outcry by academics and students seemingly minutes later. Sullivan’s saga captured the attention of Washington (and the entire higher education community) in part because she was embraced by the media as a victim in a struggle between the forces of good and evil.

The president may not have been impeccable, leaving him vulnerable. Vultures circle on campus as quickly as they do on the prairie

So what can be done to avert future “train wrecks”? Here are a few of our recommendations for search committees, search firms, board members, board chairs and would-be presidents. They aren’t dispositive, but they are likely to forestall at least some bad judgements and poor choices.

Undertake a governing board assessment. The departure of a president is a natural opportunity to pause, reflect and assess the operations of the board to ensure that it is functioning ethically and effectively.

Be selective in appointing the search committee. Its composition is often designed to satisfy demographic optics, but not necessarily to engage academic expertise.

Clarify what, if any, professional guidance and support are needed, such as whether to work with a search firm in the quest for a new leader.

Prioritise the characteristics, both personal and professional, of the next president. Without consensus the search will lack focus. However, presidential searches are an inexact art and too many preconceived expectations can be detrimental to the process.

Share information and establish mutual expectations. Derailments often occur because of unresolved issues or unexplored expectations prior to employment offers and acceptances.

Screen candidates with care. In their eagerness to narrow the candidate pool, the search committee may rush to judgement.

Speak with one voice about the appointment. When the board reaches an agreement on a candidate, it is important that the decision appears unanimous, unqualified and enthusiastic. Anything less can tear the institutional fabric.

Manage campus angst with care. Academic faculty, staff, students, alumni and friends of the institution are likely to be wary and anxious after a derailment and during the tenancy of an acting president.

Understand that every presidency is different. It can be difficult to follow a long-standing or beloved leader – and there are aspects of every presidency and institution that are impossible to anticipate.

See a presidential transition as a refreshing moment of opportunity for reflection and re-engagement with stakeholders.

Emphasise the way forward with planning. But beware the itch to start before the new president arrives.

Invest in board resources and activities. Coaching and media training, for example, may be helpful. People will besiege the new president, expecting him or her to grant their wishes – boards can help by creating an appropriate discretionary budget that allows the leader to say “yes”.

Key causes of derailment

• Ethical lapses, ranging from lavish spending to limited information sharing
• Poor interpersonal skills, including arrogant attitudes, volatile tempers and weak communication
• Inability to lead key constituencies, including board members, government officials, cabinet members and faculty
• Difficulty adapting to the institutional culture, the community context or the varied demands of an academic presidency
• Failure to meet business objectives, such as financial goals, fundraising expectations and enrolment projections
• Board shortcomings, from flawed search processes to dysfunctional dynamics or conflicts of interest.

Source: Presidencies Derailed: Why University Leaders Fail and How to Prevent It.