Text copied for fair use as a NYTimes subscriber:

Ibn Khaldun, the 14th-century North African scholar, wrote that empires tended not to last beyond three generations. The founders of the first-generation are rough men united by hardship, grit and group solidarity, a concept he called asabiyyah. The next generation preserves the achievements of their forebears. By the third or fourth generation, however, the comforts of wealth and status erode ambition and unity, leaving them vulnerable to a new generation of power seekers with fire in their bellies.

In the 1979 Iranian revolution, religious fundamentalists with fire in their bellies transformed the country into an anti-American Islamist theocracy. Today Iran is still led by one of its first-generation revolutionaries — 83-year-old Supreme Leader Ayatollah Ali Khamenei, who has ruled since 1989. Among the reasons for Mr. Khamenei’s longevity is that he rules Iran with the hyper-vigilance and brutality of a man who believes that much of his own society, and the world’s greatest superpower, aspire to unseat him.

Under Mr. Khamenei’s leadership, anti-Americanism has become central to Iran’s revolutionary identity, and indeed few nations have spent a greater percentage of their finite political and financial capital to try and topple the U.S.-led world order than Iran. On virtually every contemporary American national security concern — including the Russian invasion of Ukraine, Chinese threats against Taiwan, nuclear proliferation, and cyberwarfare — Tehran defines its own interests in opposition to the United States.

As I explained to U.S. lawmakers recently, one need only look at how Vladimir Putin’s brazen military adventures in Georgia, Crimea, and Syria convinced him he could invade Ukraine with impunity, to understand how the Islamic Republic operates. The country’s successful entrenchment of powerful proxies in Iraq, Syria, Lebanon and Yemen, coupled with America’s humiliating withdrawal from Afghanistan, have further convinced Iran of its own success as well as America’s inevitable decline. This dynamic has hampered the Biden administration’s attempts to revive the 2015 Iran nuclear deal that Donald Trump withdrew from.

Although the nuclear program has easily cost Iran over $200 billion in lost oil revenue and has not deterred Israel from reportedly carrying out brazen assassinations and acts of sabotage against Tehran’s nuclear sites, the more committed the United States has been to diplomacy, the lesser Iran’s sense of urgency to compromise. Even if the nuclear deal is revived, Tehran’s worldview will endure.

Multiple U.S. administrations have attempted to coerce or persuade Iran to reconsider its revolutionary ethos, but have failed. The reason is simple: U.S.-Iran normalization could prove deeply destabilizing to a theocratic government whose organizing principle has been premised on fighting American imperialism.

Herein lies the conundrum. By and large, the United States has sought to engage a regime that clearly doesn’t want to be engaged, and isolate a ruling regime that thrives in isolation. Yet over time, the Iranian regime has shown it’s too influential to ignore, too dogmatic to reform, too brutal to overthrow, and too large to fully contain.

A sound U.S. policy must reconcile the short-term objectives of countering Iran’s nuclear and regional ambitions without hampering the long-term goal of a representative Iranian government that is driven by the national interests of its people, rather than the revolutionary ideology of its rulers.

In the zero-sum worldview of Iran’s revolutionary elite, opening up the country could bring in competition that would undermine their private mafias. For many among Iran’s political and military elite the battle for power is not about revolutionary ideology or Islam, but about who controls the country’s vast resources.

“At the beginning of the revolution the rank and file of the regime consisted of 80 percent indoctrinated believers — ignorant of global realities — and 20 percent charlatans, and chameleons” a professor inside the country, whose students rose to senior official positions, told me. “Today it is the opposite: 20 percent are believers, and 80 percent are charlatans who flock around officials for wealth and privilege.”

U.S. policy toward Iran has for years faced a paradox that has been poorly understood: The coercive policies needed to counter the Islamic Republic’s nuclear and regional ambitions — i.e., sanctions — may inadvertently serve to strengthen, not weaken, the regime’s grip on power.

When Mr. Trump tried to entice Kim Jong-un with a vision of the riches his country could have — “You could have the best hotels in the world right there” — the North Korean president wasn’t moved to end his nuclear program. There is often a fundamental tension between the self-interest of dictatorships and the well-being of the people they rule.

Although sanctions force adversarial nations to pay a high cost, they do not, on their own — with the possible exception of South Africa — have a strong track record of unseating authoritarian regimes from power. Indeed, some even benefit from their political isolation.

The actor Sean Penn, who had met the late Cuban dictator Fidel Castro, once told me over a dinner we were both attending that “Fidel likes to joke that if America were to ever remove the embargo against Cuba, he would do something provocative the next day to get it reinstated. He understands his power is best preserved in a bubble,” sequestered from international capitalism and civil society.

Like Castro, Mr. Khamenei too understands that the greater danger to his theocracy is not global isolation but global integration. When that isolation becomes too debilitating, Mr. Khamenei is willing to consider a tactical deal to serve as a release valve. For Mr. Khamenei, the ideal position is just the right amount of isolation. Mr. Khamenei wants to be neither North Korea nor Dubai. He wants to be able to sell Iran’s oil on the global market without sanctions, but he doesn’t want Iran to be fully integrated in the global system.

The former president of Iran, Mohammad Khatami, once told me that Mr. Khamenei used to tell him that the Islamic Republic needed enmity with America. Mr. Khamenei has never hidden his cynicism about the United States. “With regard to America” he said in 2019, “no problem can be resolved and negotiations with it have nothing but economic and spiritual loss.”

While Mr. Khamenei’s animosity toward the United States is no doubt earnest, it is also in his self-interest. His commitment to the revolution’s core principles has been ironclad. Compromising any of these principles could erode the group solidarity that Ibn Khaldun long ago observed is central to the longevity of any regime.

Eric Hoffer, the American philosopher put it succinctly in his book, “The True Believer: Thoughts on the Nature of Mass Movements”: “Hatred is the most accessible and comprehensive of all unifying agents,” adding, “Mass movements can rise and spread without belief in a god, but never without belief in a devil.”

If Iran’s revolutionary elite have thrived in relative isolation, why doesn’t the United States simply restore relations with Iran? Built into this question is the assumption that America has the power to unilaterally normalize relations, and Iran has no agency whether to accept or decline.

In contrast to the Cold War, when the United States had a continuous diplomatic presence in Moscow and thousands of trained Russia specialists, the U.S. government has been absent from Iran since the 1979 seizure of the U.S. Embassy in Tehran, and boasts of little in-country expertise.

This estrangement and lack of understanding has fueled what the former U.S. National Security adviser H.R. McMaster has called “strategic narcissism,” the tendency to perceive world events solely through the prism of U.S. behavior. Liberals often argue that engaging Iran could soften its revolutionary ideology or empower regime moderates. Conservatives have argued a tougher U.S. approach could either force Iran to abandon its ideology or risk the implosion of the regime. Neither approach, on its own, has worked.

Since 1979, every U.S. administration — save for that of George W. Bush — has attempted to improve relations with Iran. Jimmy Carter’s administration tried to build confidence with Iran’s new revolutionary regime by sharing intelligence, which would go unheeded, that Iraq’s Saddam Hussein was planning to invade Iran. Ronald Reagan sent three unanswered letters to the Iranian government. George H.W. Bush’s inauguration speech included a message — “goodwill begets goodwill” — for Iran. Bill Clinton hoped to meet Iran’s reformist president Mohammad Khatami at the United Nations in 2000.

Barack Obama wrote multiple private letters to Mr. Khamenei whose response was to suggest ways America “could stop being an imperialist bully,” as Mr. Obama recalled in his latest memoir. Even Donald Trump — whose administration assassinated Maj. Gen. Qassim Suleimani of Iran in 2020 — made at least eight requests to meet with President Hassan Rouhani, according to an Iranian official.

By contrast, there is not a single known example of Iran’s supreme leader initiating a public or private dialogue with U.S. officials in the hopes of normalizing relations. Most recently, he has forbidden his diplomats to meet U.S. officials working on renegotiating the nuclear deal. Mr. Khamenei recognizes that rapprochement with the United States poses far more of an existential threat to him than continued Cold War.

To be clear, the United States has also made catastrophic errors. The 2003 Iraq war spread Iran’s Shia theocracy to Iraq, and facilitated Iran’s regional ascent. The main outcome of the Trump administration’s unilateral 2018 withdrawal from the nuclear agreement is an Iran with a far more advanced nuclear program.

If U.S. attempts to engage Iran have gone largely unreciprocated, and U.S. attempts to coerce Iran have largely backfired, where does that leave us?

There is no silver bullet that can transform the nature of the Iranian regime or the U.S.-Iran relationship. Few examples exist of Iran agreeing to meaningful compromise, but nearly all of them have been under similar circumstances: a combination of sustained global pressure and rigorous U.S. diplomacy, to achieve a specific resolution. In the case of the nuclear deal, that means restraining, rather than eliminating it. And that same formula should be applied to limit — although not eliminate — Iranian influence in the Middle East.

Robert Cooper, a decorated European diplomat who negotiated with Iran, urges strategic patience. “Revolutionary powers don’t think the way others do,” he told me. “They don’t want a different place in the world; they want a different world. It’s no good thinking you can change them, but a moment may come when they begin to doubt or to get over their revolution … then you can start something.”

Mr. Khamenei has not publicly exhibited any doubts, but he has at times shown an ability to make tactical compromises when he has feared his regime’s existence is at stake, and there is a safe path of retreat.

William J. Burns, the director of the C.I.A., and one of the diplomatic architects of the 2015 nuclear agreement with Iran, wrote that the agreement was spawned by “tough-minded diplomacy, backed up by the economic leverage of sanctions, the political leverage of an international consensus, and the military leverage of the potential use of force.” Today diplomacy has not been tough-minded, sanctions are not enforced fully, international consensus is more difficult to obtain and Tehran appears convinced that President Biden has no interest in another military conflict in the Middle East.

The clerical regime that has ruled Iran over the last four decades is terminally ill, yet it continues to endure, in part due to a lack of viable alternatives. It cannot meaningfully reform, out of well-founded fears that doing so would hasten its death. The four horsemen of Iran’s economy — inflation, corruption, mismanagement, and brain drain — are endemic. The common denominators between Iran and its regional spheres of influence — Syria, Lebanon, Yemen, and Iraq — are insecurity, economic failure, and profound unhappiness.

Crane Brinton, the author of the seminal book “The Anatomy of Revolution,” argued that most revolutions experience a radical period, the “reign of terror,” before normalcy eventually sets in. Although revolutionary fervor long ago subsided in Iran, normalcy has been elusive, partly because of powerful entrenched interests in the status quo.

The goal of Mr. Khamenei and his revolutionary cohorts — the remaining true believers — is to avoid a normal Iran, and normalization with the United States, which would deprive the Islamic Republic of the external adversary that has helped maintain the cohesion of the security forces, the asabiyyah that Ibn Khaldun wrote about. Although this is a losing strategy in the long run, the octogenarian Mr. Khamenei’s time horizon is limited. Mr. Khamenei’s priority has never been about Iran’s national interest, but it’s to keep his regime united and the international community divided.

If the four-decade history of the Islamic Republic is any guide, Mr. Khamenei may be unwilling or incapable of marshaling an internal consensus to revive the nuclear deal with the United States unless he feels regime solidarity is faltering, and societal exhaustion is beginning to fuel a new generation of power seekers. The paradox of the Islamic Republic is that it tends to compromise only under severe pressure, yet that same external pressure and isolation help keep it alive.

It is a game Mr. Khamenei has been perfecting for decades.