President Barack Obama and Defense Secretary Ashton Carter visited Saudi Arabia last week hoping to reassure nervous Gulf allies that American security commitments remain in force in the wake of last year’s Iran nuclear deal. In exchange, the White House was hoping to encourage the Gulf states to play a more active role in the fight against the Islamic State (IS). Riyadh, which last year became the world’s third-largest military spender and leads a new anti-terrorism coalition of Muslim states, would seem like an ideal partner to crush IS. But a recent large-scale exercise organized by the kingdom suggests the Saudis and their partners are hardly prepared, or even motivated, to realistically combat any terrorist threat.

In December, Saudi Arabia declared the formation of a 35-nation “Islamic military alliance” to counter terrorism. The exercise the coalition held in late February, known as Northern Thunder, was the first indication that its member states may take a more active military role in the region beyond the ongoing Saudi-led war in Yemen – particularly against IS. The coalition’s official Twitter account ambitiously claimed that Northern Thunder was aimed at “projecting real combat capabilities” to prepare member militaries for joint operations against irregular foes and the region’s “terrorist menace.” The remarks appear to support similar statements by Saudi officials in February that Riyadh would send ground troops to Syria to fight IS.

These statements should come as welcome news to Obama, who in a widely-shared recent interview with the Atlantic, expressed frustration with Arab allies, and the hope that they would “carry their weight.” But while Arab states have conducted some strikes in Syria as part of the U.S.-led coalition, it’s clear that operations in Yemen have overshadowed their commitment to the anti-IS campaign. Even there, it was not until recently that the Saudi-led coalition indicated any interest in pursuing the local al-Qaeda and IS affiliates that have grown substantially in Yemen’s year of turmoil. With Gulf states continuing to spend heavily on their militaries despite decreasing oil prices and internal unrest, it is not unreasonable for Washington to expect a more capable partner in solving the region’s military challenges.

But a closer look at Northern Thunder shows that the Saudi-led Islamic coalition is not building capacity for war against irregular threats, such as IS or the Iranian-backed Houthi rebels in Yemen. While well-armed with billions of dollars of foreign-made equipment, the participants practiced maneuvers that reflect only a small part of modern warfare. The drill did not, for example, rehearse the kind of counterinsurgency and stabilization operations that gradually weakened the Islamic State’s predecessor, al-Qaeda in Iraq, before the 2011 U.S. withdrawal. Indeed, much of the exercise appeared devoted to displaying conventional capabilities, including large tank and armored-vehicle formations backed by artillery, warplanes, and air defense systems.

This configuration of men and materiel would be ideal for battling a state army like Saddam Hussein’s forces in the 1991 Gulf War. But insurgent groups tend to avoid sustained linear battles where conventional forces are at an advantage, instead hiding among the population and harassing their opponents until an opportunity emerges to overwhelm them.

This is not to say the forces featured at Northern Thunder, including the armored units, have no value in fighting an unconventional threat. After all, the U.S.-led coalition is re-teaching the Iraqi Army combined arms tactics using tanks, close air support, and combat engineers to advance into IS-held territory. Meanwhile, the United Arab Emirates used armored columns in Yemen last summer to break the rebel siege of Aden.

In many of these cases however, armor proved useful only during an initial assault into enemy-held territory to destroy or encircle fortified positions. But once a group like IS or the Houthis lose control of territory or realize they are outgunned, they simply revert to asymmetric methods and disappear among the population.

In these scenarios, infantry backed by special forces are required to hold and secure the area – potentially for years – until local security forces can take their place. Northern Thunder offered no evidence that infantry units were training for this scenario or any urban environment where asymmetric battles frequently occur. Furthermore, while the drill featured special forces – a hallmark of counterterrorism operations – it did not include drones, except to film dramatic aerial videos. Without drones, or any other airborne reconnaissance assets, it will be extremely difficult to locate insurgent targets and give coalition commanders a clear picture of the battlefield.

The most telling absence from Northern Thunder, however, is of a mock opposing force (or OPFOR), as commonly used in U.S. and NATO drills. These role players copy the tactics of likely adversaries, prompting training units to react to unforeseen conditions. Such training is designed to instill flexibility and confidence in low-level commanders, something Arab armies have long lacked. Instead, Northern Thunder focused on overly centralized command and control to direct scripted maneuvers where the outcomes were all but certain.

This only confirms what many observers already assumed: that Northern Thunder was merely a “don’t even think about it” message to Iran, or even a “we’ll do it ourselves” message to the West. But even here, the coalition is coming up short, as the drill lacked key components that would be necessary in a potential conflict with Iran, such as ballistic missile defense and anti-submarine and counter-mine assets.

Nevertheless, Northern Thunder had some positive attributes. The logistics of moving personnel and equipment at this scale was a learning experience for participants who rarely move their forces beyond their own borders. Additionally, the opportunity to familiarize members with strengths and weaknesses of member capabilities allows the coalition to better prepare. Last, the drill’s land warfare focus could address Carter’s concerns that Gulf states were investing too heavily in their air forces at the expense of ground troops.

Ultimately, the Obama administration will find that the Saudis and their Islamic coalition are not ready to meet the region’s security challenges. With an over-reliance on high-end military hardware – and not on practical, real-world training – the Gulf states will be unlikely to provide anything more than the few airstrikes they have already contributed. It is an unfortunate reminder of why the responsibility for regional security so often falls to the United States.

Patrick Megahan is a research analyst at the Foundation for Defense of Democracies focusing on military affairs. He manages the website