The Taliban's protection racket....
It's not just sharia law that's the problem...
At least the fighting and bloodshed was over, he reasoned. But he was wrong.
First the strutting Taliban gunmen demanded 100,000 rupees (£800) a month from his family's dried fruit business. Then when they had drained it of cash they decided to make a bloody example of his two young uncles who could no longer pay.
"They took Aminullah and Sajjar away to the outskirts of the village and there they beheaded them. Afterwards they threw the bodies in the river," he said in a quiet voice.
"We have run from Swat to this new place. But now the Taliban have nearly caught up with us again."
After leaving Swat, escaping through Buner valley, the family arrived at a village near the town of Swabi, where they rented a house with what was left of their savings.
But then the Taliban invaded Buner, bringing them to within 20 miles of Mr Ali's new home.
Now the Taliban seem poised to spread like a rising tide into the fertile farmlands that lie between them and the capital Islamabad, only about 40 miles away.
Unlike the poor rural border areas where the Taliban have their main strength, the region around Swabi is one of the wealthiest in Pakistan, its rich farmlands irrigated by canals, modern roads with petrol stations every few miles, and a standard of living of which most can only dream.
Mr Ali's eyes filled with tears as he told his story, one which is all too familiar to refugees fleeing the Taliban. His family has suffered not just at their hands, but at the hands of the Pakistani security forces tasked with hunting them down. He lost 11 of his relatives in Swat in an army bombardment, unleashed after a bogus tip-off that Taliban fighters were hiding in their home. Many refugees claim that the army has killed more civilians than the Taliban when it has used heavy firepower.
A few days later another nine members of his extended family died, caught in crossfire while trying to escape fighting. The survivors fled, as more than half a million have in Pakistan, to become refugees in their own land.
Mr Ali has no doubt why his family was singled out by the Taliban. It was not because they were held to have violated some extremist Islamic doctrine, but because they owned a business.
Their Taliban persecutors were drawn from the ranks of the landless and the dispossessed, a mix of young men from Swat, Bunir and more remote valleys and a sprinkling of foreigners who are believed to be Afghans and Uzbeks. Some Talibs want to impose the extreme religious vision they have grown up hearing about in madrassas, the strict religious schools where only the Koran is taught. But many simply want to seize a once-in-a-lifetime chance to enrich themselves in the chaos by looting and extortion. Both types seem to enjoy taking revenge on the businessmen and landowners whom they have always resented.
The Taliban's propaganda makes much of their claim that they are enforcing Islamic laws by taking money from the rich and giving it to the poor. In the north of Swat, landowners have been chased out and their cherry orchards handed over to villagers. Elsewhere, the better off pay a "tax" that resembles a protection racket.