The referendum on independence of Catalonia was held, despite its invalidation by the Spanish constitutional Court.
The show of force by pro-union groups comes a week after the Catalan government went ahead and held a referendum on secession that Spain's top court had suspended and the Spanish government said was illegal.
In Cibeles Square, hundreds of others people clapped and waved their hands in the air in a crowd which included many families with young children and babies but no flags. The Government of Catalonia, besides other existing debts, owes 100 billion Euros to the Spanish Central government.
It would deprive Spain of about 16 per cent of the national population, a fifth of its economic output and more than a quarter of its exports.
More than Catalonia being oppressed by Spain, it is about an illegal and heinous plan of a minority to impose the partition of Spain on the majority.
Besides the tragedies this political, social and human disaster could cause, the most important banks and economic groups of the region have officially made a decision to leave Catalonia.
Amid the tensions, some banks and businesses declared they were relocating so regional secession would not cause them to drop out of the European Union and the single market.
The rally, created to defend the unity of Spain, was organized by the Catalan Civil Society (SCC) group, with the slogan'Let's recover our common sense!' Some people chanted "Less hate, and more understanding!"
"We are going to stop independence from happening".
People in the capital Madrid waved red and yellow Spanish flags in support of unity, while those in Barcelona, the Catalan regional capital, dressed in white to symbolise their desire for talks and a peaceful resolution to the crisis.
Until now, Rajoy has avoided saying whether or not he would use article 155 - the so-called nuclear option - of the constitution that allows him to sack the regional government and call a fresh local election.
"The ideal situation would be that I don't have to find drastic solutions, but for that to happen there will have to be some rectifications (by Catalan leaders)", Rajoy said.
The prime minister also said he planned to keep extra police deployed to Catalonia before the referendum until the crisis was over.
On Monday, the European Commission backed the Spanish government's stance that the referendum was against the law and called the police crackdown an "internal matter" for Spain.
"Spain will continue being Spain".
The vote, which was won by the Yes side, was broken up by police, who beat voters with batons.