Analysts say a big loss for the BJP would signal rural dismay and help unite opposition to Modi, though his personal popularity remains high, despite criticism that he was unable to deliver on promises to create jobs for young people and improve conditions for farmers.
The elections are also a test for Rahul Gandhi, the chief of the main opposition Congress party, who is trying to forge a broad alliance with regional groups to mount the most serious challenge to Modi yet in the election that must be held by May.
In the western state of Rajasthan, the Congress was leading in 81 seats of the 199-member assembly against the BJP's 56 in the initial round of voting, India Today TV said.
In the central state of Chhattisgarh, the Congress was ahead in 46 of the 90 seats at stake with the BJP at 22 and was holding to a slender lead in the most populous state at stake, Madhya Pradesh, the network said.
The Times Now channel also said the BJP was trailing in all three states, where it had grabbed almost all the parliamentary seats in its landslide win in the last general election in 2014.
Poll analysts cautioned that with the counting in preliminary stages, it was too early to predict the outcome of state races involving millions of voters.
Local issues usually dominate state polls, but politicians are seeing the elections as a pointer to the national vote just months away.
The Indian rupee dropped as much as 1.5 percent to 72.465 per dollar, while bond yields rose 12 basis points to 7.71 percent after the resignation of the central bank governor.
The broader NSE share index was down 1.3 percent with investors cautious ahead of the state election results.
Equity analysts had warned that Monday's surprise resignation of Urjit Patel, the governor of the Reserve Bank of India, after a long tiff with the government, could send the markets crashing.
"As the three erstwhile BJP states have a large agrarian population, the BJP's drubbing could be interpreted to mean that farm unrest is real," Nomura said in a research note.
"A rout of the BJP on its homeground states should encourage cohesion among the opposition parties to strengthen the non-BJP coalition for the general elections."
Regional parties are likely to retain two smaller states that also report results on Tuesday, southern Telangana and northeastern Mizoram, the polls show.
Gandhi, the fourth generation scion of the Nehru-Gandhi dynasty, has sought to build a coalition of regional groups, some headed by experienced firebrand, ambitious politicians.
Congress has already said it would not name Gandhi, who is seen as lacking experience, as a prime ministerial candidate."When one and one become eleven, even the mighty can be dethroned," opposition leader Akhilesh Yadav said of the prospect of growing opposition unity.