Ranil Wickremesinghe said the government believed Sunday's attacks could not have been carried out without help from terror groups abroad.
IS claimed the attack on Tuesday, although did not provide evidence.Meanwhile Sri Lanka's president vowed to "completely restructure" state security after the attacks.
In a televised address late on Tuesday, Maithripala Sirisena said he plans to change the heads of defence forces "within the next 24 hours".
He also admitted threat reports had not been shared with him, and promised to take "stern action" against security officials.
What did the prime minister say?
Sri Lanka's government has blamed the blasts on local Islamist group National Thowheed Jamath (NTJ).
But Mr Wickremesinghe said the attacks "could not have been done just locally"."There had been training given and a coordination which we are not seeing earlier," he said.
Police have now detained 40 suspects in connection with the attack, all of whom were Sri Lankan nationals. A state of emergency remains in effect to prevent further attacks.
Mr Wickremesinghe said that only Sri Lankan nationals had been arrested in connection with the attack so far, but that some of the attackers may have travelled abroad before the bombings.
''We, certainly the security apparatus, are of the view there are foreign links and some of the evidence points to that. So if the IS (Islamic State) claimed it, we will be following up on this claim," he added.
The nearly simultaneous attacks targeted three churches packed for Easter services and three major hotels in the capital, Colombo.
An attack on a fourth hotel on Sunday was foiled, Mr Wickremesinghe said. He also warned that further militants and explosives could still be "out there" following the attack.
Who could be behind the attacks?
IS said it had "targeted nationals of the crusader alliance [anti-IS US-led coalition] and Christians in Sri Lanka" via its Amaq news outlet.
It provided no evidence for the claim but shared an image on social media of eight men purported to be behind the attack.
The group's last territory fell in March but even then experts had warned it does not mean the end of IS or its ideology.
Earlier, the country's defence minister Ruwan Wijewardene told parliament that NTJ was linked to another radical Islamist group he named as JMI. He gave no further details.
He also said "preliminary investigations" indicated that the bombings were in retaliation for deadly attacks on mosques in Christchurch, New Zealand, in March.
NTJ has no history of large-scale attacks but came to prominence last year when it was blamed for damaging Buddhist statues. The group has not said it carried out Sunday's bombings.
The Sri Lankan government is facing scrutiny after it emerged the authorities were warned of about a possible attack.
Security services had been monitoring the NTJ but the prime minister and the cabinet were not warned, ministers said.
'Targets in line with IS ideology'
Analysis by BBC Security Correspondent Gordon Corera
The Sri Lankan government has said locals from two known groups carried out the attack. But from the start - because of the scale and sophistication of it - they have also said they thought there was an external role.
In the past, IS has sometimes claimed attacks that it was not involved in or which it simply inspired. But the details from IS would seem to back up the government's assessment.
The choice of targets is much more in line with IS ideology than with the traditional types of communal violence seen in Sri Lanka.
There are still questions - did the local men affiliate themselves to IS or receive direct support? Did they travel to Syria or to other countries? The Sri Lankan government has said it believes some of them had spent time abroad, but how significant was that to the plot?
Answering questions like these will be important not just for Sri Lanka but other countries as they try and understand whether other relatively small, locally focused groups could be capable of transforming a threat into violence on such a massive scale.
Who were the victims?
The first mass funeral was held on Tuesday, as Sri Lanka marked an official day of mourning for the victims.
Most of those who died were Sri Lankan nationals, including scores of Christians attending Easter Sunday church services.
Sri Lankan officials said 38 foreign nationals were among the dead, with another 14 unaccounted for. The death toll includes at least eight British citizens and at least 10 Indian nationals.
The mass funeral for about 30 victims took place at St Sebastian's church in Negombo, north of Colombo, which was one of the places targeted in Sunday's blasts. Another funeral service was scheduled for later on Tuesday.
A moment of silence was also observed at 08:30 on Tuesday, reflecting the time the first of six bombs detonated.
Flags were lowered to half-mast and people, many of them in tears, bowed their heads in respect.