"This is a time for cool heads, not macho chest-thumping," she said.
Geoffrey Tuxworth, 47, Simon Peterffy, 44, and Glen Pendlebury, 27, who belong to an environmental group called Forest Rescue, evaded razor wire and spikes to clamber aboard the Shonan Maru 2 on Saturday night.
Their apparent intention was to try to force the ship to return them to Australia, shaking it from the tail of the Steve Irwin as the protest vessel headed to disrupt the annual Japanese whale kill in the Southern Ocean.
The boarding took place outside Australian territorial waters, placing them outside Australian jurisdiction despite still being with the nation's exclusive economic zone.
Roxon said that the zone gave Australia the right to exploit its economic resources without territorial rights or legal jurisdiction, and that under laws governing maritime incidents the presumption was that Japanese law applied.
"We can't simply enforce our will on a boat which is flagged to another nation and is outside our waters," she said.
Australian National University international law expert Professor Donald Rothwell said the three activists could face a range of Japanese charges, as the unauthorised boarding of a Japanese vessel was an act of trespass wherever it occurred.
If charges were laid the three men would be sent back to Japan to face court.
Alternatively, Japan could decide against prosecution and return the men to Australia, as in 2008 when two activists, one an Australian, had been handed over to the Customs ship Oceanic Viking after boarding a whaler.
Roxon said Australia was in a difficult position. While the Government sympathised with their views the three activists would have been advised of the possible consequences and taken their action "with their eyes wide open".
"If people do take action and take the law into their own hands, the rules that apply are sometimes ones that you can't as a Government change," Roxon said.
"We are now doing all we can diplomatically to ensure that these three Australian men can be released quickly and promptly and to make sure that they are being held in conditions that are appropriate. So it is a consular issue, as well as a legal one."
Roxon said the Government did not support whaling and would continue to oppose it through diplomacy and the International Court of Justice, but that outside this relations with Japan were strong.
"Our discussions with the Japanese Government are making sure, in the strongest possible terms, that we want the Australian men to be returned to Australia promptly," she told Sky TV.
"Our first approaches have been through our embassy in Tokyo, which has a very good relationship with a lot of foreign diplomats and officials within Japan [but] of course, that will be escalated as is appropriate."
But the Government has been attacked by the Opposition, which wants a Customs ship sent to the Southern Ocean.
"These are Australian citizens in Australian waters, an Australian incident, and the Australian Government should make sure the protesters are immediately handed to Australian authorities," said shadow environment minister Greg Hunt.
The Greens have urged personal intervention by Prime Minister Julia Gillard, and Tuxworth's father, Errol Tuxworth, has called for more diplomatic pressure to protest against what he called his son's kidnapping.
