The attack on Rann, home to a camp housing tens of thousands of people displaced by the Islamist insurgency, was one of the group's bloodiest.
It came two weeks after Boko Haram had overrun the same town, driving out Nigerian soldiers and signaling its re-emergence as a force capable of capturing army bases.
The bloodshed is a challenge for President Muhammadu Buhari in particular as he seeks a second successive four-year term in an election on Feb. 16, having been elected in 2015 partly on a promise to restore security.
The military did not respond to calls and messages seeking comment.
The two attacks have driven some 40,000 people to flee, 30,000 of them into nearby Cameroon, according to aid agencies.
"This attack on civilians who have already been displaced by the bloody conflict may amount to possible war crimes," Osai Ojigho, Amnesty's Nigeria director, said in a statement.
"Witnesses told us that Nigerian soldiers abandoned their posts the day before the attack, demonstrating the authorities' utter failure to protect civilians."
Two security sources told Reuters that Nigerian troops had returned to Rann along with Cameroonian troops from a multinational task force after the first attack on Jan. 14.
After the Cameroonians went back across the border, the Nigerian soldiers left because they did not have enough troops, weapons or equipment, the sources said.
The next day, Boko Haram came. The militants set the town ablaze and executed its elders and traditional leaders, leaving one alive as a witness, one of the security sources said.
A video of the attack reviewed by Reuters showed people gathering belongings and fleeing, screaming, as the town burned.
A surge in attacks in the northeast by Boko Haram and its now more powerful offshoot Islamic State West Africa Province is throwing preparations for the presidential election into disarray.
Despite the violence, Vice President Yemi Osinbajo said on Tuesday "the current rate of return of these IDPs (internally displaced people) and refugees is encouraging, and this is due to improving security in many areas that were once completely out of bounds."