The remarks by Esper in an interview with Reuters came on the heels of a Thanksgiving trip last week to Afghanistan by President Donald Trump, who spoke of potential troop reductions and said he believed the Taliban insurgency would agree to a ceasefire in the 18-year-old war.
If honored by all sides, a ceasefire could lead to a significant reduction in violence. But US military commanders would still focus on the threats associated with two other militant groups in Afghanistan: Islamic State and al Qaeda.
Esper said the Trump administration had been discussing potential reductions in troop levels for some time, both internally and with US allies.
"I feel confident that we could reduce our numbers in Afghanistan and still ensure that place doesn't become a safe haven for terrorists who could attack the United States," Esper said, without offering a figure.
"And our allies agree we can make reductions as well."
Asked whether such reductions would necessarily be contingent on some sort of agreement with the Taliban insurgency, Esper said: "Not necessarily."
He did not elaborate.
There are currently about 13,000 U.S. forces in Afghanistan as well as thousands of other NATO troops. U.S. officials have said U.S. forces could drop to 8,600 and still carry out an effective, core counter-terrorism mission as well as some limited advising for Afghan forces.
A draft accord agreed in September would have withdrawn thousands of American troops in exchange for guarantees that Afghanistan would not be used as a base for militant attacks on the United States or its allies.Still, many US officials privately doubt the Taliban could be relied upon to prevent al Qaeda from again plotting attacks against the United States from Afghan soil.