According to the Japanese press, the 83-year-old former prime minister complained at a board of trustees meeting last week that "meetings with women take longer" because "women are competitive -- if one member raises their hand to speak, others might think they need to talk, too."
The International Olympic Committee called Mori's comments "absolutely inappropriate" and he later apologized, but initially refused to step down.
Mori's remarks have drawn international criticism and have prompted hundreds of Olympic volunteers to resign in protest.
Saburo Kawabuchi, former president of the Japan Football Association, is said to be the chief candidate to replace Mori.
Mori's departure is the latest issue for the Tokyo Summer Olympics, which are scheduled to open on July 23 after being postponed in 2020 due to the COVID-19 pandemic.
Meanwhile Japan is struggling to deal with rising COVID-19 case numbers and a recent national poll found that 77 percent of people there think the Summer Games should be canceled, according to MSN.com.