Three out of ten people are using the pension freedoms to put extra cash in bank accounts due to concerns about the economy, according to Citizens Advice.
Its research, which looks at how consumers have been reacting to the pension freedoms that came into place last April, mentions Brexit and uncertainty weighing on peoples' decisions.
The body also found one in eight accessing their pots face unexpected tax or welfare losses.
But this bad news is countered by 35% stating they believe the reforms have directly improved their retirement prospects.
Among those who said they are better off following the introduction of pension freedoms, a majority (77%) said this is because they have more control over their money.
The research is based on a ComRes poll between 17 March and 1 April of 501 people who have accessed their pensions since the flexibilities were introduced.
Citizens Advice chief executive Gillian Guy said: "With annuity rates falling, uncertainty around returns on drawdown products and the drop in interest rates many are opting to manage their savings themselves, through bank accounts or investments. Others are taking the opportunity to clear debts which would otherwise hang over their retirement."
Yet the pension freedoms are popular with consumers, he added.
Citizens Advice makes a number of recommendations to ensure the benefits of freedoms are fully realised.
Consumers should be encouraged to consider future care costs and the Financial Conduct Authority should review whether warnings are being targeted well enough.
Royal London policy director Steve Webb (pictured above) warned the big risk to the success of the freedoms is not people overspending their savings, but putting the money into bank accounts that pay little interest.
"It is vital that anyone considering taking their money out of their pension pot has access to high quality advice and guidance, which stresses the option of leaving the money invested.
"Consumers need to be made aware that putting your cash in an account paying very little interest is not a safe option and will mean that you are missing out on the returns you could get if you left your pot invested."
Intelligent Pensions head of pathways Andrew Pennie added it is important people do not make snap decisions when using their freedoms. "Mistakes are easy, often irreversible and usually come at a heavy cost."
