The scale of the problem was "huge and terrifying" with cash and property worth tens of millions of pounds taken last year, Action on Elder Abuse said.
Solicitors say the rise in financial abuse is partly due to substantial rises in property values.
Charities say a lack of awareness of the problem makes such crimes easy.
Action on Elder Abuse says it has seen a rise in the number of people who contact its helpline because of financial abuse.
But Daniel Blake, policy manager at the charity, believed that the cases it was aware of were only the tip of the iceberg.
"What we know is that this affects hundreds of thousands of older people," he said.
"The amounts of money are in the tens of millions of pounds being taken, stolen or defrauded from older people.
"We're not saying this happens in every family, but this is a widescale problem involving massive amounts of money that actually mean a lot to older people and has a severe impact on their quality of life."
The charity found that financial abuse can include:
- The direct theft of money and/or other possessions from an older person
- Benefits belonging to the older person being withheld by family members
- Older people being forced to sell their homes
Calls to the charity's helpline suggested that where property was involved, about a quarter of victims had their homes sold or taken without their consent.
And in 3% of cases, the house had been remortgaged by the abuser.
Other cases involved direct theft of money, possessions, and benefits.
Donald, not his real name, has discovered the cost of putting his trust in a close family member.
He and his wife have had around £85,000 stolen, and although around half of the money has been returned he said it has made him worry about his financial security.
"Some money was taken from the sale of the house, which is probably the major item. But there was also small bank accounts and building society accounts and allowances such as carer's allowance, that sort of thing," he told the BBC.
"I was rather disgusted to be honest. I hadn't expected it. It shook me at the time. It was very unfortunate. It should never have happened."
Last autumn the government strengthened protection for older people who suffer from dementia or other illnesses that affect their mental capacity.
But charities say that the problem of financial abuse has been allowed to grow because many people do not believe it can happen.
The number of older people who become victims was almost certainly underestimated, said Kate Joplin of Help the Aged.
"Many aged people are too embarrassed to say what has happened, or perhaps do not realise what has happened because of their mental decline," she said.
Social workers and other staff who deal with older people are trained to look for physical evidence of abuse, but not to enquire about their finances, Ms Joplin added.
"Elderly financial abuse is not recognised as an issue so it's not picked up," she said.
"If someone was being physically abused you'd expect to see bruises so you'd know what to look for.
"We need to create a culture in which people can blow the whistle when they find something suspicious because we can't allow so may older people to be defrauded."
Helen Freely, a solicitor who specialises in working with older people, said that theft involving property had been fuelled by years of rising house prices.
"It's largely because elderly people bought their properties many years ago and they have increased in value. Many elderly people are sitting on a pot of money.
"Also dementia is on the increase - we are an aging population - so that makes them vulnerable. "
Charities have warned that until the problem is taken seriously, the levels of financial abuse will not come down.