New mothers need extra support so more breastfeed.
76% of women start out breastfeeding, this falls to 50% by six weeks - and one in four by six months.
The Scientific Advisory Committee on Nutrition said rates had improved, but more awareness of the health risks of not breastfeeding was needed.
The government said a breastfeeding helpline had been set up to offer practical support and information.
The latest recommendations from the committee were based on the Infant Feeding Survey from 2005.
However, there is also a need for midwives to be given the time to talk to, advise and support mothers with breastfeeding
The survey, which has been carried out every five years since 1975, also found that well-educated, professional women aged over 30 who were first-time mothers were the most likely group to breastfeed.
Committee members said there were several areas for improvement.
They said few parents delay the introduction of solids to around six months of age, as recommended, and that labels on commercial weaning foods should be changed to reflect this advice.
Antenatal and postnatal services needed to be more accessible for hard-to-reach groups, the report warned.
In addition, breastfeeding advice and support should be widely available and all health care providers should receive proper training in how to help women who want to breastfeed, the report said.
The committee also called for the provision of adequate infant feeding facilities in the workplace and in public places.