Sounds versus Voices

I have trouble understanding what some people say, particularly young people and women. OK, most people. A while back I had my hearing checked and my hearing, as far as the tests go, is fine. Doctor told me this is a common problem with older men and the medical establishment has no explanation for it. Today I think I figured out the problem. I don't spend as much time talking with people as I used to. I spend more time just listening to sounds in my environment, and they usually don't include people's voices. Basically, I just don't get enough practice listening to people talk.