The answer to this questions is somewhat of a combination of events and action that all began many years ago.
The very first "standard specs" were chosen by individual clubmakers in the UK in the early 1900s who were in the process of converting their businesses from making one set at a time for one golfer at a time into more of a mass production basis. Up to this time, golf had not grown to be large enough for there to be a lot of golf courses with pro shops and club pros who were demanding to have sets of clubs in their pro shops to sell. Golfers of this era had to go to individual clubmakers to buy their clubs. While the clubmakers did not know all that much about individual fitting specs, they did at least know to build each set so its weights, grip size, and lengths were at least made to try to match to the physical characteristics of each golfer.
As the game grew and more courses were built in the early 1900s, more club pros got jobs at the courses and the pros decided they wanted to have clubs in their pro shops to sell to the golfers who came to the courses to play. Seeing this as a way to increase business, not all, but a few of the more enterprising clubmakers began to build sets to sell to the pros to be stocked in the pro shops and sold to the golfers. Thus this started the off the rack business model that still exists today.
The specs for these clubs were chosen by the clubmakers based on simple averages of the specs the clubmakers built for the golfers who previously had come to them to buy their clubs.
The American golf companies began to evolve in the very late 1800s and early 1900s. First were Spalding and MacGregor, followed in the 1900s by Wright & Ditson, in the 1910s by Wilson and the 1920s by Burke. Because America was so much larger than Britain and because golf evolved here such that there were no individual clubmakers in each town, almost right from the start the American golf companies pursued the "standard off the rack" business model. Their specs were adopted from having seen standard clubs made by the UK companies.
This all evolved through the 1930s to an actual series of standard specs that actually became agreed upon standards among the golf companies. Rarely did you see a golf company build their clubs with specs that were different. Men's drivers were always 43", 11*, 0 square and D2. Fwys were #2, 3, 4 and each 1" shorter than the driver, with lofts in progression of 13*, 16*, 19*. Irons were made on the basis of a 39" #2 iron at 20* loft/57* lie with all irons following at +4* loft, +1* lie and -1/2" length from there. PW had the same lie as the 9 because it was the same length at 35".
And those were the standards that no one deviated from. Until the 1980s. In the early 80s, the number of golf club companies tripled. All of a sudden competition for sales between this much larger number of golf companies became fierce.
Trying to find an edge on their competition, one company (I think it was Cobra from memory) decided to increase the length of their clubs and slightly lower the lofts of their irons. The reason was obvious - "if we make the clubs longer and lower in loft, golfers will hit the ball farther and we'll sell more clubs because everyone knows that distance sells golf clubs."
And thus started the trend of longer lengths and lower lofts that still goes on today. At first, drivers grew to 44". Then when every company pushed their driver length to 44, one of the companies jumped it to 44 1/2". Then to 45". Then to 45 1/2" and so on to where we are today at 46 1/2".
Same with iron lofts. Speaking only of the 5 iron to show the evolution, from the 80s on, the 5 iron went from 32* to 30*, then to 28*, then to 27* then to 26* and so on. Every time the majority of companies changed their lofts, some other company jumped their lofts stronger yet.
So these "standards" on clubs today most definitely were NOT determined by any research on what the "Average Golfer" needs. If you could say any one part of today's std club specs is based on some sort of average analysis of golfers, I suppose you could say the stock R flex shaft from most companies could be said to be chosen on that basis. But then too, if you look at all the stock shafts in the big company's clubs, you can find variations there.
But in terms of lengths and lofts, these are most definitely NOT chosen on the basis of what average golfers need to be able to play reasonably well. They were chosen to try to allow one company to gain a competitive sales advantage over another company, simply to sell more clubs.
What's sad is that most of the specs on std clubs today actually prevent most golfers from playing to the best of their ability.
But what's GOOD about this is that it very much does allow a good clubmaker/clubfitter to fit and build clubs for golfers that DO allow them to play better. The challenge of course is for the clubmakers to convince the golfers that what they can do for the golfer is better than what the big companies offer.
