"why not?"
Still got server 1 and 2, and if you don't like 32 players don't play it.
And what happens when the server is empty? Saint spams us invites to fill it up/we join to fill it up, all my buddies I like to play with go onto that server and I can choose between the spamtastic server with my buddies on it or a normal server without any buddies.
32 players just sucks, its way too spammy. I dunno who here enjoys getting grenade/rocket spammed and chokepoints but I sure don't. Its simply an exercise in frustration at that point, playing scout and spy is nearly pointless as well because with so many people around its just pointless you will run into someone every second. The impact of larger server on the class balance is profound and thats why I would be against it. Spy, scout, pyro, suffer the most from this, since they need to be close range and can't resort to standing far away spamming instapwn rockets/grenades nor have the health to get up close like the heavy.
More does not equal better. At a certain point more just becomes an obstacle, think about it like this. 1 barman behind the bar can serve 5 spamembers relatively fast. 2 barmans can do 12 relatively fast, 3 can do 20 relatively fast, 4 can only do 24 relatively fast since they will get in eachothers way. There is a reason valve official number is max 24 players. After that point the gameplay just starts degrading too much and too many stalemates will appear.