What's The Big Deal With Matching Serial Numbers???

OK, I can see if you spent $2500 or more on a fresh out of the dead soldiers cold hands a WW2 something, you'd like to know it wasn't altered by some bubba in his basement. But in reality, I'm sure there are/were plenty firearms used throughout that went back to the armory to get repaired and went out with mis-matching serial numbers.

As long as the parts were made for a particular type of firearm, say a 1911, and it shoots with said replacement parts, are mis-matched P/Ns that bad? I'm sure matching P/Ns command a higher price, but what's wrong with a part or 3 that don't match with the rest of the gun? If you know beforehand they don't match and what you're looking for is THE gun you've always wanted at a price you can afford, are non-matching numbers so bad?