Performance optimizations aside, and even if you concatenate all your JS files into a single file, you still have to load the library into the browser after the request has been made, and you can't do anything with it until that has finished loading.
Wouldn't it be so great if the browser had a copy of YUI or jQuery or Dojo or Prototype or GWT in memory at all times? Then your library using decision would not be influenced by the size of the library but by its speed and performance, and perhaps features and functionality provided. I mean you don't go loading new fonts and new plugins every single time you go to a new page right? Then why do we have to download the same 5 or 6 JS libraries over and over all day long as we browse the web? And if this was an option how many companies would spend time rolling out their own rather than adopting one?
I understand there would be tons of politics involved and it will probably never happen in a million years but it is worth a try. Perhaps at least a FF only version could be done through an add-on that loads the latest major version of all these libraries and checks for updates on startup, similar to how other plugins get updated in FireFox today. The browser would have to namespace the libraries to avoid confusion. Developers would then just do a test to see if the browser has a required library in memory, and if not then make a server call for it as a fail safe.