Having worked on the Internet for over ten years I suppose I should know the answer to this, but I don't :o
Why is it that using a fast broadband connection (8Mps) which will download a huge file in seconds, some days my browser usually hangs for 5-10 seconds loading each simple HTML page, and quite often comes back with helpful comments like "sorry, www.google.co.uk doesn't exist". :curse:
Don't get me wrong, I use Firefox on a daily basis and will only use IE if I really have to. Firefox seems to be a power hungry beast these days, ensuring you don't have the Firebug add-on enabled vastly increases Firefox response time and page loading times, which is often my problem. Disable all other Add-ons too (Tools>Add-Ons) to see if it makes any difference.
additionally, change your dns to those servers provided by opendns and ensure that nobody on your network is using anything that ties up a lot of ports / bandwidth.
like online games, cloud backup solutions, streaming video, bittorrent, and any other p2p software.