high #http-requests slows down (mainly s.gif)

[rblon]

Browser: FireFox/3.5.3 & Google Chrome/3.0.195.21
Webserver: lighttpd/1.4.23 & Apache/2.2.13

Hi,

I am designing a front-end for a web-application and started with the portal template. At the moment I have two tree menu's and couple of grids, which are filled with static data from json files at load time. However, loading the page takes around 5 seconds, which I consider pretty poor as there is (almost) no network latency.

I had a look at the server access log and notice that loading leads to around 160 requests, and 100 of them are for "s.gif". Using Chrome instead of FireFox leads to a much better performance, as in that case there are only a few "s.gif" request.

A similar behaviour can be observed in the "./examples/tree/two-trees.html" sample, where I see around 50 request, with more than 30 for "s.gif".

Even though there are >150 simultaneous requests, 5 seconds still seems very slow, which I thought indicated a web server problem. However, I have tried both lighttpd and Apache, and I observe similar loading times

I have two questions:
* Is such a high number of requests a known "issue"? It appears browser specific, but I was wondering if there any methods to bring it down (100+ "s.gif requests doesn't seem efficient...)
* Any views on what are other reasons for slow loading times? The two-trees sample loads slow as well in my opinion, so perhaps we can discuss using that example.

[Animal]

Shouldn't be any requests for that now from the better browsers. By default it is a data URL.

[rblon]

ok, but lets take "./examples/tree/two-trees.html" as example.
On the serverside I am running lighttpd under FreeBSD (don't think that matters though).
I see the following on reloading (shift-reload button):

Win XP with FireFox/3.5.3: 52 requests, 37 for s.gif
Win XP with Chrome/3.0.195.21: 14 requests, 2 for s.gif
Win XP with IE/8.0: 15 requests, 3 for s.gif
Win XP with Safari/4.0.3: 29 requests, 5 for s.gif
Mac OS X 10.6.1 with FireFox/3.5.3: 60 requests, 41 for s.gif
Mac OS X 10.6.1 with Safari/4.0.3: 8 requests, 1 for s.gif

These numbers are non-producable, i.e. they change every time I do a re-load, but it does suggest a FireFox issue (and less severe Safari for Windows).

I would consider an up-to-date FireFox in general one of the "better browsers", so I am wondering if other people observe something similar.

Also, on second thoughts, I do think this offers some explanation for slow load time. Because even though the server might be able to handle >1000 requests per second, the browser will request them sequentially.

[Animal]

Any one page should only request an image once however many times it is used on a page. This is what happens:

http://i131.photobucket.com/albums/p286/TimeTrialAnimal/s_gif_requests.jpg

As you see, only one request for s.gif

[rblon]

can I ask what browser & debug tool you are using? I see in Firebug only XHR requests (or I don't how to see all requests).

below my lighttpd access.log, which is showing all s.gif requests (I have changed my domain to example.com before pasting). It might be an issue with logging, but my first impression is that these requests actually take place

[I have attached now the log file instead of copy-pasting it]

[Animal]

That's Firebug. The Net tab has a series of buttons to select which network requests to show. You see them along the top in that picture. I have "All" clicked.

[Condor]

You should configure your HTTP server to send an Expires header for images (and probably also for all other static resources).

ps. Try running YSlow for your webpage.

[rblon]

Animal, i have tried Firebug - net tab, and I am seeing many "s.gif" requests. Both under Win XP as Mac OS X. So same observation as lighttpd access logs.

Condor, I have mapped in lighttpd ".gif" to "image/gif", which seems to work (in Firebug I see Content-Type image/gif in Response Header for GET s.gif request)

I am going to give now YSlow a go (haven't used it before)

[Condor]

Did you configure an expires header as I suggest? (see here for instructions (http://www.cyberciti.biz/tips/lighttpd-client-side-cache-with-mod_expire.html)).

ps. I bulk-approved all your messages. Could you delete the duplicates?

[rblon]

Hi Condor,

I have added "mod_expire" to lighttpd, but still see many "s.gif" requests
This is a response header

Expires Wed, 30 Sep 2009 12:59:08 GMT
Cache-Control max-age=604800
Content-Type image/gif
Accept-Ranges bytes
Etag "1314839837"
Last-Modified Mon, 06 Jul 2009 23:09:20 GMT
Content-Length 43
Date Wed, 23 Sep 2009 12:59:08 GMT
Server lighttpd/1.4.23

and this the request header

Host dev.example.com
User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language en-gb,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 300
Connection keep-alive
Referer http://dev.example.com/extjs/examples/tree/two-trees.html
Pragma no-cache
Cache-Control no-cache

[nlotz]

Did you check the ETags?

Note that with Apache the ETag is calculated from the number of the inode, the filesize and the last-modified time of the resource. This can lead to the same resource being fetched multiple times when it is served by multiple webservers (load balancing). That is because the same resource will have different inode-numbers on different filesystems. You can fix that by telling Apache to calculate the ETag from the size and last-modified time of the resource, only:

FileETag MTime Size
Reference: Apache 2.2 - FileETag (http://httpd.apache.org/docs/2.2/mod/core.html#fileetag)

[rblon]

yes I think Lighttpd is similar, but that is not an issue here as I am using only 1 webserver.

It all goes fine when I do a normal reload (F5), but when I do a shift-reload (Ctrl+F5) then I see multiple requests for the same file (mainly s.gif). It seems if FireFox adds new requests to the queue even if a request for the same file is already outstanding.

Perhaps it is not a huge problem, as it will only affects new users to the website (or users who haven't visited for a while).

Still wondering if other people see the same with FireFox, or that is a web server configuration problem after all.

[joeri]

AFAIK, shift-reload reloads the page without using the cache at all. If the cache is entirely bypassed, then you're going to see the same resource downloaded multiple times. I suggest using regular reload. If you are seeing issues where some resources in the page are stale, either configure expire headers to expire immediately (explicit expires header), or change "browser.cache.check_doc_frequency" to "1" in about:cache.

If you want to simulate a new site experience, just use the "Clear Recent History" menu option to clear all cookies and the entire cache.

[Animal]

It really should not load the same resource multiple times.

Bypassing the cache should reload all resources sure, but if a page references http://foo.com/s.gif twenty times, the browser will only make one HTTP request for it. To do otherwise is pathological behaviour and should be reported to the browser manufacturer.

[joeri]

Good point. So, if it is the case that firefox loads the same image multiple times, then that's a bug. Still, shift-reload as far as I know bypasses the cache.

[Condor]

Why is your client sending the following headers:

Pragma no-cache
Cache-Control no-cache

ps. Did you by any chance enable the 'Disable Cache' option in the Firefox Web Developer plugin?

[rblon]

So my Firefox behaves as Joeri explains, with Ctrl+F5 cache is completely bypassed and there are multiple request for 1 file. I agree with Animal that behaviour should be considered a bug. Did one of you test if you see the same in Firefox?

Condor, no I didn't disable cache, I think those request headers are related to Ctrl+F5. If I do a normal F5, I don't have those headers.

I think I have most of it is clear now. Thanks for all your help. However, a related problem is how to get rid of the validation requests? The "two-trees" example has 20 requests, most of them for data that hardly anytime changes. So it could have a far-in-the-future expires date, while updates are controlled with folder/filename versioning. The template I'm working on has currently around 60 initial requests. To validate each file is a waste of http connections.

I have tried to use pre-check and post-check in the Cache-Control header, but no success. If I have the following response header, validation still takes place on reload:

Cache-Control max-age=2592000, post-check=2592000, pre-check=2592000
Content-Type image/gif
Accept-Ranges bytes
Etag "1314839837"
Last-Modified Mon, 06 Jul 2009 23:09:20 GMT
Content-Length 43
Date Wed, 23 Sep 2009 20:36:09 GMT
Server lighttpd/1.4.23

Perhaps the following lighttpd bug (http://redmine.lighttpd.net/issues/1386) is the reason it is not possible.

Btw, sorry if it now becomes a bit off-topic (ExtJS). On the other hand ExtJS library is generating many http requests, which makes it relevant how to minimize these.

[joeri]

An explanation of browser caching as I understand it (please correct me if I'm wrong):

Basic cache principles:
- Last-Modified or Etag must be set to enable caching
- Once cached, Expires or Cache-Control:max-age set your expiration date (the latter overrides the former)
- Before the expiration date is reached, the browser will not request the file from the server
- After the expiration date is reached, the browser will do conditional gets (validate requests)

Some other tidbits:
- Cache-Control: "no-store" / "no-cache" prevent caching by the browser, "private" says it can be cached by the browser, "public" says it can be cached by the browsers and proxies. Getting fancy with Cache-Control (using anything besides "no-cache" and "public") can get you into trouble because there are browser bugs and proxy bugs related to these headers.
- "Cache-Control: no-cache" is the only correct way to prevent caching. Dynamic languages (e.g. PHP) implicitly send this header.
- Pragma headers are deprecated and should not be used
- If no expiration date is specified, the browser will assume one based on the browser caching settings. You don't want this, because you don't know in that case whether the browser will send requests to the server or not.

So, as recommendations you get:
- Always include Last-Modified and/or Etag
- Always include Expires and/or Cache-Control: max-age headers
- Either set expires in the past (Expires: -1) to ensure freshness of resources in the browser cache
- Or adapt your deployment so new versions are deployed at different URL's, and set expires far into the future. In that case, browsers will not request updated versions until the expiration date is reached, so you must always change URL's to force fresh downloads.

[rblon]

What you're saying makes sense

"- Before the expiration date is reached, the browser will not request the file from the server
- After the expiration date is reached, the browser will do conditional gets (validate requests)"

but it is not what I observe.

If I use the following response headers (using "access 30 days")

Expires Sat, 24 Oct 2009 09:03:41 GMT
Cache-Control max-age=2592000
Content-Type image/gif
Accept-Ranges bytes
Etag "1314839837"
Last-Modified Mon, 06 Jul 2009 23:09:20 GMT
Content-Length 43
Date Thu, 24 Sep 2009 09:03:41 GMT
Server lighttpd/1.4.23

I still see a validation request on the next re-load (F5).

Host dev.example.com
User-Agent Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language en-gb,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 300
Connection keep-alive
Referer http://dev.example.com/extjs/examples/tree/two-trees.html
If-Modified-Since Mon, 06 Jul 2009 23:09:20 GMT
If-None-Match "1314839837"
Cache-Control max-age=0

So my Firefox is doing validation request before the expires date (and is putting Cache-Control max-age=0 in the request headers).

I have checked with "access 10 seconds" to see what happens after expires date. Then Firefox is putting in a validation request as well (so basically same as before expires date).

[Condor]

Are you sure you don't have a disabled Firefox cache?

This can be the case with 'private browsing' or if you specifically disabled it (e.g. with the Web Developer plugin).

[rblon]

I don't think the cache is disabled.
On the validation request I get "304 Not Modified" response, which implies the file is available in the cache.
Also, I see the same on the Mac, which is a pretty standard installation (ie I haven't done much tweaking there).
I tried to test the example on the extjs website (http://www.extjs.com/deploy/dev/examples/tree/two-trees.html), but there is no Expires header in the response.

[joeri]

Can you check the expiration date for those resources in about:cache?

I don't use cache-control max-age myself, but I've read on the interwebs that it overrides expires. Tests on my system with just the expires headers did bear out what I've posted.

Btw, if you don't have max-age or expires headers, the browsers fall back to default settings. For firefox this is in about:config, the http://kb.mozillazine.org/Browser.cache.check_doc_frequency setting, which you have to set to 1. For IE this is on the browser history settings page.

[rblon]

the expiration dates in about:cache match the response headers...
browser.cache.check_doc_frequency is set to 3 (default), but that shouldn't matter as the responses have Expires & Cache-Control headers.
Anyway, it seems I am not going to find out the problem, and I shouldn't bother you anymore as it really seems something strange with my system.
Thanks for your help and if I find a solution after all I will post it here.

[rblon]

One more post to (hopefully) close this issue.
I realized (a bit late) that reload (F5) is not the same as entering the url in the address bar (or accessing the page via a link).
My observations are now as follows


With reload (F5), Firefox validates each file, regardless the expires date.
When browsing to the website (normal situation), Firefox only validates / download files that haven't expired.
And finally (as I started out this thread) with shift-reload (Ctrl+F5), Firefox downloads all files (and some even multiple times).

[Michal Jelen]

I have experienced the same problem. My Firefox 3.5 was loading s.gif many dozen times for one single page. Besides stressing the web server, it was also causing ugly rendering of trees, because they were waiting up to several seconds for their s.gifs to arrive.

In my case, the problem was resolved by uninstalling HttpFox Firefox Add-On.

[evanc]

when you shift-reload that's telling Firefox not to use the cache. If you do a normal reload, you won't experience that.

Isn't this a Firefox behavior? What's this got to do with Ext?