Updating a feed

Topics: Argotic.Web
Jul 17, 2008 at 10:43 AM

I have been using the ConditionalGet function in the SyndicationDiscoveryUtility class and that works fine for updating a feed.
However, sometimes the eTag is not present meaning that the ConditionalGet function will not work as it takes this as a parameter.

Is there another way to do a partial update, I don't want to re-download the entire feed again as it wastes bandwidth.

Jul 17, 2008 at 2:37 PM
Per http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.25, it appears that technically you should be able to send just the cached value of the Last-Modified header and get back a 304 status indicating the resource has not been modified.

Most guidelines I have seen recommend sending both the If-Modified-Since and ETag headers, as I believe it covers both HTTP 1.0 and HTTP 1.1 scenarios.

I will look into modifying the ConditionalGet method to support performing a conditional GET using only a the Last-Modified header value cached froma previous request.

You may want to review http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.19 and http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.25 to learn how to perform conditional GEToperations in your application in the interim.
Jul 23, 2008 at 12:10 PM
I tried setting the httpRequest.IfModifiedSince property to the previously cached lastmodified date taken from the source but it does not make a difference.
The http response code is set to 200 OK and the entire content is returned back, instead of a partial update.

So far I have simply tested it on the google news rss feed at http://news.google.co.uk/nwshp?hl=en&tab=wn&output=rss

I have looked at the links which you've posted but also found this one http://nick.typepad.com/blog/2004/09/rss_bandwidth_c.html

The most important section is this:

"Almost all aggregators store the date/time that a feed was last updated, and they pass this to the HTTP server via the If-Modified-Since HTTP
header the next time they request the feed.

If the feed hasn't changed since that date/time, the server returns an HTTP status code 304 to let the aggregator know the feed hasn't changed."

Jul 28, 2008 at 5:46 PM
Unfortunately, not all web applications are created equal. For static content/files, you can rely on IIS, Apache, etc. to handle a conditional GET request in the documented manner; but when consuming content that is dynamically generated like many feed soures are, it is up to the publisher to responsibly handle conditional GET requests. You may face a situation where you are properly sending a conditional GET request with appropriate header values and still not receive the proper HTTP status code, as conditional GET requests are not handled/supported.

It can sometimes be difficult to determine if the problem exists on the slient or server when debugging conditional GET operations. Usually if you are recieving an eTag you can assume the resource supports conditional GET requests, otherwise you may have to try using one or more requests that use the various HTTP headers that support a conditional GET operation. If that fails, you are probably looking at a web resource that does not support conditional GET requests.

I will look into trying to provide a single conditional GET request call that will attempt to cover the standard set of conditional GET requests, but in the end it is the web resource provider and the underlying platform they use to serve web content that determines if conditional GET requests will be supported.
Jul 28, 2008 at 6:22 PM
Yes, in my implementation I rely on checking if the feed location has an eTag if it has then a conditional
get is possible, if not I download the entire feed.

I was pretty surprised that such a large corporation like Google does not support eTags in their
news feeds.

On another note:

As I have been developing my rss reader application I have looked at many feeds. Several are using Argotic
to generate their feeds. These include CNN, Wired and several more. It just shows that the Argotic framework
is pretty popular!

Anyway thanks for all your help.