Back to OCMPFeeds? main page.

This page will outline the design decisions and process for the newsfeed plugin for the OCMP application framework. While the title of the page calls this an RSS plugin, it may actually apply to any one of a number of newsfeed publishing technologies.

Using the Plugin

(8/10/2006) The key is the feeds.conf file on the proxy. For now, this must be hand-edited. Write a file in this format:

CONFIG
FEED <url>
FEED <url>
FEED <url>

And place it in $OCMPHOME/src/feeds.conf

Now, run the proxy, run the directoryAPI watcher, and then run the proxyfeed.py daemon. It will first sync your feeds.conf file with the client, and then look for updates. Finally, run the client OCMP proxy, and client directory watcher.

Overview of Concepts and Beavhior

The newsfeed plugin has some characteristics that make it significantly different from other OCMP plugins so far. In particular, the goal of the plugin is to allow mobile clients to have access to news at any time, regardless of connectivity status. Furthermore, news should be updated as quickly as possible without taxing the resources of the mobile device. These goals result in the proxy needing to run in a daemon mode, independently of requests from a mobile user for data. All OCMP plugins so far operate only on a per-request basis. The proxy will also need to manage cached data at a higher level than the per-packet level that is currently maintained in OCMP. In particular, the proxy will work with data units that are the size of XML newsfeed items. Similarly, the client proxy on the mobile device will also need to manage data at the newsfeed-item level, as well.

The plugin will be implemented as a ProtocolServicePlugin (PSP)under the HTTP Application plugin. The client and proxy will maintain a synchronized feed configuration file, which can be configured through text editing, through a web page interface, or through a later protocol-triggered method. When the client plugin receives an HTTP request, it triggers the newsfeed PSP which checks the config file to find out if the current requested HTML document is actually a subscribed feed. If so, then the client does not initiate any remote communications, but instead, it looks in a local cache for updated feed items. The proxy plugin runs as a daemon, and manages the polling of various newsfeeds. It pushes news information out to the client, which quietly receives and caches this information for the user to consume at a later time. Functional Paths

• Proxy-initiated feed configuration synch (proxy feed list updated... send to client)
• Client-initiated feed configuration synch (client feed list updated... send to proxy)
• Proxy polling of remote feed sites (proxy connect to remote feed sites, download data)
• Proxy in-band client notification (proxy prepares OCMP App packet for next client connect)
• Proxy OOB client notification (client sends OOB (sms, email) notification to client)
• Client response to OOB notification (connect, get in-band notification)
• Client response to in-band notification (get feed data)
• Client gets request from application program (send feed data)

Client Plugin Design Discussion

The client plugin performs the following functions:

   1. Synchronize a list of subscribed feeds with the proxy
   2. Respond to out-of-band messages from the proxy
   3. Maintain a cached version of recent news items 

Implementation

The first part of the implementation is the hook that decides whether or not to trigger the newsfeed management behavior. Newsfeeds are almost always retreived via an HTTP interface. Within the current OCMP interface, protcols are detected by sniffing the first few characters that the client sends. As a result, the current OCMP implementation can not differentiate a news-feed request from a standard HTTP request.

HTTP has become a base protocol for a number of technologies. Considering this trend, we've decided that the newsfeed plugin will be developed as a sub-plugin of the HTTP plugin, rather than a completely independent plugin. This will allow the HTTP plugin to continue to handle lower-level concerns, so that the RSS (and future HTTP-based tehcnology plugins) will continue to benefit from the future improvements to the HTTP plugin. This decision results in the creation of a new component in the OCMP framework. Since this is an orthogonal development that the newsfeed plugin will simply be using, we address it in a different area: ProtocolServicePlugin.

Client Behavior

The client newsfeed PSP is triggered on two types of events:

1. An out-of-band notification from the proxy, indicating that new feed data is available. Data is written to the feed store.

2. A request from a client application for feed data. Data is read from the feed store.

The newsfeed plugin behaves in a such a way that the client's newsreader/aggregator application receives feeds completely aysnchronously from actual network activity. That is, the plugin receives data only based on signals from the proxy, and the plugin only sends data based on signals from an application. This may change in the future, to allow applications to request to poll a feed immediately. Out of Band Notification In case of an out-of-band notification, the client will connect to the remote proxy. The remote proxy will have prepared a packet announcing the desire to send newsfeed data back to the client (a message like "FEED REVERSE"). The message says reverse, because this is a proxy-triggered application plugin load. The client connects the pool of connections. This should trigger the remote proxy to send a packet that will trigger the client to instantiate the necessary plugins. The client will receive data from the proxy, and store it in the local feed store.

Proxy Behavior

The proxy simply runs as a daemon, if any feed configuration exists. Peridocally, a polling agent connects to each feed website, and looks for updates. If updates exist, they are stored in a proxy-side feed store. On certain conditions, like keywords, or a polling frequency, a notification might be send via an OOB channel to the client. Also, App packets may simply be queued by the proxy to alert the client that new data is availble. Proxy Plugin Design Discussion

The proxy plugin performs the following functions:

   1. Synchronize a list of subscribed feeds with the client
   2. Poll the subscribed feeds as necessary to get updates
   3. Maintain a cached version of recent news
   4. Notify the client of news based on various events.
         1. Time
         2. keywords/tags
         3. popularity 
   5. Manage news items in such a way that news reading on the mobile device is low overhead
         1. Refrain from sending redundant items
         2. Use a unified format so that all news feeds use the same internal representation 

Implementation: DirectoryAPI

The DirectoryAPI plugin for OCMP makes adding some types of functionality a breeze. The DirectoryAPI is a watcher that simply watches for files in a particular directory, and sends whatever files appear. When the other end receives the file, there is a configuration file that lets the device know how to handle the file. So, the proxy feed manager is implemented as a standalone daemon that polls websites, and stores diffs of feeds in a DirectoryAPI file. The client side receives these feeds and diffs, and places them in the feeds directory, where the ProtocolServicePlugin for the client can find them.