Pushlets - Protocol Specification

Author: Just van den Broecke
Organization: Just Objects B.V.
Email: just[AT]justobjects.nl

FileID: $Id: protocol.xml,v 1.5 2006/05/25 11:20:26 justb Exp $
Date: $Date: 2006/05/25 11:20:26 $

This document provides the Pushlet protocol specification. WORK IN PROGRESS. FOR NOW SOME HIGHLIGHTS.


1. Concepts

The Pushlet protocol facilitates publishing and streaming of data on top of the standard HTTP protocol This section introduces the main Pushlet protocol concepts.

1.1. Example

As a start, below is an example of Pushlet protocol exchange between a client and a server (various details omitted).

1: (client to server: start session) 2: GET /pushlet/pushlet.srv?p_event=join&p_format=xml HTTP/1.1 3: 4: (server to client: ack session create) 5: HTTP/1.1 200 OK 6: 7: <event p_id="faruhofoja" p_event="join-ack" p_format="xml" /> 8: 9: 10: (client to server: open data stream) 11: GET /pushlet/pushlet.srv?p_event=listen&p_id=faruhofoja&p_mode=stream&p_subject=/temperature HTTP/1.1 12: 13: (server to client: send data stream) 14: HTTP/1.1 200 OK 15: 16: <event p_sid="dujut" p_id="faruhofoja" p_mode="stream" p_event="listen-ack" p_format="xml" /> 17: 18: <event p_sid="dujut" p_subject="/temperature" p_event="data" city="twente" value="8" /> 19: 20: <event p_sid="dujut" p_subject="/temperature" p_event="data" city="amsterdam" value="8" /> 21: 22: <event p_sid="dujut" p_subject="/temperature" p_event="data" city="leeuwarden" value="6" /> 23: 24: <event p_sid="dujut" p_subject="/temperature" p_event="data" city="limburg" value="12" /> 25:

The above example shows a client starting a session through a join request. This request is acknowledged with an XML event join-ack. Subsequently a data stream is opened using the listen request. The listen-ack is returned followed by the data events over the data channel.

1.2. Streaming

Concepts used in the Pushlet protocol are somewhat similar to those used in the Real-Time Streaming Protocol (RTSP).

Like RTSP the Pushlet protocol provides both a control channel and a data channel. The client uses the control channel to start/stop sessions (join/leave), subscribe/unsubscribe to/from subjects, publish data etc. The actual streaming of data is transmitted out of band over a separate (HTTP) connection.

Since the HTTP protocol is request/response oriented, data streaming is achieved similar to audio/video streaming over HTTP, i.e. as a very long response. Since not all clients may support long-lived HTTP responses, data streaming may also be conducted in the so called pull and poll modes as controlled by the client or enforced by the server. In these latter modes the server will end the HTTP response document after one or more events are returned and instruct the client to wait and renew (refresh) the data streaming request.

1.3. Events

Protocol control and data messages exchanged between client and server are called events. Pushlet events are currently defined as a flat set of name/value pairs. The Pushlet protocol reserves names starting with p_ to discern protocol-specific names from user-supplied names.

Note: future versions of the protocol will introduce a more versatile event structure in which applications may use arbitrary payload data.

1.4. Event Encoding

To encode events (control responses and data) sent to the client, the Pushlet protocol supports several encoding formats. Browser clients supporting DHTML receive events encoded as JavaScript calls. This technique is similar to what Apple describes as Remote Scripting. Other clients, such as Java clients can receive events as XML elements as a stream (format="xml" or as a complete document (format="xml-strict"). The preferred encoding format is specified by the client.

Control events to the server can be specified as HTTP query parameters when using GET or as XML in a POST body.

1.5. Session Management

The Pushlet protocol provides a simple form of session management. A client starts with creating a Pushlet session using a join request. This request is acknowledged by the server through a join-ack response. Within the response a session id ( p_id) is returned that the client should supply in all other requests in this session. Note that Pushlet session management is independent from the standard HTTP session management (often with cookies) as to be independent of specific server implementations or (browser) client settings.

Since not all applications may require session management, the join-listen service is supplied that allows clients to join, subscribe and listen in one request. This allows for very simple clients (like even Unix telnet) to receive data without the need to keep state.

1.6. Topic Subscription

After a client has created a Pushlet session or even while streaming data, it can subscribe to and unsubscribe from topics, historically called subjects.

1.7. Event Publication

After a client has created a Pushlet session or even while streaming data, it use the publish service to send data events to the server. The server will route the event to those clients with a matching subscription (multicast). The event may also be directed to one specific client by supplying that client's session id.

2. Protocol Services

Below is a summary of the Pushlet protocol services.

Service
Description
join
Start a Pushlet session
leave
Stop a Pushlet session
subscribe
Subscribe to one or more subjects
unsubscribe
Unsubscribe from one or more or all subjects
listen
Open data channel and starts data streaming in one of three modes: stream, pull or poll. In pull or poll mode the server may provide so called refresh indications that the client should refresh this data stream request
join-listen
One-request service to immediately join, subscribe and start streaming data events. After this service the state is similar as after listen. Simple and RESTful clients may use this one-shot service.
publish
Publish data event to be dispatched by the server. Clients may use this service to multicast or unicast events through the server.
heartbeat
Indicate peer is alive

3. Implementation

The server-part of the Pushlet protocol is controlled by the Pushlet web application as part of the distribution. For the client-side several libraries are provides, most notably JavaScript and Java.

See js-pushlet-client.js (JavaScript library) and PushletClient.java

Other client-types used are Macromedia Flash and even telnet.

4. Raw Protocol Exchange

Below some samples of captured with tcpflow, a very valuable tool to capture/analyse protocol traffic..

4.1. JavaScript Encoding

The example below shows HTTP requests exchanged between a browser and the server. The encoding is JavaSCript and the mode is stream.

1: GET /pushlet/pushlet.srv?p_event=join HTTP/1.1 2: Host: sumatra:8080 3: User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.2) Gecko/20040803 4: Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 5: Accept-Language: en-us,en;q=0.5 6: Accept-Encoding: gzip,deflate 7: Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 8: Keep-Alive: 300 9: Connection: keep-alive 10: Referer: http://sumatra:8080/pushlet/examples/weather/nl-temperature.html 11: Cookie: JSESSIONID=37E7A128DADF6F6C5E8C2124389E75BA 12: 13: HTTP/1.1 200 OK 14: Expires: Sat, 6 May 1995 12:00:00 GMT 15: Cache-Control: no-store, no-cache, must-revalidate 16: Cache-Control: post-check=0, pre-check=0 17: Pragma: no-cache 18: Content-Type: text/html;charset=ISO-8859-1 19: Transfer-Encoding: chunked 20: Date: Mon, 28 Feb 2005 20:14:01 GMT 21: Server: Apache-Coyote/1.1 22: 23: f2 24: <html><head><meta http-equiv="Pragma" content="no-cache"><meta http-equiv="Expires" content="Tue, 31 Dec 1997 23:59:59 GMT"></head><body> 25: <script language="JavaScript"> var url=" "; 26: function refresh() { document.location.href=url; }</script> 27: 28: 87 29: <script language="JavaScript">parent.push('p_id', "becyrulufo",'p_time', "1109621641",'p_event', "join-ack",'p_format', "js");</script> 30: 31: GET /pushlet/pushlet.srv?p_id=becyrulufo&p_event=listen&p_subject=/temperature HTTP/1.1 32: Host: sumatra:8080 33: User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7.2) Gecko/20040803 34: Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 35: Accept-Language: en-us,en;q=0.5 36: Accept-Encoding: gzip,deflate 37: Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 38: Keep-Alive: 300 39: Connection: keep-alive 40: Referer: http://sumatra:8080/pushlet/examples/weather/nl-temperature.html 41: Cookie: JSESSIONID=37E7A128DADF6F6C5E8C2124389E75BA 42: 43: 44: HTTP/1.1 200 OK 45: Expires: Sat, 6 May 1995 12:00:00 GMT 46: Cache-Control: no-store, no-cache, must-revalidate 47: Cache-Control: post-check=0, pre-check=0 48: Pragma: no-cache 49: Content-Type: text/html;charset=ISO-8859-1 50: Transfer-Encoding: chunked 51: Date: Mon, 28 Feb 2005 20:14:01 GMT 52: Server: Apache-Coyote/1.1 53: 54: f2 55: <html><head><meta http-equiv="Pragma" content="no-cache"><meta http-equiv="Expires" content="Tue, 31 Dec 1997 23:59:59 GMT"></head><body> 56: <script language="JavaScript"> var url=" "; 57: function refresh() { document.location.href=url; }</script> 58: 59: ad 60: <script language="JavaScript">parent.push('p_sid', "jycab",'p_id', "becyrulufo",'p_mode', "stream",'p_time', "1109621641",'p_event', "listen-ack",'p_format', "js");</script> 61: 62: c4 63: <script language="JavaScript">parent.push('value', "10",'p_sid', "jycab",'p_seq', "1",'p_time', "1109621644",'p_subject', "/temperature",'p_event', "data",'city', "twente",'number', "3");</script> 64: 65: c7 66: <script language="JavaScript">parent.push('value', "9",'p_sid', "jycab",'p_seq', "2",'p_time', "1109621648",'p_subject', "/temperature",'p_event', "data",'city', "leeuwarden",'number', "2");</script> 67: 68: c3 69: <script language="JavaScript">parent.push('value', "6",'p_sid', "jycab",'p_seq', "3",'p_time', "1109621651",'p_subject', "/temperature",'p_event', "data",'city', "twente",'number', "3");</script> 70: 71: c4 72: <script language="JavaScript">parent.push('value', "10",'p_sid', "jycab",'p_seq', "4",'p_time', "1109621655",'p_subject', "/temperature",'p_event', "data",'city', "twente",'number', "3");</script> 73: 74: c7 75: <script language="JavaScript">parent.push('value', "9",'p_sid', "jycab",'p_seq', "5",'p_time', "1109621659",'p_subject', "/temperature",'p_event', "data",'city', "leeuwarden",'number', "2");</script> 76: 77: c4 78: <script language="JavaScript">parent.push('value', "10",'p_sid', "jycab",'p_seq', "6",'p_time', "1109621662",'p_subject', "/temperature",'p_event', "data",'city', "twente",'number', "3");</script> 79: 80: c3 81: <script language="JavaScript">parent.push('value', "8",'p_sid', "jycab",'p_seq', "7",'p_time', "1109621666",'p_subject', "/temperature",'p_event', "data",'city', "twente",'number', "3");</script>

4.2. XML Encoding

The next example below shows HTTP requests exchanged between a browser and the server. The encoding is XML and the mode is stream. The exchange was captured from the PushletPingApplication, a test application that both publishes and subscribes to the topic "/test/ping". The tcpflow output shows who is sending receiving with lines like 010.000.000.200.08080-010.000.000.030.52200: , denoting the sender IP/port-receiver IP/port. In our example the client resides on IP 10.0.0.30 and the server on 10.0.0.200.

1: 2: 010.000.000.030.52200-010.000.000.200.08080: 3: GET /pushlet/pushlet.srv?p_event=join&p_format=xml HTTP/1.1 4: Cache-Control: no-cache 5: Pragma: no-cache 6: User-Agent: Java/1.4.2_05 7: Host: pundit:8080 8: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 9: Connection: keep-alive 10: 11: 12: 010.000.000.200.08080-010.000.000.030.52200: 13: HTTP/1.1 200 OK 14: Expires: Sat, 6 May 1995 12:00:00 GMT 15: Cache-Control: no-store, no-cache, must-revalidate 16: Cache-Control: post-check=0, pre-check=0 17: Pragma: no-cache 18: Content-Type: text/plain 19: Transfer-Encoding: chunked 20: Date: Wed, 02 Mar 2005 14:25:39 GMT 21: Server: Apache-Coyote/1.1 22: 23: 51 24: <event p_id="fezibylina" p_time="1109773539" p_event="join-ack" p_format="xml" /> 25: 26: 010.000.000.200.08080-010.000.000.030.52200: 0 27: 28: 29: 010.000.000.030.52200-010.000.000.200.08080: 30: GET /pushlet/pushlet.srv?p_event=listen&p_id=fezibylina&p_mode=stream HTTP/1.1 31: Cache-Control: no-cache 32: Pragma: no-cache 33: User-Agent: Java/1.4.2_05 34: Host: pundit:8080 35: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 36: Connection: keep-alive 37: 38: 39: 010.000.000.200.08080-010.000.000.030.52200: 40: HTTP/1.1 200 OK 41: Expires: Sat, 6 May 1995 12:00:00 GMT 42: Cache-Control: no-store, no-cache, must-revalidate 43: Cache-Control: post-check=0, pre-check=0 44: Pragma: no-cache 45: Content-Type: text/plain 46: Transfer-Encoding: chunked 47: Date: Wed, 02 Mar 2005 14:25:39 GMT 48: Server: Apache-Coyote/1.1 49: 50: 63 51: <event p_id="fezibylina" p_mode="stream" p_time="1109773539" p_event="listen-ack" p_format="xml" /> 52: 53: 010.000.000.030.52201-010.000.000.200.08080: 54: GET /pushlet/pushlet.srv?p_id=fezibylina&p_time=1109773581&p_event=subscribe&p_subject=/test/ping HTTP/1.1 55: Cache-Control: no-cache 56: Pragma: no-cache 57: User-Agent: Java/1.4.2_05 58: Host: pundit:8080 59: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 60: Connection: keep-alive 61: 62: 63: 010.000.000.200.08080-010.000.000.030.52201: 64: HTTP/1.1 200 OK 65: Expires: Sat, 6 May 1995 12:00:00 GMT 66: Cache-Control: no-store, no-cache, must-revalidate 67: Cache-Control: post-check=0, pre-check=0 68: Pragma: no-cache 69: Content-Type: text/plain 70: Transfer-Encoding: chunked 71: Date: Wed, 02 Mar 2005 14:25:39 GMT 72: Server: Apache-Coyote/1.1 73: 74: 6c 75: <event p_sid="lilib" p_id="fezibylina" p_time="1109773539" p_event="subscribe-ack" p_subject="/test/ping" /> 76: 77: 010.000.000.030.52202-010.000.000.200.08080: 78: GET /pushlet/pushlet.srv?p_id=fezibylina&p_sid=lilib&p_time=1109773581&p_event=unsubscribe HTTP/1.1 79: Cache-Control: no-cache 80: Pragma: no-cache 81: User-Agent: Java/1.4.2_05 82: Host: pundit:8080 83: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 84: Connection: keep-alive 85: 86: 87: 010.000.000.200.08080-010.000.000.030.52202: 88: HTTP/1.1 200 OK 89: Expires: Sat, 6 May 1995 12:00:00 GMT 90: Cache-Control: no-store, no-cache, must-revalidate 91: Cache-Control: post-check=0, pre-check=0 92: Pragma: no-cache 93: Content-Type: text/plain 94: Transfer-Encoding: chunked 95: Date: Wed, 02 Mar 2005 14:25:39 GMT 96: Server: Apache-Coyote/1.1 97: 98: 6e 99: <event p_id="fezibylina" p_sid="lilib" p_time="1109773539" p_subject="/test/ping" p_event="unsubscribe-ack" /> 100: 101: 102: 103: 010.000.000.030.52203-010.000.000.200.08080: 104: GET /pushlet/pushlet.srv?p_id=fezibylina&p_time=1109773581&p_event=subscribe&p_subject=/test/ping HTTP/1.1 105: Cache-Control: no-cache 106: Pragma: no-cache 107: User-Agent: Java/1.4.2_05 108: Host: pundit:8080 109: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 110: Connection: keep-alive 111: 112: 113: 010.000.000.200.08080-010.000.000.030.52203: 114: HTTP/1.1 200 OK 115: Expires: Sat, 6 May 1995 12:00:00 GMT 116: Cache-Control: no-store, no-cache, must-revalidate 117: Cache-Control: post-check=0, pre-check=0 118: Pragma: no-cache 119: Content-Type: text/plain 120: Transfer-Encoding: chunked 121: Date: Wed, 02 Mar 2005 14:25:39 GMT 122: Server: Apache-Coyote/1.1 123: 124: 6c 125: <event p_sid="naziq" p_id="fezibylina" p_time="1109773539" p_event="subscribe-ack" p_subject="/test/ping" /> 126: 127: 010.000.000.030.52204-010.000.000.200.08080: 128: GET /pushlet/pushlet.srv?p_id=fezibylina&seqNr=1&p_time=1109773581&p_subject=/test/ping&time=1109773581776&p_event=publish HTTP/1.1 129: Cache-Control: no-cache 130: Pragma: no-cache 131: User-Agent: Java/1.4.2_05 132: Host: pundit:8080 133: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 134: Connection: keep-alive 135: 136: 137: 010.000.000.200.08080-010.000.000.030.52200: 138: a0 139: <event p_id="fezibylina" time="1109773581776" p_sid="naziq" seqNr="1" p_from="fezibylina" p_seq="1" p_time="1109773581" p_event="data" p_subject="/test/ping" /> 140: 141: 010.000.000.200.08080-010.000.000.030.52204: 142: HTTP/1.1 200 OK 143: Expires: Sat, 6 May 1995 12:00:00 GMT 144: Cache-Control: no-store, no-cache, must-revalidate 145: Cache-Control: post-check=0, pre-check=0 146: Pragma: no-cache 147: Content-Type: text/plain 148: Transfer-Encoding: chunked 149: Date: Wed, 02 Mar 2005 14:25:39 GMT 150: Server: Apache-Coyote/1.1 151: 152: 33 153: <event p_time="1109773539" p_event="publish-ack" /> 154: 155: 010.000.000.200.08080-010.000.000.030.52204: 156: 0 157: 158: 159: 010.000.000.030.52204-010.000.000.200.08080: 160: GET /pushlet/pushlet.srv?p_id=fezibylina&seqNr=2&p_time=1109773584&p_subject=/test/ping&time=1109773584808&p_event=publish HTTP/1.1 161: Cache-Control: no-cache 162: Pragma: no-cache 163: User-Agent: Java/1.4.2_05 164: Host: pundit:8080 165: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 166: Connection: keep-alive 167: 168: 169: 010.000.000.200.08080-010.000.000.030.52204: 170: HTTP/1.1 200 OK 171: Expires: Sat, 6 May 1995 12:00:00 GMT 172: Cache-Control: no-store, no-cache, must-revalidate 173: Cache-Control: post-check=0, pre-check=0 174: Pragma: no-cache 175: Content-Type: text/plain 176: Transfer-Encoding: chunked 177: Date: Wed, 02 Mar 2005 14:25:42 GMT 178: Server: Apache-Coyote/1.1 179: 180: 33 181: <event p_time="1109773542" p_event="publish-ack" /> 182: 183: 010.000.000.200.08080-010.000.000.030.52204: 0 184: 185: 186: 010.000.000.200.08080-010.000.000.030.52200: 187: a0 188: <event p_id="fezibylina" time="1109773584808" p_sid="naziq" seqNr="2" p_from="fezibylina" p_seq="2" p_time="1109773584" p_event="data" p_subject="/test/ping" /> 189: 190: 010.000.000.030.52204-010.000.000.200.08080: 191: GET /pushlet/pushlet.srv?p_id=fezibylina&seqNr=3&p_time=1109773587&p_subject=/test/ping&time=1109773587830&p_event=publish HTTP/1.1 192: Cache-Control: no-cache 193: Pragma: no-cache 194: User-Agent: Java/1.4.2_05 195: Host: pundit:8080 196: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 197: Connection: keep-alive 198: 199: 200: 010.000.000.200.08080-010.000.000.030.52204: 201: HTTP/1.1 200 OK 202: Expires: Sat, 6 May 1995 12:00:00 GMT 203: Cache-Control: no-store, no-cache, must-revalidate 204: Cache-Control: post-check=0, pre-check=0 205: Pragma: no-cache 206: Content-Type: text/plain 207: Transfer-Encoding: chunked 208: Date: Wed, 02 Mar 2005 14:25:46 GMT 209: Server: Apache-Coyote/1.1 210: 211: 33 212: <event p_time="1109773546" p_event="publish-ack" /> 213: 214: 010.000.000.200.08080-010.000.000.030.52204: 215: 0 216: 217: 218: 010.000.000.200.08080-010.000.000.030.52200: 219: a0 220: <event p_id="fezibylina" time="1109773587830" p_sid="naziq" seqNr="3" p_from="fezibylina" p_seq="3" p_time="1109773587" p_event="data" p_subject="/test/ping" /> 221: 222: 010.000.000.030.52204-010.000.000.200.08080: 223: GET /pushlet/pushlet.srv?p_id=fezibylina&seqNr=4&p_time=1109773590&p_subject=/test/ping&time=1109773590846&p_event=publish HTTP/1.1 224: Cache-Control: no-cache 225: Pragma: no-cache 226: User-Agent: Java/1.4.2_05 227: Host: pundit:8080 228: Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 229: Connection: keep-alive 230: 231: 232: 010.000.000.200.08080-010.000.000.030.52204: 233: HTTP/1.1 200 OK 234: Expires: Sat, 6 May 1995 12:00:00 GMT 235: Cache-Control: no-store, no-cache, must-revalidate 236: Cache-Control: post-check=0, pre-check=0 237: Pragma: no-cache 238: Content-Type: text/plain 239: Transfer-Encoding: chunked 240: Date: Wed, 02 Mar 2005 14:25:49 GMT 241: Server: Apache-Coyote/1.1 242: 243: 33 244: <event p_time="1109773549" p_event="publish-ack" /> 245: 246: 010.000.000.200.08080-010.000.000.030.52204: 247: 0 248: 249: 250: 010.000.000.200.08080-010.000.000.030.52200: 251: a0 252: <event p_id="fezibylina" time="1109773590846" p_sid="naziq" seqNr="4" p_from="fezibylina" p_seq="4" p_time="1109773590" p_event="data" p_subject="/test/ping" /> 253: 254: tcpflow[7140]: terminating