Mention in changelog that we disabled presence.

This happpens with room URL like http://127.0.0.1:8080/matrix/client/#/room/#public:localhost. The second hash part is transferred to the next page when using $location.path.

CHANGES.rst

@@ -0,0 +1,28 @@
+Changes in synapse 0.0.1 (2014-08-22)
+=====================================
+Presence has been disabled in this release due to a bug that caused the
+homeserver to spam other remote homeservers.
+
+Homeserver:
+ * Completely change the database schema to support generic event types.
+ * Improve presence reliability.
+ * Improve reliability of joining remote rooms.
+ * Fix bug where room join events were duplicated.
+ * Improve initial sync API to return more information to the client.
+ * Stop generating fake messages for room membership events.
+
+Webclient:
+ * Add tab completion of names.
+ * Add ability to upload and send images.
+ * Add profile pages.
+ * Improve CSS layout of room.
+ * Disambiguate identical display names.
+ * Don't get remote users display names and avatars individually.
+ * Use the new initial sync API to reduce number of round trips to the homeserver.
+ * Change url scheme to use room aliases instead of room ids where known.
+ * Increase longpoll timeout.
+
+Changes in synapse 0.0.0 (2014-08-13)
+=====================================
+
+ * Initial alpha release

README.rst

@@ -24,11 +24,8 @@ To get up and running:
     
     - To run your own **private** homeserver on localhost:8080, install synapse 
       with ``python setup.py develop --user`` and then run one with
-      ``python synapse/app/homeserver.py``
-      
-    - To run your own webclient:
-      ``cd webclient; python -m SimpleHTTPServer`` and hit http://localhost:8000
-      in your web browser (a recent Chrome, Safari or Firefox for now,
+      ``python synapse/app/homeserver.py`` - you will find a webclient running
+      at http://localhost:8080 (use a recent Chrome, Safari or Firefox for now,
       please...)
              
     - To make the homeserver **public** and let it exchange messages with 
@@ -36,7 +33,8 @@ To get up and running:
       up port 8080 and run ``python synapse/app/homeserver.py --host 
       machine.my.domain.name``.  Then come join ``#matrix:matrix.org`` and
       say hi! :)
-    
+
+   
 About Matrix
 ============
 
@@ -120,6 +118,10 @@ may need to also run:
     $ sudo apt-get install python-pip
     $ sudo pip install --upgrade setuptools
     
+If you don't have access to github, then you may need to install ``syutil``
+manually by checking it out and running ``python setup.py develop --user`` on it
+too.
+    
 If you get errors about ``sodium.h`` being missing, you may also need to
 manually install a newer PyNaCl via pip as setuptools installs an old one. Or
 you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and
@@ -142,6 +144,13 @@ This should end with a 'PASSED' result::
     PASSED (successes=143)
 
 
+Upgrading an existing homeserver
+================================
+
+Before upgrading an existing homeserver to a new version, please refer to
+UPGRADE.rst for any additional instructions.
+ 
+
 Setting up Federation
 =====================
 
@@ -189,22 +198,15 @@ Running a Demo Federation of Homeservers
 
 If you want to get up and running quickly with a trio of homeservers in a
 private federation (``localhost:8080``, ``localhost:8081`` and
-``localhost:8082``) which you can then access through the webclient running at http://localhost:8080.  Simply run::
+``localhost:8082``) which you can then access through the webclient running at
+http://localhost:8080. Simply run::
 
     $ demo/start.sh
 
 Running The Demo Web Client
 ===========================
 
-At the present time, the web client is not directly served by the homeserver's
-HTTP server. To serve this in a form the web browser can reach, arrange for the
-'webclient' sub-directory to be made available by any sort of HTTP server that
-can serve static files. For example, python's SimpleHTTPServer will suffice::
-
-    $ cd webclient
-    $ python -m SimpleHTTPServer
-
-You can now point your browser at  http://localhost:8000/  to find the client.
+The homeserver runs a web client by default at http://localhost:8080.
 
 If this is the first time you have used the client from that browser (it uses
 HTML5 local storage to remember its config), you will need to log in to your

UPGRADE.rst

@@ -0,0 +1,24 @@
+Upgrading to v0.0.1
+===================
+
+This release completely changes the database schema and so requires upgrading
+it before starting the new version of the homeserver.
+
+The script "database-prepare-for-0.0.1.sh" should be used to upgrade the
+database. This will save all user information, such as logins and profiles, 
+but will otherwise purge the database. This includes messages, which
+rooms the home server was a member of and room alias mappings.
+
+Before running the command the homeserver should be first completely 
+shutdown. To run it, simply specify the location of the database, e.g.:
+
+  ./database-prepare-for-0.0.1.sh "homeserver.db"
+
+Once this has successfully completed it will be safe to restart the 
+homeserver. You may notice that the homeserver takes a few seconds longer to 
+restart than usual as it reinitializes the database.
+
+On startup of the new version, users can either rejoin remote rooms using room
+aliases or by being reinvited. Alternatively, if any other homeserver sends a
+message to a room that the homeserver was previously in the local HS will 
+automatically rejoin the room.

VERSION

@@ -0,0 +1 @@
+0.0.1

cmdclient/console.py

@@ -233,56 +233,68 @@ class SynapseCmd(cmd.Cmd):
             defer.returnValue(False)
         defer.returnValue(True)
 
-    def do_3pidrequest(self, line):
+    def do_emailrequest(self, line):
         """Requests the association of a third party identifier
-        <medium> The medium of the identifer (currently only 'email')
-        <address> The address of the identifer (ie. the email address)
+        <address> The email address)
+        <clientSecret> A string of characters generated when requesting an email that you'll supply in subsequent calls to identify yourself
+        <sendAttempt> The number of times the user has requested an email. Leave this the same between requests to retry the request at the transport level. Increment it to request that the email be sent again.
         """
-        args = self._parse(line, ['medium', 'address'])
+        args = self._parse(line, ['address', 'clientSecret', 'sendAttempt'])
 
-        if not args['medium'] == 'email':
-            print "Only email is supported currently"
-            return
+        postArgs = {'email': args['address'], 'clientSecret': args['clientSecret'], 'sendAttempt': args['sendAttempt']}
 
-        postArgs = {'email': args['address'], 'clientSecret': '____'}
-
-        reactor.callFromThread(self._do_3pidrequest, postArgs)
+        reactor.callFromThread(self._do_emailrequest, postArgs)
 
     @defer.inlineCallbacks
-    def _do_3pidrequest(self, args):
+    def _do_emailrequest(self, args):
         url = self._identityServerUrl()+"/matrix/identity/api/v1/validate/email/requestToken"
 
         json_res = yield self.http_client.do_request("POST", url, data=urllib.urlencode(args), jsonreq=False,
                                                      headers={'Content-Type': ['application/x-www-form-urlencoded']})
         print json_res
-        if 'tokenId' in json_res:
-            print "Token ID %s sent" % (json_res['tokenId'])
+        if 'sid' in json_res:
+            print "Token sent. Your session ID is %s" % (json_res['sid'])
 
-    def do_3pidvalidate(self, line):
+    def do_emailvalidate(self, line):
         """Validate and associate a third party ID
-        <medium> The medium of the identifer (currently only 'email')
-        <tokenId> The identifier iof the token given in 3pidrequest
+        <sid> The session ID (sid) given to you in the response to requestToken
         <token> The token sent to your third party identifier address
+        <clientSecret> The same clientSecret you supplied in requestToken
         """
-        args = self._parse(line, ['medium', 'tokenId', 'token'])
+        args = self._parse(line, ['sid', 'token', 'clientSecret'])
 
-        if not args['medium'] == 'email':
-            print "Only email is supported currently"
-            return
+        postArgs = { 'sid' : args['sid'], 'token' : args['token'], 'clientSecret': args['clientSecret'] }
 
-        postArgs = { 'tokenId' : args['tokenId'], 'token' : args['token'] }
-        postArgs['mxId'] = self.config["user"]
-
-        reactor.callFromThread(self._do_3pidvalidate, postArgs)
+        reactor.callFromThread(self._do_emailvalidate, postArgs)
 
     @defer.inlineCallbacks
-    def _do_3pidvalidate(self, args):
+    def _do_emailvalidate(self, args):
         url = self._identityServerUrl()+"/matrix/identity/api/v1/validate/email/submitToken"
 
         json_res = yield self.http_client.do_request("POST", url, data=urllib.urlencode(args), jsonreq=False,
                                                      headers={'Content-Type': ['application/x-www-form-urlencoded']})
         print json_res
 
+    def do_3pidbind(self, line):
+        """Validate and associate a third party ID
+        <sid> The session ID (sid) given to you in the response to requestToken
+        <clientSecret> The same clientSecret you supplied in requestToken
+        """
+        args = self._parse(line, ['sid', 'clientSecret'])
+
+        postArgs = { 'sid' : args['sid'], 'clientSecret': args['clientSecret'] }
+        postArgs['mxid'] = self.config["user"]
+
+        reactor.callFromThread(self._do_3pidbind, postArgs)
+
+    @defer.inlineCallbacks
+    def _do_3pidbind(self, args):
+        url = self._identityServerUrl()+"/matrix/identity/api/v1/3pid/bind"
+
+        json_res = yield self.http_client.do_request("POST", url, data=urllib.urlencode(args), jsonreq=False,
+                                                     headers={'Content-Type': ['application/x-www-form-urlencoded']})
+        print json_res
+
     def do_join(self, line):
         """Joins a room: "join <roomid>" """
         try:

database-prepare-for-0.0.1.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# This is will prepare a synapse database for running with v0.0.1 of synapse. 
+# It will store all the user information, but will *delete* all messages and
+# room data.
+
+set -e
+
+cp "$1" "$1.bak"
+
+DUMP=$(sqlite3 "$1" << 'EOF'
+.dump users
+.dump access_tokens
+.dump presence
+.dump profiles
+EOF
+)
+
+rm "$1"
+
+sqlite3 "$1" <<< "$DUMP" 

database-save.sh

@@ -8,7 +8,7 @@
 #
 #   $ sqlite3 homeserver.db < table-save.sql
 
-sqlite3 homeserver.db <<'EOF' >table-save.sql
+sqlite3 "$1" <<'EOF' >table-save.sql
 .dump users
 .dump access_tokens
 .dump presence

demo/webserver.py

@@ -2,9 +2,32 @@ import argparse
 import BaseHTTPServer
 import os
 import SimpleHTTPServer
+import cgi, logging
 
 from daemonize import Daemonize
 
+class SimpleHTTPRequestHandlerWithPOST(SimpleHTTPServer.SimpleHTTPRequestHandler):
+    UPLOAD_PATH = "upload"
+
+    """
+    Accept all post request as file upload
+    """
+    def do_POST(self):
+
+        path = os.path.join(self.UPLOAD_PATH, os.path.basename(self.path))
+        length = self.headers['content-length']
+        data = self.rfile.read(int(length))
+
+        with open(path, 'wb') as fh:
+            fh.write(data)
+
+        self.send_response(200)
+        self.send_header('Content-Type', 'application/json')
+        self.end_headers()
+
+        # Return the absolute path of the uploaded file
+        self.wfile.write('{"url":"/%s"}' % path)
+
 
 def setup():
     parser = argparse.ArgumentParser()
@@ -19,7 +42,7 @@ def setup():
 
     httpd = BaseHTTPServer.HTTPServer(
         ('', args.port),
-        SimpleHTTPServer.SimpleHTTPRequestHandler
+        SimpleHTTPRequestHandlerWithPOST
     )
 
     def run():

docs/client-server/howto.rst

@@ -0,0 +1,303 @@
+TODO(kegan): Tweak joinalias API keys/path? Event stream historical > live needs
+a token (currently doesn't). im/sync responses include outdated event formats
+(room membership change messages). Room config (specifically: message history,
+public rooms). /register seems super simplistic compared to /login, maybe it
+would be better if /register used the same technique as /login? /register should
+be "user" not "user_id".
+
+
+How to use the client-server API
+================================
+
+This guide focuses on how the client-server APIs *provided by the reference 
+home server* can be used. Since this is specific to a home server 
+implementation, there may be variations in relation to registering/logging in
+which are not covered in extensive detail in this guide.
+
+If you haven't already, get a home server up and running on 
+``http://localhost:8080``.
+
+
+Accounts
+========
+Before you can send and receive messages, you must **register** for an account. 
+If you already have an account, you must **login** into it.
+
+**Try out the fiddle: http://jsfiddle.net/jrf1h02d/**
+
+Registration
+------------
+The aim of registration is to get a user ID and access token which you will need
+when accessing other APIs::
+
+    curl -XPOST -d '{"user_id":"example", "password":"wordpass"}' "http://localhost:8080/matrix/client/api/v1/register"
+
+    {
+        "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.AqdSzFmFYrLrTmteXc", 
+        "home_server": "localhost", 
+        "user_id": "@example:localhost"
+    }
+
+NB: If a ``user_id`` is not specified, one will be randomly generated for you. 
+If you do not specify a ``password``, you will be unable to login to the account
+if you forget the ``access_token``.
+
+Implementation note: The matrix specification does not enforce how users 
+register with a server. It just specifies the URL path and absolute minimum 
+keys. The reference home server uses a username/password to authenticate user,
+but other home servers may use different methods.
+
+Login
+-----
+The aim when logging in is to get an access token for your existing user ID::
+
+    curl -XGET "http://localhost:8080/matrix/client/api/v1/login"
+
+    {
+        "type": "m.login.password"
+    }
+
+    curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "http://localhost:8080/matrix/client/api/v1/login"
+
+    {
+        "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd", 
+        "home_server": "localhost", 
+        "user_id": "@example:localhost"
+    }
+    
+Implementation note: Different home servers may implement different methods for 
+logging in to an existing account. In order to check that you know how to login 
+to this home server, you must perform a ``GET`` first and make sure you 
+recognise the login type. If you do not know how to login, you can 
+``GET /login/fallback`` which will return a basic webpage which you can use to 
+login. The reference home server implementation support username/password login,
+but other home servers may support different login methods (e.g. OAuth2).
+
+
+Communicating
+=============
+
+In order to communicate with another user, you must **create a room** with that 
+user and **send a message** to that room. 
+
+**Try out the fiddle: http://jsfiddle.net/jnwqcshc/**
+
+Creating a room
+---------------
+If you want to send a message to someone, you have to be in a room with them. To
+create a room::
+
+    curl -XPOST -d '{"room_alias_name":"tutorial"}' "http://localhost:8080/matrix/client/api/v1/rooms?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
+
+    {
+        "room_alias": "#tutorial:localhost", 
+        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+    }
+    
+The "room alias" is a human-readable string which can be shared with other users
+so they can join a room, rather than the room ID which is a randomly generated
+string. You can have multiple room aliases per room.
+
+TODO(kegan): How to add/remove aliases from an existing room.
+    
+
+Sending messages
+----------------
+You can now send messages to this room::
+
+    curl -XPUT -d '{"msgtype":"m.text", "body":"hello"}' "http://localhost:8080/matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/messages/%40example%3Alocalhost/msgid1?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
+    
+NB: There are no limitations to the types of messages which can be exchanged.
+The only requirement is that ``"msgtype"`` is specified.
+
+NB: Depending on the room config, users who join the room may be able to see
+message history from before they joined.
+
+Users and rooms
+===============
+
+Each room can be configured to allow or disallow certain rules. In particular,
+these rules may specify if you require an **invitation** from someone already in
+the room in order to **join the room**. In addition, you may also be able to 
+join a room **via a room alias** if one was set up.
+
+**Try out the fiddle: http://jsfiddle.net/og1xokcr/**
+
+Inviting a user to a room
+-------------------------
+You can directly invite a user to a room like so::
+
+    curl -XPUT -d '{"membership":"invite"}' "http://localhost:8080/matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
+    
+This informs ``@myfriend:localhost`` of the room ID 
+``!CvcvRuDYDzTOzfKKgh:localhost`` and allows them to join the room.
+
+Joining a room via an invite
+----------------------------
+If you receive an invite, you can join the room by changing the membership to
+join::
+
+    curl -XPUT -d '{"membership":"join"}' "http://localhost:8080/matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
+    
+NB: Only the person invited (``@myfriend:localhost``) can change the membership
+state to ``"join"``.
+
+Joining a room via an alias
+---------------------------
+Alternatively, if you know the room alias for this room and the room config 
+allows it, you can directly join a room via the alias::
+
+    curl -XPUT -d '{}' "http://localhost:8080/matrix/client/api/v1/join/%23tutorial%3Alocalhost?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
+    
+    {
+        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+    }
+    
+You will need to use the room ID when sending messages, not the room alias.
+
+NB: If the room is configured to be an invite-only room, you will still require
+an invite in order to join the room even though you know the room alias. As a
+result, it is more common to see a room alias in relation to a public room, 
+which do not require invitations.
+
+Getting events
+==============
+An event is some interesting piece of data that a client may be interested in. 
+It can be a message in a room, a room invite, etc. There are many different ways
+of getting events, depending on what the client already knows.
+
+**Try out the fiddle: http://jsfiddle.net/5uk4dqe2/**
+
+Getting all state
+-----------------
+If the client doesn't know any information on the rooms the user is 
+invited/joined on, they can get all the user's state for all rooms::
+
+    curl -XGET "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
+    
+    [
+        {
+            "membership": "join", 
+            "messages": {
+                "chunk": [
+                    {
+                        "content": {
+                            "body": "@example:localhost joined the room.", 
+                            "hsob_ts": 1408444664249, 
+                            "membership": "join", 
+                            "membership_source": "@example:localhost", 
+                            "membership_target": "@example:localhost", 
+                            "msgtype": "m.text"
+                        }, 
+                        "event_id": "lZjmmlrEvo", 
+                        "msg_id": "m1408444664249", 
+                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
+                        "type": "m.room.message", 
+                        "user_id": "_homeserver_"
+                    }, 
+                    {
+                        "content": {
+                            "body": "hello", 
+                            "hsob_ts": 1408445405672, 
+                            "msgtype": "m.text"
+                        }, 
+                        "event_id": "BiBJqamISg", 
+                        "msg_id": "msgid1", 
+                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
+                        "type": "m.room.message", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    [...]
+                    {
+                        "content": {
+                            "body": "@myfriend:localhost joined the room.", 
+                            "hsob_ts": 1408446501661, 
+                            "membership": "join", 
+                            "membership_source": "@myfriend:localhost", 
+                            "membership_target": "@myfriend:localhost", 
+                            "msgtype": "m.text"
+                        }, 
+                        "event_id": "IMmXbOzFAa", 
+                        "msg_id": "m1408446501661", 
+                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
+                        "type": "m.room.message", 
+                        "user_id": "_homeserver_"
+                    }
+                ], 
+                "end": "20", 
+                "start": "0"
+            }, 
+            "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+        }
+    ]
+    
+This returns all the room IDs of rooms the user is invited/joined on, as well as
+all of the messages and feedback for these rooms. This can be a LOT of data. You
+may just want the most recent message for each room. This can be achieved by 
+applying pagination stream parameters to this request::
+
+    curl -XGET "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END&to=START&limit=1"
+    
+    [
+        {
+            "membership": "join", 
+            "messages": {
+                "chunk": [
+                    {
+                        "content": {
+                            "body": "@myfriend:localhost joined the room.", 
+                            "hsob_ts": 1408446501661, 
+                            "membership": "join", 
+                            "membership_source": "@myfriend:localhost", 
+                            "membership_target": "@myfriend:localhost", 
+                            "msgtype": "m.text"
+                        }, 
+                        "event_id": "IMmXbOzFAa", 
+                        "msg_id": "m1408446501661", 
+                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
+                        "type": "m.room.message", 
+                        "user_id": "_homeserver_"
+                    }
+                ], 
+                "end": "20", 
+                "start": "21"
+            }, 
+            "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+        }
+    ]
+
+Getting live state
+------------------
+Once you know which rooms the client has previously interacted with, you need to
+listen for incoming events. This can be done like so::
+
+    curl -XGET "http://localhost:8080/matrix/client/api/v1/events?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END"
+    
+    {
+        "chunk": [], 
+        "end": "215", 
+        "start": "215"
+    }
+    
+This will block waiting for an incoming event, timing out after several seconds.
+Even if there are no new events (as in the example above), there will be some
+pagination stream response keys. The client should make subsequent requests 
+using the value of the ``"end"`` key (in this case ``215``) as the ``from`` 
+query parameter. This value should be stored so when the client reopens your app
+after a period of inactivity, you can resume from where you got up to in the 
+event stream. If it has been a long period of inactivity, there may be LOTS of 
+events waiting for the user. In this case, you may wish to get all state instead
+and then resume getting live state from a newer end token.
+
+NB: The timeout can be changed by adding a ``timeout`` query parameter, which is
+in milliseconds. A timeout of 0 will not block.
+
+
+Example application
+-------------------
+The following example demonstrates registration and login, live event streaming,
+creating and joining rooms, sending messages, getting member lists and getting 
+historical messages for a room. This covers most functionality of a messaging
+application.
+
+**Try out the fiddle: http://jsfiddle.net/L8r3o1wr/**

docs/client-server/specification.rst

@@ -1,6 +1,6 @@
-=========================
-Synapse Client-Server API
-=========================
+========================
+Matrix Client-Server API
+========================
 
 The following specification outlines how a client can send and receive data from 
 a home server.
@@ -262,7 +262,10 @@ the error, but the keys 'error' and 'errcode' will always be present.
 Some standard error codes are below:
 
 M_FORBIDDEN:
-Forbidden access, e.g. bad access token, failed login.
+Forbidden access, e.g. joining a room without permission, failed login.
+
+M_UNKNOWN_TOKEN:
+The access token specified was not recognised.
 
 M_BAD_JSON:
 Request contained valid JSON, but it was malformed in some way, e.g. missing
@@ -411,6 +414,9 @@ The server checks this, finds it is valid, and returns:
 {
   "access_token": "abcdef0123456789"
 }
+The server may optionally return "user_id" to confirm or change the user's ID.
+This is particularly useful if the home server wishes to support localpart entry
+of usernames (e.g. "bob" rather than "@bob:matrix.org").
 
 OAuth2-based
 ------------
@@ -688,6 +694,16 @@ Invite/Joining/Leaving a room
   Required keys:
     membership : [join|invite] - The membership state of $user_id in room 
                                  $room_id.
+  Optional keys:
+    displayname,
+    avatar_url : String fields from the member user's profile
+    state,
+    status_msg,
+    mtime_age : Presence information
+
+  These optional keys provide extra information that the client is likely to
+  be interested in so it doesn't have to perform an additional profile or
+  presence information fetch.
 
 Where:
   join - Indicate you ($user_id) are joining the room $room_id.

docs/client-server/swagger_matrix/api-docs

@@ -0,0 +1,38 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "apis": [
+    {
+      "path": "/login",
+      "description": "Login operations"
+    },
+    {
+      "path": "/registration",
+      "description": "Registration operations"
+    },
+    {
+      "path": "/rooms",
+      "description": "Room operations"
+    },
+    {
+      "path": "/profile",
+      "description": "Profile operations"
+    },
+    {
+      "path": "/presence",
+      "description": "Presence operations"
+    }
+  ],
+  "authorizations": {
+    "token": {
+      "scopes": []
+    }
+  },
+  "info": {
+    "title": "Matrix Client-Server API Reference",
+    "description": "This contains the client-server API for the reference implementation of the home server",
+    "termsOfServiceUrl": "http://matrix.org",
+    "license": "Apache 2.0",
+    "licenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.html"
+  }
+}

docs/client-server/swagger_matrix/events

@@ -0,0 +1,299 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "basePath": "http://petstore.swagger.wordnik.com/api",
+  "resourcePath": "/user",
+  "produces": [
+    "application/json"
+  ],
+  "apis": [
+    {
+      "path": "/user",
+      "operations": [
+        {
+          "method": "POST",
+          "summary": "Create user",
+          "notes": "This can only be done by the logged in user.",
+          "type": "void",
+          "nickname": "createUser",
+          "authorizations": {
+            "oauth2": [
+              {
+                "scope": "test:anything",
+                "description": "anything"
+              }
+            ]
+          },
+          "parameters": [
+            {
+              "name": "body",
+              "description": "Created user object",
+              "required": true,
+              "type": "User",
+              "paramType": "body"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/user/logout",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Logs out current logged in user session",
+          "notes": "",
+          "type": "void",
+          "nickname": "logoutUser",
+          "authorizations": {},
+          "parameters": []
+        }
+      ]
+    },
+    {
+      "path": "/user/createWithArray",
+      "operations": [
+        {
+          "method": "POST",
+          "summary": "Creates list of users with given input array",
+          "notes": "",
+          "type": "void",
+          "nickname": "createUsersWithArrayInput",
+          "authorizations": {
+            "oauth2": [
+              {
+                "scope": "test:anything",
+                "description": "anything"
+              }
+            ]
+          },
+          "parameters": [
+            {
+              "name": "body",
+              "description": "List of user object",
+              "required": true,
+              "type": "array",
+              "items": {
+                "$ref": "User"
+              },
+              "paramType": "body"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/user/createWithList",
+      "operations": [
+        {
+          "method": "POST",
+          "summary": "Creates list of users with given list input",
+          "notes": "",
+          "type": "void",
+          "nickname": "createUsersWithListInput",
+          "authorizations": {
+            "oauth2": [
+              {
+                "scope": "test:anything",
+                "description": "anything"
+              }
+            ]
+          },
+          "parameters": [
+            {
+              "name": "body",
+              "description": "List of user object",
+              "required": true,
+              "type": "array",
+              "items": {
+                "$ref": "User"
+              },
+              "paramType": "body"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/user/{username}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Updated user",
+          "notes": "This can only be done by the logged in user.",
+          "type": "void",
+          "nickname": "updateUser",
+          "authorizations": {
+            "oauth2": [
+              {
+                "scope": "test:anything",
+                "description": "anything"
+              }
+            ]
+          },
+          "parameters": [
+            {
+              "name": "username",
+              "description": "name that need to be deleted",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "body",
+              "description": "Updated user object",
+              "required": true,
+              "type": "User",
+              "paramType": "body"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "Invalid username supplied"
+            },
+            {
+              "code": 404,
+              "message": "User not found"
+            }
+          ]
+        },
+        {
+          "method": "DELETE",
+          "summary": "Delete user",
+          "notes": "This can only be done by the logged in user.",
+          "type": "void",
+          "nickname": "deleteUser",
+          "authorizations": {
+            "oauth2": [
+              {
+                "scope": "test:anything",
+                "description": "anything"
+              }
+            ]
+          },
+          "parameters": [
+            {
+              "name": "username",
+              "description": "The name that needs to be deleted",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "Invalid username supplied"
+            },
+            {
+              "code": 404,
+              "message": "User not found"
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get user by user name",
+          "notes": "",
+          "type": "User",
+          "nickname": "getUserByName",
+          "authorizations": {},
+          "parameters": [
+            {
+              "name": "username",
+              "description": "The name that needs to be fetched. Use user1 for testing.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "Invalid username supplied"
+            },
+            {
+              "code": 404,
+              "message": "User not found"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/user/login",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Logs user into the system",
+          "notes": "",
+          "type": "string",
+          "nickname": "loginUser",
+          "authorizations": {},
+          "parameters": [
+            {
+              "name": "username",
+              "description": "The user name for login",
+              "required": true,
+              "type": "string",
+              "paramType": "query"
+            },
+            {
+              "name": "password",
+              "description": "The password for login in clear text",
+              "required": true,
+              "type": "string",
+              "paramType": "query"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "Invalid username and password combination"
+            }
+          ]
+        }
+      ]
+    }
+  ],
+  "models": {
+    "User": {
+      "id": "User",
+      "properties": {
+        "id": {
+          "type": "integer",
+          "format": "int64"
+        },
+        "firstName": {
+          "type": "string"
+        },
+        "username": {
+          "type": "string"
+        },
+        "lastName": {
+          "type": "string"
+        },
+        "email": {
+          "type": "string"
+        },
+        "password": {
+          "type": "string"
+        },
+        "phone": {
+          "type": "string"
+        },
+        "userStatus": {
+          "type": "integer",
+          "format": "int32",
+          "description": "User Status",
+          "enum": [
+            "1-registered",
+            "2-active",
+            "3-closed"
+          ]
+        }
+      }
+    }
+  }
+}

docs/client-server/swagger_matrix/login

@@ -0,0 +1,102 @@
+{
+  "apiVersion": "1.0.0", 
+  "apis": [
+    {
+      "operations": [
+        {
+          "method": "GET", 
+          "nickname": "get_login_info", 
+          "notes": "All login stages MUST be mentioned if there is >1 login type.", 
+          "summary": "Get the login mechanism to use when logging in.", 
+          "type": "LoginInfo"
+        }, 
+        {
+          "method": "POST", 
+          "nickname": "submit_login", 
+          "notes": "If this is part of a multi-stage login, there MUST be a 'session' key.", 
+          "parameters": [
+            {
+              "description": "A login submission", 
+              "name": "body", 
+              "paramType": "body", 
+              "required": true, 
+              "type": "LoginSubmission"
+            }
+          ], 
+          "responseMessages": [
+            {
+              "code": 400, 
+              "message": "Bad login type"
+            }, 
+            {
+              "code": 400, 
+              "message": "Missing JSON keys"
+            }
+          ], 
+          "summary": "Submit a login action.", 
+          "type": "LoginResult"
+        }
+      ], 
+      "path": "/login"
+    }
+  ], 
+  "basePath": "http://localhost:8080/matrix/client/api/v1", 
+  "consumes": [
+    "application/json"
+  ], 
+  "models": {
+    "LoginInfo": {
+      "id": "LoginInfo", 
+      "properties": {
+        "stages": {
+          "description": "Multi-stage login only: An array of all the login types required to login.", 
+          "format": "string", 
+          "type": "array"
+        }, 
+        "type": {
+          "description": "The login type that must be used when logging in.", 
+          "type": "string"
+        }
+      }
+    }, 
+    "LoginResult": {
+      "id": "LoginResult", 
+      "properties": {
+        "access_token": {
+          "description": "The access token for this user's login if this is the final stage of the login process.", 
+          "type": "string"
+        }, 
+        "next": {
+          "description": "Multi-stage login only: The next login type to submit.", 
+          "type": "string"
+        },
+        "session": {
+          "description": "Multi-stage login only: The session token to send when submitting the next login type.",
+          "type": "string"
+        }
+      }
+    }, 
+    "LoginSubmission": {
+      "id": "LoginSubmission", 
+      "properties": {
+        "type": {
+          "description": "The type of login being submitted.", 
+          "type": "string"
+        },
+        "session": {
+          "description": "Multi-stage login only: The session token from an earlier login stage.",
+          "type": "string"
+        },
+        "_login_type_defined_keys_": {
+          "description": "Keys as defined by the specified login type, e.g. \"user\", \"password\""
+        }
+      }
+    }
+  }, 
+  "produces": [
+    "application/json"
+  ], 
+  "resourcePath": "/login", 
+  "swaggerVersion": "1.2"
+}
+

docs/client-server/swagger_matrix/presence

@@ -0,0 +1,164 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "basePath": "http://localhost:8080/matrix/client/api/v1",
+  "resourcePath": "/presence",
+  "produces": [
+    "application/json"
+  ],
+  "consumes": [
+    "application/json"
+  ],
+  "apis": [
+    {
+      "path": "/presence/{userId}/status",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Update this user's presence state.",
+          "notes": "This can only be done by the logged in user.",
+          "type": "void",
+          "nickname": "update_presence",
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The new presence state",
+              "required": true,
+              "type": "PresenceUpdate",
+              "paramType": "body"
+            },
+            {
+              "name": "userId",
+              "description": "The user whose presence to set.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get this user's presence state.",
+          "notes": "Get this user's presence state.",
+          "type": "PresenceUpdate",
+          "nickname": "get_presence",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose presence to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/presence_list/{userId}",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Retrieve a list of presences for all of this user's friends.",
+          "notes": "",
+          "type": "array",
+          "items": {
+            "$ref": "Presence"
+          },
+          "nickname": "get_presence_list",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose presence list to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "POST",
+          "summary": "Add or remove users from this presence list.",
+          "notes": "Add or remove users from this presence list.",
+          "type": "void",
+          "nickname": "modify_presence_list",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose presence list is being modified.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "body",
+              "description": "The modifications to make to this presence list.",
+              "required": true,
+              "type": "PresenceListModifications",
+              "paramType": "body"
+            }
+          ]
+        }
+      ]
+    }
+  ],
+  "models": {
+    "PresenceUpdate": {
+      "id": "PresenceUpdate",
+      "properties": {
+        "state": {
+          "type": "string",
+          "description": "Enum: The presence state.",
+          "enum": [
+            "offline",
+            "unavailable",
+            "online",
+            "free_for_chat"
+          ]
+        },
+        "status_msg": {
+          "type": "string",
+          "description": "The user-defined message associated with this presence state."
+        }
+      },
+      "subTypes": [
+        "Presence"
+      ]
+    },
+    "Presence": {
+      "id": "Presence",
+      "properties": {
+        "mtime_age": {
+          "type": "integer",
+          "format": "int64",
+          "description": "The last time this user's presence state changed, in milliseconds."
+        },
+        "user_id": {
+          "type": "string",
+          "description": "The fully qualified user ID"
+        }
+      }
+    },
+    "PresenceListModifications": {
+      "id": "PresenceListModifications",
+      "properties": {
+        "invite": {
+          "type": "array",
+          "description": "A list of user IDs to add to the list.",
+          "items": {
+            "type": "string",
+            "description": "A fully qualified user ID."
+          }
+        },
+        "drop": {
+          "type": "array",
+          "description": "A list of user IDs to remove from the list.",
+          "items": {
+            "type": "string",
+            "description": "A fully qualified user ID."
+          }
+        }
+      }
+    }
+  }
+}

docs/client-server/swagger_matrix/profile

@@ -0,0 +1,122 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "basePath": "http://localhost:8080/matrix/client/api/v1",
+  "resourcePath": "/profile",
+  "produces": [
+    "application/json"
+  ],
+  "consumes": [
+    "application/json"
+  ],
+  "apis": [
+    {
+      "path": "/profile/{userId}/displayname",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Set a display name.",
+          "notes": "This can only be done by the logged in user.",
+          "type": "void",
+          "nickname": "set_display_name",
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The new display name for this user.",
+              "required": true,
+              "type": "DisplayName",
+              "paramType": "body"
+            },
+            {
+              "name": "userId",
+              "description": "The user whose display name to set.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get a display name.",
+          "notes": "This can be done by anyone.",
+          "type": "DisplayName",
+          "nickname": "get_display_name",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose display name to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/profile/{userId}/avatar_url",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Set an avatar URL.",
+          "notes": "This can only be done by the logged in user.",
+          "type": "void",
+          "nickname": "set_avatar_url",
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The new avatar url for this user.",
+              "required": true,
+              "type": "AvatarUrl",
+              "paramType": "body"
+            },
+            {
+              "name": "userId",
+              "description": "The user whose avatar url to set.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get an avatar url.",
+          "notes": "This can be done by anyone.",
+          "type": "AvatarUrl",
+          "nickname": "get_avatar_url",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose avatar url to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    }
+  ],
+  "models": {
+    "DisplayName": {
+      "id": "DisplayName",
+      "properties": {
+        "displayname": {
+          "type": "string",
+          "description": "The textual display name"
+        }
+      }
+    },
+    "AvatarUrl": {
+      "id": "AvatarUrl",
+      "properties": {
+        "avatar_url": {
+          "type": "string",
+          "description": "A url to an image representing an avatar."
+        }
+      }
+    }
+  }
+}

docs/client-server/swagger_matrix/registration

@@ -0,0 +1,75 @@
+{
+  "apiVersion": "1.0.0", 
+  "apis": [
+    {
+      "operations": [
+        {
+          "method": "POST", 
+          "nickname": "register", 
+          "notes": "Volatile: This API is likely to change.", 
+          "parameters": [
+            {
+              "description": "A registration request", 
+              "name": "body", 
+              "paramType": "body", 
+              "required": true, 
+              "type": "RegistrationRequest"
+            }
+          ], 
+          "responseMessages": [
+            {
+              "code": 400, 
+              "message": "No JSON object."
+            }, 
+            {
+              "code": 400, 
+              "message": "User ID must only contain characters which do not require url encoding."
+            },
+            {
+              "code": 400, 
+              "message": "User ID already taken."
+            }
+          ], 
+          "summary": "Register with the home server.", 
+          "type": "RegistrationResponse"
+        }
+      ], 
+      "path": "/register"
+    }
+  ], 
+  "basePath": "http://localhost:8080/matrix/client/api/v1", 
+  "consumes": [
+    "application/json"
+  ], 
+  "models": {
+    "RegistrationResponse": {
+      "id": "RegistrationResponse", 
+      "properties": {
+        "access_token": {
+          "description": "The access token for this user.", 
+          "type": "string"
+        }, 
+        "user_id": {
+          "description": "The fully-qualified user ID.", 
+          "type": "string"
+        }
+      }
+    }, 
+    "RegistrationRequest": {
+      "id": "RegistrationRequest", 
+      "properties": {
+        "user_id": {
+          "description": "The desired user ID. If not specified, a random user ID will be allocated.", 
+          "type": "string",
+          "required": false
+        }
+      }
+    }
+  }, 
+  "produces": [
+    "application/json"
+  ], 
+  "resourcePath": "/register", 
+  "swaggerVersion": "1.2"
+}
+

docs/client-server/swagger_matrix/rooms

@@ -0,0 +1,807 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "basePath": "http://localhost:8080/matrix/client/api/v1", 
+  "resourcePath": "/rooms",
+  "produces": [
+    "application/json"
+  ],
+  "consumes": [
+    "application/json"
+  ],
+  "authorizations": {
+    "token": []
+  },
+  "apis": [
+    {
+      "path": "/rooms/{roomId}/messages/{userId}/{messageId}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Send a message in this room.",
+          "notes": "Send a message in this room.",
+          "type": "void",
+          "nickname": "send_message",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The message contents",
+              "required": true,
+              "type": "Message",
+              "paramType": "body"
+            },
+            {
+              "name": "roomId",
+              "description": "The room to send the message in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "userId",
+              "description": "The fully qualified message sender's user ID.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "messageId",
+              "description": "A message ID which is unique for each room and user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 403,
+              "message": "Must send messages as yourself."
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get a message from this room.",
+          "notes": "Get a message from this room.",
+          "type": "Message",
+          "nickname": "get_message",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to send the message in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "userId",
+              "description": "The fully qualified message sender's user ID.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "messageId",
+              "description": "A message ID which is unique for each room and user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 404,
+              "message": "Message not found."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/topic",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Set the topic for this room.",
+          "notes": "Set the topic for this room.",
+          "type": "void",
+          "nickname": "set_topic",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The topic contents",
+              "required": true,
+              "type": "Topic",
+              "paramType": "body"
+            },
+            {
+              "name": "roomId",
+              "description": "The room to set the topic in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 403,
+              "message": "Must send messages as yourself."
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get the topic for this room.",
+          "notes": "Get the topic for this room.",
+          "type": "Topic",
+          "nickname": "get_topic",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to get topic in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 404,
+              "message": "Topic not found."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/messages/{msgSenderId}/{messageId}/feedback/{senderId}/{feedbackType}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Send feedback to a message.",
+          "notes": "Send feedback to a message.",
+          "type": "void",
+          "nickname": "send_feedback",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The feedback contents",
+              "required": true,
+              "type": "Feedback",
+              "paramType": "body"
+            },
+            {
+              "name": "roomId",
+              "description": "The room to send the feedback in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "msgSenderId",
+              "description": "The fully qualified message sender's user ID.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "messageId",
+              "description": "A message ID which is unique for each room and user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "senderId",
+              "description": "The fully qualified feedback sender's user ID.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "feedbackType",
+              "description": "The type of feedback being sent.",
+              "required": true,
+              "type": "string",
+              "paramType": "path",
+              "enum": [
+                "d",
+                "r"
+              ]
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 403,
+              "message": "Must send feedback as yourself."
+            },
+            {
+              "code": 400,
+              "message": "Bad feedback type."
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get feedback for a message.",
+          "notes": "Get feedback for a message.",
+          "type": "Feedback",
+          "nickname": "get_feedback",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to send the message in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "msgSenderId",
+              "description": "The fully qualified message sender's user ID.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "messageId",
+              "description": "A message ID which is unique for each room and user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "senderId",
+              "description": "The fully qualified feedback sender's user ID.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "feedbackType",
+              "description": "Enum: The type of feedback being sent.",
+              "required": true,
+              "type": "string",
+              "paramType": "path",
+              "enum": [
+                "d",
+                "r"
+              ]
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 404,
+              "message": "Feedback not found."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/members/{userId}/state",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Change the membership state for a user in a room.",
+          "notes": "Change the membership state for a user in a room.",
+          "type": "void",
+          "nickname": "set_membership",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The new membership state",
+              "required": true,
+              "type": "Member",
+              "paramType": "body"
+            },
+            {
+              "name": "userId",
+              "description": "The user whose membership is being changed.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "roomId",
+              "description": "The room which has this user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "No membership key."
+            },
+            {
+              "code": 400,
+              "message": "Bad membership value."
+            },
+            {
+              "code": 403,
+              "message": "When inviting: You are not in the room."
+            },
+            {
+              "code": 403,
+              "message": "When inviting: <target> is already in the room."
+            },
+            {
+              "code": 403,
+              "message": "When joining: Cannot force another user to join."
+            },
+            {
+              "code": 403,
+              "message": "When joining: You are not invited to this room."
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get the membership state of a user in a room.",
+          "notes": "Get the membership state of a user in a room.",
+          "type": "Member",
+          "nickname": "get_membership",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose membership state you want to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "roomId",
+              "description": "The room which has this user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 404,
+              "message": "Member not found."
+            }
+          ]
+        },
+        {
+          "method": "DELETE",
+          "summary": "Leave a room.",
+          "notes": "Leave a room.",
+          "type": "void",
+          "nickname": "remove_membership",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user who is leaving.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "roomId",
+              "description": "The room which has this user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 403,
+              "message": "You are not in the room."
+            },
+            {
+              "code": 403,
+              "message": "Cannot force another user to leave."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/join/{roomAlias}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Join a room via a room alias.",
+          "notes": "Join a room via a room alias.",
+          "type": "RoomInfo",
+          "nickname": "join_room_via_alias",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "roomAlias",
+              "description": "The room alias to join.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "Bad room alias."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms",
+      "operations": [
+        {
+          "method": "POST",
+          "summary": "Create a room.",
+          "notes": "Create a room.",
+          "type": "RoomInfo",
+          "nickname": "create_room",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The desired configuration for the room.",
+              "required": true,
+              "type": "RoomConfig",
+              "paramType": "body"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "Body must be JSON."
+            },
+            {
+              "code": 400,
+              "message": "Room alias already taken."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/messages/list",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get a list of messages for this room.",
+          "notes": "Get a list of messages for this room.",
+          "type": "MessagePaginationChunk",
+          "nickname": "get_messages",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to get messages in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "from",
+              "description": "The token to start getting results from.",
+              "required": false,
+              "type": "string",
+              "paramType": "query"
+            },
+            {
+              "name": "to",
+              "description": "The token to stop getting results at.",
+              "required": false,
+              "type": "string",
+              "paramType": "query"
+            },
+            {
+              "name": "limit",
+              "description": "The maximum number of messages to return.",
+              "required": false,
+              "type": "integer",
+              "paramType": "query"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/members/list",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get a list of members for this room.",
+          "notes": "Get a list of members for this room.",
+          "type": "MemberPaginationChunk",
+          "nickname": "get_members",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to get a list of members from.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "from",
+              "description": "The token to start getting results from.",
+              "required": false,
+              "type": "string",
+              "paramType": "query"
+            },
+            {
+              "name": "to",
+              "description": "The token to stop getting results at.",
+              "required": false,
+              "type": "string",
+              "paramType": "query"
+            },
+            {
+              "name": "limit",
+              "description": "The maximum number of members to return.",
+              "required": false,
+              "type": "integer",
+              "paramType": "query"
+            }
+          ]
+        }
+      ]
+    }
+  ],
+  "models": {
+    "Topic": {
+      "id": "Topic",
+      "properties": {
+        "topic": {
+          "type": "string",
+          "description": "The topic text"
+        }
+      }
+    },
+    "Message": {
+      "id": "Message",
+      "properties": {
+        "msgtype": {
+          "type": "string",
+          "description": "The type of message being sent, e.g. \"m.text\"",
+          "required": true
+        },
+        "_msgtype_defined_keys_": {
+          "description": "Additional keys as defined by the msgtype, e.g. \"body\""
+        }
+      }
+    },
+    "Feedback": {
+      "id": "Feedback",
+      "properties": {
+      }
+    },
+    "Member": {
+      "id": "Member",
+      "properties": {
+        "membership": {
+          "type": "string",
+          "description": "Enum: The membership state of this member.",
+          "enum": [
+            "invite",
+            "join",
+            "leave",
+            "knock"
+          ]
+        }
+      }
+    },
+    "RoomInfo": {
+      "id": "RoomInfo",
+      "properties": {
+        "room_id": {
+          "type": "string",
+          "description": "The allocated room ID.",
+          "required": true
+        },
+        "room_alias": {
+          "type": "string",
+          "description": "The alias for the room.",
+          "required": false
+        }
+      }
+    },
+    "RoomConfig": {
+      "id": "RoomConfig",
+      "properties": {
+        "visibility": {
+          "type": "string",
+          "description": "Enum: The room visibility.",
+          "required": false,
+          "enum": [
+            "public",
+            "private"
+          ]
+        },
+        "room_alias_name": {
+          "type": "string",
+          "description": "The alias to give the new room.",
+          "required": false
+        }
+      }
+    },
+    "PaginationRequest": {
+      "id": "PaginationRequest",
+      "properties": {
+        "from": {
+          "type": "string",
+          "description": "The token to start getting results from."
+        },
+        "to": {
+          "type": "string",
+          "description": "The token to stop getting results at."
+        },
+        "limit": {
+          "type": "integer",
+          "description": "The maximum number of entries to return."
+        }
+      }
+    },
+    "PaginationChunk": {
+      "id": "PaginationChunk",
+      "properties": {
+        "start": {
+          "type": "string",
+          "description": "A token which correlates to the first value in \"chunk\" for paginating.",
+          "required": true
+        },
+        "end": {
+          "type": "string",
+          "description": "A token which correlates to the last value in \"chunk\" for paginating.",
+          "required": true
+        }
+      },
+      "subTypes": [
+        "MessagePaginationChunk"
+      ]
+    },
+    "MessagePaginationChunk": {
+      "id": "MessagePaginationChunk",
+      "properties": {
+        "chunk": {
+          "type": "array",
+          "description": "A list of message events.",
+          "items": {
+            "$ref": "MessageEvent"
+          },
+          "required": true
+        }
+      }
+    },
+    "MemberPaginationChunk": {
+      "id": "MemberPaginationChunk",
+      "properties": {
+        "chunk": {
+          "type": "array",
+          "description": "A list of member events.",
+          "items": {
+            "$ref": "MemberEvent"
+          },
+          "required": true
+        }
+      }
+    },
+    "Event": {
+      "id": "Event",
+      "properties": {
+        "event_id": {
+          "type": "string",
+          "description": "An ID which uniquely identifies this event.",
+          "required": true
+        },
+        "room_id": {
+          "type": "string",
+          "description": "The room in which this event occurred.",
+          "required": true
+        }
+      },
+      "subTypes": [
+        "MessageEvent"
+      ]
+    },
+    "MessageEvent": {
+      "id": "MessageEvent",
+      "properties": {
+        "content": {
+          "type": "Message"
+        }
+      }
+    },
+    "MemberEvent": {
+      "id": "MemberEvent",
+      "properties": {
+        "content": {
+          "type": "Member"
+        }
+      }
+    },
+    "Tag": {
+      "id": "Tag",
+      "properties": {
+        "id": {
+          "type": "integer",
+          "format": "int64"
+        },
+        "name": {
+          "type": "string"
+        }
+      }
+    },
+    "Pet": {
+      "id": "Pet",
+      "required": [
+        "id",
+        "name"
+      ],
+      "properties": {
+        "id": {
+          "type": "integer",
+          "format": "int64",
+          "description": "unique identifier for the pet",
+          "minimum": "0.0",
+          "maximum": "100.0"
+        },
+        "category": {
+          "$ref": "Category"
+        },
+        "name": {
+          "type": "string"
+        },
+        "photoUrls": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        },
+        "tags": {
+          "type": "array",
+          "items": {
+            "$ref": "Tag"
+          }
+        },
+        "status": {
+          "type": "string",
+          "description": "pet status in the store",
+          "enum": [
+            "available",
+            "pending",
+            "sold"
+          ]
+        }
+      }
+    },
+    "Category": {
+      "id": "Category",
+      "properties": {
+        "id": {
+          "type": "integer",
+          "format": "int64"
+        },
+        "name": {
+          "type": "string"
+        },
+        "pet": {
+          "$ref": "Pet"
+        }
+      }
+    }
+  }
+}

docs/server-server/specification.rst

@@ -1,8 +1,8 @@
-============================
-Synapse Server-to-Server API
-============================
+===========================
+Matrix Server-to-Server API
+===========================
 
-A description of the protocol used to communicate between Synapse home servers;
+A description of the protocol used to communicate between Matrix home servers;
 also known as Federation.
 
 
@@ -10,7 +10,7 @@ Overview
 ========
 
 The server-server API is a mechanism by which two home servers can exchange
-Synapse event messages, both as a real-time push of current events, and as a
+Matrix event messages, both as a real-time push of current events, and as a
 historic fetching mechanism to synchronise past history for clients to view. It
 uses HTTP connections between each pair of servers involved as the underlying
 transport. Messages are exchanged between servers in real-time by active pushing
@@ -19,7 +19,7 @@ historic data for the purpose of back-filling scrollback buffers and the like
 can also be performed.
 
 
- { Synapse entities }                            { Synapse entities }
+  { Matrix clients }                              { Matrix clients }
      ^          |                                    ^          |
      |  events  |                                    |  events  |
      |          V                                    |          V
@@ -29,27 +29,53 @@ can also be performed.
  |                  |<--------( HTTP )-----------|                  |
  +------------------+                            +------------------+
 
+There are three main kinds of communication that occur between home servers:
 
-Transactions and PDUs
-=====================
+ * Queries
+   These are single request/response interactions between a given pair of
+   servers, initiated by one side sending an HTTP request to obtain some
+   information, and responded by the other. They are not persisted and contain
+   no long-term significant history. They simply request a snapshot state at the
+   instant the query is made.
 
-The communication between home servers is performed by a bidirectional exchange
-of messages. These messages are called Transactions, and are encoded as JSON
-objects with a dict as the top-level element, passed over HTTP. A Transaction is
-meaningful only to the pair of home servers that exchanged it; they are not
-globally-meaningful.
+ * EDUs - Ephemeral Data Units
+   These are notifications of events that are pushed from one home server to
+   another. They are not persisted and contain no long-term significant history,
+   nor does the receiving home server have to reply to them.
 
-Each transaction has an opaque ID and timestamp (UNIX epoch time in miliseconds)
-generated by its origin server, an origin and destination server name, a list of
-"previous IDs", and a list of PDUs - the actual message payload that the
-Transaction carries.
+ * PDUs - Persisted Data Units
+   These are notifications of events that are broadcast from one home server to
+   any others that are interested in the same "context" (namely, a Room ID).
+   They are persisted to long-term storage and form the record of history for
+   that context.
+
+Where Queries are presented directly across the HTTP connection as GET requests
+to specific URLs, EDUs and PDUs are further wrapped in an envelope called a
+Transaction, which is transferred from the origin to the destination home server
+using a PUT request.
+
+
+Transactions and EDUs/PDUs
+==========================
+
+The transfer of EDUs and PDUs between home servers is performed by an exchange
+of Transaction messages, which are encoded as JSON objects with a dict as the
+top-level element, passed over an HTTP PUT request. A Transaction is meaningful
+only to the pair of home servers that exchanged it; they are not globally-
+meaningful.
+
+Each transaction has an opaque ID and timestamp (UNIX epoch time in
+milliseconds) generated by its origin server, an origin and destination server
+name, a list of "previous IDs", and a list of PDUs - the actual message payload
+that the Transaction carries.
 
  {"transaction_id":"916d630ea616342b42e98a3be0b74113",
   "ts":1404835423000,
   "origin":"red",
   "destination":"blue",
   "prev_ids":["e1da392e61898be4d2009b9fecce5325"],
-  "pdus":[...]}
+  "pdus":[...],
+  "edus":[...]}
 
 The "previous IDs" field will contain a list of previous transaction IDs that
 the origin server has sent to this destination. Its purpose is to act as a
@@ -58,7 +84,9 @@ successfully received that Transaction, or ask for a retransmission if not.
 
 The "pdus" field of a transaction is a list, containing zero or more PDUs.[*]
 Each PDU is itself a dict containing a number of keys, the exact details of
-which will vary depending on the type of PDU.
+which will vary depending on the type of PDU. Similarly, the "edus" field is
+another list containing the EDUs. This key may be entirely absent if there are
+no EDUs to transfer.
 
 (* Normally the PDU list will be non-empty, but the server should cope with
 receiving an "empty" transaction, as this is useful for informing peers of other
@@ -86,7 +114,7 @@ field of a PDU refers to PDUs that any origin server has sent, rather than
 previous IDs that this origin has sent. This list may refer to other PDUs sent
 by the same origin as the current one, or other origins.
 
-Because of the distributed nature of participants in a Synapse conversation, it
+Because of the distributed nature of participants in a Matrix conversation, it
 is impossible to establish a globally-consistent total ordering on the events.
 However, by annotating each outbound PDU at its origin with IDs of other PDUs it
 has received, a partial ordering can be constructed allowing causallity
@@ -112,6 +140,15 @@ so on. This part needs refining. And writing in its own document as the details
 relate to the server/system as a whole, not specifically to server-server
 federation.]]
 
+EDUs, by comparison to PDUs, do not have an ID, a context, or a list of
+"previous" IDs. The only mandatory fields for these are the type, origin and
+destination home server names, and the actual nested content.
+
+ {"edu_type":"m.presence",
+  "origin":"blue",
+  "destination":"orange",
+  "content":...}
+
 
 Protocol URLs
 =============
@@ -179,3 +216,16 @@ To stream events all the events:
   Retrieves all of the transactions later than any version given by the "v"
   arguments. [[TODO(paul): I'm not sure what the "origin" argument does because
   I think at some point in the code it's got swapped around.]]
+
+
+To make a query:
+
+  GET .../query/:query_type
+    Query args: as specified by the individual query types
+
+  Response: JSON encoding of a response object
+
+  Performs a single query request on the receiving home server. The Query Type
+  part of the path specifies the kind of query being made, and its query
+  arguments have a meaning specific to that kind of query. The response is a
+  JSON-encoded object whose meaning also depends on the kind of query.

graph/graph.py

@@ -113,7 +113,7 @@ def make_graph(pdus, room, filename_prefix):
                 graph.add_edge(state_edge)
 
     graph.write('%s.dot' % filename_prefix, format='raw', prog='dot')
-    graph.write_png("%s.png" % filename_prefix, prog='dot')
+#    graph.write_png("%s.png" % filename_prefix, prog='dot')
     graph.write_svg("%s.svg" % filename_prefix, prog='dot')
 
 

jsfiddles/create_room_send_msg/demo.css

@@ -0,0 +1,17 @@
+.loggedin {
+    visibility: hidden;
+}
+
+p {
+    font-family: monospace;
+}
+
+table
+{
+    border-spacing:5px;
+}
+
+th,td
+{
+    padding:5px;
+}

jsfiddles/create_room_send_msg/demo.html

@@ -0,0 +1,30 @@
+<div>
+    <p>This room creation / message sending demo requires a home server to be running on http://localhost:8080</p>
+</div>
+<form class="loginForm">
+    <input type="text" id="userLogin" placeholder="Username"></input>
+    <input type="password" id="passwordLogin" placeholder="Password"></input>
+    <input type="button" class="login" value="Login"></input>
+</form>
+<div class="loggedin">
+    <form class="createRoomForm">
+        <input type="text" id="roomAlias" placeholder="Room alias (optional)"></input>
+        <input type="button" class="createRoom" value="Create Room"></input>
+    </form>
+    <form class="sendMessageForm">
+        <input type="text" id="roomId" placeholder="Room ID"></input>
+        <input type="text" id="messageBody" placeholder="Message body"></input>
+        <input type="button" class="sendMessage" value="Send Message"></input>
+    </form>
+    <table id="rooms">
+        <tbody>
+            <tr>
+                <th>Room ID</th>
+                <th>My state</th>
+                <th>Room Alias</th>
+                <th>Latest message</th>
+            </tr>
+        </tbody>
+    </table>
+</div>
+    

jsfiddles/create_room_send_msg/demo.js

@@ -0,0 +1,109 @@
+var accountInfo = {};
+
+var showLoggedIn = function(data) {
+    accountInfo = data;
+    getCurrentRoomList();
+    $(".loggedin").css({visibility: "visible"});
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var getCurrentRoomList = function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.length; ++i) {
+            data[i].latest_message = data[i].messages.chunk[0].content.body;
+            addRoom(data[i]);   
+        }
+    }).fail(function(err) {
+        alert(JSON.stringify($.parseJSON(err.responseText)));
+    });
+};
+
+$('.createRoom').live('click', function() {
+    var roomAlias = $("#roomAlias").val();
+    var data = {};
+    if (roomAlias.length > 0) {
+        data.room_alias_name = roomAlias;   
+    }
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/rooms?access_token="+accountInfo.access_token,
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            data.membership = "join"; // you are automatically joined into every room you make.
+            data.latest_message = "";
+            addRoom(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var addRoom = function(data) {
+    row = "<tr>" +
+        "<td>"+data.room_id+"</td>" +
+        "<td>"+data.membership+"</td>" +
+        "<td>"+data.room_alias+"</td>" +
+        "<td>"+data.latest_message+"</td>" +
+        "</tr>";
+    $("#rooms").append(row);
+};
+
+$('.sendMessage').live('click', function() {
+    var roomId = $("#roomId").val();
+    var body = $("#messageBody").val();
+    var msgId = $.now();
+    
+    if (roomId.length === 0 || body.length === 0) {
+        return;
+    }
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/messages/$user/$msgid?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomid", encodeURIComponent(roomId));
+    url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
+    url = url.replace("$msgid", msgId);
+    
+    var data = {
+        msgtype: "m.text",
+        body: body
+    };
+    
+    $.ajax({
+        url: url,
+        type: "PUT",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            $("#messageBody").val("");
+            // wipe the table and reload it. Using the event stream would be the best
+            // solution but that is out of scope of this fiddle.
+            $("#rooms").find("tr:gt(0)").remove();
+            getCurrentRoomList();
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});

jsfiddles/event_stream/demo.css

@@ -0,0 +1,17 @@
+.loggedin {
+    visibility: hidden;
+}
+
+p {
+    font-family: monospace;
+}
+
+table
+{
+    border-spacing:5px;
+}
+
+th,td
+{
+    padding:5px;
+}

jsfiddles/event_stream/demo.html

@@ -0,0 +1,23 @@
+<div>
+    <p>This event stream demo requires a home server to be running on http://localhost:8080</p>
+</div>
+<form class="loginForm">
+    <input type="text" id="userLogin" placeholder="Username"></input>
+    <input type="password" id="passwordLogin" placeholder="Password"></input>
+    <input type="button" class="login" value="Login"></input>
+</form>
+<div class="loggedin">
+    <form class="sendMessageForm">
+        <input type="button" class="sendMessage" value="Send random message"></input>
+    </form>
+    <p id="streamErrorText"></p>
+    <table id="rooms">
+        <tbody>
+            <tr>
+                <th>Room ID</th>
+                <th>Latest message</th>
+            </tr>
+        </tbody>
+    </table>
+</div>
+    

jsfiddles/event_stream/demo.js

@@ -0,0 +1,142 @@
+var accountInfo = {};
+
+var eventStreamInfo = {
+    from: "END"
+};
+
+var roomInfo = [];
+
+var longpollEventStream = function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/events?access_token=$token&from=$from";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$from", eventStreamInfo.from);
+    
+    $.getJSON(url, function(data) {
+        eventStreamInfo.from = data.end;
+        
+        var hasNewLatestMessage = false;
+        for (var i=0; i<data.chunk.length; ++i) {
+            if (data.chunk[i].type === "m.room.message") {
+                for (var j=0; j<roomInfo.length; ++j) {
+                    if (roomInfo[j].room_id === data.chunk[i].room_id) {
+                        roomInfo[j].latest_message = data.chunk[i].content.body;
+                        hasNewLatestMessage = true;
+                    }
+                }
+            }
+        }
+        
+        if (hasNewLatestMessage) {
+           setRooms(roomInfo);
+        }
+        $("#streamErrorText").text("");
+        longpollEventStream();
+    }).fail(function(err) {
+        $("#streamErrorText").text("Event stream error: "+JSON.stringify($.parseJSON(err.responseText)));
+        setTimeout(longpollEventStream, 5000);
+    });
+};
+
+var showLoggedIn = function(data) {
+    accountInfo = data;
+    longpollEventStream();
+    getCurrentRoomList();
+    $(".loggedin").css({visibility: "visible"});
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            $("#rooms").find("tr:gt(0)").remove();
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var getCurrentRoomList = function() {
+    $("#roomId").val("");
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.length; ++i) {
+            if ("messages" in data[i]) {
+                data[i].latest_message = data[i].messages.chunk[0].content.body;   
+            }
+        }
+        roomInfo = data;
+        setRooms(roomInfo);  
+    }).fail(function(err) {
+        alert(JSON.stringify($.parseJSON(err.responseText)));
+    });
+};
+
+$('.sendMessage').live('click', function() {
+    if (roomInfo.length === 0) {
+        alert("There is no room to send a message to!");
+        return;
+    }
+    
+    var index = Math.floor(Math.random() * roomInfo.length);
+    
+    sendMessage(roomInfo[index].room_id);
+});
+
+var sendMessage = function(roomId) {
+    var body = "jsfiddle message @" + $.now();
+    var msgId = $.now();
+    
+    if (roomId.length === 0) {
+        return;
+    }
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/messages/$user/$msgid?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomid", encodeURIComponent(roomId));
+    url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
+    url = url.replace("$msgid", msgId);
+    
+    var data = {
+        msgtype: "m.text",
+        body: body
+    };
+    
+    $.ajax({
+        url: url,
+        type: "PUT",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            $("#messageBody").val("");
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+};
+
+var setRooms = function(roomList) {
+    // wipe existing entries
+    $("#rooms").find("tr:gt(0)").remove();
+    
+    var rows = "";
+    for (var i=0; i<roomList.length; ++i) {
+        row = "<tr>" +
+            "<td>"+roomList[i].room_id+"</td>" +
+            "<td>"+roomList[i].latest_message+"</td>" +
+            "</tr>";  
+        rows += row;
+    }
+    
+    $("#rooms").append(rows);
+};
+

jsfiddles/example_app/demo.css

@@ -0,0 +1,43 @@
+.roomListDashboard, .roomContents, .sendMessageForm {
+    visibility: hidden;
+}
+
+.roomList {
+    background-color: #909090;
+}
+
+.messageWrapper {
+    background-color: #EEEEEE;
+    height: 400px;
+    overflow: scroll;
+}
+
+.membersWrapper {
+    background-color: #EEEEEE;
+    height: 200px;
+    width: 50%;
+    overflow: scroll;
+}
+
+.textEntry {
+    width: 100%
+}
+
+p {
+    font-family: monospace;
+}
+
+table
+{
+    border-spacing:5px;
+}
+
+th,td
+{
+    padding:5px;
+}
+
+.roomList tr:not(:first-child):hover {
+    background-color: orange;
+    cursor: pointer;
+}

jsfiddles/example_app/demo.html

@@ -0,0 +1,56 @@
+<div class="signUp">
+    <p>Matrix example application: Requires a local home server running at http://localhost:8080</p>
+    <form class="registrationForm">
+        <p>No account? Register:</p>
+        <input type="text" id="userReg" placeholder="Username"></input>
+        <input type="password" id="passwordReg" placeholder="Password"></input>
+        <input type="button" class="register" value="Register"></input>
+    </form>
+    <form class="loginForm">
+        <p>Got an account? Login:</p>
+        <input type="text" id="userLogin" placeholder="Username"></input>
+        <input type="password" id="passwordLogin" placeholder="Password"></input>
+        <input type="button" class="login" value="Login"></input>
+    </form>
+</div>
+
+<div class="roomListDashboard">
+    <form class="createRoomForm">
+        <input type="text" id="roomAlias" placeholder="Room alias"></input>
+        <input type="button" class="createRoom" value="Create Room"></input>
+    </form>
+    <table id="rooms" class="roomList">
+        <tbody>
+            <tr>
+                <th>Room</th>
+                <th>My state</th>
+                <th>Latest message</th>
+            </tr>
+        </tbody>
+    </table>
+</div>
+
+<div class="roomContents">
+    <p id="roomName">Select a room</p>
+    <div class="messageWrapper">
+        <table id="messages">
+            <tbody>
+            </tbody>
+        </table>
+    </div>
+    <form class="sendMessageForm">
+        <input type="text" class="textEntry" id="body" placeholder="Enter text here..." onkeydown="javascript:if (event.keyCode == 13) document.getElementById('sendMsg').focus()"></input>
+        <input type="button" class="sendMessage" id="sendMsg" value="Send"></input>
+    </form>
+</div>
+
+<div>
+    <p>Member list:</p>
+    <div class="membersWrapper">
+        <table id="members">
+            <tbody>
+            </tbody>
+        </table>
+    </div>
+</div>
+    

jsfiddles/example_app/demo.js

@@ -0,0 +1,303 @@
+var accountInfo = {};
+
+var eventStreamInfo = {
+    from: "END"
+};
+
+var roomInfo = [];
+var memberInfo = [];
+var viewingRoomId;
+
+// ************** Event Streaming **************
+var longpollEventStream = function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/events?access_token=$token&from=$from";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$from", eventStreamInfo.from);
+
+    $.getJSON(url, function(data) {
+        eventStreamInfo.from = data.end;
+        
+        var hasNewLatestMessage = false;
+        var updatedMemberList = false;
+        var i=0;
+        var j=0;
+        for (i=0; i<data.chunk.length; ++i) {
+            if (data.chunk[i].type === "m.room.message") {
+                console.log("Got new message: " + JSON.stringify(data.chunk[i]));
+                if (viewingRoomId === data.chunk[i].room_id) {
+                    addMessage(data.chunk[i]);
+                }
+                
+                for (j=0; j<roomInfo.length; ++j) {
+                    if (roomInfo[j].room_id === data.chunk[i].room_id) {
+                        roomInfo[j].latest_message = data.chunk[i].content.body;
+                        hasNewLatestMessage = true;
+                    }
+                }
+            }
+            else if (data.chunk[i].type === "m.room.member") {
+                if (viewingRoomId === data.chunk[i].room_id) {
+                    console.log("Got new member: " + JSON.stringify(data.chunk[i]));
+                    for (j=0; j<memberInfo.length; ++j) {
+                        if (memberInfo[j].target_user_id === data.chunk[i].target_user_id) {
+                            memberInfo[j] = data.chunk[i];
+                            updatedMemberList = true;
+                            break;
+                        }
+                    }
+                    if (!updatedMemberList) {
+                        memberInfo.push(data.chunk[i]);  
+                        updatedMemberList = true;
+                    }
+                }
+                if (data.chunk[i].target_user_id === accountInfo.user_id) {
+                    getCurrentRoomList(); // update our join/invite list
+                }
+            }
+            else {
+                console.log("Discarding: " + JSON.stringify(data.chunk[i]));
+            }
+        }
+        
+        if (hasNewLatestMessage) {
+           setRooms(roomInfo);
+        }
+        if (updatedMemberList) {
+            $("#members").empty();
+            for (i=0; i<memberInfo.length; ++i) { 
+                addMember(memberInfo[i]);
+            }
+        }
+        longpollEventStream();
+    }).fail(function(err) {
+        setTimeout(longpollEventStream, 5000);
+    });
+};
+
+// ************** Registration and Login **************
+var onLoggedIn = function(data) {
+    accountInfo = data;
+    longpollEventStream();
+    getCurrentRoomList();
+    $(".roomListDashboard").css({visibility: "visible"});
+    $(".roomContents").css({visibility: "visible"});
+    $(".signUp").css({display: "none"});
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            onLoggedIn(data);
+        },
+        error: function(err) {
+            alert("Unable to login: is the home server running?");  
+        }
+    }); 
+});
+
+$('.register').live('click', function() {
+    var user = $("#userReg").val();
+    var password = $("#passwordReg").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/register",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user_id: user, password: password }),
+        dataType: "json",
+        success: function(data) {
+            onLoggedIn(data);
+        },
+        error: function(err) {
+            var msg = "Is the home server running?";
+            var errJson = $.parseJSON(err.responseText);
+            if (errJson !== null) {
+                msg = errJson.error;   
+            }
+            alert("Unable to register: "+msg);  
+        }
+    });
+});
+
+// ************** Creating a room ******************
+$('.createRoom').live('click', function() {
+    var roomAlias = $("#roomAlias").val();
+    var data = {};
+    if (roomAlias.length > 0) {
+        data.room_alias_name = roomAlias;   
+    }
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/rooms?access_token="+accountInfo.access_token,
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(response) {
+            $("#roomAlias").val("");
+            response.membership = "join"; // you are automatically joined into every room you make.
+            response.latest_message = "";
+            
+            roomInfo.push(response);
+            setRooms(roomInfo);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+// ************** Getting current state **************
+var getCurrentRoomList = function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.length; ++i) {
+            if ("messages" in data[i]) {
+                data[i].latest_message = data[i].messages.chunk[0].content.body;   
+            }
+        }
+        roomInfo = data;
+        setRooms(roomInfo);  
+    }).fail(function(err) {
+        alert(JSON.stringify($.parseJSON(err.responseText)));
+    });
+};
+
+var loadRoomContent = function(roomId) {
+    console.log("loadRoomContent " + roomId);
+    viewingRoomId = roomId;
+    $("#roomName").text("Room: "+roomId);
+    $(".sendMessageForm").css({visibility: "visible"});
+    getMessages(roomId);
+    getMemberList(roomId);
+};
+
+var getMessages = function(roomId) {
+    $("#messages").empty();
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/" + roomId + "/messages/list?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=10";
+    $.getJSON(url, function(data) {
+        for (var i=data.chunk.length-1; i>=0; --i) {
+            addMessage(data.chunk[i]);   
+        }
+    });
+};
+
+var getMemberList = function(roomId) {
+    $("#members").empty();
+    memberInfo = [];
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/" + roomId + "/members/list?access_token=" + accountInfo.access_token;
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.chunk.length; ++i) {
+            memberInfo.push(data.chunk[i]);
+            addMember(data.chunk[i]);   
+        }
+    });
+};
+
+// ************** Sending messages **************
+$('.sendMessage').live('click', function() {
+    if (viewingRoomId === undefined) {
+        alert("There is no room to send a message to!");
+        return;
+    }
+    var body = $("#body").val();
+    sendMessage(viewingRoomId, body);
+});
+
+var sendMessage = function(roomId, body) {
+    var msgId = $.now();
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/messages/$user/$msgid?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomid", encodeURIComponent(roomId));
+    url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
+    url = url.replace("$msgid", msgId);
+    
+    var data = {
+        msgtype: "m.text",
+        body: body
+    };
+    
+    $.ajax({
+        url: url,
+        type: "PUT",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            $("#body").val("");
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    });
+};
+
+// ************** Navigation and DOM manipulation **************
+var setRooms = function(roomList) {
+    // wipe existing entries
+    $("#rooms").find("tr:gt(0)").remove();
+    
+    var rows = "";
+    for (var i=0; i<roomList.length; ++i) {
+        row = "<tr>" +
+              "<td>"+roomList[i].room_id+"</td>" +
+              "<td>"+roomList[i].membership+"</td>" +
+              "<td>"+roomList[i].latest_message+"</td>" +
+              "</tr>";  
+        rows += row;
+    }
+    
+    $("#rooms").append(rows);
+    
+    $('#rooms').find("tr").click(function(){
+        var roomId = $(this).find('td:eq(0)').text();
+        var membership = $(this).find('td:eq(1)').text();
+        if (membership !== "join") {
+            console.log("Joining room " + roomId); 
+            var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/members/$user/state?access_token=$token";
+            url = url.replace("$token", accountInfo.access_token);
+            url = url.replace("$roomid", encodeURIComponent(roomId));
+            url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
+            $.ajax({
+                url: url,
+                type: "PUT",
+                contentType: "application/json; charset=utf-8",
+                data: JSON.stringify({membership: "join"}),
+                dataType: "json",
+                success: function(data) {
+                    loadRoomContent(roomId);
+                    getCurrentRoomList();
+                },
+                error: function(err) {
+                    alert(JSON.stringify($.parseJSON(err.responseText)));  
+                }
+            });
+        }
+        else {
+            loadRoomContent(roomId);
+        }
+    });
+};
+
+var addMessage = function(data) {
+    var row = "<tr>" +
+              "<td>"+data.user_id+"</td>" +
+              "<td>"+data.content.body+"</td>" +
+              "</tr>"; 
+    $("#messages").append(row);
+};
+
+var addMember = function(data) {
+    var row = "<tr>" +
+              "<td>"+data.target_user_id+"</td>" +
+              "<td>"+data.content.membership+"</td>" +
+              "</tr>"; 
+    $("#members").append(row);
+};
+

jsfiddles/register_login/demo.css

@@ -0,0 +1,7 @@
+.loggedin {
+    visibility: hidden;
+}
+
+p {
+    font-family: monospace;
+}

jsfiddles/register_login/demo.html

@@ -0,0 +1,20 @@
+<div>
+    <p>This registration/login demo requires a home server to be running on http://localhost:8080</p>
+</div>
+<form class="registrationForm">
+    <input type="text" id="user" placeholder="Username"></input>
+    <input type="password" id="password" placeholder="Password"></input>
+    <input type="button" class="register" value="Register"></input>
+</form>
+<form class="loginForm">
+    <input type="text" id="userLogin" placeholder="Username"></input>
+    <input type="password" id="passwordLogin" placeholder="Password"></input>
+    <input type="button" class="login" value="Login"></input>
+</form>
+<div class="loggedin">
+    <p id="welcomeText"></p>
+    <input type="button" class="testToken" value="Test token"></input>
+    <input type="button" class="logout" value="Logout"></input>
+    <p id="imSyncText"></p>
+</div>
+    

jsfiddles/register_login/demo.js

@@ -0,0 +1,69 @@
+var accountInfo = {};
+
+var showLoggedIn = function(data) {
+    accountInfo = data;
+    $(".loggedin").css({visibility: "visible"});
+    $("#welcomeText").text("Welcome " + accountInfo.user_id+". Your access token is:  " +
+                           accountInfo.access_token);    
+};
+
+$('.register').live('click', function() {
+    var user = $("#user").val();
+    var password = $("#password").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/register",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user_id: user, password: password }),
+        dataType: "json",
+        success: function(data) {
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    });
+});
+
+var login = function(user, password) {
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    });  
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.getJSON("http://localhost:8080/matrix/client/api/v1/login", function(data) {
+        if (data.type !== "m.login.password") {
+            alert("I don't know how to login with this type: " + data.type);
+            return;
+        }
+        login(user, password);
+    });
+});
+
+$('.logout').live('click', function() {
+    accountInfo = {};
+    $("#imSyncText").text("");
+    $(".loggedin").css({visibility: "hidden"});
+});
+
+$('.testToken').live('click', function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+         $("#imSyncText").text(JSON.stringify(data, undefined, 2));
+    }).fail(function(err) {
+        $("#imSyncText").text(JSON.stringify($.parseJSON(err.responseText)));
+    });
+});

jsfiddles/room_memberships/demo.css

@@ -0,0 +1,17 @@
+.loggedin {
+    visibility: hidden;
+}
+
+p {
+    font-family: monospace;
+}
+
+table
+{
+    border-spacing:5px;
+}
+
+th,td
+{
+    padding:5px;
+}

jsfiddles/room_memberships/demo.html

@@ -0,0 +1,37 @@
+<div>
+    <p>This room membership demo requires a home server to be running on http://localhost:8080</p>
+</div>
+<form class="loginForm">
+    <input type="text" id="userLogin" placeholder="Username"></input>
+    <input type="password" id="passwordLogin" placeholder="Password"></input>
+    <input type="button" class="login" value="Login"></input>
+</form>
+<div class="loggedin">
+    <form class="createRoomForm">
+        <input type="button" class="createRoom" value="Create Room"></input>
+    </form>
+    <form class="changeMembershipForm">
+        <input type="text" id="roomId" placeholder="Room ID"></input>
+        <input type="text" id="targetUser" placeholder="Target User ID"></input>
+        <select id="membership">
+            <option value="invite">Invite</option>
+            <option value="join">Join</option>
+            <option value="leave">Leave</option>
+        </select>
+        <input type="button" class="changeMembership" value="Change Membership"></input>
+    </form>
+    <form class="joinAliasForm">
+        <input type="text" id="roomAlias" placeholder="Room Alias (#name:domain)"></input>
+        <input type="button" class="joinAlias" value="Join via Alias"></input>
+    </form>
+    <table id="rooms">
+        <tbody>
+            <tr>
+                <th>Room ID</th>
+                <th>My state</th>
+                <th>Room Alias</th>
+            </tr>
+        </tbody>
+    </table>
+</div>
+    

jsfiddles/room_memberships/demo.js

@@ -0,0 +1,139 @@
+var accountInfo = {};
+
+var showLoggedIn = function(data) {
+    accountInfo = data;
+    getCurrentRoomList();
+    $(".loggedin").css({visibility: "visible"});
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            $("#rooms").find("tr:gt(0)").remove();
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var getCurrentRoomList = function() {
+    $("#roomId").val("");
+    // wipe the table and reload it. Using the event stream would be the best
+    // solution but that is out of scope of this fiddle.
+    $("#rooms").find("tr:gt(0)").remove();
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.length; ++i) {
+            addRoom(data[i]);   
+        }
+    }).fail(function(err) {
+        alert(JSON.stringify($.parseJSON(err.responseText)));
+    });
+};
+
+$('.createRoom').live('click', function() {
+    var data = {};
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/rooms?access_token="+accountInfo.access_token,
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            data.membership = "join"; // you are automatically joined into every room you make.
+            data.latest_message = "";
+            addRoom(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var addRoom = function(data) {
+    row = "<tr>" +
+        "<td>"+data.room_id+"</td>" +
+        "<td>"+data.membership+"</td>" +
+        "<td>"+data.room_alias+"</td>" +
+        "</tr>";
+    $("#rooms").append(row);
+};
+
+$('.changeMembership').live('click', function() {
+    var roomId = $("#roomId").val();
+    var member = $("#targetUser").val();
+    var membership = $("#membership").val();
+    
+    if (roomId.length === 0) {
+        return;
+    }
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/members/$user/state?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomid", encodeURIComponent(roomId));
+    url = url.replace("$user", encodeURIComponent(member));
+    
+    if (membership === "leave") {
+        $.ajax({
+            url: url,
+            type: "DELETE",
+            contentType: "application/json; charset=utf-8",
+            dataType: "json",
+            success: function(data) {
+                getCurrentRoomList();
+            },
+            error: function(err) {
+                alert(JSON.stringify($.parseJSON(err.responseText)));  
+            }
+        });
+    }
+    else {
+        var data = {
+            membership: membership
+        };
+        
+        $.ajax({
+            url: url,
+            type: "PUT",
+            contentType: "application/json; charset=utf-8",
+            data: JSON.stringify(data),
+            dataType: "json",
+            success: function(data) {
+                getCurrentRoomList();
+            },
+            error: function(err) {
+                alert(JSON.stringify($.parseJSON(err.responseText)));  
+            }
+        }); 
+    }
+});
+
+$('.joinAlias').live('click', function() {
+    var roomAlias = $("#roomAlias").val();
+    var url = "http://localhost:8080/matrix/client/api/v1/join/$roomalias?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomalias", encodeURIComponent(roomAlias));
+    $.ajax({
+        url: url,
+        type: "PUT",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({}),
+        dataType: "json",
+        success: function(data) {
+            getCurrentRoomList();
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});

nuke-room-from-db.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+## CAUTION:
+## This script will remove (hopefully) all trace of the given room ID from
+## your homeserver.db
+
+## Do not run it lightly.
+
+ROOMID="$1"
+
+sqlite3 homeserver.db <<EOF
+DELETE FROM context_depth WHERE context = '$ROOMID';
+DELETE FROM current_state WHERE context = '$ROOMID';
+DELETE FROM feedback WHERE room_id = '$ROOMID';
+DELETE FROM messages WHERE room_id = '$ROOMID';
+DELETE FROM pdu_backward_extremities WHERE context = '$ROOMID';
+DELETE FROM pdu_edges WHERE context = '$ROOMID';
+DELETE FROM pdu_forward_extremities WHERE context = '$ROOMID';
+DELETE FROM pdus WHERE context = '$ROOMID';
+DELETE FROM room_data WHERE room_id = '$ROOMID';
+DELETE FROM room_memberships WHERE room_id = '$ROOMID';
+DELETE FROM rooms WHERE room_id = '$ROOMID';
+DELETE FROM state_pdus WHERE context = '$ROOMID';
+EOF

setup.py

@@ -25,7 +25,7 @@ def read(fname):
 
 setup(
     name="SynapseHomeServer",
-    version="0.1",
+    version="0.0.1",
     packages=find_packages(exclude=["tests"]),
     description="Reference Synapse Home Server",
     install_requires=[

synapse/__init__.py

@@ -15,3 +15,5 @@
 
 """ This is a reference implementation of a synapse home server.
 """
+
+__version__ = "0.0.1"

synapse/api/auth.py

@@ -18,7 +18,7 @@
 from twisted.internet import defer
 
 from synapse.api.constants import Membership
-from synapse.api.errors import AuthError, StoreError
+from synapse.api.errors import AuthError, StoreError, Codes
 from synapse.api.events.room import (RoomTopicEvent, RoomMemberEvent,
                                      MessageEvent, FeedbackEvent)
 
@@ -163,4 +163,5 @@ class Auth(object):
             user_id = yield self.store.get_user_by_token(token=token)
             defer.returnValue(self.hs.parse_userid(user_id))
         except StoreError:
-            raise AuthError(403, "Unrecognised access token.")
+            raise AuthError(403, "Unrecognised access token.",
+                            errcode=Codes.UNKNOWN_TOKEN)

synapse/api/errors.py

@@ -27,6 +27,7 @@ class Codes(object):
     BAD_PAGINATION = "M_BAD_PAGINATION"
     UNKNOWN = "M_UNKNOWN"
     NOT_FOUND = "M_NOT_FOUND"
+    UNKNOWN_TOKEN = "M_UNKNOWN_TOKEN"
 
 
 class CodeMessageException(Exception):
@@ -74,7 +75,10 @@ class AuthError(SynapseError):
 
 class EventStreamError(SynapseError):
     """An error raised when there a problem with the event stream."""
-    pass
+    def __init__(self, *args, **kwargs):
+        if "errcode" not in kwargs:
+            kwargs["errcode"] = Codes.BAD_PAGINATION
+        super(EventStreamError, self).__init__(*args, **kwargs)
 
 
 class LoginError(SynapseError):

synapse/api/events/__init__.py

@@ -51,6 +51,7 @@ class SynapseEvent(JsonEncodedObject):
         "depth",
         "destinations",
         "origin",
+        "outlier",
     ]
 
     required_keys = [

synapse/api/events/factory.py

@@ -15,7 +15,7 @@
 
 from synapse.api.events.room import (
     RoomTopicEvent, MessageEvent, RoomMemberEvent, FeedbackEvent,
-    InviteJoinEvent, RoomConfigEvent
+    InviteJoinEvent, RoomConfigEvent, RoomNameEvent, GenericEvent,
 )
 
 from synapse.util.stringutils import random_string
@@ -25,6 +25,7 @@ class EventFactory(object):
 
     _event_classes = [
         RoomTopicEvent,
+        RoomNameEvent,
         MessageEvent,
         RoomMemberEvent,
         FeedbackEvent,
@@ -32,20 +33,24 @@ class EventFactory(object):
         RoomConfigEvent
     ]
 
-    def __init__(self):
+    def __init__(self, hs):
         self._event_list = {}  # dict of TYPE to event class
         for event_class in EventFactory._event_classes:
             self._event_list[event_class.TYPE] = event_class
 
+        self.clock = hs.get_clock()
+
     def create_event(self, etype=None, **kwargs):
         kwargs["type"] = etype
         if "event_id" not in kwargs:
             kwargs["event_id"] = random_string(10)
 
-        try:
+        if "ts" not in kwargs:
+            kwargs["ts"] = int(self.clock.time_msec())
+
+        if etype in self._event_list:
             handler = self._event_list[etype]
-        except KeyError:  # unknown event type
-            # TODO allow custom event types.
-            raise NotImplementedError("Unknown etype=%s" % etype)
+        else:
+            handler = GenericEvent
 
         return handler(**kwargs)

synapse/api/events/room.py

@@ -16,17 +16,45 @@
 from . import SynapseEvent
 
 
+class GenericEvent(SynapseEvent):
+    def get_content_template(self):
+        return {}
+
+
 class RoomTopicEvent(SynapseEvent):
     TYPE = "m.room.topic"
 
+    internal_keys = SynapseEvent.internal_keys + [
+        "topic",
+    ]
+
     def __init__(self, **kwargs):
         kwargs["state_key"] = ""
+        if "topic" in kwargs["content"]:
+            kwargs["topic"] = kwargs["content"]["topic"]
         super(RoomTopicEvent, self).__init__(**kwargs)
 
     def get_content_template(self):
         return {"topic": u"string"}
 
 
+class RoomNameEvent(SynapseEvent):
+    TYPE = "m.room.name"
+
+    internal_keys = SynapseEvent.internal_keys + [
+        "name",
+    ]
+
+    def __init__(self, **kwargs):
+        kwargs["state_key"] = ""
+        if "name" in kwargs["content"]:
+            kwargs["name"] = kwargs["content"]["name"]
+        super(RoomNameEvent, self).__init__(**kwargs)
+
+    def get_content_template(self):
+        return {"name": u"string"}
+
+
 class RoomMemberEvent(SynapseEvent):
     TYPE = "m.room.member"
 
@@ -38,6 +66,8 @@ class RoomMemberEvent(SynapseEvent):
     def __init__(self, **kwargs):
         if "target_user_id" in kwargs:
             kwargs["state_key"] = kwargs["target_user_id"]
+        if "membership" not in kwargs:
+            kwargs["membership"] = kwargs.get("content", {}).get("membership")
         super(RoomMemberEvent, self).__init__(**kwargs)
 
     def get_content_template(self):

synapse/api/notifier.py

@@ -15,6 +15,7 @@
 
 from synapse.api.constants import Membership
 from synapse.api.events.room import RoomMemberEvent
+from synapse.api.streams.event import EventsStreamData
 
 from twisted.internet import defer
 from twisted.internet import reactor
@@ -56,13 +57,17 @@ class Notifier(object):
         if (event.type == RoomMemberEvent.TYPE and
                 event.content["membership"] == Membership.INVITE):
             member_list.append(event.target_user_id)
+        # similarly, LEAVEs must be sent to the person leaving
+        if (event.type == RoomMemberEvent.TYPE and
+                event.content["membership"] == Membership.LEAVE):
+            member_list.append(event.target_user_id)
 
         for user_id in member_list:
             if user_id in self.stored_event_listeners:
                 self._notify_and_callback(
                     user_id=user_id,
                     event_data=event.get_dict(),
-                    stream_type=event.type,
+                    stream_type=EventsStreamData.EVENT_TYPE,
                     store_id=store_id)
 
     def on_new_user_event(self, user_id, event_data, stream_type, store_id):

synapse/api/streams/__init__.py

@@ -20,23 +20,24 @@ class PaginationConfig(object):
 
     """A configuration object which stores pagination parameters."""
 
-    def __init__(self, from_tok=None, to_tok=None, limit=0):
+    def __init__(self, from_tok=None, to_tok=None, direction='f', limit=0):
         self.from_tok = from_tok
         self.to_tok = to_tok
+        self.direction = direction
         self.limit = limit
 
     @classmethod
     def from_request(cls, request, raise_invalid_params=True):
         params = {
-            "from_tok": PaginationStream.TOK_START,
-            "to_tok": PaginationStream.TOK_END,
-            "limit": 0
+            "from_tok": "END",
+            "direction": 'f',
         }
 
         query_param_mappings = [  # 3-tuple of qp_key, attribute, rules
             ("from", "from_tok", lambda x: type(x) == str),
             ("to", "to_tok", lambda x: type(x) == str),
-            ("limit", "limit", lambda x: x.isdigit())
+            ("limit", "limit", lambda x: x.isdigit()),
+            ("dir", "direction", lambda x: x == 'f' or x == 'b'),
         ]
 
         for qp, attr, is_valid in query_param_mappings:
@@ -48,12 +49,17 @@ class PaginationConfig(object):
 
         return PaginationConfig(**params)
 
+    def __str__(self):
+        return (
+            "<PaginationConfig from_tok=%s, to_tok=%s, "
+            "direction=%s, limit=%s>"
+        ) % (self.from_tok, self.to_tok, self.direction, self.limit)
+
 
 class PaginationStream(object):
 
     """ An interface for streaming data as chunks. """
 
-    TOK_START = "START"
     TOK_END = "END"
 
     def get_chunk(self, config=None):
@@ -76,7 +82,7 @@ class StreamData(object):
         self.hs = hs
         self.store = hs.get_datastore()
 
-    def get_rows(self, user_id, from_pkey, to_pkey, limit):
+    def get_rows(self, user_id, from_pkey, to_pkey, limit, direction):
         """ Get event stream data between the specified pkeys.
 
         Args:

synapse/api/streams/event.py

@@ -18,6 +18,7 @@
 from twisted.internet import defer
 
 from synapse.api.errors import EventStreamError
+from synapse.api.events import SynapseEvent
 from synapse.api.events.room import (
     RoomMemberEvent, MessageEvent, FeedbackEvent, RoomTopicEvent
 )
@@ -28,17 +29,17 @@ import logging
 logger = logging.getLogger(__name__)
 
 
-class MessagesStreamData(StreamData):
-    EVENT_TYPE = MessageEvent.TYPE
+class EventsStreamData(StreamData):
+    EVENT_TYPE = "EventsStream"
 
     def __init__(self, hs, room_id=None, feedback=False):
-        super(MessagesStreamData, self).__init__(hs)
+        super(EventsStreamData, self).__init__(hs)
         self.room_id = room_id
         self.with_feedback = feedback
 
     @defer.inlineCallbacks
-    def get_rows(self, user_id, from_key, to_key, limit):
-        (data, latest_ver) = yield self.store.get_message_stream(
+    def get_rows(self, user_id, from_key, to_key, limit, direction):
+        data, latest_ver = yield self.store.get_room_events(
             user_id=user_id,
             from_key=from_key,
             to_key=to_key,
@@ -50,74 +51,7 @@ class MessagesStreamData(StreamData):
 
     @defer.inlineCallbacks
     def max_token(self):
-        val = yield self.store.get_max_message_id()
-        defer.returnValue(val)
-
-
-class RoomMemberStreamData(StreamData):
-    EVENT_TYPE = RoomMemberEvent.TYPE
-
-    @defer.inlineCallbacks
-    def get_rows(self, user_id, from_key, to_key, limit):
-        (data, latest_ver) = yield self.store.get_room_member_stream(
-            user_id=user_id,
-            from_key=from_key,
-            to_key=to_key
-        )
-
-        defer.returnValue((data, latest_ver))
-
-    @defer.inlineCallbacks
-    def max_token(self):
-        val = yield self.store.get_max_room_member_id()
-        defer.returnValue(val)
-
-
-class FeedbackStreamData(StreamData):
-    EVENT_TYPE = FeedbackEvent.TYPE
-
-    def __init__(self, hs, room_id=None):
-        super(FeedbackStreamData, self).__init__(hs)
-        self.room_id = room_id
-
-    @defer.inlineCallbacks
-    def get_rows(self, user_id, from_key, to_key, limit):
-        (data, latest_ver) = yield self.store.get_feedback_stream(
-            user_id=user_id,
-            from_key=from_key,
-            to_key=to_key,
-            limit=limit,
-            room_id=self.room_id
-        )
-        defer.returnValue((data, latest_ver))
-
-    @defer.inlineCallbacks
-    def max_token(self):
-        val = yield self.store.get_max_feedback_id()
-        defer.returnValue(val)
-
-
-class RoomDataStreamData(StreamData):
-    EVENT_TYPE = RoomTopicEvent.TYPE  # TODO need multiple event types
-
-    def __init__(self, hs, room_id=None):
-        super(RoomDataStreamData, self).__init__(hs)
-        self.room_id = room_id
-
-    @defer.inlineCallbacks
-    def get_rows(self, user_id, from_key, to_key, limit):
-        (data, latest_ver) = yield self.store.get_room_data_stream(
-            user_id=user_id,
-            from_key=from_key,
-            to_key=to_key,
-            limit=limit,
-            room_id=self.room_id
-        )
-        defer.returnValue((data, latest_ver))
-
-    @defer.inlineCallbacks
-    def max_token(self):
-        val = yield self.store.get_max_room_data_id()
+        val = yield self.store.get_room_events_max_id()
         defer.returnValue(val)
 
 
@@ -136,6 +70,15 @@ class EventStream(PaginationStream):
             pagination_config.from_tok)
         pagination_config.to_tok = yield self.fix_token(
             pagination_config.to_tok)
+
+        if (
+            not pagination_config.to_tok
+            and pagination_config.direction == 'f'
+        ):
+            pagination_config.to_tok = yield self.get_current_max_token()
+
+        logger.debug("pagination_config: %s", pagination_config)
+
         defer.returnValue(pagination_config)
 
     @defer.inlineCallbacks
@@ -147,39 +90,42 @@ class EventStream(PaginationStream):
         Returns:
             The fixed-up token, which may == token.
         """
-        # replace TOK_START and TOK_END with 0_0_0 or -1_-1_-1 depending.
-        replacements = [
-            (PaginationStream.TOK_START, "0"),
-            (PaginationStream.TOK_END, "-1")
-        ]
-        for magic_token, key in replacements:
-            if magic_token == token:
-                token = EventStream.SEPARATOR.join(
-                    [key] * len(self.stream_data)
-                )
+        if token == PaginationStream.TOK_END:
+            new_token = yield self.get_current_max_token()
 
-        # replace -1 values with an actual pkey
-        token_segments = self._split_token(token)
-        for i, tok in enumerate(token_segments):
-            if tok == -1:
-                # add 1 to the max token because results are EXCLUSIVE from the
-                # latest version.
-                token_segments[i] = 1 + (yield self.stream_data[i].max_token())
-        defer.returnValue(EventStream.SEPARATOR.join(
-            str(x) for x in token_segments
-        ))
+            logger.debug("fix_token: From %s to %s", token, new_token)
+
+            token = new_token
+
+        defer.returnValue(token)
 
     @defer.inlineCallbacks
-    def get_chunk(self, config=None):
+    def get_current_max_token(self):
+        new_token_parts = []
+        for s in self.stream_data:
+            mx = yield s.max_token()
+            new_token_parts.append(str(mx))
+
+        new_token = EventStream.SEPARATOR.join(new_token_parts)
+
+        logger.debug("get_current_max_token: %s", new_token)
+
+        defer.returnValue(new_token)
+
+    @defer.inlineCallbacks
+    def get_chunk(self, config):
         # no support for limit on >1 streams, makes no sense.
         if config.limit and len(self.stream_data) > 1:
             raise EventStreamError(
                 400, "Limit not supported on multiplexed streams."
             )
 
-        (chunk_data, next_tok) = yield self._get_chunk_data(config.from_tok,
-                                                            config.to_tok,
-                                                            config.limit)
+        chunk_data, next_tok = yield self._get_chunk_data(
+            config.from_tok,
+            config.to_tok,
+            config.limit,
+            config.direction,
+        )
 
         defer.returnValue({
             "chunk": chunk_data,
@@ -188,7 +134,7 @@ class EventStream(PaginationStream):
         })
 
     @defer.inlineCallbacks
-    def _get_chunk_data(self, from_tok, to_tok, limit):
+    def _get_chunk_data(self, from_tok, to_tok, limit, direction):
         """ Get event data between the two tokens.
 
         Tokens are SEPARATOR separated values representing pkey values of
@@ -206,11 +152,12 @@ class EventStream(PaginationStream):
             EventStreamError if something went wrong.
         """
         # sanity check
-        if (from_tok.count(EventStream.SEPARATOR) !=
-                to_tok.count(EventStream.SEPARATOR) or
-                (from_tok.count(EventStream.SEPARATOR) + 1) !=
-                len(self.stream_data)):
-            raise EventStreamError(400, "Token lengths don't match.")
+        if to_tok is not None:
+            if (from_tok.count(EventStream.SEPARATOR) !=
+                    to_tok.count(EventStream.SEPARATOR) or
+                    (from_tok.count(EventStream.SEPARATOR) + 1) !=
+                    len(self.stream_data)):
+                raise EventStreamError(400, "Token lengths don't match.")
 
         chunk = []
         next_ver = []
@@ -224,10 +171,13 @@ class EventStream(PaginationStream):
                 continue
 
             (event_chunk, max_pkey) = yield self.stream_data[i].get_rows(
-                self.user_id, from_pkey, to_pkey, limit
+                self.user_id, from_pkey, to_pkey, limit, direction,
             )
 
-            chunk += event_chunk
+            chunk.extend([
+                e.get_dict() if isinstance(e, SynapseEvent) else e
+                for e in event_chunk
+            ])
             next_ver.append(str(max_pkey))
 
         defer.returnValue((chunk, EventStream.SEPARATOR.join(next_ver)))
@@ -240,9 +190,8 @@ class EventStream(PaginationStream):
         Returns:
             A list of ints.
         """
-        segments = token.split(EventStream.SEPARATOR)
-        try:
-            int_segments = [int(x) for x in segments]
-        except ValueError:
-            raise EventStreamError(400, "Bad token: %s" % token)
-        return int_segments
+        if token:
+            segments = token.split(EventStream.SEPARATOR)
+        else:
+            segments = [None] * len(self.stream_data)
+        return segments

synapse/api/urls.py

@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 matrix.org
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Contains the URL paths to prefix various aspects of the server with. """
+
+CLIENT_PREFIX = "/matrix/client/api/v1"
+FEDERATION_PREFIX = "/matrix/federation/v1"
+WEB_CLIENT_PREFIX = "/matrix/client"
+CONTENT_REPO_PREFIX = "/matrix/content"

synapse/app/homeserver.py

@@ -21,8 +21,14 @@ from synapse.server import HomeServer
 from twisted.internet import reactor
 from twisted.enterprise import adbapi
 from twisted.python.log import PythonLoggingObserver
-from synapse.http.server import TwistedHttpServer
+from twisted.web.resource import Resource
+from twisted.web.static import File
+from twisted.web.server import Site
+from synapse.http.server import JsonResource, RootRedirect, ContentRepoResource
 from synapse.http.client import TwistedHttpClient
+from synapse.api.urls import (
+    CLIENT_PREFIX, FEDERATION_PREFIX, WEB_CLIENT_PREFIX, CONTENT_REPO_PREFIX
+)
 
 from daemonize import Daemonize
 
@@ -30,17 +36,28 @@ import argparse
 import logging
 import logging.config
 import sqlite3
+import os
 
 logger = logging.getLogger(__name__)
 
 
 class SynapseHomeServer(HomeServer):
-    def build_http_server(self):
-        return TwistedHttpServer()
 
     def build_http_client(self):
         return TwistedHttpClient()
 
+    def build_resource_for_client(self):
+        return JsonResource()
+
+    def build_resource_for_federation(self):
+        return JsonResource()
+
+    def build_resource_for_web_client(self):
+        return File("webclient")  # TODO configurable?
+
+    def build_resource_for_content_repo(self):
+        return ContentRepoResource(self, self.upload_dir, self.auth)
+
     def build_db_pool(self):
         """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we
         don't have to worry about overwriting existing content.
@@ -73,6 +90,99 @@ class SynapseHomeServer(HomeServer):
 
         return pool
 
+    def create_resource_tree(self, web_client, redirect_root_to_web_client):
+        """Create the resource tree for this Home Server.
+
+        This in unduly complicated because Twisted does not support putting
+        child resources more than 1 level deep at a time.
+
+        Args:
+            web_client (bool): True to enable the web client.
+            redirect_root_to_web_client (bool): True to redirect '/' to the
+            location of the web client. This does nothing if web_client is not
+            True.
+        """
+        # list containing (path_str, Resource) e.g:
+        # [ ("/aaa/bbb/cc", Resource1), ("/aaa/dummy", Resource2) ]
+        desired_tree = [
+            (CLIENT_PREFIX, self.get_resource_for_client()),
+            (FEDERATION_PREFIX, self.get_resource_for_federation()),
+            (CONTENT_REPO_PREFIX, self.get_resource_for_content_repo())
+        ]
+        if web_client:
+            logger.info("Adding the web client.")
+            desired_tree.append((WEB_CLIENT_PREFIX,
+                                self.get_resource_for_web_client()))
+
+        if web_client and redirect_root_to_web_client:
+            self.root_resource = RootRedirect(WEB_CLIENT_PREFIX)
+        else:
+            self.root_resource = Resource()
+
+        # ideally we'd just use getChild and putChild but getChild doesn't work
+        # unless you give it a Request object IN ADDITION to the name :/ So
+        # instead, we'll store a copy of this mapping so we can actually add
+        # extra resources to existing nodes. See self._resource_id for the key.
+        resource_mappings = {}
+        for (full_path, resource) in desired_tree:
+            logging.info("Attaching %s to path %s", resource, full_path)
+            last_resource = self.root_resource
+            for path_seg in full_path.split('/')[1:-1]:
+                if not path_seg in last_resource.listNames():
+                    # resource doesn't exist, so make a "dummy resource"
+                    child_resource = Resource()
+                    last_resource.putChild(path_seg, child_resource)
+                    res_id = self._resource_id(last_resource, path_seg)
+                    resource_mappings[res_id] = child_resource
+                    last_resource = child_resource
+                else:
+                    # we have an existing Resource, use that instead.
+                    res_id = self._resource_id(last_resource, path_seg)
+                    last_resource = resource_mappings[res_id]
+
+            # ===========================
+            # now attach the actual desired resource
+            last_path_seg = full_path.split('/')[-1]
+
+            # if there is already a resource here, thieve its children and
+            # replace it
+            res_id = self._resource_id(last_resource, last_path_seg)
+            if res_id in resource_mappings:
+                # there is a dummy resource at this path already, which needs
+                # to be replaced with the desired resource.
+                existing_dummy_resource = resource_mappings[res_id]
+                for child_name in existing_dummy_resource.listNames():
+                    child_res_id = self._resource_id(existing_dummy_resource,
+                                                     child_name)
+                    child_resource = resource_mappings[child_res_id]
+                    # steal the children
+                    resource.putChild(child_name, child_resource)
+
+            # finally, insert the desired resource in the right place
+            last_resource.putChild(last_path_seg, resource)
+            res_id = self._resource_id(last_resource, last_path_seg)
+            resource_mappings[res_id] = resource
+
+        return self.root_resource
+
+    def _resource_id(self, resource, path_seg):
+        """Construct an arbitrary resource ID so you can retrieve the mapping
+        later.
+
+        If you want to represent resource A putChild resource B with path C,
+        the mapping should looks like _resource_id(A,C) = B.
+
+        Args:
+            resource (Resource): The *parent* Resource
+            path_seg (str): The name of the child Resource to be attached.
+        Returns:
+            str: A unique string which can be a key to the child Resource.
+        """
+        return "%s-%s" % (resource, path_seg)
+
+    def start_listening(self, port):
+        reactor.listenTCP(port, Site(self.root_resource))
+
 
 def setup_logging(verbosity=0, filename=None, config_path=None):
     """ Sets up logging with verbosity levels.
@@ -125,15 +235,21 @@ def setup():
     parser.add_argument('--pid-file', dest="pid", help="When running as a "
                         "daemon, the file to store the pid in",
                         default="hs.pid")
-    parser.add_argument("-w", "--webclient", dest="webclient",
-                        action="store_true", help="Host the web client.")
+    parser.add_argument("-W", "--webclient", dest="webclient", default=True,
+                        action="store_false", help="Don't host a web client.")
     args = parser.parse_args()
 
     verbosity = int(args.verbose) if args.verbose else None
 
+    # Because if/when we daemonize we change to root dir.
+    db_name = os.path.abspath(args.db)
+    log_file = args.log_file
+    if log_file:
+        log_file = os.path.abspath(log_file)
+
     setup_logging(
         verbosity=verbosity,
-        filename=args.log_file,
+        filename=log_file,
         config_path=args.log_config,
     )
 
@@ -141,7 +257,8 @@ def setup():
 
     hs = SynapseHomeServer(
         args.host,
-        db_name=args.db
+        upload_dir=os.path.abspath("uploads"),
+        db_name=db_name,
     )
 
     # This object doesn't need to be saved because it's set as the handler for
@@ -150,7 +267,10 @@ def setup():
 
     hs.register_servlets()
 
-    hs.get_http_server().start_listening(args.port)
+    hs.create_resource_tree(
+        web_client=args.webclient,
+        redirect_root_to_web_client=True)
+    hs.start_listening(args.port)
 
     hs.build_db_pool()
 

synapse/federation/__init__.py

@@ -23,7 +23,7 @@ from .transport import TransportLayer
 def initialize_http_replication(homeserver):
     transport = TransportLayer(
         homeserver.hostname,
-        server=homeserver.get_http_server(),
+        server=homeserver.get_resource_for_federation(),
         client=homeserver.get_http_client()
     )
 

synapse/federation/handler.py

@@ -63,7 +63,7 @@ class FederationEventHandler(object):
             Deferred: Resolved when it has successfully been queued for
             processing.
         """
-        yield self._fill_out_prev_events(event)
+        yield self.fill_out_prev_events(event)
 
         pdu = self.pdu_codec.pdu_from_event(event)
 
@@ -74,10 +74,18 @@ class FederationEventHandler(object):
 
     @log_function
     @defer.inlineCallbacks
-    def backfill(self, room_id, limit):
-        # TODO: Work out which destinations to ask for backfill
-        # self.replication_layer.backfill(dest, room_id, limit)
-        pass
+    def backfill(self, dest, room_id, limit):
+        pdus = yield self.replication_layer.backfill(dest, room_id, limit)
+
+        if not pdus:
+            defer.returnValue([])
+
+        events = [
+            self.pdu_codec.event_from_pdu(pdu)
+            for pdu in pdus
+        ]
+
+        defer.returnValue(events)
 
     @log_function
     def get_state_for_room(self, destination, room_id):
@@ -87,7 +95,7 @@ class FederationEventHandler(object):
 
     @log_function
     @defer.inlineCallbacks
-    def on_receive_pdu(self, pdu):
+    def on_receive_pdu(self, pdu, backfilled):
         """ Called by the ReplicationLayer when we have a new pdu. We need to
         do auth checks and put it throught the StateHandler.
         """
@@ -95,7 +103,7 @@ class FederationEventHandler(object):
 
         try:
             with (yield self.lock_manager.lock(pdu.context)):
-                if event.is_state:
+                if event.is_state and not backfilled:
                     is_new_state = yield self.state_handler.handle_new_state(
                         pdu
                     )
@@ -104,7 +112,7 @@ class FederationEventHandler(object):
                 else:
                     is_new_state = False
 
-            yield self.event_handler.on_receive(event, is_new_state)
+            yield self.event_handler.on_receive(event, is_new_state, backfilled)
 
         except AuthError:
             # TODO: Implement something in federation that allows us to
@@ -129,7 +137,7 @@ class FederationEventHandler(object):
         yield self.event_handler.on_receive(new_state_event)
 
     @defer.inlineCallbacks
-    def _fill_out_prev_events(self, event):
+    def fill_out_prev_events(self, event):
         if hasattr(event, "prev_events"):
             return
 

synapse/federation/replication.py

@@ -158,6 +158,7 @@ class ReplicationLayer(object):
 
         # TODO, add errback, etc.
         self._transaction_queue.enqueue_edu(edu)
+        return defer.succeed(None)
 
     @log_function
     def make_query(self, destination, query_type, args):
@@ -208,7 +209,7 @@ class ReplicationLayer(object):
 
         pdus = [Pdu(outlier=False, **p) for p in transaction.pdus]
         for pdu in pdus:
-            yield self._handle_new_pdu(pdu)
+            yield self._handle_new_pdu(pdu, backfilled=True)
 
         defer.returnValue(pdus)
 
@@ -415,7 +416,7 @@ class ReplicationLayer(object):
 
     @defer.inlineCallbacks
     @log_function
-    def _handle_new_pdu(self, pdu):
+    def _handle_new_pdu(self, pdu, backfilled=False):
         # We reprocess pdus when we have seen them only as outliers
         existing = yield self._get_persisted_pdu(pdu.pdu_id, pdu.origin)
 
@@ -451,7 +452,10 @@ class ReplicationLayer(object):
         # Persist the Pdu, but don't mark it as processed yet.
         yield self.pdu_actions.persist_received(pdu)
 
-        ret = yield self.handler.on_receive_pdu(pdu)
+        if not backfilled:
+            ret = yield self.handler.on_receive_pdu(pdu, backfilled=backfilled)
+        else:
+            ret = None
 
         yield self.pdu_actions.mark_as_processed(pdu)
 

synapse/federation/transport.py

@@ -23,6 +23,7 @@ over a different (albeit still reliable) protocol.
 
 from twisted.internet import defer
 
+from synapse.api.urls import FEDERATION_PREFIX as PREFIX
 from synapse.util.logutils import log_function
 
 import logging
@@ -33,9 +34,6 @@ import re
 logger = logging.getLogger(__name__)
 
 
-PREFIX = "/matrix/federation/v1"
-
-
 class TransportLayer(object):
     """This is a basic implementation of the transport layer that translates
     transactions and other requests to/from HTTP.

synapse/handlers/_base.py

@@ -24,4 +24,5 @@ class BaseHandler(object):
         self.notifier = hs.get_notifier()
         self.room_lock = hs.get_room_lock_manager()
         self.state_handler = hs.get_state_handler()
+        self.distributor = hs.get_distributor()
         self.hs = hs

synapse/handlers/directory.py

@@ -20,17 +20,11 @@ from ._base import BaseHandler
 from synapse.api.errors import SynapseError
 
 import logging
-import json
-import urllib
 
 
 logger = logging.getLogger(__name__)
 
 
-# TODO(erikj): This needs to be factored out somewere
-PREFIX = "/matrix/client/api/v1"
-
-
 class DirectoryHandler(BaseHandler):
 
     def __init__(self, hs):

synapse/handlers/events.py

@@ -17,8 +17,7 @@ from twisted.internet import defer
 
 from ._base import BaseHandler
 from synapse.api.streams.event import (
-    EventStream, MessagesStreamData, RoomMemberStreamData, FeedbackStreamData,
-    RoomDataStreamData
+    EventStream, EventsStreamData
 )
 from synapse.handlers.presence import PresenceStreamData
 
@@ -26,10 +25,7 @@ from synapse.handlers.presence import PresenceStreamData
 class EventStreamHandler(BaseHandler):
 
     stream_data_classes = [
-        MessagesStreamData,
-        RoomMemberStreamData,
-        FeedbackStreamData,
-        RoomDataStreamData,
+        EventsStreamData,
         PresenceStreamData,
     ]
 

synapse/handlers/federation.py

@@ -32,10 +32,19 @@ logger = logging.getLogger(__name__)
 class FederationHandler(BaseHandler):
 
     """Handles events that originated from federation."""
+    def __init__(self, hs):
+        super(FederationHandler, self).__init__(hs)
+
+        self.distributor.observe(
+            "user_joined_room",
+            self._on_user_joined
+        )
+
+        self.waiting_for_join_list = {}
 
     @log_function
     @defer.inlineCallbacks
-    def on_receive(self, event, is_new_state):
+    def on_receive(self, event, is_new_state, backfilled):
         if hasattr(event, "state_key") and not is_new_state:
             logger.debug("Ignoring old state.")
             return
@@ -70,6 +79,115 @@ class FederationHandler(BaseHandler):
 
         else:
             with (yield self.room_lock.lock(event.room_id)):
-                store_id = yield self.store.persist_event(event)
+                store_id = yield self.store.persist_event(event, backfilled)
 
-            yield self.notifier.on_new_room_event(event, store_id)
+            room = yield self.store.get_room(event.room_id)
+
+            if not room:
+                # Huh, let's try and get the current state
+                try:
+                    federation = self.hs.get_federation()
+                    yield federation.get_state_for_room(
+                        event.origin, event.room_id
+                    )
+
+                    hosts = yield self.store.get_joined_hosts_for_room(
+                        event.room_id
+                    )
+                    if self.hs.hostname in hosts:
+                        try:
+                            yield self.store.store_room(
+                                event.room_id,
+                                "",
+                                is_public=False
+                            )
+                        except:
+                            pass
+                except:
+                    logger.exception(
+                        "Failed to get current state for room %s",
+                        event.room_id
+                    )
+
+            if not backfilled:
+                yield self.notifier.on_new_room_event(event, store_id)
+
+        if event.type == RoomMemberEvent.TYPE:
+            if event.membership == Membership.JOIN:
+                user = self.hs.parse_userid(event.target_user_id)
+                self.distributor.fire(
+                    "user_joined_room", user=user, room_id=event.room_id
+                )
+
+
+    @log_function
+    @defer.inlineCallbacks
+    def backfill(self, dest, room_id, limit):
+        events = yield self.hs.get_federation().backfill(dest, room_id, limit)
+
+        for event in events:
+            try:
+                yield self.store.persist_event(event, backfilled=True)
+            except:
+                logger.exception("Failed to persist event: %s", event)
+
+        defer.returnValue(events)
+
+    @log_function
+    @defer.inlineCallbacks
+    def do_invite_join(self, target_host, room_id, joinee, content):
+        federation = self.hs.get_federation()
+
+        hosts = yield self.store.get_joined_hosts_for_room(room_id)
+        if self.hs.hostname in hosts:
+            # We are already in the room.
+            logger.debug("We're already in the room apparently")
+            defer.returnValue(False)
+
+        # First get current state to see if we are already joined.
+        try:
+            yield federation.get_state_for_room(target_host, room_id)
+
+            hosts = yield self.store.get_joined_hosts_for_room(room_id)
+            if self.hs.hostname in hosts:
+                # Oh, we were actually in the room already.
+                logger.debug("We're already in the room apparently")
+                defer.returnValue(False)
+        except Exception:
+            logger.exception("Failed to get current state")
+
+        new_event = self.event_factory.create_event(
+            etype=InviteJoinEvent.TYPE,
+            target_host=target_host,
+            room_id=room_id,
+            user_id=joinee,
+            content=content
+        )
+
+        new_event.destinations = [target_host]
+
+        yield federation.handle_new_event(new_event)
+
+        # TODO (erikj): Time out here.
+        d = defer.Deferred()
+        self.waiting_for_join_list.setdefault((joinee, room_id), []).append(d)
+        yield d
+
+        try:
+            yield self.store.store_room(
+                event.room_id,
+                "",
+                is_public=False
+            )
+        except:
+            pass
+
+
+        defer.returnValue(True)
+
+
+    @log_function
+    def _on_user_joined(self, user, room_id):
+        waiters = self.waiting_for_join_list.get((user.to_string(), room_id), [])
+        while waiters:
+            waiters.pop().callback(None)

synapse/handlers/login.py

@@ -16,7 +16,7 @@
 from twisted.internet import defer
 
 from ._base import BaseHandler
-from synapse.api.errors import LoginError
+from synapse.api.errors import LoginError, Codes
 
 import bcrypt
 import logging
@@ -51,7 +51,7 @@ class LoginHandler(BaseHandler):
         user_info = yield self.store.get_user_by_id(user_id=user)
         if not user_info:
             logger.warn("Attempted to login as %s but they do not exist.", user)
-            raise LoginError(403, "")
+            raise LoginError(403, "", errcode=Codes.FORBIDDEN)
 
         stored_hash = user_info[0]["password_hash"]
         if bcrypt.checkpw(password, stored_hash):
@@ -62,4 +62,4 @@ class LoginHandler(BaseHandler):
             defer.returnValue(token)
         else:
             logger.warn("Failed password login for user %s", user)
-            raise LoginError(403, "")
+            raise LoginError(403, "", errcode=Codes.FORBIDDEN)

synapse/handlers/presence.py

@@ -142,6 +142,10 @@ class PresenceHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def is_presence_visible(self, observer_user, observed_user):
+        defer.returnValue(True)
+        return
+        # FIXME (erikj): This code path absolutely kills the database.
+
         assert(observed_user.is_mine)
 
         if observer_user == observed_user:
@@ -155,7 +159,10 @@ class PresenceHandler(BaseHandler):
         if allowed_by_subscription:
             defer.returnValue(True)
 
-        # TODO(paul): Check same channel
+        rm_handler = self.homeserver.get_handlers().room_member_handler
+        for room_id in (yield rm_handler.get_rooms_for_user(observer_user)):
+            if observed_user in (yield rm_handler.get_room_members(room_id)):
+                defer.returnValue(True)
 
         defer.returnValue(False)
 
@@ -176,12 +183,18 @@ class PresenceHandler(BaseHandler):
             # TODO(paul): Have remote server send us permissions set
             state = self._get_or_offline_usercache(target_user).get_state()
 
-        if "mtime" in state:
-            state["mtime_age"] = self.clock.time_msec() - state.pop("mtime")
+        if "mtime" in state and (state["mtime"] is not None):
+            state["mtime_age"] = int(
+                self.clock.time_msec() - state.pop("mtime")
+            )
         defer.returnValue(state)
 
     @defer.inlineCallbacks
     def set_state(self, target_user, auth_user, state):
+        return
+        # TODO (erikj): Turn this back on. Why did we end up sending EDUs
+        # everywhere?
+
         if not target_user.is_mine:
             raise SynapseError(400, "User is not hosted on this Home Server")
 
@@ -226,9 +239,6 @@ class PresenceHandler(BaseHandler):
         #   we don't have to do this all the time
         self.changed_presencelike_data(target_user, state)
 
-        if not now_online:
-            del self._user_cachemap[target_user]
-
     def changed_presencelike_data(self, user, state):
         statuscache = self._get_or_make_usercache(user)
 
@@ -367,7 +377,9 @@ class PresenceHandler(BaseHandler):
             p["observed_user"] = observed_user
             p.update(self._get_or_offline_usercache(observed_user).get_state())
             if "mtime" in p:
-                p["mtime_age"] = self.clock.time_msec() - p.pop("mtime")
+                p["mtime_age"] = int(
+                    self.clock.time_msec() - p.pop("mtime")
+                )
 
         defer.returnValue(presence)
 
@@ -376,14 +388,23 @@ class PresenceHandler(BaseHandler):
         logger.debug("Start polling for presence from %s", user)
 
         if target_user:
-            target_users = [target_user]
+            target_users = set([target_user])
         else:
             presence = yield self.store.get_presence_list(
                 user.localpart, accepted=True
             )
-            target_users = [
+            target_users = set([
                 self.hs.parse_userid(x["observed_user_id"]) for x in presence
-            ]
+            ])
+
+            # Also include people in all my rooms
+
+            rm_handler = self.homeserver.get_handlers().room_member_handler
+            room_ids = yield rm_handler.get_rooms_for_user(user)
+
+            for room_id in room_ids:
+                for member in (yield rm_handler.get_room_members(room_id)):
+                    target_users.add(member)
 
         if state is None:
             state = yield self.store.get_presence_state(user.localpart)
@@ -447,9 +468,13 @@ class PresenceHandler(BaseHandler):
         deferreds = []
 
         if target_user:
-            raise NotImplementedError("TODO: remove one user")
+            if target_user not in self._remote_recvmap:
+                return
+            target_users = set([target_user])
+        else:
+            target_users = self._remote_recvmap.keys()
 
-        remoteusers = [u for u in self._remote_recvmap
+        remoteusers = [u for u in target_users
                        if user in self._remote_recvmap[u]]
         remoteusers_by_domain = partition(remoteusers, lambda u: u.domain)
 
@@ -560,7 +585,9 @@ class PresenceHandler(BaseHandler):
 
         if "mtime" in state:
             state = dict(state)
-            state["mtime_age"] = self.clock.time_msec() - state.pop("mtime")
+            state["mtime_age"] = int(
+                self.clock.time_msec() - state.pop("mtime")
+            )
 
         yield self.federation.send_edu(
             destination=destination,
@@ -598,7 +625,9 @@ class PresenceHandler(BaseHandler):
             del state["user_id"]
 
             if "mtime_age" in state:
-                state["mtime"] = self.clock.time_msec() - state.pop("mtime_age")
+                state["mtime"] = int(
+                    self.clock.time_msec() - state.pop("mtime_age")
+                )
 
             statuscache = self._get_or_make_usercache(user)
 
@@ -664,7 +693,10 @@ class PresenceStreamData(StreamData):
         super(PresenceStreamData, self).__init__(hs)
         self.presence = hs.get_handlers().presence_handler
 
-    def get_rows(self, user_id, from_key, to_key, limit):
+    def get_rows(self, user_id, from_key, to_key, limit, direction):
+        from_key = int(from_key)
+        to_key = int(to_key)
+
         cachemap = self.presence._user_cachemap
 
         # TODO(paul): limit, and filter by visibility
@@ -720,6 +752,8 @@ class UserPresenceCache(object):
         content["user_id"] = user.to_string()
 
         if "mtime" in content:
-            content["mtime_age"] = clock.time_msec() - content.pop("mtime")
+            content["mtime_age"] = int(
+                clock.time_msec() - content.pop("mtime")
+            )
 
         return {"type": "m.presence", "content": content}

synapse/handlers/room.py

@@ -23,7 +23,8 @@ from synapse.api.events.room import (
     RoomTopicEvent, MessageEvent, InviteJoinEvent, RoomMemberEvent,
     RoomConfigEvent
 )
-from synapse.api.streams.event import EventStream, MessagesStreamData
+from synapse.api.streams.event import EventStream, EventsStreamData
+from synapse.handlers.presence import PresenceStreamData
 from synapse.util import stringutils
 from ._base import BaseHandler
 
@@ -59,12 +60,14 @@ class MessageHandler(BaseHandler):
         yield self.auth.check_joined_room(room_id, user_id)
 
         # Pull out the message from the db
-        msg = yield self.store.get_message(room_id=room_id,
-                                           msg_id=msg_id,
-                                           user_id=sender_id)
+#        msg = yield self.store.get_message(
+#            room_id=room_id,
+#            msg_id=msg_id,
+#            user_id=sender_id
+#        )
+
+        # TODO (erikj): Once we work out the correct c-s api we need to think on how to do this.
 
-        if msg:
-            defer.returnValue(msg)
         defer.returnValue(None)
 
     @defer.inlineCallbacks
@@ -94,10 +97,10 @@ class MessageHandler(BaseHandler):
                 event.room_id
             )
 
-            yield self.hs.get_federation().handle_new_event(event)
-
             self.notifier.on_new_room_event(event, store_id)
 
+        yield self.hs.get_federation().handle_new_event(event)
+
     @defer.inlineCallbacks
     def get_messages(self, user_id=None, room_id=None, pagin_config=None,
                      feedback=False):
@@ -114,8 +117,9 @@ class MessageHandler(BaseHandler):
         """
         yield self.auth.check_joined_room(room_id, user_id)
 
-        data_source = [MessagesStreamData(self.hs, room_id=room_id,
-                                          feedback=feedback)]
+        data_source = [
+            EventsStreamData(self.hs, room_id=room_id, feedback=feedback)
+        ]
         event_stream = EventStream(user_id, data_source)
         pagin_config = yield event_stream.fix_tokens(pagin_config)
         data_chunk = yield event_stream.get_chunk(config=pagin_config)
@@ -141,12 +145,7 @@ class MessageHandler(BaseHandler):
             yield self.state_handler.handle_new_event(event)
 
             # store in db
-            store_id = yield self.store.store_room_data(
-                room_id=event.room_id,
-                etype=event.type,
-                state_key=event.state_key,
-                content=json.dumps(event.content)
-            )
+            store_id = yield self.store.persist_event(event)
 
             event.destinations = yield self.store.get_joined_hosts_for_room(
                 event.room_id
@@ -201,19 +200,17 @@ class MessageHandler(BaseHandler):
                 raise RoomError(
                     403, "Member does not meet private room rules.")
 
-        data = yield self.store.get_room_data(room_id, event_type, state_key)
+        data = yield self.store.get_current_state(
+            room_id, event_type, state_key
+        )
         defer.returnValue(data)
 
     @defer.inlineCallbacks
-    def get_feedback(self, room_id=None, msg_sender_id=None, msg_id=None,
-                     user_id=None, fb_sender_id=None, fb_type=None):
-        yield self.auth.check_joined_room(room_id, user_id)
+    def get_feedback(self, event_id):
+        # yield self.auth.check_joined_room(room_id, user_id)
 
         # Pull out the feedback from the db
-        fb = yield self.store.get_feedback(
-            room_id=room_id, msg_id=msg_id, msg_sender_id=msg_sender_id,
-            fb_sender_id=fb_sender_id, fb_type=fb_type
-        )
+        fb = yield self.store.get_feedback(event_id)
 
         if fb:
             defer.returnValue(fb)
@@ -260,20 +257,59 @@ class MessageHandler(BaseHandler):
             user_id=user_id,
             membership_list=[Membership.INVITE, Membership.JOIN]
         )
-        for room_info in room_list:
-            if room_info["membership"] != Membership.JOIN:
+
+        rooms_ret = []
+
+        now_rooms_token = yield self.store.get_room_events_max_id()
+
+        # FIXME (erikj): Fix this.
+        presence_stream = PresenceStreamData(self.hs)
+        now_presence_token = yield presence_stream.max_token()
+        presence = yield presence_stream.get_rows(
+            user_id, 0, now_presence_token, None, None
+        )
+
+        # FIXME (erikj): We need to not generate this token,
+        now_token = "%s_%s" % (now_rooms_token, now_presence_token)
+
+        for event in room_list:
+            d = {
+                "room_id": event.room_id,
+                "membership": event.membership,
+            }
+
+            if event.membership == Membership.INVITE:
+                d["inviter"] = event.user_id
+
+            rooms_ret.append(d)
+
+            if event.membership != Membership.JOIN:
                 continue
             try:
-                event_chunk = yield self.get_messages(
-                    user_id=user_id,
-                    pagin_config=pagin_config,
-                    feedback=feedback,
-                    room_id=room_info["room_id"]
+                messages, token = yield self.store.get_recent_events_for_room(
+                    event.room_id,
+                    limit=10,
+                    end_token=now_rooms_token,
                 )
-                room_info["messages"] = event_chunk
+
+                d["messages"] = {
+                    "chunk": [m.get_dict() for m in messages],
+                    "start": token[0],
+                    "end": token[1],
+                }
+
+                current_state = yield self.store.get_current_state(event.room_id)
+                d["state"] = [c.get_dict() for c in current_state]
             except:
-                pass
-        defer.returnValue(room_list)
+                logger.exception("Failed to get snapshot")
+
+        user = self.hs.parse_userid(user_id)
+
+        ret = {"rooms": rooms_ret, "presence": presence[0], "end": now_token}
+
+        # logger.debug("snapshot_all_rooms returning: %s", ret)
+
+        defer.returnValue(ret)
 
 
 class RoomCreationHandler(BaseHandler):
@@ -372,7 +408,6 @@ class RoomCreationHandler(BaseHandler):
 
         yield self.hs.get_handlers().room_member_handler.change_membership(
             join_event,
-            broadcast_msg=True,
             do_auth=False
         )
 
@@ -451,11 +486,11 @@ class RoomMemberHandler(BaseHandler):
 
         member_list = yield self.store.get_room_members(room_id=room_id)
         event_list = [
-            entry.as_event(self.event_factory).get_dict()
+            entry.get_dict()
             for entry in member_list
         ]
         chunk_data = {
-            "start": "START",
+            "start": "START",  # FIXME (erikj): START is no longer a valid value
             "end": "END",
             "chunk": event_list
         }
@@ -484,29 +519,28 @@ class RoomMemberHandler(BaseHandler):
         defer.returnValue(member)
 
     @defer.inlineCallbacks
-    def change_membership(self, event=None, broadcast_msg=False, do_auth=True):
+    def change_membership(self, event=None, do_auth=True):
         """ Change the membership status of a user in a room.
 
         Args:
             event (SynapseEvent): The membership event
-            broadcast_msg (bool): True to inject a membership message into this
-                room on success.
         Raises:
             SynapseError if there was a problem changing the membership.
         """
 
-        #broadcast_msg = False
-
         prev_state = yield self.store.get_room_member(
             event.target_user_id, event.room_id
         )
 
-        if prev_state and prev_state.membership == event.membership:
-            # treat this event as a NOOP.
-            if do_auth:  # This is mainly to fix a unit test.
-                yield self.auth.check(event, raises=True)
-            defer.returnValue({})
-            return
+        if prev_state:
+            event.content["prev"] = prev_state.membership
+
+#        if prev_state and prev_state.membership == event.membership:
+#            # treat this event as a NOOP.
+#            if do_auth:  # This is mainly to fix a unit test.
+#                yield self.auth.check(event, raises=True)
+#            defer.returnValue({})
+#            return
 
         room_id = event.room_id
 
@@ -514,9 +548,7 @@ class RoomMemberHandler(BaseHandler):
         # if this HS is not currently in the room, i.e. we have to do the
         # invite/join dance.
         if event.membership == Membership.JOIN:
-            yield self._do_join(
-                event, do_auth=do_auth, broadcast_msg=broadcast_msg
-            )
+            yield self._do_join(event, do_auth=do_auth)
         else:
             # This is not a JOIN, so we can handle it normally.
             if do_auth:
@@ -534,7 +566,6 @@ class RoomMemberHandler(BaseHandler):
             yield self._do_local_membership_update(
                 event,
                 membership=event.content["membership"],
-                broadcast_msg=broadcast_msg,
             )
 
         defer.returnValue({"room_id": room_id})
@@ -569,14 +600,14 @@ class RoomMemberHandler(BaseHandler):
         defer.returnValue({"room_id": room_id})
 
     @defer.inlineCallbacks
-    def _do_join(self, event, room_host=None, do_auth=True, broadcast_msg=True):
+    def _do_join(self, event, room_host=None, do_auth=True):
         joinee = self.hs.parse_userid(event.target_user_id)
         # room_id = RoomID.from_string(event.room_id, self.hs)
         room_id = event.room_id
 
         # If event doesn't include a display name, add one.
-        yield self._fill_out_join_content(
-            joinee, event.content
+        yield self.distributor.fire(
+            "collect_presencelike_data", joinee, event.content
         )
 
         # XXX: We don't do an auth check if we are doing an invite
@@ -584,9 +615,9 @@ class RoomMemberHandler(BaseHandler):
         # that we are allowed to join when we decide whether or not we
         # need to do the invite/join dance.
 
-        room = yield self.store.get_room(room_id)
+        hosts = yield self.store.get_joined_hosts_for_room(room_id)
 
-        if room:
+        if self.hs.hostname in hosts:
             should_do_dance = False
         elif room_host:
             should_do_dance = True
@@ -598,7 +629,7 @@ class RoomMemberHandler(BaseHandler):
             if prev_state and prev_state.membership == Membership.INVITE:
                 room = yield self.store.get_room(room_id)
                 inviter = UserID.from_string(
-                    prev_state.sender, self.hs
+                    prev_state.user_id, self.hs
                 )
 
                 should_do_dance = not inviter.is_mine and not room
@@ -606,8 +637,15 @@ class RoomMemberHandler(BaseHandler):
             else:
                 should_do_dance = False
 
+        have_joined = False
+        if should_do_dance:
+            handler = self.hs.get_handlers().federation_handler
+            have_joined = yield handler.do_invite_join(
+                room_host, room_id, event.user_id, event.content
+            )
+
         # We want to do the _do_update inside the room lock.
-        if not should_do_dance:
+        if not have_joined:
             logger.debug("Doing normal join")
 
             if do_auth:
@@ -617,16 +655,6 @@ class RoomMemberHandler(BaseHandler):
             yield self._do_local_membership_update(
                 event,
                 membership=event.content["membership"],
-                broadcast_msg=broadcast_msg,
-            )
-
-
-        if should_do_dance:
-            yield self._do_invite_join_dance(
-                room_id=room_id,
-                joinee=event.user_id,
-                target_host=room_host,
-                content=event.content,
             )
 
         user = self.hs.parse_userid(event.user_id)
@@ -634,32 +662,6 @@ class RoomMemberHandler(BaseHandler):
             "user_joined_room", user=user, room_id=room_id
         )
 
-    @defer.inlineCallbacks
-    def _fill_out_join_content(self, user_id, content):
-        # If event doesn't include a display name, add one.
-        profile_handler = self.hs.get_handlers().profile_handler
-        if "displayname" not in content:
-            try:
-                display_name = yield profile_handler.get_displayname(
-                    user_id
-                )
-
-                if display_name:
-                    content["displayname"] = display_name
-            except:
-                logger.exception("Failed to set display_name")
-
-        if "avatar_url" not in content:
-            try:
-                avatar_url = yield profile_handler.get_avatar_url(
-                    user_id
-                )
-
-                if avatar_url:
-                    content["avatar_url"] = avatar_url
-            except:
-                logger.exception("Failed to set display_name")
-
     @defer.inlineCallbacks
     def _should_invite_join(self, room_id, prev_state, do_auth):
         logger.debug("_should_invite_join: room_id: %s", room_id)
@@ -694,18 +696,12 @@ class RoomMemberHandler(BaseHandler):
             user_id=user.to_string(), membership_list=membership_list
         )
 
-        defer.returnValue([r["room_id"] for r in rooms])
+        defer.returnValue([r.room_id for r in rooms])
 
     @defer.inlineCallbacks
-    def _do_local_membership_update(self, event, membership, broadcast_msg):
+    def _do_local_membership_update(self, event, membership):
         # store membership
-        store_id = yield self.store.store_room_member(
-            user_id=event.target_user_id,
-            sender=event.user_id,
-            room_id=event.room_id,
-            content=event.content,
-            membership=membership
-        )
+        store_id = yield self.store.persist_event(event)
 
         # Send a PDU to all hosts who have joined the room.
         destinations = yield self.store.get_joined_hosts_for_room(
@@ -732,78 +728,11 @@ class RoomMemberHandler(BaseHandler):
         yield self.hs.get_federation().handle_new_event(event)
         self.notifier.on_new_room_event(event, store_id)
 
-        if broadcast_msg:
-            yield self._inject_membership_msg(
-                source=event.user_id,
-                target=event.target_user_id,
-                room_id=event.room_id,
-                membership=event.content["membership"]
-            )
-
-    @defer.inlineCallbacks
-    def _do_invite_join_dance(self, room_id, joinee, target_host, content):
-        logger.debug("Doing remote join dance")
-
-        # do invite join dance
-        federation = self.hs.get_federation()
-        new_event = self.event_factory.create_event(
-            etype=InviteJoinEvent.TYPE,
-            target_host=target_host,
-            room_id=room_id,
-            user_id=joinee,
-            content=content
-        )
-
-        new_event.destinations = [target_host]
-
-        yield self.store.store_room(
-            room_id, "", is_public=False
-        )
-
-        #yield self.state_handler.handle_new_event(event)
-        yield federation.handle_new_event(new_event)
-        yield federation.get_state_for_room(
-            target_host, room_id
-        )
-
-    @defer.inlineCallbacks
-    def _inject_membership_msg(self, room_id=None, source=None, target=None,
-                               membership=None):
-        # TODO this should be a different type of message, not m.text
-        if membership == Membership.INVITE:
-            body = "%s invited %s to the room." % (source, target)
-        elif membership == Membership.JOIN:
-            body = "%s joined the room." % (target)
-        elif membership == Membership.LEAVE:
-            body = "%s left the room." % (target)
-        else:
-            raise RoomError(500, "Unknown membership value %s" % membership)
-
-        membership_json = {
-            "msgtype": u"m.text",
-            "body": body,
-            "membership_source": source,
-            "membership_target": target,
-            "membership": membership,
-        }
-
-        msg_id = "m%s" % int(self.clock.time_msec())
-
-        event = self.event_factory.create_event(
-            etype=MessageEvent.TYPE,
-            room_id=room_id,
-            user_id="_homeserver_",
-            msg_id=msg_id,
-            content=membership_json
-        )
-
-        handler = self.hs.get_handlers().message_handler
-        yield handler.send_message(event, suppress_auth=True)
-
 
 class RoomListHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def get_public_room_list(self):
-        chunk = yield self.store.get_rooms(is_public=True, with_topics=True)
+        chunk = yield self.store.get_rooms(is_public=True)
+        # FIXME (erikj): START is no longer a valid value
         defer.returnValue({"start": "START", "end": "END", "chunk": chunk})

synapse/http/server.py

@@ -17,15 +17,23 @@
 from syutil.jsonutil import (
     encode_canonical_json, encode_pretty_printed_json
 )
-from synapse.api.errors import cs_exception, CodeMessageException
+from synapse.api.errors import (
+    cs_exception, SynapseError, CodeMessageException, Codes, cs_error
+)
+from synapse.util.stringutils import random_string
 
 from twisted.internet import defer, reactor
+from twisted.protocols.basic import FileSender
 from twisted.web import server, resource
 from twisted.web.server import NOT_DONE_YET
+from twisted.web.util import redirectTo
 
+import base64
 import collections
+import json
 import logging
-
+import os
+import re
 
 logger = logging.getLogger(__name__)
 
@@ -52,10 +60,9 @@ class HttpServer(object):
         pass
 
 
-# The actual HTTP server impl, using twisted http server
-class TwistedHttpServer(HttpServer, resource.Resource):
-    """ This wraps the twisted HTTP server, and triggers the correct callbacks
-    on the transport_layer.
+class JsonResource(HttpServer, resource.Resource):
+    """ This implements the HttpServer interface and provides JSON support for
+    Resources.
 
     Register callbacks via register_path()
     """
@@ -125,7 +132,11 @@ class TwistedHttpServer(HttpServer, resource.Resource):
                 {"error": "Unrecognized request"}
             )
         except CodeMessageException as e:
-            logger.exception(e)
+            if isinstance(e, SynapseError):
+                logger.error("%s SynapseError: %s - %s", request, e.code,
+                             e.msg)
+            else:
+                logger.exception(e)
             self._send_response(
                 request,
                 e.code,
@@ -140,6 +151,14 @@ class TwistedHttpServer(HttpServer, resource.Resource):
             )
 
     def _send_response(self, request, code, response_json_object):
+        # could alternatively use request.notifyFinish() and flip a flag when
+        # the Deferred fires, but since the flag is RIGHT THERE it seems like
+        # a waste.
+        if request._disconnected:
+            logger.warn(
+                "Not sending response to request %s, already disconnected.",
+                request)
+            return
 
         if not self._request_user_agent_is_curl(request):
             json_bytes = encode_canonical_json(response_json_object)
@@ -160,6 +179,171 @@ class TwistedHttpServer(HttpServer, resource.Resource):
         return False
 
 
+class RootRedirect(resource.Resource):
+    """Redirects the root '/' path to another path."""
+
+    def __init__(self, path):
+        resource.Resource.__init__(self)
+        self.url = path
+
+    def render_GET(self, request):
+        return redirectTo(self.url, request)
+
+    def getChild(self, name, request):
+        if len(name) == 0:
+            return self  # select ourselves as the child to render
+        return resource.Resource.getChild(self, name, request)
+
+
+class ContentRepoResource(resource.Resource):
+    """Provides file uploading and downloading.
+
+    Uploads are POSTed to wherever this Resource is linked to. This resource
+    returns a "content token" which can be used to GET this content again. The
+    token is typically a path, but it may not be. Tokens can expire, be one-time
+    uses, etc.
+
+    In this case, the token is a path to the file and contains 3 interesting
+    sections:
+        - User ID base64d (for namespacing content to each user)
+        - random 24 char string
+        - Content type base64d (so we can return it when clients GET it)
+
+    """
+    isLeaf = True
+
+    def __init__(self, hs, directory, auth):
+        resource.Resource.__init__(self)
+        self.hs = hs
+        self.directory = directory
+        self.auth = auth
+
+        if not os.path.isdir(self.directory):
+            os.mkdir(self.directory)
+            logger.info("ContentRepoResource : Created %s directory.",
+                        self.directory)
+
+    @defer.inlineCallbacks
+    def map_request_to_name(self, request):
+        # auth the user
+        auth_user = yield self.auth.get_user_by_req(request)
+
+        # namespace all file uploads on the user
+        prefix = base64.urlsafe_b64encode(
+            auth_user.to_string()
+        ).replace('=', '')
+
+        # use a random string for the main portion
+        main_part = random_string(24)
+
+        # suffix with a file extension if we can make one. This is nice to
+        # provide a hint to clients on the file information. We will also reuse
+        # this info to spit back the content type to the client.
+        suffix = ""
+        if request.requestHeaders.hasHeader("Content-Type"):
+            content_type = request.requestHeaders.getRawHeaders(
+                "Content-Type")[0]
+            suffix = "." + base64.urlsafe_b64encode(content_type)
+            if (content_type.split("/")[0].lower() in
+                    ["image", "video", "audio"]):
+                file_ext = content_type.split("/")[-1]
+                # be a little paranoid and only allow a-z
+                file_ext = re.sub("[^a-z]", "", file_ext)
+                suffix += "." + file_ext
+
+        file_name = prefix + main_part + suffix
+        file_path = os.path.join(self.directory, file_name)
+        logger.info("User %s is uploading a file to path %s",
+                    auth_user.to_string(),
+                    file_path)
+
+        # keep trying to make a non-clashing file, with a sensible max attempts
+        attempts = 0
+        while os.path.exists(file_path):
+            main_part = random_string(24)
+            file_name = prefix + main_part + suffix
+            file_path = os.path.join(self.directory, file_name)
+            attempts += 1
+            if attempts > 25:  # really? Really?
+                raise SynapseError(500, "Unable to create file.")
+
+        defer.returnValue(file_path)
+
+    def render_GET(self, request):
+        # no auth here on purpose, to allow anyone to view, even across home
+        # servers.
+
+        # TODO: A little crude here, we could do this better.
+        filename = request.path.split('/')[-1]
+        # be paranoid
+        filename = re.sub("[^0-9A-z.-_]", "", filename)
+
+        file_path = self.directory + "/" + filename
+
+        logger.debug("Searching for %s", file_path)
+
+        if os.path.isfile(file_path):
+            # filename has the content type
+            base64_contentype = filename.split(".")[1]
+            content_type = base64.urlsafe_b64decode(base64_contentype)
+            logger.info("Sending file %s", file_path)
+            f = open(file_path, 'rb')
+            request.setHeader('Content-Type', content_type)
+            d = FileSender().beginFileTransfer(f, request)
+
+            # after the file has been sent, clean up and finish the request
+            def cbFinished(ignored):
+                f.close()
+                request.finish()
+            d.addCallback(cbFinished)
+        else:
+            respond_with_json_bytes(
+                request,
+                404,
+                json.dumps(cs_error("Not found", code=Codes.NOT_FOUND)),
+                send_cors=True)
+
+        return server.NOT_DONE_YET
+
+    def render_POST(self, request):
+        self._async_render(request)
+        return server.NOT_DONE_YET
+
+    def render_OPTIONS(self, request):
+        respond_with_json_bytes(request, 200, {}, send_cors=True)
+        return server.NOT_DONE_YET
+
+    @defer.inlineCallbacks
+    def _async_render(self, request):
+        try:
+            fname = yield self.map_request_to_name(request)
+
+            # TODO I have a suspcious feeling this is just going to block
+            with open(fname, "wb") as f:
+                f.write(request.content.read())
+
+
+            # FIXME (erikj): These should use constants.
+            file_name = os.path.basename(fname)
+            url = "http://%s/matrix/content/%s" % (self.hs.hostname, file_name)
+
+            respond_with_json_bytes(request, 200,
+                                    json.dumps({"content_token": url}),
+                                    send_cors=True)
+
+        except CodeMessageException as e:
+            logger.exception(e)
+            respond_with_json_bytes(request, e.code,
+                                    json.dumps(cs_exception(e)))
+        except Exception as e:
+            logger.error("Failed to store file: %s" % e)
+            respond_with_json_bytes(
+                request,
+                500,
+                json.dumps({"error": "Internal server error"}),
+                send_cors=True)
+
+
 def respond_with_json_bytes(request, code, json_bytes, send_cors=False):
     """Sends encoded JSON in response to the given request.
 

synapse/rest/__init__.py

@@ -15,8 +15,7 @@
 
 
 from . import (
-    room, events, register, login, profile, public, presence, im, directory,
-    webclient
+    room, events, register, login, profile, public, presence, im, directory
 )
 
 
@@ -32,19 +31,15 @@ class RestServletFactory(object):
     """
 
     def __init__(self, hs):
-        http_server = hs.get_http_server()
+        client_resource = hs.get_resource_for_client()
 
         # TODO(erikj): There *must* be a better way of doing this.
-        room.register_servlets(hs, http_server)
-        events.register_servlets(hs, http_server)
-        register.register_servlets(hs, http_server)
-        login.register_servlets(hs, http_server)
-        profile.register_servlets(hs, http_server)
-        public.register_servlets(hs, http_server)
-        presence.register_servlets(hs, http_server)
-        im.register_servlets(hs, http_server)
-        directory.register_servlets(hs, http_server)
-
-    def register_web_client(self, hs):
-        http_server = hs.get_http_server()
-        webclient.register_servlets(hs, http_server)
+        room.register_servlets(hs, client_resource)
+        events.register_servlets(hs, client_resource)
+        register.register_servlets(hs, client_resource)
+        login.register_servlets(hs, client_resource)
+        profile.register_servlets(hs, client_resource)
+        public.register_servlets(hs, client_resource)
+        presence.register_servlets(hs, client_resource)
+        im.register_servlets(hs, client_resource)
+        directory.register_servlets(hs, client_resource)

synapse/rest/base.py

@@ -14,6 +14,7 @@
 # limitations under the License.
 
 """ This module contains base REST classes for constructing REST servlets. """
+from synapse.api.urls import CLIENT_PREFIX
 import re
 
 
@@ -27,7 +28,7 @@ def client_path_pattern(path_regex):
     Returns:
         SRE_Pattern
     """
-    return re.compile("^/matrix/client/api/v1" + path_regex)
+    return re.compile("^" + CLIENT_PREFIX + path_regex)
 
 
 class RestServlet(object):

synapse/rest/events.py

@@ -24,7 +24,7 @@ from synapse.rest.base import RestServlet, client_path_pattern
 class EventStreamRestServlet(RestServlet):
     PATTERN = client_path_pattern("/events$")
 
-    DEFAULT_LONGPOLL_TIME_MS = 5000
+    DEFAULT_LONGPOLL_TIME_MS = 30000
 
     @defer.inlineCallbacks
     def on_GET(self, request):

synapse/rest/login.py

@@ -16,6 +16,7 @@
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError
+from synapse.types import UserID
 from base import RestServlet, client_path_pattern
 
 import json
@@ -45,12 +46,17 @@ class LoginRestServlet(RestServlet):
 
     @defer.inlineCallbacks
     def do_password_login(self, login_submission):
+        if not login_submission["user"].startswith('@'):
+            login_submission["user"] = UserID.create_local(
+                login_submission["user"], self.hs).to_string()
+
         handler = self.handlers.login_handler
         token = yield handler.login(
             user=login_submission["user"],
             password=login_submission["password"])
 
         result = {
+            "user_id": login_submission["user"],  # may have changed
             "access_token": token,
             "home_server": self.hs.hostname,
         }

synapse/rest/room.py

@@ -115,7 +115,7 @@ class RoomTopicRestServlet(RestServlet):
 
         if not data:
             raise SynapseError(404, "Topic not found.", errcode=Codes.NOT_FOUND)
-        defer.returnValue((200, json.loads(data.content)))
+        defer.returnValue((200, data.content))
 
     @defer.inlineCallbacks
     def on_PUT(self, request, room_id):
@@ -170,12 +170,14 @@ class RoomMemberRestServlet(RestServlet):
         user = yield self.auth.get_user_by_req(request)
 
         handler = self.handlers.room_member_handler
-        member = yield handler.get_room_member(room_id, target_user_id,
-                                               user.to_string())
+        member = yield handler.get_room_member(
+            room_id,
+            urllib.unquote(target_user_id),
+            user.to_string())
         if not member:
             raise SynapseError(404, "Member not found.",
                                errcode=Codes.NOT_FOUND)
-        defer.returnValue((200, json.loads(member.content)))
+        defer.returnValue((200, member.content))
 
     @defer.inlineCallbacks
     def on_DELETE(self, request, roomid, target_user_id):
@@ -183,7 +185,7 @@ class RoomMemberRestServlet(RestServlet):
 
         event = self.event_factory.create_event(
             etype=self.get_event_type(),
-            target_user_id=target_user_id,
+            target_user_id=urllib.unquote(target_user_id),
             room_id=urllib.unquote(roomid),
             user_id=user.to_string(),
             membership=Membership.LEAVE,
@@ -191,7 +193,7 @@ class RoomMemberRestServlet(RestServlet):
             )
 
         handler = self.handlers.room_member_handler
-        yield handler.change_membership(event, broadcast_msg=True)
+        yield handler.change_membership(event)
         defer.returnValue((200, ""))
 
     @defer.inlineCallbacks
@@ -210,7 +212,7 @@ class RoomMemberRestServlet(RestServlet):
 
         event = self.event_factory.create_event(
             etype=self.get_event_type(),
-            target_user_id=target_user_id,
+            target_user_id=urllib.unquote(target_user_id),
             room_id=urllib.unquote(roomid),
             user_id=user.to_string(),
             membership=content["membership"],
@@ -218,8 +220,8 @@ class RoomMemberRestServlet(RestServlet):
             )
 
         handler = self.handlers.room_member_handler
-        result = yield handler.change_membership(event, broadcast_msg=True)
-        defer.returnValue((200, result))
+        yield handler.change_membership(event)
+        defer.returnValue((200, ""))
 
 
 class MessageRestServlet(RestServlet):
@@ -235,7 +237,7 @@ class MessageRestServlet(RestServlet):
 
         msg_handler = self.handlers.message_handler
         msg = yield msg_handler.get_message(room_id=urllib.unquote(room_id),
-                                            sender_id=sender_id,
+                                            sender_id=urllib.unquote(sender_id),
                                             msg_id=msg_id,
                                             user_id=user.to_string(),
                                             )
@@ -250,7 +252,7 @@ class MessageRestServlet(RestServlet):
     def on_PUT(self, request, room_id, sender_id, msg_id):
         user = yield self.auth.get_user_by_req(request)
 
-        if user.to_string() != sender_id:
+        if user.to_string() != urllib.unquote(sender_id):
             raise SynapseError(403, "Must send messages as yourself.",
                                errcode=Codes.FORBIDDEN)
 
@@ -285,25 +287,28 @@ class FeedbackRestServlet(RestServlet):
                feedback_type):
         user = yield (self.auth.get_user_by_req(request))
 
-        if feedback_type not in Feedback.LIST:
-            raise SynapseError(400, "Bad feedback type.",
-                               errcode=Codes.BAD_JSON)
+        # TODO (erikj): Implement this?
+        raise NotImplementedError("Getting feedback is not supported")
 
-        msg_handler = self.handlers.message_handler
-        feedback = yield msg_handler.get_feedback(
-            room_id=urllib.unquote(room_id),
-            msg_sender_id=msg_sender_id,
-            msg_id=msg_id,
-            user_id=user.to_string(),
-            fb_sender_id=fb_sender_id,
-            fb_type=feedback_type
-        )
-
-        if not feedback:
-            raise SynapseError(404, "Feedback not found.",
-                               errcode=Codes.NOT_FOUND)
-
-        defer.returnValue((200, json.loads(feedback.content)))
+#        if feedback_type not in Feedback.LIST:
+#            raise SynapseError(400, "Bad feedback type.",
+#                               errcode=Codes.BAD_JSON)
+#
+#        msg_handler = self.handlers.message_handler
+#        feedback = yield msg_handler.get_feedback(
+#            room_id=urllib.unquote(room_id),
+#            msg_sender_id=msg_sender_id,
+#            msg_id=msg_id,
+#            user_id=user.to_string(),
+#            fb_sender_id=fb_sender_id,
+#            fb_type=feedback_type
+#        )
+#
+#        if not feedback:
+#            raise SynapseError(404, "Feedback not found.",
+#                               errcode=Codes.NOT_FOUND)
+#
+#        defer.returnValue((200, json.loads(feedback.content)))
 
     @defer.inlineCallbacks
     def on_PUT(self, request, room_id, sender_id, msg_id, fb_sender_id,
@@ -348,6 +353,17 @@ class RoomMemberListRestServlet(RestServlet):
             room_id=urllib.unquote(room_id),
             user_id=user.to_string())
 
+        for event in members["chunk"]:
+            target_user = self.hs.parse_userid(event["target_user_id"])
+            # Presence is an optional cache; don't fail if we can't fetch it
+            try:
+                presence_state = yield self.handlers.presence_handler.get_state(
+                    target_user=target_user, auth_user=user
+                )
+                event["content"].update(presence_state)
+            except:
+                pass
+
         defer.returnValue((200, members))
 
 
@@ -369,6 +385,21 @@ class RoomMessageListRestServlet(RestServlet):
         defer.returnValue((200, msgs))
 
 
+class RoomTriggerBackfill(RestServlet):
+    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/backfill$")
+
+    @defer.inlineCallbacks
+    def on_GET(self, request, room_id):
+        remote_server = urllib.unquote(request.args["remote"][0])
+        room_id = urllib.unquote(room_id)
+        limit = int(request.args["limit"][0])
+
+        handler = self.handlers.federation_handler
+        events = yield handler.backfill(remote_server, room_id, limit)
+
+        res = [event.get_dict() for event in events]
+        defer.returnValue((200, res))
+
 def _parse_json(request):
     try:
         content = json.loads(request.content.read())
@@ -389,3 +420,4 @@ def register_servlets(hs, http_server):
     RoomMemberListRestServlet(hs).register(http_server)
     RoomMessageListRestServlet(hs).register(http_server)
     JoinRoomAliasServlet(hs).register(http_server)
+    RoomTriggerBackfill(hs).register(http_server)

synapse/rest/webclient.py

@@ -1,45 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.rest.base import RestServlet
-
-import logging
-import re
-
-logger = logging.getLogger(__name__)
-
-
-class WebClientRestServlet(RestServlet):
-    # No PATTERN; we have custom dispatch rules here
-
-    def register(self, http_server):
-        http_server.register_path("GET",
-                                  re.compile("^/$"),
-                                  self.on_GET_redirect)
-        http_server.register_path("GET",
-                                  re.compile("^/matrix/client$"),
-                                  self.on_GET)
-
-    def on_GET(self, request):
-        return (200, "not implemented")
-
-    def on_GET_redirect(self, request):
-        request.setHeader("Location", request.uri + "matrix/client")
-        return (302, None)
-
-
-def register_servlets(hs, http_server):
-    logger.info("Registering web client.")
-    WebClientRestServlet(hs).register(http_server)

synapse/server.py

@@ -55,7 +55,6 @@ class BaseHomeServer(object):
 
     DEPENDENCIES = [
         'clock',
-        'http_server',
         'http_client',
         'db_pool',
         'persistence_service',
@@ -70,6 +69,10 @@ class BaseHomeServer(object):
         'room_lock_manager',
         'notifier',
         'distributor',
+        'resource_for_client',
+        'resource_for_federation',
+        'resource_for_web_client',
+        'resource_for_content_repo',
     ]
 
     def __init__(self, hostname, **kwargs):
@@ -135,7 +138,10 @@ class HomeServer(BaseHomeServer):
     required.
 
     It still requires the following to be specified by the caller:
-        http_server
+        resource_for_client
+        resource_for_web_client
+        resource_for_federation
+        resource_for_content_repo
         http_client
         db_pool
     """
@@ -153,7 +159,7 @@ class HomeServer(BaseHomeServer):
         return DataStore(self)
 
     def build_event_factory(self):
-        return EventFactory()
+        return EventFactory(self)
 
     def build_handlers(self):
         return Handlers(self)
@@ -178,9 +184,6 @@ class HomeServer(BaseHomeServer):
 
     def register_servlets(self):
         """ Register all servlets associated with this HomeServer.
-
-        Args:
-            host_web_client (bool): True to host the web client as well.
         """
         # Simply building the ServletFactory is sufficient to have it register
-        factory = self.get_rest_servlet_factory()
+        self.get_rest_servlet_factory()

synapse/state.py

@@ -86,7 +86,7 @@ class StateHandler(object):
         else:
             event.depth = 0
 
-        current_state = yield self.store.get_current_state(
+        current_state = yield self.store.get_current_state_pdu(
             key.context, key.type, key.state_key
         )
 
@@ -157,7 +157,10 @@ class StateHandler(object):
             defer.returnValue(True)
             return
 
-        if new_branch[-1] == current_branch[-1]:
+        n = new_branch[-1]
+        c = current_branch[-1]
+
+        if n.pdu_id == c.pdu_id and n.origin == c.origin:
             # We have all the PDUs we need, so we can just do the conflict
             # resolution.
 
@@ -188,10 +191,18 @@ class StateHandler(object):
                 key=lambda x: x.depth
             )
 
+            pdu_id = missing_prev.prev_state_id
+            origin = missing_prev.prev_state_origin
+
+            is_missing = yield self.store.get_pdu(pdu_id, origin) is None
+
+            if not is_missing:
+                raise Exception("Conflict resolution failed.")
+
             yield self._replication.get_pdu(
                 destination=missing_prev.origin,
-                pdu_origin=missing_prev.prev_state_origin,
-                pdu_id=missing_prev.prev_state_id,
+                pdu_origin=origin,
+                pdu_id=pdu_id,
                 outlier=True
             )
 

synapse/storage/__init__.py

@@ -13,30 +13,35 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
 
 from synapse.api.events.room import (
     RoomMemberEvent, MessageEvent, RoomTopicEvent, FeedbackEvent,
-    RoomConfigEvent
+    RoomConfigEvent, RoomNameEvent,
 )
 
+from synapse.util.logutils import log_function
+
 from .directory import DirectoryStore
 from .feedback import FeedbackStore
-from .message import MessageStore
 from .presence import PresenceStore
 from .profile import ProfileStore
 from .registration import RegistrationStore
 from .room import RoomStore
 from .roommember import RoomMemberStore
-from .roomdata import RoomDataStore
 from .stream import StreamStore
 from .pdu import StatePduStore, PduStore
 from .transactions import TransactionStore
 
 import json
+import logging
 import os
 
 
-class DataStore(RoomDataStore, RoomMemberStore, MessageStore, RoomStore,
+logger = logging.getLogger(__name__)
+
+
+class DataStore(RoomMemberStore, RoomStore,
                 RegistrationStore, StreamStore, ProfileStore, FeedbackStore,
                 PresenceStore, PduStore, StatePduStore, TransactionStore,
                 DirectoryStore):
@@ -44,51 +49,147 @@ class DataStore(RoomDataStore, RoomMemberStore, MessageStore, RoomStore,
     def __init__(self, hs):
         super(DataStore, self).__init__(hs)
         self.event_factory = hs.get_event_factory()
+        self.hs = hs
 
-    def persist_event(self, event):
-        if event.type == MessageEvent.TYPE:
-            return self.store_message(
-                user_id=event.user_id,
-                room_id=event.room_id,
-                msg_id=event.msg_id,
-                content=json.dumps(event.content)
-            )
-        elif event.type == RoomMemberEvent.TYPE:
-            return self.store_room_member(
-                user_id=event.target_user_id,
-                sender=event.user_id,
-                room_id=event.room_id,
-                content=event.content,
-                membership=event.content["membership"]
-            )
+        self.min_token_deferred = self._get_min_token()
+        self.min_token = None
+
+    @defer.inlineCallbacks
+    @log_function
+    def persist_event(self, event, backfilled=False):
+        if event.type == RoomMemberEvent.TYPE:
+            yield self._store_room_member(event)
         elif event.type == FeedbackEvent.TYPE:
-            return self.store_feedback(
-                room_id=event.room_id,
-                msg_id=event.msg_id,
-                msg_sender_id=event.msg_sender_id,
-                fb_sender_id=event.user_id,
-                fb_type=event.feedback_type,
-                content=json.dumps(event.content)
-            )
+            yield self._store_feedback(event)
+#        elif event.type == RoomConfigEvent.TYPE:
+#            yield self._store_room_config(event)
+        elif event.type == RoomNameEvent.TYPE:
+            yield self._store_room_name(event)
         elif event.type == RoomTopicEvent.TYPE:
-            return self.store_room_data(
-                room_id=event.room_id,
-                etype=event.type,
-                state_key=event.state_key,
-                content=json.dumps(event.content)
-            )
-        elif event.type == RoomConfigEvent.TYPE:
-            if "visibility" in event.content:
-                visibility = event.content["visibility"]
-                return self.store_room_config(
-                    room_id=event.room_id,
-                    visibility=visibility
-                )
+            yield self._store_room_topic(event)
 
+        ret = yield self._store_event(event, backfilled)
+        defer.returnValue(ret)
+
+    @defer.inlineCallbacks
+    def get_event(self, event_id):
+        events_dict = yield self._simple_select_one(
+            "events",
+            {"event_id": event_id},
+            [
+                "event_id",
+                "type",
+                "sender",
+                "room_id",
+                "content",
+                "unrecognized_keys"
+            ],
+        )
+
+        event = self._parse_event_from_row(events_dict)
+        defer.returnValue(event)
+
+    @defer.inlineCallbacks
+    @log_function
+    def _store_event(self, event, backfilled):
+        # FIXME (erikj): This should be removed when we start amalgamating
+        # event and pdu storage
+        yield self.hs.get_federation().fill_out_prev_events(event)
+
+        vals = {
+            "topological_ordering": event.depth,
+            "event_id": event.event_id,
+            "type": event.type,
+            "room_id": event.room_id,
+            "content": json.dumps(event.content),
+            "processed": True,
+        }
+
+        if hasattr(event, "outlier"):
+            vals["outlier"] = event.outlier
         else:
-            raise NotImplementedError(
-                "Don't know how to persist type=%s" % event.type
+            vals["outlier"] = False
+
+        if backfilled:
+            if not self.min_token_deferred.called:
+                yield self.min_token_deferred
+            self.min_token -= 1
+            vals["stream_ordering"] = self.min_token
+
+        unrec = {
+            k: v
+            for k, v in event.get_full_dict().items()
+            if k not in vals.keys()
+        }
+        vals["unrecognized_keys"] = json.dumps(unrec)
+
+        try:
+            yield self._simple_insert("events", vals)
+        except:
+            logger.exception(
+                "Failed to persist, probably duplicate: %s",
+                event.event_id
             )
+            return
+
+        if not backfilled and hasattr(event, "state_key"):
+            vals = {
+                "event_id": event.event_id,
+                "room_id": event.room_id,
+                "type": event.type,
+                "state_key": event.state_key,
+            }
+
+            if hasattr(event, "prev_state"):
+                vals["prev_state"] = event.prev_state
+
+            yield self._simple_insert("state_events", vals)
+
+            yield self._simple_insert(
+                "current_state_events",
+                {
+                    "event_id": event.event_id,
+                    "room_id": event.room_id,
+                    "type": event.type,
+                    "state_key": event.state_key,
+                }
+            )
+
+        latest = yield self.get_room_events_max_id()
+        defer.returnValue(latest)
+
+    @defer.inlineCallbacks
+    def get_current_state(self, room_id, event_type=None, state_key=""):
+        sql = (
+            "SELECT e.* FROM events as e "
+            "INNER JOIN current_state_events as c ON e.event_id = c.event_id "
+            "INNER JOIN state_events as s ON e.event_id = s.event_id "
+            "WHERE c.room_id = ? "
+        )
+
+        if event_type:
+            sql += " AND s.type = ? AND s.state_key = ? "
+            args = (room_id, event_type, state_key)
+        else:
+            args = (room_id, )
+
+        results = yield self._execute_and_decode(sql, *args)
+
+        defer.returnValue([self._parse_event_from_row(r) for r in results])
+
+    @defer.inlineCallbacks
+    def _get_min_token(self):
+        row = yield self._execute(
+            None,
+            "SELECT MIN(stream_ordering) FROM events"
+        )
+
+        self.min_token = row[0][0] if row and row[0] and row[0][0] else -1
+        self.min_token = min(self.min_token, -1)
+
+        logger.debug("min_token is: %s", self.min_token)
+
+        defer.returnValue(self.min_token)
 
 
 def schema_path(schema):

synapse/storage/_base.py

@@ -12,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
 import logging
 
 from twisted.internet import defer
@@ -20,6 +19,9 @@ from twisted.internet import defer
 from synapse.api.errors import StoreError
 
 import collections
+import copy
+import json
+
 
 logger = logging.getLogger(__name__)
 
@@ -29,6 +31,7 @@ class SQLBaseStore(object):
     def __init__(self, hs):
         self.hs = hs
         self._db_pool = hs.get_db_pool()
+        self.event_factory = hs.get_event_factory()
         self._clock = hs.get_clock()
 
     def cursor_to_dict(self, cursor):
@@ -57,14 +60,22 @@ class SQLBaseStore(object):
             The result of decoder(results)
         """
         logger.debug(
-            "[SQL] %s  Args=%s Func=%s", query, args, decoder.__name__
+            "[SQL] %s  Args=%s Func=%s",
+            query, args, decoder.__name__ if decoder else None
         )
 
         def interaction(txn):
             cursor = txn.execute(query, args)
-            return decoder(cursor)
+            if decoder:
+                return decoder(cursor)
+            else:
+                return cursor.fetchall()
+
         return self._db_pool.runInteraction(interaction)
 
+    def _execute_and_decode(self, query, *args):
+        return self._execute(self.cursor_to_dict, query, *args)
+
     # "Simple" SQL API methods that operate on a single table with no JOINs,
     # no complex WHERE clauses, just a dict of values for columns.
 
@@ -281,6 +292,22 @@ class SQLBaseStore(object):
 
         return self._db_pool.runInteraction(func)
 
+    def _parse_event_from_row(self, row_dict):
+        d = copy.deepcopy({k: v for k, v in row_dict.items() if v})
+
+        d.pop("stream_ordering", None)
+        d.pop("topological_ordering", None)
+        d.pop("processed", None)
+
+        d.update(json.loads(row_dict["unrecognized_keys"]))
+        d["content"] = json.loads(d["content"])
+        del d["unrecognized_keys"]
+
+        return self.event_factory.create_event(
+            etype=d["type"],
+            **d
+        )
+
 
 class Table(object):
     """ A base class used to store information about a particular table.

synapse/storage/feedback.py

@@ -13,6 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
+
 from ._base import SQLBaseStore, Table
 from synapse.api.events.room import FeedbackEvent
 
@@ -22,54 +24,28 @@ import json
 
 class FeedbackStore(SQLBaseStore):
 
-    def store_feedback(self, room_id, msg_id, msg_sender_id,
-                       fb_sender_id, fb_type, content):
-        return self._simple_insert(FeedbackTable.table_name, dict(
-            room_id=room_id,
-            msg_id=msg_id,
-            msg_sender_id=msg_sender_id,
-            fb_sender_id=fb_sender_id,
-            fb_type=fb_type,
-            content=content,
-        ))
+    def _store_feedback(self, event):
+        return self._simple_insert("feedback", {
+            "event_id": event.event_id,
+            "feedback_type": event.feedback_type,
+            "room_id": event.room_id,
+            "target_event_id": event.target_event,
+            "sender": event.user_id,
+        })
 
-    def get_feedback(self, room_id=None, msg_id=None, msg_sender_id=None,
-                     fb_sender_id=None, fb_type=None):
-        query = FeedbackTable.select_statement(
-            "msg_sender_id = ? AND room_id = ? AND msg_id = ? " +
-            "AND fb_sender_id = ? AND feedback_type = ? " +
-            "ORDER BY id DESC LIMIT 1")
-        return self._execute(
-            FeedbackTable.decode_single_result,
-            query, msg_sender_id, room_id, msg_id, fb_sender_id, fb_type,
+    @defer.inlineCallbacks
+    def get_feedback_for_event(self, event_id):
+        sql = (
+            "SELECT events.* FROM events INNER JOIN feedback "
+            "ON events.event_id = feedback.event_id "
+            "WHERE feedback.target_event_id = ? "
         )
 
-    def get_max_feedback_id(self):
-        return self._simple_max_id(FeedbackTable.table_name)
+        rows = yield self._execute_and_decode(sql, event_id)
 
-
-class FeedbackTable(Table):
-    table_name = "feedback"
-
-    fields = [
-        "id",
-        "content",
-        "feedback_type",
-        "fb_sender_id",
-        "msg_id",
-        "room_id",
-        "msg_sender_id"
-    ]
-
-    class EntryType(collections.namedtuple("FeedbackEntry", fields)):
-
-        def as_event(self, event_factory):
-            return event_factory.create_event(
-                etype=FeedbackEvent.TYPE,
-                room_id=self.room_id,
-                msg_id=self.msg_id,
-                msg_sender_id=self.msg_sender_id,
-                user_id=self.fb_sender_id,
-                feedback_type=self.feedback_type,
-                content=json.loads(self.content),
-            )
+        defer.returnValue(
+            [
+                self._parse_event_from_row(r)
+                for r in rows
+            ]
+        )

synapse/storage/message.py

@@ -1,81 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import SQLBaseStore, Table
-from synapse.api.events.room import MessageEvent
-
-import collections
-import json
-
-
-class MessageStore(SQLBaseStore):
-
-    def get_message(self, user_id, room_id, msg_id):
-        """Get a message from the store.
-
-        Args:
-            user_id (str): The ID of the user who sent the message.
-            room_id (str): The room the message was sent in.
-            msg_id (str): The unique ID for this user/room combo.
-        """
-        query = MessagesTable.select_statement(
-            "user_id = ? AND room_id = ? AND msg_id = ? " +
-            "ORDER BY id DESC LIMIT 1")
-        return self._execute(
-            MessagesTable.decode_single_result,
-            query, user_id, room_id, msg_id,
-        )
-
-    def store_message(self, user_id, room_id, msg_id, content):
-        """Store a message in the store.
-
-        Args:
-            user_id (str): The ID of the user who sent the message.
-            room_id (str): The room the message was sent in.
-            msg_id (str): The unique ID for this user/room combo.
-            content (str): The content of the message (JSON)
-        """
-        return self._simple_insert(MessagesTable.table_name, dict(
-            user_id=user_id,
-            room_id=room_id,
-            msg_id=msg_id,
-            content=content,
-        ))
-
-    def get_max_message_id(self):
-        return self._simple_max_id(MessagesTable.table_name)
-
-
-class MessagesTable(Table):
-    table_name = "messages"
-
-    fields = [
-        "id",
-        "user_id",
-        "room_id",
-        "msg_id",
-        "content"
-    ]
-
-    class EntryType(collections.namedtuple("MessageEntry", fields)):
-
-        def as_event(self, event_factory):
-            return event_factory.create_event(
-                etype=MessageEvent.TYPE,
-                room_id=self.room_id,
-                user_id=self.user_id,
-                msg_id=self.msg_id,
-                content=json.loads(self.content),
-            )

synapse/storage/pdu.py

@@ -13,6 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
+
 from ._base import SQLBaseStore, Table, JoinHelper
 
 from synapse.util.logutils import log_function
@@ -319,6 +321,7 @@ class PduStore(SQLBaseStore):
 
         return [(row[0], row[1], row[2]) for row in results]
 
+    @defer.inlineCallbacks
     def get_oldest_pdus_in_context(self, context):
         """Get a list of Pdus that we haven't backfilled beyond yet (and haven't    
         seen). This list is used when we want to backfill backwards and is the 
@@ -331,17 +334,14 @@ class PduStore(SQLBaseStore):
         Returns:
             list: A list of PduIdTuple.
         """
-        return self._db_pool.runInteraction(
-            self._get_oldest_pdus_in_context, context
-        )
-
-    def _get_oldest_pdus_in_context(self, txn, context):
-        txn.execute(
+        results = yield self._execute(
+            None,
             "SELECT pdu_id, origin FROM %(back)s WHERE context = ?"
             % {"back": PduBackwardExtremitiesTable.table_name, },
-            (context,)
+            context
         )
-        return [PduIdTuple(i, o) for i, o in txn.fetchall()]
+
+        defer.returnValue([PduIdTuple(i, o) for i, o in results])
 
     def is_pdu_new(self, pdu_id, origin, context, depth):
         """For a given Pdu, try and figure out if it's 'new', i.e., if it's
@@ -580,7 +580,7 @@ class StatePduStore(SQLBaseStore):
 
         txn.execute(query, query_args)
 
-    def get_current_state(self, context, pdu_type, state_key):
+    def get_current_state_pdu(self, context, pdu_type, state_key):
         """For a given context, pdu_type, state_key 3-tuple, return what is
         currently considered the current state.
 
@@ -595,10 +595,10 @@ class StatePduStore(SQLBaseStore):
         """
 
         return self._db_pool.runInteraction(
-            self._get_current_state, context, pdu_type, state_key
+            self._get_current_state_pdu, context, pdu_type, state_key
         )
 
-    def _get_current_state(self, txn, context, pdu_type, state_key):
+    def _get_current_state_pdu(self, txn, context, pdu_type, state_key):
         return self._get_current_interaction(txn, context, pdu_type, state_key)
 
     def _get_current_interaction(self, txn, context, pdu_type, state_key):

synapse/storage/presence.py

@@ -67,6 +67,7 @@ class PresenceStore(SQLBaseStore):
             table="presence_allow_inbound",
             keyvalues={"observed_user_id": observed_localpart,
                        "observer_user_id": observer_userid},
+            retcols=["observed_user_id"],
             allow_none=True,
         )
 

synapse/storage/registration.py

@@ -17,7 +17,7 @@ from twisted.internet import defer
 
 from sqlite3 import IntegrityError
 
-from synapse.api.errors import StoreError
+from synapse.api.errors import StoreError, Codes
 
 from ._base import SQLBaseStore
 
@@ -73,7 +73,7 @@ class RegistrationStore(SQLBaseStore):
                         "VALUES (?,?,?)",
                         [user_id, password_hash, now])
         except IntegrityError:
-            raise StoreError(400, "User ID already taken.")
+            raise StoreError(400, "User ID already taken.", errcode=Codes.USER_IN_USE)
 
         # it's possible for this to get a conflict, but only for a single user
         # since tokens are namespaced based on their user ID

synapse/storage/room.py

@@ -76,49 +76,80 @@ class RoomStore(SQLBaseStore):
         )
 
     @defer.inlineCallbacks
-    def get_rooms(self, is_public, with_topics):
+    def get_rooms(self, is_public):
         """Retrieve a list of all public rooms.
 
         Args:
             is_public (bool): True if the rooms returned should be public.
-            with_topics (bool): True to include the current topic for the room
-            in the response.
         Returns:
-            A list of room dicts containing at least a "room_id" key, and a
-            "topic" key if one is set and with_topic=True.
+            A list of room dicts containing at least a "room_id" key, a
+            "topic" key if one is set, and a "name" key if one is set
         """
-        room_data_type = RoomTopicEvent.TYPE
-        public = 1 if is_public else 0
 
-        latest_topic = ("SELECT max(room_data.id) FROM room_data WHERE "
-                        + "room_data.type = ? GROUP BY room_id")
-
-        query = ("SELECT rooms.*, room_data.content, room_alias FROM rooms "
-                 + "LEFT JOIN "
-                 + "room_aliases ON room_aliases.room_id = rooms.room_id "
-                 + "LEFT JOIN "
-                 + "room_data ON rooms.room_id = room_data.room_id WHERE "
-                 + "(room_data.id IN (" + latest_topic + ") "
-                 + "OR room_data.id IS NULL) AND rooms.is_public = ?")
-
-        res = yield self._execute(
-            self.cursor_to_dict, query, room_data_type, public
+        topic_subquery = (
+            "SELECT topics.event_id as event_id, "
+            "topics.room_id as room_id, topic "
+            "FROM topics "
+            "INNER JOIN current_state_events as c "
+            "ON c.event_id = topics.event_id "
         )
 
-        # return only the keys the specification expects
-        ret_keys = ["room_id", "topic", "room_alias"]
+        name_subquery = (
+            "SELECT room_names.event_id as event_id, "
+            "room_names.room_id as room_id, name "
+            "FROM room_names "
+            "INNER JOIN current_state_events as c "
+            "ON c.event_id = room_names.event_id "
+        )
 
-        # extract topic from the json (icky) FIXME
-        for i, room_row in enumerate(res):
-            try:
-                content_json = json.loads(room_row["content"])
-                room_row["topic"] = content_json["topic"]
-            except:
-                pass  # no topic set
-            # filter the dict based on ret_keys
-            res[i] = {k: v for k, v in room_row.iteritems() if k in ret_keys}
+        # We use non printing ascii character US () as a seperator
+        sql = (
+            "SELECT r.room_id, n.name, t.topic, "
+            "group_concat(a.room_alias, '') "
+            "FROM rooms AS r "
+            "LEFT JOIN (%(topic)s) AS t ON t.room_id = r.room_id "
+            "LEFT JOIN (%(name)s) AS n ON n.room_id = r.room_id "
+            "INNER JOIN room_aliases AS a ON a.room_id = r.room_id "
+            "WHERE r.is_public = ? "
+            "GROUP BY r.room_id "
+        ) % {
+            "topic": topic_subquery,
+            "name": name_subquery,
+        }
 
-        defer.returnValue(res)
+        rows = yield self._execute(None, sql, is_public)
+
+        ret = [
+            {
+                "room_id": r[0],
+                "name": r[1],
+                "topic": r[2],
+                "aliases": r[3].split(""),
+            }
+            for r in rows
+        ]
+
+        defer.returnValue(ret)
+
+    def _store_room_topic(self, event):
+        return self._simple_insert(
+            "topics",
+            {
+                "event_id": event.event_id,
+                "room_id": event.room_id,
+                "topic": event.topic,
+            }
+        )
+
+    def _store_room_name(self, event):
+        return self._simple_insert(
+            "room_names",
+            {
+                "event_id": event.event_id,
+                "room_id": event.room_id,
+                "name": event.name,
+            }
+        )
 
 
 class RoomsTable(Table):

synapse/storage/roomdata.py

@@ -1,85 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import SQLBaseStore, Table
-
-import collections
-import json
-
-
-class RoomDataStore(SQLBaseStore):
-
-    """Provides various CRUD operations for Room Events. """
-
-    def get_room_data(self, room_id, etype, state_key=""):
-        """Retrieve the data stored under this type and state_key.
-
-        Args:
-            room_id (str)
-            etype (str)
-            state_key (str)
-        Returns:
-            namedtuple: Or None if nothing exists at this path.
-        """
-        query = RoomDataTable.select_statement(
-            "room_id = ? AND type = ? AND state_key = ? "
-            "ORDER BY id DESC LIMIT 1"
-        )
-        return self._execute(
-            RoomDataTable.decode_single_result,
-            query, room_id, etype, state_key,
-        )
-
-    def store_room_data(self, room_id, etype, state_key="", content=None):
-        """Stores room specific data.
-
-        Args:
-            room_id (str)
-            etype (str)
-            state_key (str)
-            data (str)- The data to store for this path in JSON.
-        Returns:
-            The store ID for this data.
-        """
-        return self._simple_insert(RoomDataTable.table_name, dict(
-            etype=etype,
-            state_key=state_key,
-            room_id=room_id,
-            content=content,
-        ))
-
-    def get_max_room_data_id(self):
-        return self._simple_max_id(RoomDataTable.table_name)
-
-
-class RoomDataTable(Table):
-    table_name = "room_data"
-
-    fields = [
-        "id",
-        "room_id",
-        "type",
-        "state_key",
-        "content"
-    ]
-
-    class EntryType(collections.namedtuple("RoomDataEntry", fields)):
-
-        def as_event(self, event_factory):
-            return event_factory.create_event(
-                etype=self.type,
-                room_id=self.room_id,
-                content=json.loads(self.content),
-            )

synapse/storage/roommember.py

@@ -31,6 +31,38 @@ logger = logging.getLogger(__name__)
 
 class RoomMemberStore(SQLBaseStore):
 
+    @defer.inlineCallbacks
+    def _store_room_member(self, event):
+        """Store a room member in the database.
+        """
+        domain = self.hs.parse_userid(event.target_user_id).domain
+
+        yield self._simple_insert(
+            "room_memberships",
+            {
+                "event_id": event.event_id,
+                "user_id": event.target_user_id,
+                "sender": event.user_id,
+                "room_id": event.room_id,
+                "membership": event.membership,
+            }
+        )
+
+        # Update room hosts table
+        if event.membership == Membership.JOIN:
+            sql = (
+                "INSERT OR IGNORE INTO room_hosts (room_id, host) "
+                "VALUES (?, ?)"
+            )
+            yield self._execute(None, sql, event.room_id, domain)
+        else:
+            sql = (
+                "DELETE FROM room_hosts WHERE room_id = ? AND host = ?"
+            )
+
+            yield self._execute(None, sql, event.room_id, domain)
+
+    @defer.inlineCallbacks
     def get_room_member(self, user_id, room_id):
         """Retrieve the current state of a room member.
 
@@ -38,36 +70,15 @@ class RoomMemberStore(SQLBaseStore):
             user_id (str): The member's user ID.
             room_id (str): The room the member is in.
         Returns:
-            namedtuple: The room member from the database, or None if this
-            member does not exist.
+            Deferred: Results in a MembershipEvent or None.
         """
-        query = RoomMemberTable.select_statement(
-            "room_id = ? AND user_id = ? ORDER BY id DESC LIMIT 1")
-        return self._execute(
-            RoomMemberTable.decode_single_result,
-            query, room_id, user_id,
-        )
+        rows = yield self._get_members_by_dict({
+            "e.room_id": room_id,
+            "m.user_id": user_id,
+        })
 
-    def store_room_member(self, user_id, sender, room_id, membership, content):
-        """Store a room member in the database.
+        defer.returnValue(rows[0] if rows else None)
 
-        Args:
-            user_id (str): The member's user ID.
-            room_id (str): The room in relation to the member.
-            membership (synapse.api.constants.Membership): The new membership
-            state.
-            content (dict): The content of the membership (JSON).
-        """
-        content_json = json.dumps(content)
-        return self._simple_insert(RoomMemberTable.table_name, dict(
-            user_id=user_id,
-            sender=sender,
-            room_id=room_id,
-            membership=membership,
-            content=content_json,
-        ))
-
-    @defer.inlineCallbacks
     def get_room_members(self, room_id, membership=None):
         """Retrieve the current room member list for a room.
 
@@ -79,17 +90,12 @@ class RoomMemberStore(SQLBaseStore):
         Returns:
             list of namedtuples representing the members in this room.
         """
-        query = RoomMemberTable.select_statement(
-            "id IN (SELECT MAX(id) FROM " + RoomMemberTable.table_name
-            + " WHERE room_id = ? GROUP BY user_id)"
-        )
-        res = yield self._execute(
-            RoomMemberTable.decode_results, query, room_id,
-        )
-        # strip memberships which don't match
+
+        where = {"m.room_id": room_id}
         if membership:
-            res = [entry for entry in res if entry.membership == membership]
-        defer.returnValue(res)
+            where["m.membership"] = membership
+
+        return self._get_members_by_dict(where)
 
     def get_rooms_for_user_where_membership_is(self, user_id, membership_list):
         """ Get all the rooms for this user where the membership for this user
@@ -106,67 +112,40 @@ class RoomMemberStore(SQLBaseStore):
             return defer.succeed(None)
 
         args = [user_id]
-        membership_placeholder = ["membership=?"] * len(membership_list)
-        where_membership = "(" + " OR ".join(membership_placeholder) + ")"
-        for membership in membership_list:
-            args.append(membership)
+        args.extend(membership_list)
 
-        query = ("SELECT room_id, membership FROM room_memberships"
-                 + " WHERE user_id=? AND " + where_membership
-                 + " GROUP BY room_id ORDER BY id DESC")
-        return self._execute(
-            self.cursor_to_dict, query, *args
+        where_clause = "user_id = ? AND (%s)" % (
+            " OR ".join(["membership = ?" for _ in membership_list]),
         )
 
+        return self._get_members_query(where_clause, args)
+
+    def get_joined_hosts_for_room(self, room_id):
+        return self._simple_select_onecol(
+            "room_hosts",
+            {"room_id": room_id},
+            "host"
+        )
+
+    def _get_members_by_dict(self, where_dict):
+        clause = " AND ".join("%s = ?" % k for k in where_dict.keys())
+        vals = where_dict.values()
+        return self._get_members_query(clause, vals)
+
     @defer.inlineCallbacks
-    def get_joined_hosts_for_room(self, room_id):
-        query = RoomMemberTable.select_statement(
-            "id IN (SELECT MAX(id) FROM " + RoomMemberTable.table_name
-            + " WHERE room_id = ? GROUP BY user_id)"
-        )
+    def _get_members_query(self, where_clause, where_values):
+        sql = (
+            "SELECT e.* FROM events as e "
+            "INNER JOIN room_memberships as m "
+            "ON e.event_id = m.event_id "
+            "INNER JOIN current_state_events as c "
+            "ON m.event_id = c.event_id "
+            "WHERE %s "
+        ) % (where_clause,)
 
-        res = yield self._execute(
-            RoomMemberTable.decode_results, query, room_id,
-        )
+        rows = yield self._execute_and_decode(sql, *where_values)
 
-        def host_from_user_id_string(user_id):
-            domain = UserID.from_string(entry.user_id, self.hs).domain
-            return domain
+        # logger.debug("_get_members_query Got rows %s", rows)
 
-        # strip memberships which don't match
-        hosts = [
-            host_from_user_id_string(entry.user_id)
-            for entry in res
-            if entry.membership == Membership.JOIN
-        ]
-
-        logger.debug("Returning hosts: %s from results: %s", hosts, res)
-
-        defer.returnValue(hosts)
-
-    def get_max_room_member_id(self):
-        return self._simple_max_id(RoomMemberTable.table_name)
-
-
-class RoomMemberTable(Table):
-    table_name = "room_memberships"
-
-    fields = [
-        "id",
-        "user_id",
-        "sender",
-        "room_id",
-        "membership",
-        "content"
-    ]
-
-    class EntryType(collections.namedtuple("RoomMemberEntry", fields)):
-
-        def as_event(self, event_factory):
-            return event_factory.create_event(
-                etype=RoomMemberEvent.TYPE,
-                room_id=self.room_id,
-                target_user_id=self.user_id,
-                user_id=self.sender,
-                content=json.loads(self.content),
-            )
+        results = [self._parse_event_from_row(r) for r in rows]
+        defer.returnValue(results)

synapse/storage/schema/im.sql

@@ -12,43 +12,91 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
+
+CREATE TABLE IF NOT EXISTS events(
+    stream_ordering INTEGER PRIMARY KEY AUTOINCREMENT,
+    topological_ordering INTEGER NOT NULL,
+    event_id TEXT NOT NULL,
+    type TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    content TEXT NOT NULL,
+    unrecognized_keys TEXT,
+    processed BOOL NOT NULL,
+    outlier BOOL NOT NULL,
+    CONSTRAINT ev_uniq UNIQUE (event_id)
+);
+
+CREATE INDEX IF NOT EXISTS events_event_id ON events (event_id);
+CREATE INDEX IF NOT EXISTS events_stream_ordering ON events (stream_ordering);
+CREATE INDEX IF NOT EXISTS events_topological_ordering ON events (topological_ordering);
+CREATE INDEX IF NOT EXISTS events_room_id ON events (room_id);
+
+CREATE TABLE IF NOT EXISTS state_events(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    type TEXT NOT NULL,
+    state_key TEXT NOT NULL,
+    prev_state TEXT
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS state_events_event_id ON state_events (event_id);
+CREATE INDEX IF NOT EXISTS state_events_room_id ON state_events (room_id);
+CREATE INDEX IF NOT EXISTS state_events_type ON state_events (type);
+CREATE INDEX IF NOT EXISTS state_events_state_key ON state_events (state_key);
+
+
+CREATE TABLE IF NOT EXISTS current_state_events(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    type TEXT NOT NULL,
+    state_key TEXT NOT NULL,
+    CONSTRAINT curr_uniq UNIQUE (room_id, type, state_key) ON CONFLICT REPLACE
+);
+
+CREATE INDEX IF NOT EXISTS curr_events_event_id ON current_state_events (event_id);
+CREATE INDEX IF NOT EXISTS current_state_events_room_id ON current_state_events (room_id);
+CREATE INDEX IF NOT EXISTS current_state_events_type ON current_state_events (type);
+CREATE INDEX IF NOT EXISTS current_state_events_state_key ON current_state_events (state_key);
+
+CREATE TABLE IF NOT EXISTS room_memberships(
+    event_id TEXT NOT NULL,
+    user_id TEXT NOT NULL,
+    sender TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    membership TEXT NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS room_memberships_event_id ON room_memberships (event_id);
+CREATE INDEX IF NOT EXISTS room_memberships_room_id ON room_memberships (room_id);
+CREATE INDEX IF NOT EXISTS room_memberships_user_id ON room_memberships (user_id);
+
+CREATE TABLE IF NOT EXISTS feedback(
+    event_id TEXT NOT NULL,
+    feedback_type TEXT,
+    target_event_id TEXT,
+    sender TEXT,
+    room_id TEXT
+);
+
+CREATE TABLE IF NOT EXISTS topics(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    topic TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS room_names(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    name TEXT NOT NULL
+);
+
 CREATE TABLE IF NOT EXISTS rooms(
     room_id TEXT PRIMARY KEY NOT NULL,
     is_public INTEGER,
     creator TEXT
 );
 
-CREATE TABLE IF NOT EXISTS room_memberships(
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    user_id TEXT NOT NULL, -- no foreign key to users table, it could be an id belonging to another home server
-    sender TEXT NOT NULL,
+CREATE TABLE IF NOT EXISTS room_hosts(
     room_id TEXT NOT NULL,
-    membership TEXT NOT NULL,
-    content TEXT NOT NULL
-);
-
-CREATE TABLE IF NOT EXISTS messages(
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    user_id TEXT, 
-    room_id TEXT,
-    msg_id TEXT,
-    content TEXT
-);
-
-CREATE TABLE IF NOT EXISTS feedback(
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    content TEXT,
-    feedback_type TEXT,
-    fb_sender_id TEXT,
-    msg_id TEXT,
-    room_id TEXT,
-    msg_sender_id TEXT
-);
-
-CREATE TABLE IF NOT EXISTS room_data(
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    room_id TEXT NOT NULL,
-    type TEXT NOT NULL,
-    state_key TEXT NOT NULL,
-    content TEXT
+    host TEXT NOT NULL
 );

synapse/storage/stream.py

@@ -13,271 +13,287 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+""" This module is responsible for getting events from the DB for pagination
+and event streaming.
+
+The order it returns events in depend on whether we are streaming forwards or
+are paginating backwards. We do this because we want to handle out of order
+messages nicely, while still returning them in the correct order when we
+paginate bacwards.
+
+This is implemented by keeping two ordering columns: stream_ordering and
+topological_ordering. Stream ordering is basically insertion/received order
+(except for events from backfill requests). The topolgical_ordering is a
+weak ordering of events based on the pdu graph.
+
+This means that we have to have two different types of tokens, depending on
+what sort order was used:
+    - stream tokens are of the form: "s%d", which maps directly to the column
+    - topological tokems: "t%d-%d", where the integers map to the topological
+      and stream ordering columns respectively.
+"""
+
+from twisted.internet import defer
 
 from ._base import SQLBaseStore
-from .message import MessagesTable
-from .feedback import FeedbackTable
-from .roomdata import RoomDataTable
-from .roommember import RoomMemberTable
+from synapse.api.errors import SynapseError
+from synapse.api.constants import Membership
+from synapse.util.logutils import log_function
 
 import json
 import logging
 
+
 logger = logging.getLogger(__name__)
 
 
+MAX_STREAM_SIZE = 1000
+
+
+_STREAM_TOKEN = "stream"
+_TOPOLOGICAL_TOKEN = "topological"
+
+
+def _parse_stream_token(string):
+    try:
+        if string[0] != 's':
+            raise
+        return int(string[1:])
+    except:
+        raise SynapseError(400, "Invalid token")
+
+
+def _parse_topological_token(string):
+    try:
+        if string[0] != 't':
+            raise
+        parts = string[1:].split('-', 1)
+        return (int(parts[0]), int(parts[1]))
+    except:
+        raise SynapseError(400, "Invalid token")
+
+
+def is_stream_token(string):
+    try:
+        _parse_stream_token(string)
+        return True
+    except:
+        return False
+
+
+def is_topological_token(string):
+    try:
+        _parse_topological_token(string)
+        return True
+    except:
+        return False
+
+
+def _get_token_bound(token, comparison):
+    try:
+        s = _parse_stream_token(token)
+        return "%s %s %d" % ("stream_ordering", comparison, s)
+    except:
+        pass
+
+    try:
+        top, stream = _parse_topological_token(token)
+        return "%s %s %d AND %s %s %d" % (
+            "topological_ordering", comparison, top,
+            "stream_ordering", comparison, stream,
+        )
+    except:
+        pass
+
+    raise SynapseError(400, "Invalid token")
+
+
 class StreamStore(SQLBaseStore):
+    @log_function
+    def get_room_events(self, user_id, from_key, to_key, room_id, limit=0,
+                        direction='f', with_feedback=False):
+        # We deal with events request in two different ways depending on if
+        # this looks like an /events request or a pagination request.
+        is_events = (
+            direction == 'f'
+            and user_id
+            and is_stream_token(from_key)
+            and to_key and is_stream_token(to_key)
+        )
 
-    def get_message_stream(self, user_id, from_key, to_key, room_id, limit=0,
-                           with_feedback=False):
-        """Get all messages for this user between the given keys.
-
-        Args:
-            user_id (str): The user who is requesting messages.
-            from_key (int): The ID to start returning results from (exclusive).
-            to_key (int): The ID to stop returning results (exclusive).
-            room_id (str): Gets messages only for this room. Can be None, in
-            which case all room messages will be returned.
-        Returns:
-            A tuple of rows (list of namedtuples), new_id(int)
-        """
-        if with_feedback and room_id:  # with fb MUST specify a room ID
-            return self._db_pool.runInteraction(
-                self._get_message_rows_with_feedback,
-                user_id, from_key, to_key, room_id, limit
+        if is_events:
+            return self.get_room_events_stream(
+                user_id=user_id,
+                from_key=from_key,
+                to_key=to_key,
+                room_id=room_id,
+                limit=limit,
+                with_feedback=with_feedback,
             )
         else:
-            return self._db_pool.runInteraction(
-                self._get_message_rows,
-                user_id, from_key, to_key, room_id, limit
+            return self.paginate_room_events(
+                from_key=from_key,
+                to_key=to_key,
+                room_id=room_id,
+                limit=limit,
+                with_feedback=with_feedback,
             )
 
-    def _get_message_rows(self, txn, user_id, from_pkey, to_pkey, room_id,
-                          limit):
-        # work out which rooms this user is joined in on and join them with
-        # the room id on the messages table, bounded by the specified pkeys
+    @defer.inlineCallbacks
+    @log_function
+    def get_room_events_stream(self, user_id, from_key, to_key, room_id,
+                               limit=0, with_feedback=False):
+        # TODO (erikj): Handle compressed feedback
 
-        # get all messages where the *current* membership state is 'join' for
-        # this user in that room.
-        query = ("SELECT messages.* FROM messages WHERE ? IN"
-                 + " (SELECT membership from room_memberships WHERE user_id=?"
-                 + " AND room_id = messages.room_id ORDER BY id DESC LIMIT 1)")
-        query_args = ["join", user_id]
-
-        if room_id:
-            query += " AND messages.room_id=?"
-            query_args.append(room_id)
-
-        (query, query_args) = self._append_stream_operations(
-            "messages", query, query_args, from_pkey, to_pkey, limit=limit
+        current_room_membership_sql = (
+            "SELECT m.room_id FROM room_memberships as m "
+            "INNER JOIN current_state_events as c ON m.event_id = c.event_id "
+            "WHERE m.user_id = ?"
         )
 
-        logger.debug("[SQL] %s : %s", query, query_args)
-        cursor = txn.execute(query, query_args)
-        return self._as_events(cursor, MessagesTable, from_pkey)
-
-    def _get_message_rows_with_feedback(self, txn, user_id, from_pkey, to_pkey,
-                                        room_id, limit):
-        # this col represents the compressed feedback JSON as per spec
-        compressed_feedback_col = (
-            "'[' || group_concat('{\"sender_id\":\"' || f.fb_sender_id"
-            + " || '\",\"feedback_type\":\"' || f.feedback_type"
-            + " || '\",\"content\":' || f.content || '}') || ']'"
+        # We also want to get any membership events about that user, e.g.
+        # invites or leave notifications.
+        membership_sql = (
+            "SELECT m.event_id FROM room_memberships as m "
+            "INNER JOIN current_state_events as c ON m.event_id = c.event_id "
+            "WHERE m.user_id = ? "
         )
 
-        global_msg_id_join = ("f.room_id = messages.room_id"
-                              + " and f.msg_id = messages.msg_id"
-                              + " and messages.user_id = f.msg_sender_id")
+        if limit:
+            limit = max(limit, MAX_STREAM_SIZE)
+        else:
+            limit = MAX_STREAM_SIZE
 
-        select_query = (
-            "SELECT messages.*, f.content AS fb_content, f.fb_sender_id"
-            + ", " + compressed_feedback_col + " AS compressed_fb"
-            + " FROM messages LEFT JOIN feedback f ON " + global_msg_id_join)
+        # From and to keys should be integers from ordering.
+        from_id = _parse_stream_token(from_key)
+        to_id = _parse_stream_token(to_key)
 
-        current_membership_sub_query = (
-            "(SELECT membership from room_memberships rm"
-            + " WHERE user_id=? AND room_id = rm.room_id"
-            + " ORDER BY id DESC LIMIT 1)")
+        if from_key == to_key:
+            defer.returnValue(([], to_key))
+            return
 
-        where = (" WHERE ? IN " + current_membership_sub_query
-                 + " AND messages.room_id=?")
+        sql = (
+            "SELECT * FROM events as e WHERE "
+            "((room_id IN (%(current)s)) OR "
+            "(event_id IN (%(invites)s))) "
+            "AND e.stream_ordering > ? AND e.stream_ordering < ? "
+            "AND e.outlier = 0 "
+            "ORDER BY stream_ordering ASC LIMIT %(limit)d "
+        ) % {
+            "current": current_room_membership_sql,
+            "invites": membership_sql,
+            "limit": limit
+        }
 
-        query = select_query + where
-        query_args = ["join", user_id, room_id]
-
-        (query, query_args) = self._append_stream_operations(
-            "messages", query, query_args, from_pkey, to_pkey,
-            limit=limit, group_by=" GROUP BY messages.id "
+        rows = yield self._execute_and_decode(
+            sql,
+            user_id, user_id, from_id, to_id
         )
 
-        logger.debug("[SQL] %s : %s", query, query_args)
-        cursor = txn.execute(query, query_args)
+        ret = [self._parse_event_from_row(r) for r in rows]
 
-        # convert the result set into events
-        entries = self.cursor_to_dict(cursor)
-        events = []
-        for entry in entries:
-            # TODO we should spec the cursor > event mapping somewhere else.
-            event = {}
-            straight_mappings = ["msg_id", "user_id", "room_id"]
-            for key in straight_mappings:
-                event[key] = entry[key]
-            event["content"] = json.loads(entry["content"])
-            if entry["compressed_fb"]:
-                event["feedback"] = json.loads(entry["compressed_fb"])
-            events.append(event)
+        if rows:
+            key = "s%d" % max([r["stream_ordering"] for r in rows])
+        else:
+            # Assume we didn't get anything because there was nothing to get.
+            key = to_key
 
-        latest_pkey = from_pkey if len(entries) == 0 else entries[-1]["id"]
+        defer.returnValue((ret, key))
 
-        return (events, latest_pkey)
+    @defer.inlineCallbacks
+    @log_function
+    def paginate_room_events(self, room_id, from_key, to_key=None,
+                             direction='b', limit=-1,
+                             with_feedback=False):
+        # TODO (erikj): Handle compressed feedback
 
-    def get_room_member_stream(self, user_id, from_key, to_key):
-        """Get all room membership events for this user between the given keys.
+        from_comp = '<' if direction =='b' else '>'
+        to_comp = '>' if direction =='b' else '<'
+        order = "DESC" if direction == 'b' else "ASC"
 
-        Args:
-            user_id (str): The user who is requesting membership events.
-            from_key (int): The ID to start returning results from (exclusive).
-            to_key (int): The ID to stop returning results (exclusive).
-        Returns:
-            A tuple of rows (list of namedtuples), new_id(int)
-        """
-        return self._db_pool.runInteraction(
-            self._get_room_member_rows, user_id, from_key, to_key
+        args = [room_id]
+
+        bounds = _get_token_bound(from_key, from_comp)
+        if to_key:
+            bounds = "%s AND %s" % (bounds, _get_token_bound(to_key, to_comp))
+
+        if int(limit) > 0:
+            args.append(int(limit))
+            limit_str = " LIMIT ?"
+        else:
+            limit_str = ""
+
+        sql = (
+            "SELECT * FROM events "
+            "WHERE outlier = 0 AND room_id = ? AND %(bounds)s "
+            "ORDER BY topological_ordering %(order)s, stream_ordering %(order)s %(limit)s "
+        ) % {"bounds": bounds, "order": order, "limit": limit_str}
+
+        rows = yield self._execute_and_decode(
+            sql,
+            *args
         )
 
-    def _get_room_member_rows(self, txn, user_id, from_pkey, to_pkey):
-        # get all room membership events for rooms which the user is
-        # *currently* joined in on, or all invite events for this user.
-        current_membership_sub_query = (
-            "(SELECT membership FROM room_memberships"
-            + " WHERE user_id=? AND room_id = rm.room_id"
-            + " ORDER BY id DESC LIMIT 1)")
+        if rows:
+            topo = rows[-1]["topological_ordering"]
+            toke = rows[-1]["stream_ordering"]
+            next_token = "t%s-%s" % (topo, toke)
+        else:
+            # TODO (erikj): We should work out what to do here instead.
+            next_token = to_key if to_key else from_key
 
-        query = ("SELECT rm.* FROM room_memberships rm "
-                 # all membership events for rooms you've currently joined.
-                 + " WHERE (? IN " + current_membership_sub_query
-                 # all invite membership events for this user
-                 + " OR rm.membership=? AND user_id=?)"
-                 + " AND rm.id > ?")
-        query_args = ["join", user_id, "invite", user_id, from_pkey]
-
-        if to_pkey != -1:
-            query += " AND rm.id < ?"
-            query_args.append(to_pkey)
-
-        cursor = txn.execute(query, query_args)
-        return self._as_events(cursor, RoomMemberTable, from_pkey)
-
-    def get_feedback_stream(self, user_id, from_key, to_key, room_id, limit=0):
-        return self._db_pool.runInteraction(
-            self._get_feedback_rows,
-            user_id, from_key, to_key, room_id, limit
+        defer.returnValue(
+            (
+                [self._parse_event_from_row(r) for r in rows],
+                next_token
+            )
         )
 
-    def _get_feedback_rows(self, txn, user_id, from_pkey, to_pkey, room_id,
-                           limit):
-        # work out which rooms this user is joined in on and join them with
-        # the room id on the feedback table, bounded by the specified pkeys
+    @defer.inlineCallbacks
+    def get_recent_events_for_room(self, room_id, limit, end_token,
+                                   with_feedback=False):
+        # TODO (erikj): Handle compressed feedback
 
-        # get all messages where the *current* membership state is 'join' for
-        # this user in that room.
-        query = (
-            "SELECT feedback.* FROM feedback WHERE ? IN "
-            + "(SELECT membership from room_memberships WHERE user_id=?"
-            + " AND room_id = feedback.room_id ORDER BY id DESC LIMIT 1)")
-        query_args = ["join", user_id]
-
-        if room_id:
-            query += " AND feedback.room_id=?"
-            query_args.append(room_id)
-
-        (query, query_args) = self._append_stream_operations(
-            "feedback", query, query_args, from_pkey, to_pkey, limit=limit
+        sql = (
+            "SELECT * FROM events "
+            "WHERE room_id = ? AND stream_ordering <= ? "
+            "ORDER BY topological_ordering, stream_ordering DESC LIMIT ? "
         )
 
-        logger.debug("[SQL] %s : %s", query, query_args)
-        cursor = txn.execute(query, query_args)
-        return self._as_events(cursor, FeedbackTable, from_pkey)
-
-    def get_room_data_stream(self, user_id, from_key, to_key, room_id,
-                             limit=0):
-        return self._db_pool.runInteraction(
-            self._get_room_data_rows,
-            user_id, from_key, to_key, room_id, limit
+        rows = yield self._execute_and_decode(
+            sql,
+            room_id, end_token, limit
         )
 
-    def _get_room_data_rows(self, txn, user_id, from_pkey, to_pkey, room_id,
-                            limit):
-        # work out which rooms this user is joined in on and join them with
-        # the room id on the feedback table, bounded by the specified pkeys
+        rows.reverse()  # As we selected with reverse ordering
 
-        # get all messages where the *current* membership state is 'join' for
-        # this user in that room.
-        query = (
-            "SELECT room_data.* FROM room_data WHERE ? IN "
-            + "(SELECT membership from room_memberships WHERE user_id=?"
-            + " AND room_id = room_data.room_id ORDER BY id DESC LIMIT 1)")
-        query_args = ["join", user_id]
+        if rows:
+            topo = rows[0]["topological_ordering"]
+            toke = rows[0]["stream_ordering"]
+            start_token = "t%s-%s" % (topo, toke)
 
-        if room_id:
-            query += " AND room_data.room_id=?"
-            query_args.append(room_id)
+            token = (start_token, end_token)
+        else:
+            token = (end_token, end_token)
 
-        (query, query_args) = self._append_stream_operations(
-            "room_data", query, query_args, from_pkey, to_pkey, limit=limit
+        defer.returnValue(
+            (
+                [self._parse_event_from_row(r) for r in rows],
+                token
+            )
         )
 
-        logger.debug("[SQL] %s : %s", query, query_args)
-        cursor = txn.execute(query, query_args)
-        return self._as_events(cursor, RoomDataTable, from_pkey)
+    @defer.inlineCallbacks
+    def get_room_events_max_id(self):
+        res = yield self._execute_and_decode(
+            "SELECT MAX(stream_ordering) as m FROM events"
+        )
 
-    def _append_stream_operations(self, table_name, query, query_args,
-                                  from_pkey, to_pkey, limit=None,
-                                  group_by=""):
-        LATEST_ROW = -1
-        order_by = ""
-        if to_pkey > from_pkey:
-            if from_pkey != LATEST_ROW:
-                # e.g. from=5 to=9 >> from 5 to 9 >> id>5 AND id<9
-                query += (" AND %s.id > ? AND %s.id < ?" %
-                         (table_name, table_name))
-                query_args.append(from_pkey)
-                query_args.append(to_pkey)
-            else:
-                # e.g. from=-1 to=5 >> from now to 5 >> id>5 ORDER BY id DESC
-                query += " AND %s.id > ? " % table_name
-                order_by = "ORDER BY id DESC"
-                query_args.append(to_pkey)
-        elif from_pkey > to_pkey:
-            if to_pkey != LATEST_ROW:
-                # from=9 to=5 >> from 9 to 5 >> id>5 AND id<9 ORDER BY id DESC
-                query += (" AND %s.id > ? AND %s.id < ? " %
-                          (table_name, table_name))
-                order_by = "ORDER BY id DESC"
-                query_args.append(to_pkey)
-                query_args.append(from_pkey)
-            else:
-                # from=5 to=-1 >> from 5 to now >> id>5
-                query += " AND %s.id > ?" % table_name
-                query_args.append(from_pkey)
+        logger.debug("get_room_events_max_id: %s", res)
 
-        query += group_by + order_by
+        if not res or not res[0] or not res[0]["m"]:
+            defer.returnValue("s1")
+            return
 
-        if limit and limit > 0:
-            query += " LIMIT ?"
-            query_args.append(str(limit))
-
-        return (query, query_args)
-
-    def _as_events(self, cursor, table, from_pkey):
-        data_entries = table.decode_results(cursor)
-        last_pkey = from_pkey
-        if data_entries:
-            last_pkey = data_entries[-1].id
-
-        events = [
-            entry.as_event(self.event_factory).get_dict()
-            for entry in data_entries
-        ]
-
-        return (events, last_pkey)
+        key = res[0]["m"] + 1
+        defer.returnValue("s%d" % (key,))

synapse/types.py

@@ -32,6 +32,20 @@ class DomainSpecificString(
             HomeServer as being its own
     """
 
+    # Deny iteration because it will bite you if you try to create a singleton
+    # set by:
+    #    users = set(user)
+    def __iter__(self):
+        raise ValueError("Attempted to iterate a %s" % (type(self).__name__))
+
+    # Because this class is a namedtuple of strings and booleans, it is deeply
+    # immutable.
+    def __copy__(self):
+        return self
+
+    def __deepcopy__(self, memo):
+        return self
+
     @classmethod
     def from_string(cls, s, hs):
         """Parse the string given by 's' into a structure object."""

synapse/util/lockutils.py

@@ -24,9 +24,10 @@ logger = logging.getLogger(__name__)
 
 class Lock(object):
 
-    def __init__(self, deferred):
+    def __init__(self, deferred, key):
         self._deferred = deferred
         self.released = False
+        self.key = key
 
     def release(self):
         self.released = True
@@ -38,9 +39,10 @@ class Lock(object):
             self.release()
 
     def __enter__(self):
-            return self
+        return self
 
     def __exit__(self, type, value, traceback):
+        logger.debug("Releasing lock for key=%r", self.key)
         self.release()
 
 
@@ -63,6 +65,10 @@ class LockManager(object):
         self._lock_deferreds[key] = new_deferred
 
         if old_deferred:
+            logger.debug("Queueing on lock for key=%r", key)
             yield old_deferred
+            logger.debug("Obtained lock for key=%r", key)
+        else:
+            logger.debug("Entering uncontended lock for key=%r", key)
 
-        defer.returnValue(Lock(new_deferred))
+        defer.returnValue(Lock(new_deferred, key))

tests/federation/test_federation.py

@@ -20,7 +20,7 @@ from twisted.trial import unittest
 from mock import Mock
 import logging
 
-from ..utils import MockHttpServer, MockClock
+from ..utils import MockHttpResource, MockClock
 
 from synapse.server import HomeServer
 from synapse.federation import initialize_http_replication
@@ -50,7 +50,7 @@ def make_pdu(prev_pdus=[], **kwargs):
 
 class FederationTestCase(unittest.TestCase):
     def setUp(self):
-        self.mock_http_server = MockHttpServer()
+        self.mock_resource = MockHttpResource()
         self.mock_http_client = Mock(spec=[
             "get_json",
             "put_json",
@@ -70,7 +70,7 @@ class FederationTestCase(unittest.TestCase):
         )
         self.clock = MockClock()
         hs = HomeServer("test",
-                http_server=self.mock_http_server,
+                resource_for_federation=self.mock_resource,
                 http_client=self.mock_http_client,
                 db_pool=None,
                 datastore=self.mock_persistence,
@@ -86,7 +86,7 @@ class FederationTestCase(unittest.TestCase):
         )
 
         # Empty context initially
-        (code, response) = yield self.mock_http_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/matrix/federation/v1/state/my-context/", None)
         self.assertEquals(200, code)
         self.assertFalse(response["pdus"])
@@ -111,7 +111,7 @@ class FederationTestCase(unittest.TestCase):
             ])
         )
 
-        (code, response) = yield self.mock_http_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/matrix/federation/v1/state/my-context/", None)
         self.assertEquals(200, code)
         self.assertEquals(1, len(response["pdus"]))
@@ -122,7 +122,7 @@ class FederationTestCase(unittest.TestCase):
             defer.succeed(None)
         )
 
-        (code, response) = yield self.mock_http_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/matrix/federation/v1/pdu/red/abc123def456/", None)
         self.assertEquals(404, code)
 
@@ -141,7 +141,7 @@ class FederationTestCase(unittest.TestCase):
             )
         )
 
-        (code, response) = yield self.mock_http_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/matrix/federation/v1/pdu/red/abc123def456/", None)
         self.assertEquals(200, code)
         self.assertEquals(1, len(response["pdus"]))
@@ -225,7 +225,7 @@ class FederationTestCase(unittest.TestCase):
 
         self.federation.register_edu_handler("m.test", recv_observer)
 
-        yield self.mock_http_server.trigger("PUT",
+        yield self.mock_resource.trigger("PUT",
                 "/matrix/federation/v1/send/1001000/",
                 """{
                     "origin": "remote",
@@ -272,7 +272,7 @@ class FederationTestCase(unittest.TestCase):
 
         self.federation.register_query_handler("a-question", recv_handler)
 
-        code, response = yield self.mock_http_server.trigger("GET",
+        code, response = yield self.mock_resource.trigger("GET",
             "/matrix/federation/v1/query/a-question?three=3&four=4", None)
 
         self.assertEquals(200, code)

tests/handlers/test_directory.py

@@ -51,7 +51,7 @@ class DirectoryTestCase(unittest.TestCase):
                 "get_association_from_room_alias",
             ]),
             http_client=None,
-            http_server=Mock(),
+            resource_for_federation=Mock(),
             replication_layer=self.mock_federation,
         )
         hs.handlers = DirectoryHandlers(hs)

tests/handlers/test_federation.py

@@ -1,4 +1,3 @@
-# -*- coding: utf-8 -*-
 # Copyright 2014 matrix.org
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -41,8 +40,9 @@ class FederationTestCase(unittest.TestCase):
             datastore=NonCallableMock(spec_set=[
                 "persist_event",
                 "store_room",
+                "get_room",
             ]),
-            http_server=NonCallableMock(),
+            resource_for_federation=NonCallableMock(),
             http_client=NonCallableMock(spec_set=[]),
             notifier=NonCallableMock(spec_set=["on_new_room_event"]),
             handlers=NonCallableMock(spec_set=[
@@ -69,10 +69,11 @@ class FederationTestCase(unittest.TestCase):
 
         store_id = "ASD"
         self.datastore.persist_event.return_value = defer.succeed(store_id)
+        self.datastore.get_room.return_value = defer.succeed(True)
 
-        yield self.handlers.federation_handler.on_receive(event, False)
+        yield self.handlers.federation_handler.on_receive(event, False, False)
 
-        self.datastore.persist_event.assert_called_once_with(event)
+        self.datastore.persist_event.assert_called_once_with(event, False)
         self.notifier.on_new_room_event.assert_called_once_with(
                 event, store_id)
 
@@ -89,7 +90,7 @@ class FederationTestCase(unittest.TestCase):
             content={},
         )
 
-        yield self.handlers.federation_handler.on_receive(event, False)
+        yield self.handlers.federation_handler.on_receive(event, False, False)
 
         mem_handler = self.handlers.room_member_handler
         self.assertEquals(1, mem_handler.change_membership.call_count)
@@ -115,7 +116,7 @@ class FederationTestCase(unittest.TestCase):
             content={},
         )
 
-        yield self.handlers.federation_handler.on_receive(event, False)
+        yield self.handlers.federation_handler.on_receive(event, False, False)
 
         mem_handler = self.handlers.room_member_handler
         self.assertEquals(0, mem_handler.change_membership.call_count)

tests/handlers/test_presence.py

@@ -19,8 +19,9 @@ from twisted.internet import defer
 
 from mock import Mock, call, ANY
 import logging
+import json
 
-from ..utils import MockClock
+from ..utils import MockHttpResource, MockClock, DeferredMockCallable
 
 from synapse.server import HomeServer
 from synapse.api.constants import PresenceState
@@ -34,17 +35,27 @@ ONLINE = PresenceState.ONLINE
 
 
 logging.getLogger().addHandler(logging.NullHandler())
+#logging.getLogger().addHandler(logging.StreamHandler())
+#logging.getLogger().setLevel(logging.DEBUG)
 
 
-class MockReplication(object):
-    def __init__(self):
-        self.edu_handlers = {}
+def _expect_edu(destination, edu_type, content, origin="test"):
+    return {
+        "origin": origin,
+        "ts": 1000000,
+        "pdus": [],
+        "edus": [
+            {
+                "origin": origin,
+                "destination": destination,
+                "edu_type": edu_type,
+                "content": content,
+            }
+        ],
+    }
 
-    def register_edu_handler(self, edu_type, handler):
-        self.edu_handlers[edu_type] = handler
-
-    def received_edu(self, origin, edu_type, content):
-        self.edu_handlers[edu_type](origin, content)
+def _make_edu_json(origin, edu_type, content):
+    return json.dumps(_expect_edu("test", edu_type, content, origin=origin))
 
 
 class JustPresenceHandlers(object):
@@ -66,7 +77,7 @@ class PresenceStateTestCase(unittest.TestCase):
                     "set_presence_list_accepted",
                 ]),
                 handlers=None,
-                http_server=Mock(),
+                resource_for_federation=Mock(),
                 http_client=None,
             )
         hs.handlers = JustPresenceHandlers(hs)
@@ -80,6 +91,14 @@ class PresenceStateTestCase(unittest.TestCase):
             return defer.succeed(allow)
         self.datastore.is_presence_visible = is_presence_visible
 
+        # Mock the RoomMemberHandler
+        room_member_handler = Mock(spec=[
+            "get_rooms_for_user",
+            "get_room_members",
+        ])
+        hs.handlers.room_member_handler = room_member_handler
+        logging.getLogger().debug("Mocking room_member_handler=%r", room_member_handler)
+
         # Some local users to test with
         self.u_apple = hs.parse_userid("@apple:test")
         self.u_banana = hs.parse_userid("@banana:test")
@@ -87,11 +106,21 @@ class PresenceStateTestCase(unittest.TestCase):
 
         self.handler = hs.get_handlers().presence_handler
 
-        hs.handlers.room_member_handler = Mock(spec=[
-            "get_rooms_for_user",
-        ])
-        hs.handlers.room_member_handler.get_rooms_for_user = (
-                lambda u: defer.succeed([]))
+        self.room_members = []
+
+        def get_rooms_for_user(user):
+            if user in self.room_members:
+                return defer.succeed(["a-room"])
+            else:
+                return defer.succeed([])
+        room_member_handler.get_rooms_for_user = get_rooms_for_user
+
+        def get_room_members(room_id):
+            if room_id == "a-room":
+                return defer.succeed(self.room_members)
+            else:
+                return defer.succeed([])
+        room_member_handler.get_room_members = get_room_members
 
         self.mock_start = Mock()
         self.mock_stop = Mock()
@@ -131,6 +160,21 @@ class PresenceStateTestCase(unittest.TestCase):
         )
         mocked_get.assert_called_with("apple")
 
+    @defer.inlineCallbacks
+    def test_get_same_room_state(self):
+        mocked_get = self.datastore.get_presence_state
+        mocked_get.return_value = defer.succeed(
+            {"state": ONLINE, "status_msg": "Online"}
+        )
+
+        self.room_members = [self.u_apple, self.u_clementine]
+
+        state = yield self.handler.get_state(
+            target_user=self.u_apple, auth_user=self.u_clementine
+        )
+
+        self.assertEquals({"state": ONLINE, "status_msg": "Online"}, state)
+
     @defer.inlineCallbacks
     def test_get_disallowed_state(self):
         mocked_get = self.datastore.get_presence_state
@@ -138,12 +182,15 @@ class PresenceStateTestCase(unittest.TestCase):
             {"state": ONLINE, "status_msg": "Online"}
         )
 
+        self.room_members = []
+
         yield self.assertFailure(
             self.handler.get_state(
                 target_user=self.u_apple, auth_user=self.u_clementine
             ),
             SynapseError
         )
+    test_get_disallowed_state.skip = "Presence polling is disabled"
 
     @defer.inlineCallbacks
     def test_set_my_state(self):
@@ -168,16 +215,20 @@ class PresenceStateTestCase(unittest.TestCase):
                 state={"state": OFFLINE})
 
         self.mock_stop.assert_called_with(self.u_apple)
+    test_set_my_state.skip = "Presence polling is disabled"
 
 
 class PresenceInvitesTestCase(unittest.TestCase):
     """ Tests presence management. """
 
     def setUp(self):
-        self.replication = MockReplication()
-        self.replication.send_edu = Mock()
+        self.mock_http_client = Mock(spec=[])
+        self.mock_http_client.put_json = DeferredMockCallable()
+
+        self.mock_federation_resource = MockHttpResource()
 
         hs = HomeServer("test",
+                clock=MockClock(),
                 db_pool=None,
                 datastore=Mock(spec=[
                     "has_presence_state",
@@ -186,11 +237,17 @@ class PresenceInvitesTestCase(unittest.TestCase):
                     "set_presence_list_accepted",
                     "get_presence_list",
                     "del_presence_list",
+
+                    # Bits that Federation needs
+                    "prep_send_transaction",
+                    "delivered_txn",
+                    "get_received_txn_response",
+                    "set_received_txn_response",
                 ]),
                 handlers=None,
-                http_server=Mock(),
-                http_client=None,
-                replication_layer=self.replication
+                resource_for_client=Mock(),
+                resource_for_federation=self.mock_federation_resource,
+                http_client=self.mock_http_client,
             )
         hs.handlers = JustPresenceHandlers(hs)
 
@@ -201,6 +258,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
                 user_localpart in ("apple", "banana"))
         self.datastore.has_presence_state = has_presence_state
 
+        def get_received_txn_response(*args):
+            return defer.succeed(None)
+        self.datastore.get_received_txn_response = get_received_txn_response
+
         # Some local users to test with
         self.u_apple = hs.parse_userid("@apple:test")
         self.u_banana = hs.parse_userid("@banana:test")
@@ -248,7 +309,19 @@ class PresenceInvitesTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_invite_remote(self):
-        self.replication.send_edu.return_value = defer.succeed((200, "OK"))
+        put_json = self.mock_http_client.put_json
+        put_json.expect_call_and_return(
+            call("elsewhere",
+                path="/matrix/federation/v1/send/1000000/",
+                data=_expect_edu("elsewhere", "m.presence_invite",
+                    content={
+                        "observer_user": "@apple:test",
+                        "observed_user": "@cabbage:elsewhere",
+                    }
+                )
+            ),
+            defer.succeed((200, "OK"))
+        )
 
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_cabbage)
@@ -256,67 +329,79 @@ class PresenceInvitesTestCase(unittest.TestCase):
         self.datastore.add_presence_list_pending.assert_called_with(
                 "apple", "@cabbage:elsewhere")
 
-        self.replication.send_edu.assert_called_with(
-                destination="elsewhere",
-                edu_type="m.presence_invite",
-                content={
-                    "observer_user": "@apple:test",
-                    "observed_user": "@cabbage:elsewhere",
-                }
-        )
+        yield put_json.await_calls()
 
     @defer.inlineCallbacks
     def test_accept_remote(self):
         # TODO(paul): This test will likely break if/when real auth permissions
         # are added; for now the HS will always accept any invite
-        self.replication.send_edu.return_value = defer.succeed((200, "OK"))
+        put_json = self.mock_http_client.put_json
+        put_json.expect_call_and_return(
+            call("elsewhere",
+                path="/matrix/federation/v1/send/1000000/",
+                data=_expect_edu("elsewhere", "m.presence_accept",
+                    content={
+                        "observer_user": "@cabbage:elsewhere",
+                        "observed_user": "@apple:test",
+                    }
+                )
+            ),
+            defer.succeed((200, "OK"))
+        )
 
-        yield self.replication.received_edu(
-                "elsewhere", "m.presence_invite", {
+        yield self.mock_federation_resource.trigger("PUT",
+            "/matrix/federation/v1/send/1000000/",
+            _make_edu_json("elsewhere", "m.presence_invite",
+                content={
                     "observer_user": "@cabbage:elsewhere",
                     "observed_user": "@apple:test",
                 }
+            )
         )
 
         self.datastore.allow_presence_visible.assert_called_with(
                 "apple", "@cabbage:elsewhere")
 
-        self.replication.send_edu.assert_called_with(
-                destination="elsewhere",
-                edu_type="m.presence_accept",
-                content={
-                    "observer_user": "@cabbage:elsewhere",
-                    "observed_user": "@apple:test",
-                }
-        )
+        yield put_json.await_calls()
 
     @defer.inlineCallbacks
     def test_invited_remote_nonexistant(self):
-        self.replication.send_edu.return_value = defer.succeed((200, "OK"))
-
-        yield self.replication.received_edu(
-                "elsewhere", "m.presence_invite", {
-                    "observer_user": "@cabbage:elsewhere",
-                    "observed_user": "@durian:test",
-                }
+        put_json = self.mock_http_client.put_json
+        put_json.expect_call_and_return(
+            call("elsewhere",
+                path="/matrix/federation/v1/send/1000000/",
+                data=_expect_edu("elsewhere", "m.presence_deny",
+                    content={
+                        "observer_user": "@cabbage:elsewhere",
+                        "observed_user": "@durian:test",
+                    }
+                )
+            ),
+            defer.succeed((200, "OK"))
         )
 
-        self.replication.send_edu.assert_called_with(
-                destination="elsewhere",
-                edu_type="m.presence_deny",
+        yield self.mock_federation_resource.trigger("PUT",
+            "/matrix/federation/v1/send/1000000/",
+            _make_edu_json("elsewhere", "m.presence_invite",
                 content={
                     "observer_user": "@cabbage:elsewhere",
                     "observed_user": "@durian:test",
                 }
+            )
         )
 
+        yield put_json.await_calls()
+
     @defer.inlineCallbacks
     def test_accepted_remote(self):
-        yield self.replication.received_edu(
-                "elsewhere", "m.presence_accept", {
+        yield self.mock_federation_resource.trigger("PUT",
+            "/matrix/federation/v1/send/1000000/",
+            _make_edu_json("elsewhere", "m.presence_accept",
+                content={
                     "observer_user": "@apple:test",
                     "observed_user": "@cabbage:elsewhere",
                 }
+            )
         )
 
         self.datastore.set_presence_list_accepted.assert_called_with(
@@ -327,11 +412,14 @@ class PresenceInvitesTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_denied_remote(self):
-        yield self.replication.received_edu(
-                "elsewhere", "m.presence_deny", {
+        yield self.mock_federation_resource.trigger("PUT",
+            "/matrix/federation/v1/send/1000000/",
+            _make_edu_json("elsewhere", "m.presence_deny",
+                content={
                     "observer_user": "@apple:test",
                     "observed_user": "@eggplant:elsewhere",
                 }
+            )
         )
 
         self.datastore.del_presence_list.assert_called_with(
@@ -348,6 +436,14 @@ class PresenceInvitesTestCase(unittest.TestCase):
         self.mock_stop.assert_called_with(
                 self.u_apple, target_user=self.u_banana)
 
+    @defer.inlineCallbacks
+    def test_drop_remote(self):
+        yield self.handler.drop(
+                observer_user=self.u_apple, observed_user=self.u_cabbage)
+
+        self.datastore.del_presence_list.assert_called_with(
+                "apple", "@cabbage:elsewhere")
+
     @defer.inlineCallbacks
     def test_get_presence_list(self):
         self.datastore.get_presence_list.return_value = defer.succeed(
@@ -389,22 +485,29 @@ class PresencePushTestCase(unittest.TestCase):
     BE WARNED...
     """
     def setUp(self):
-        self.replication = MockReplication()
-        self.replication.send_edu = Mock()
-        self.replication.send_edu.return_value = defer.succeed((200, "OK"))
-
         self.clock = MockClock()
 
+        self.mock_http_client = Mock(spec=[])
+        self.mock_http_client.put_json = DeferredMockCallable()
+
+        self.mock_federation_resource = MockHttpResource()
+
         hs = HomeServer("test",
                 clock=self.clock,
                 db_pool=None,
                 datastore=Mock(spec=[
                     "set_presence_state",
+
+                    # Bits that Federation needs
+                    "prep_send_transaction",
+                    "delivered_txn",
+                    "get_received_txn_response",
+                    "set_received_txn_response",
                 ]),
                 handlers=None,
-                http_server=Mock(),
-                http_client=None,
-                replication_layer=self.replication,
+                resource_for_client=Mock(),
+                resource_for_federation=self.mock_federation_resource,
+                http_client=self.mock_http_client,
             )
         hs.handlers = JustPresenceHandlers(hs)
 
@@ -412,6 +515,11 @@ class PresencePushTestCase(unittest.TestCase):
         self.mock_update_client.return_value = defer.succeed(None)
 
         self.datastore = hs.get_datastore()
+
+        def get_received_txn_response(*args):
+            return defer.succeed(None)
+        self.datastore.get_received_txn_response = get_received_txn_response
+
         self.handler = hs.get_handlers().presence_handler
         self.handler.push_update_to_clients = self.mock_update_client
 
@@ -547,13 +655,47 @@ class PresencePushTestCase(unittest.TestCase):
                     observed_user=self.u_banana,
                     statuscache=ANY), # self-reflection
         ]) # and no others...
+    test_push_local.skip = "Presence polling is disabled"
 
     @defer.inlineCallbacks
     def test_push_remote(self):
+        put_json = self.mock_http_client.put_json
+        put_json.expect_call_and_return(
+            call("remote",
+                path=ANY,  # Can't guarantee which txn ID will be which
+                data=_expect_edu("remote", "m.presence",
+                    content={
+                        "push": [
+                            {"user_id": "@apple:test",
+                             "state": "online",
+                             "mtime_age": 0},
+                        ],
+                    }
+                )
+            ),
+            defer.succeed((200, "OK"))
+        )
+        put_json.expect_call_and_return(
+            call("farm",
+                path=ANY,  # Can't guarantee which txn ID will be which
+                data=_expect_edu("farm", "m.presence",
+                    content={
+                        "push": [
+                            {"user_id": "@apple:test",
+                             "state": "online",
+                             "mtime_age": 0},
+                        ],
+                    }
+                )
+            ),
+            defer.succeed((200, "OK"))
+        )
+
         self.room_members = [self.u_apple, self.u_onion]
 
         self.datastore.set_presence_state.return_value = defer.succeed(
-                {"state": ONLINE})
+            {"state": ONLINE}
+        )
 
         # TODO(paul): Gut-wrenching
         self.handler._user_cachemap[self.u_apple] = UserPresenceCache()
@@ -561,30 +703,11 @@ class PresencePushTestCase(unittest.TestCase):
         apple_set.add(self.u_potato.domain)
 
         yield self.handler.set_state(self.u_apple, self.u_apple,
-                {"state": ONLINE})
+            {"state": ONLINE}
+        )
 
-        self.replication.send_edu.assert_has_calls([
-                call(
-                    destination="remote",
-                    edu_type="m.presence",
-                    content={
-                        "push": [
-                            {"user_id": "@apple:test",
-                             "state": "online",
-                             "mtime_age": 0},
-                        ],
-                    }),
-                call(
-                    destination="farm",
-                    edu_type="m.presence",
-                    content={
-                        "push": [
-                            {"user_id": "@apple:test",
-                             "state": "online",
-                             "mtime_age": 0},
-                        ],
-                    })
-        ], any_order=True)
+        yield put_json.await_calls()
+    test_push_remote.skip = "Presence polling is disabled"
 
     @defer.inlineCallbacks
     def test_recv_remote(self):
@@ -595,14 +718,17 @@ class PresencePushTestCase(unittest.TestCase):
 
         self.room_members = [self.u_banana, self.u_potato]
 
-        yield self.replication.received_edu(
-                "remote", "m.presence", {
+        yield self.mock_federation_resource.trigger("PUT",
+            "/matrix/federation/v1/send/1000000/",
+            _make_edu_json("elsewhere", "m.presence",
+                content={
                     "push": [
                         {"user_id": "@potato:remote",
                          "state": "online",
                          "mtime_age": 1000},
                     ],
                 }
+            )
         )
 
         self.mock_update_client.assert_has_calls([
@@ -648,6 +774,35 @@ class PresencePushTestCase(unittest.TestCase):
     @defer.inlineCallbacks
     def test_join_room_remote(self):
         ## Sending local user state to a newly-joined remote user
+        put_json = self.mock_http_client.put_json
+        put_json.expect_call_and_return(
+            call("remote",
+                path=ANY,  # Can't guarantee which txn ID will be which
+                data=_expect_edu("remote", "m.presence",
+                    content={
+                        "push": [
+                            {"user_id": "@apple:test",
+                            "state": "online"},
+                        ],
+                    }
+                ),
+            ),
+            defer.succeed((200, "OK"))
+        )
+        put_json.expect_call_and_return(
+            call("remote",
+                path=ANY,  # Can't guarantee which txn ID will be which
+                data=_expect_edu("remote", "m.presence",
+                    content={
+                        "push": [
+                            {"user_id": "@banana:test",
+                            "state": "offline"},
+                        ],
+                    }
+                ),
+            ),
+            defer.succeed((200, "OK"))
+        )
 
         # TODO(paul): Gut-wrenching
         self.handler._user_cachemap[self.u_apple] = UserPresenceCache()
@@ -659,31 +814,25 @@ class PresencePushTestCase(unittest.TestCase):
             "a-room"
         )
 
-        self.replication.send_edu.assert_has_calls([
-                call(
-                    destination="remote",
-                    edu_type="m.presence",
-                    content={
-                        "push": [
-                            {"user_id": "@apple:test",
-                            "state": "online"},
-                        ],
-                    }),
-                call(
-                    destination="remote",
-                    edu_type="m.presence",
-                    content={
-                        "push": [
-                            {"user_id": "@banana:test",
-                            "state": "offline"},
-                        ],
-                    }),
-        ], any_order=True)
-
-        self.replication.send_edu.reset_mock()
+        yield put_json.await_calls()
 
         ## Sending newly-joined local user state to remote users
 
+        put_json.expect_call_and_return(
+            call("remote",
+                path="/matrix/federation/v1/send/1000002/",
+                data=_expect_edu("remote", "m.presence",
+                    content={
+                        "push": [
+                            {"user_id": "@clementine:test",
+                            "state": "online"},
+                        ],
+                    }
+                ),
+            ),
+            defer.succeed((200, "OK"))
+        )
+
         self.handler._user_cachemap[self.u_clementine] = UserPresenceCache()
         self.handler._user_cachemap[self.u_clementine].update(
                 {"state": ONLINE}, self.u_clementine)
@@ -693,17 +842,7 @@ class PresencePushTestCase(unittest.TestCase):
             "a-room"
         )
 
-        self.replication.send_edu.assert_has_calls(
-                call(
-                    destination="remote",
-                    edu_type="m.presence",
-                    content={
-                        "push": [
-                            {"user_id": "@clementine:test",
-                            "state": "online"},
-                        ],
-                    }),
-        )
+        put_json.await_calls()
 
 
 class PresencePollingTestCase(unittest.TestCase):
@@ -720,21 +859,34 @@ class PresencePollingTestCase(unittest.TestCase):
 
 
     def setUp(self):
-        self.replication = MockReplication()
-        self.replication.send_edu = Mock()
+        self.mock_http_client = Mock(spec=[])
+        self.mock_http_client.put_json = DeferredMockCallable()
+
+        self.mock_federation_resource = MockHttpResource()
 
         hs = HomeServer("test",
+                clock=MockClock(),
                 db_pool=None,
-                datastore=Mock(spec=[]),
+                datastore=Mock(spec=[
+                    # Bits that Federation needs
+                    "prep_send_transaction",
+                    "delivered_txn",
+                    "get_received_txn_response",
+                    "set_received_txn_response",
+                ]),
                 handlers=None,
-                http_server=Mock(),
-                http_client=None,
-                replication_layer=self.replication,
+                resource_for_client=Mock(),
+                resource_for_federation=self.mock_federation_resource,
+                http_client=self.mock_http_client,
             )
         hs.handlers = JustPresenceHandlers(hs)
 
         self.datastore = hs.get_datastore()
 
+        def get_received_txn_response(*args):
+            return defer.succeed(None)
+        self.datastore.get_received_txn_response = get_received_txn_response
+
         self.mock_update_client = Mock()
         self.mock_update_client.return_value = defer.succeed(None)
 
@@ -792,8 +944,9 @@ class PresencePollingTestCase(unittest.TestCase):
     def test_push_local(self):
         # apple goes online
         yield self.handler.set_state(
-                target_user=self.u_apple, auth_user=self.u_apple,
-                state={"state": ONLINE})
+            target_user=self.u_apple, auth_user=self.u_apple,
+            state={"state": ONLINE}
+        )
 
         # apple should see both banana and clementine currently offline
         self.mock_update_client.assert_has_calls([
@@ -847,71 +1000,98 @@ class PresencePollingTestCase(unittest.TestCase):
 
         self.assertFalse("banana" in self.handler._local_pushmap)
         self.assertFalse("clementine" in self.handler._local_pushmap)
+    test_push_local.skip = "Presence polling is disabled"
+
 
     @defer.inlineCallbacks
     def test_remote_poll_send(self):
+        put_json = self.mock_http_client.put_json
+        put_json.expect_call_and_return(
+            call("remote",
+                path="/matrix/federation/v1/send/1000000/",
+                data=_expect_edu("remote", "m.presence",
+                    content={
+                        "poll": [ "@potato:remote" ],
+                    },
+                ),
+            ),
+            defer.succeed((200, "OK"))
+        )
+
         # clementine goes online
         yield self.handler.set_state(
                 target_user=self.u_clementine, auth_user=self.u_clementine,
                 state={"state": ONLINE})
 
-        self.replication.send_edu.assert_called_with(
-                destination="remote",
-                edu_type="m.presence",
-                content={
-                    "poll": [ "@potato:remote" ],
-                },
-        )
+        yield put_json.await_calls()
 
         # Gut-wrenching tests
         self.assertTrue(self.u_potato in self.handler._remote_recvmap)
         self.assertTrue(self.u_clementine in
                 self.handler._remote_recvmap[self.u_potato])
 
-        self.replication.send_edu.reset_mock()
+        put_json.expect_call_and_return(
+            call("remote",
+                path="/matrix/federation/v1/send/1000001/",
+                data=_expect_edu("remote", "m.presence",
+                    content={
+                        "unpoll": [ "@potato:remote" ],
+                    },
+                ),
+            ),
+            defer.succeed((200, "OK"))
+        )
 
         # clementine goes offline
         yield self.handler.set_state(
                 target_user=self.u_clementine, auth_user=self.u_clementine,
                 state={"state": OFFLINE})
 
-        self.replication.send_edu.assert_called_with(
-                destination="remote",
-                edu_type="m.presence",
-                content={
-                    "unpoll": [ "@potato:remote" ],
-                },
-        )
+        put_json.await_calls()
 
         self.assertFalse(self.u_potato in self.handler._remote_recvmap)
+    test_remote_poll_send.skip = "Presence polling is disabled"
 
     @defer.inlineCallbacks
     def test_remote_poll_receive(self):
-        yield self.replication.received_edu(
-                "remote", "m.presence", {
-                    "poll": [ "@banana:test" ],
-                }
+        put_json = self.mock_http_client.put_json
+        put_json.expect_call_and_return(
+            call("remote",
+                path="/matrix/federation/v1/send/1000000/",
+                data=_expect_edu("remote", "m.presence",
+                    content={
+                        "push": [
+                            {"user_id": "@banana:test",
+                             "state": "offline",
+                             "status_msg": None},
+                        ],
+                    },
+                ),
+            ),
+            defer.succeed((200, "OK"))
         )
 
+        yield self.mock_federation_resource.trigger("PUT",
+            "/matrix/federation/v1/send/1000000/",
+            _make_edu_json("remote", "m.presence",
+                content={
+                    "poll": [ "@banana:test" ],
+                },
+            )
+        )
+
+        yield put_json.await_calls()
+
         # Gut-wrenching tests
         self.assertTrue(self.u_banana in self.handler._remote_sendmap)
 
-        self.replication.send_edu.assert_called_with(
-                destination="remote",
-                edu_type="m.presence",
+        yield self.mock_federation_resource.trigger("PUT",
+            "/matrix/federation/v1/send/1000001/",
+            _make_edu_json("remote", "m.presence",
                 content={
-                    "push": [
-                        {"user_id": "@banana:test",
-                         "state": "offline",
-                         "status_msg": None},
-                    ],
-                },
-        )
-
-        yield self.replication.received_edu(
-                "remote", "m.presence", {
                     "unpoll": [ "@banana:test" ],
                 }
+            )
         )
 
         # Gut-wrenching tests

tests/handlers/test_presencelike.py

@@ -71,7 +71,7 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
                     "set_profile_displayname",
                 ]),
                 handlers=None,
-                http_server=Mock(),
+                resource_for_federation=Mock(),
                 http_client=None,
                 replication_layer=MockReplication(),
             )
@@ -135,6 +135,7 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
 
         mocked_set.assert_called_with("apple",
                 {"state": UNAVAILABLE, "status_msg": "Away"})
+    test_set_my_state.skip = "Presence polling is disabled"
 
     @defer.inlineCallbacks
     def test_push_local(self):
@@ -209,6 +210,8 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
             "displayname": "I am an Apple",
             "avatar_url": "http://foo",
         }, statuscache.state)
+    test_push_local.skip = "Presence polling is disabled"
+
 
     @defer.inlineCallbacks
     def test_push_remote(self):
@@ -239,6 +242,7 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
                     ],
                 },
         )
+    test_push_remote.skip = "Presence polling is disabled"
 
     @defer.inlineCallbacks
     def test_recv_remote(self):

tests/handlers/test_profile.py

@@ -56,7 +56,7 @@ class ProfileTestCase(unittest.TestCase):
                     "set_profile_avatar_url",
                 ]),
                 handlers=None,
-                http_server=Mock(),
+                resource_for_federation=Mock(),
                 replication_layer=self.mock_federation,
             )
         hs.handlers = ProfileHandlers(hs)
@@ -139,7 +139,7 @@ class ProfileTestCase(unittest.TestCase):
         mocked_set = self.datastore.set_profile_avatar_url
         mocked_set.return_value = defer.succeed(())
 
-        yield self.handler.set_avatar_url(self.frank, self.frank, 
+        yield self.handler.set_avatar_url(self.frank, self.frank,
                 "http://my.server/pic.gif")
 
         mocked_set.assert_called_with("1234ABCD", "http://my.server/pic.gif")

tests/handlers/test_room.py

@@ -40,13 +40,13 @@ class RoomMemberHandlerTestCase(unittest.TestCase):
             self.hostname,
             db_pool=None,
             datastore=NonCallableMock(spec_set=[
-                "store_room_member",
+                "persist_event",
                 "get_joined_hosts_for_room",
                 "get_room_member",
                 "get_room",
                 "store_room",
             ]),
-            http_server=NonCallableMock(),
+            resource_for_federation=NonCallableMock(),
             http_client=NonCallableMock(spec_set=[]),
             notifier=NonCallableMock(spec_set=["on_new_room_event"]),
             handlers=NonCallableMock(spec_set=[
@@ -69,6 +69,8 @@ class RoomMemberHandlerTestCase(unittest.TestCase):
         self.distributor = hs.get_distributor()
         self.hs = hs
 
+        self.distributor.declare("collect_presencelike_data")
+
         self.handlers.room_member_handler = RoomMemberHandler(self.hs)
         self.handlers.profile_handler = ProfileHandler(self.hs)
         self.room_member_handler = self.handlers.room_member_handler
@@ -97,7 +99,7 @@ class RoomMemberHandlerTestCase(unittest.TestCase):
         )
 
         store_id = "store_id_fooo"
-        self.datastore.store_room_member.return_value = defer.succeed(store_id)
+        self.datastore.persist_event.return_value = defer.succeed(store_id)
 
         # Actual invocation
         yield self.room_member_handler.change_membership(event)
@@ -110,12 +112,8 @@ class RoomMemberHandlerTestCase(unittest.TestCase):
             set(event.destinations)
         )
 
-        self.datastore.store_room_member.assert_called_once_with(
-            user_id=target_user_id,
-            sender=user_id,
-            room_id=room_id,
-            content=content,
-            membership=Membership.INVITE,
+        self.datastore.persist_event.assert_called_once_with(
+            event
         )
         self.notifier.on_new_room_event.assert_called_once_with(
                 event, store_id)
@@ -144,12 +142,14 @@ class RoomMemberHandlerTestCase(unittest.TestCase):
         joined = ["red", "green"]
 
         self.state_handler.handle_new_event.return_value = defer.succeed(True)
-        self.datastore.get_joined_hosts_for_room.return_value = (
-            defer.succeed(joined)
-        )
+
+        def get_joined(*args):
+            return defer.succeed(joined)
+
+        self.datastore.get_joined_hosts_for_room.side_effect = get_joined
 
         store_id = "store_id_fooo"
-        self.datastore.store_room_member.return_value = defer.succeed(store_id)
+        self.datastore.persist_event.return_value = defer.succeed(store_id)
         self.datastore.get_room.return_value = defer.succeed(1)  # Not None.
 
         prev_state = NonCallableMock()
@@ -171,12 +171,8 @@ class RoomMemberHandlerTestCase(unittest.TestCase):
             set(event.destinations)
         )
 
-        self.datastore.store_room_member.assert_called_once_with(
-            user_id=target_user_id,
-            sender=user_id,
-            room_id=room_id,
-            content=content,
-            membership=Membership.JOIN,
+        self.datastore.persist_event.assert_called_once_with(
+            event
         )
         self.notifier.on_new_room_event.assert_called_once_with(
                 event, store_id)
@@ -317,7 +313,6 @@ class RoomCreationTest(unittest.TestCase):
             datastore=NonCallableMock(spec_set=[
                 "store_room",
             ]),
-            http_server=NonCallableMock(),
             http_client=NonCallableMock(spec_set=[]),
             notifier=NonCallableMock(spec_set=["on_new_room_event"]),
             handlers=NonCallableMock(spec_set=[

tests/rest/test_events.py

@@ -29,7 +29,7 @@ from synapse.server import HomeServer
 import json
 import logging
 
-from ..utils import MockHttpServer, MemoryDataStore
+from ..utils import MockHttpResource, MemoryDataStore
 from .utils import RestTestCase
 
 from mock import Mock
@@ -116,7 +116,7 @@ class EventStreamPermissionsTestCase(RestTestCase):
 
     @defer.inlineCallbacks
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
 
         state_handler = Mock(spec=["handle_new_event"])
         state_handler.handle_new_event.return_value = True
@@ -142,9 +142,9 @@ class EventStreamPermissionsTestCase(RestTestCase):
         hs.get_clock().time_msec.return_value = 1000000
 
         hs.datastore = MemoryDataStore()
-        synapse.rest.register.register_servlets(hs, self.mock_server)
-        synapse.rest.events.register_servlets(hs, self.mock_server)
-        synapse.rest.room.register_servlets(hs, self.mock_server)
+        synapse.rest.register.register_servlets(hs, self.mock_resource)
+        synapse.rest.events.register_servlets(hs, self.mock_resource)
+        synapse.rest.room.register_servlets(hs, self.mock_resource)
 
         # register an account
         self.user_id = "sid1"
@@ -164,12 +164,12 @@ class EventStreamPermissionsTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_stream_basic_permissions(self):
         # invalid token, expect 403
-        (code, response) = yield self.mock_server.trigger_get(
+        (code, response) = yield self.mock_resource.trigger_get(
                            "/events?access_token=%s" % ("invalid" + self.token))
         self.assertEquals(403, code, msg=str(response))
 
         # valid token, expect content
-        (code, response) = yield self.mock_server.trigger_get(
+        (code, response) = yield self.mock_resource.trigger_get(
                            "/events?access_token=%s&timeout=0" % (self.token))
         self.assertEquals(200, code, msg=str(response))
         self.assertTrue("chunk" in response)
@@ -186,13 +186,11 @@ class EventStreamPermissionsTestCase(RestTestCase):
         # invited to room (expect no content for room)
         yield self.invite(room_id, src=self.other_user, targ=self.user_id,
                           tok=self.other_token)
-        (code, response) = yield self.mock_server.trigger_get(
+        (code, response) = yield self.mock_resource.trigger_get(
                            "/events?access_token=%s&timeout=0" % (self.token))
         self.assertEquals(200, code, msg=str(response))
 
-        # First message is a reflection of my own presence status change
-        self.assertEquals(1, len(response["chunk"]))
-        self.assertEquals("m.presence", response["chunk"][0]["type"])
+        self.assertEquals(0, len(response["chunk"]))
 
         # joined room (expect all content for room)
         yield self.join(room=room_id, user=self.user_id, tok=self.token)

tests/rest/test_presence.py

@@ -21,9 +21,10 @@ from twisted.internet import defer
 from mock import Mock
 import logging
 
-from ..utils import MockHttpServer
+from ..utils import MockHttpResource
 
 from synapse.api.constants import PresenceState
+from synapse.handlers.presence import PresenceHandler
 from synapse.server import HomeServer
 
 
@@ -39,27 +40,48 @@ myid = "@apple:test"
 PATH_PREFIX = "/matrix/client/api/v1"
 
 
+class JustPresenceHandlers(object):
+    def __init__(self, hs):
+        self.presence_handler = PresenceHandler(hs)
+
+
 class PresenceStateTestCase(unittest.TestCase):
 
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
-        self.mock_handler = Mock(spec=[
-            "get_state",
-            "set_state",
-        ])
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
 
         hs = HomeServer("test",
             db_pool=None,
+            datastore=Mock(spec=[
+                "get_presence_state",
+                "set_presence_state",
+            ]),
             http_client=None,
-            http_server=self.mock_server,
+            resource_for_client=self.mock_resource,
+            resource_for_federation=self.mock_resource,
         )
+        hs.handlers = JustPresenceHandlers(hs)
+
+        self.datastore = hs.get_datastore()
+
+        def get_presence_list(*a, **kw):
+            return defer.succeed([])
+        self.datastore.get_presence_list = get_presence_list
 
         def _get_user_by_token(token=None):
             return hs.parse_userid(myid)
 
         hs.get_auth().get_user_by_token = _get_user_by_token
 
-        hs.get_handlers().presence_handler = self.mock_handler
+        room_member_handler = hs.handlers.room_member_handler = Mock(
+            spec=[
+                "get_rooms_for_user",
+            ]
+        )
+
+        def get_rooms_for_user(user):
+            return defer.succeed([])
+        room_member_handler.get_rooms_for_user = get_rooms_for_user
 
         hs.register_servlets()
 
@@ -67,56 +89,75 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_get_my_status(self):
-        mocked_get = self.mock_handler.get_state
+        mocked_get = self.datastore.get_presence_state
         mocked_get.return_value = defer.succeed(
-                {"state": ONLINE, "status_msg": "Available"})
+            {"state": ONLINE, "status_msg": "Available"}
+        )
 
-        (code, response) = yield self.mock_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/presence/%s/status" % (myid), None)
 
         self.assertEquals(200, code)
         self.assertEquals({"state": ONLINE, "status_msg": "Available"},
                 response)
-        mocked_get.assert_called_with(target_user=self.u_apple,
-                auth_user=self.u_apple)
+        mocked_get.assert_called_with("apple")
 
     @defer.inlineCallbacks
     def test_set_my_status(self):
-        mocked_set = self.mock_handler.set_state
-        mocked_set.return_value = defer.succeed(())
+        mocked_set = self.datastore.set_presence_state
+        mocked_set.return_value = defer.succeed({"state": OFFLINE})
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                 "/presence/%s/status" % (myid),
                 '{"state": "unavailable", "status_msg": "Away"}')
 
         self.assertEquals(200, code)
-        mocked_set.assert_called_with(target_user=self.u_apple,
-                auth_user=self.u_apple,
-                state={"state": UNAVAILABLE, "status_msg": "Away"})
+        mocked_set.assert_called_with("apple",
+                {"state": UNAVAILABLE, "status_msg": "Away"})
+    test_set_my_status.skip = "Presence polling is disabled"
 
 
 class PresenceListTestCase(unittest.TestCase):
 
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
-        self.mock_handler = Mock(spec=[
-            "get_presence_list",
-            "send_invite",
-            "drop",
-        ])
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
 
         hs = HomeServer("test",
             db_pool=None,
+            datastore=Mock(spec=[
+                "has_presence_state",
+                "get_presence_state",
+                "allow_presence_visible",
+                "is_presence_visible",
+                "add_presence_list_pending",
+                "set_presence_list_accepted",
+                "del_presence_list",
+                "get_presence_list",
+            ]),
             http_client=None,
-            http_server=self.mock_server,
+            resource_for_client=self.mock_resource,
+            resource_for_federation=self.mock_resource
         )
+        hs.handlers = JustPresenceHandlers(hs)
+
+        self.datastore = hs.get_datastore()
+
+        def has_presence_state(user_localpart):
+            return defer.succeed(
+                user_localpart in ("apple", "banana",)
+            )
+        self.datastore.has_presence_state = has_presence_state
 
         def _get_user_by_token(token=None):
             return hs.parse_userid(myid)
 
-        hs.get_auth().get_user_by_token = _get_user_by_token
+        room_member_handler = hs.handlers.room_member_handler = Mock(
+            spec=[
+                "get_rooms_for_user",
+            ]
+        )
 
-        hs.get_handlers().presence_handler = self.mock_handler
+        hs.get_auth().get_user_by_token = _get_user_by_token
 
         hs.register_servlets()
 
@@ -125,52 +166,66 @@ class PresenceListTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_get_my_list(self):
-        self.mock_handler.get_presence_list.return_value = defer.succeed(
-                [{"observed_user": self.u_banana}]
+        self.datastore.get_presence_list.return_value = defer.succeed(
+            [{"observed_user_id": "@banana:test"}],
         )
 
-        (code, response) = yield self.mock_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/presence_list/%s" % (myid), None)
 
         self.assertEquals(200, code)
-        self.assertEquals([{"user_id": "@banana:test"}], response)
+        self.assertEquals(
+            [{"user_id": "@banana:test", "state": OFFLINE}], response
+        )
+
+        self.datastore.get_presence_list.assert_called_with(
+            "apple", accepted=True
+        )
 
     @defer.inlineCallbacks
     def test_invite(self):
-        self.mock_handler.send_invite.return_value = defer.succeed(())
+        self.datastore.add_presence_list_pending.return_value = (
+            defer.succeed(())
+        )
+        self.datastore.is_presence_visible.return_value = defer.succeed(
+            True
+        )
 
-        (code, response) = yield self.mock_server.trigger("POST",
-                "/presence_list/%s" % (myid),
-                """{
-                    "invite": ["@banana:test"]
-                }""")
+        (code, response) = yield self.mock_resource.trigger("POST",
+            "/presence_list/%s" % (myid),
+            """{"invite": ["@banana:test"]}"""
+        )
 
         self.assertEquals(200, code)
 
-        self.mock_handler.send_invite.assert_called_with(
-                observer_user=self.u_apple, observed_user=self.u_banana)
+        self.datastore.add_presence_list_pending.assert_called_with(
+            "apple", "@banana:test"
+        )
+        self.datastore.set_presence_list_accepted.assert_called_with(
+            "apple", "@banana:test"
+        )
 
     @defer.inlineCallbacks
     def test_drop(self):
-        self.mock_handler.drop.return_value = defer.succeed(())
+        self.datastore.del_presence_list.return_value = (
+            defer.succeed(())
+        )
 
-        (code, response) = yield self.mock_server.trigger("POST",
-                "/presence_list/%s" % (myid),
-                """{
-                    "drop": ["@banana:test"]
-                }""")
+        (code, response) = yield self.mock_resource.trigger("POST",
+            "/presence_list/%s" % (myid),
+            """{"drop": ["@banana:test"]}"""
+        )
 
         self.assertEquals(200, code)
 
-        self.mock_handler.drop.assert_called_with(
-                observer_user=self.u_apple, observed_user=self.u_banana)
+        self.datastore.del_presence_list.assert_called_with(
+            "apple", "@banana:test"
+        )
 
 
 class PresenceEventStreamTestCase(unittest.TestCase):
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
-
-        # TODO: mocked data store
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
 
         # HIDEOUS HACKERY
         # TODO(paul): This should be injected in via the HomeServer DI system
@@ -183,7 +238,8 @@ class PresenceEventStreamTestCase(unittest.TestCase):
         hs = HomeServer("test",
             db_pool=None,
             http_client=None,
-            http_server=self.mock_server,
+            resource_for_client=self.mock_resource,
+            resource_for_federation=self.mock_resource,
             datastore=Mock(spec=[
                 "set_presence_state",
                 "get_presence_list",
@@ -223,7 +279,7 @@ class PresenceEventStreamTestCase(unittest.TestCase):
         self.mock_datastore.get_presence_list.return_value = defer.succeed(
                 [])
 
-        (code, response) = yield self.mock_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/events?timeout=0", None)
 
         self.assertEquals(200, code)
@@ -232,14 +288,7 @@ class PresenceEventStreamTestCase(unittest.TestCase):
         # all be ours
 
         # I'll already get my own presence state change
-        self.assertEquals({"start": "0", "end": "1", "chunk": [
-            {"type": "m.presence",
-             "content": {
-                 "user_id": "@apple:test",
-                 "state": ONLINE,
-                 "mtime_age": 0,
-            }},
-        ]}, response)
+        self.assertEquals({"start": "1", "end": "1", "chunk": []}, response)
 
         self.mock_datastore.set_presence_state.return_value = defer.succeed(
                 {"state": ONLINE})
@@ -249,7 +298,7 @@ class PresenceEventStreamTestCase(unittest.TestCase):
         yield self.presence.set_state(self.u_banana, self.u_banana,
                 state={"state": ONLINE})
 
-        (code, response) = yield self.mock_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/events?from=1&timeout=0", None)
 
         self.assertEquals(200, code)
@@ -261,3 +310,4 @@ class PresenceEventStreamTestCase(unittest.TestCase):
                  "mtime_age": 0,
             }},
         ]}, response)
+    test_shortpoll.skip = "Presence polling is disabled"

tests/rest/test_profile.py

@@ -20,7 +20,7 @@ from twisted.internet import defer
 
 from mock import Mock
 
-from ..utils import MockHttpServer
+from ..utils import MockHttpResource
 
 from synapse.api.errors import SynapseError, AuthError
 from synapse.server import HomeServer
@@ -32,7 +32,7 @@ class ProfileTestCase(unittest.TestCase):
     """ Tests profile management. """
 
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
         self.mock_handler = Mock(spec=[
             "get_displayname",
             "set_displayname",
@@ -43,9 +43,10 @@ class ProfileTestCase(unittest.TestCase):
         hs = HomeServer("test",
             db_pool=None,
             http_client=None,
-            http_server=self.mock_server,
+            resource_for_client=self.mock_resource,
             federation=Mock(),
             replication_layer=Mock(),
+            datastore=None,
         )
 
         def _get_user_by_token(token=None):
@@ -62,7 +63,7 @@ class ProfileTestCase(unittest.TestCase):
         mocked_get = self.mock_handler.get_displayname
         mocked_get.return_value = defer.succeed("Frank")
 
-        (code, response) = yield self.mock_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/profile/%s/displayname" % (myid), None)
 
         self.assertEquals(200, code)
@@ -74,7 +75,7 @@ class ProfileTestCase(unittest.TestCase):
         mocked_set = self.mock_handler.set_displayname
         mocked_set.return_value = defer.succeed(())
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                 "/profile/%s/displayname" % (myid),
                 '{"displayname": "Frank Jr."}')
 
@@ -88,7 +89,7 @@ class ProfileTestCase(unittest.TestCase):
         mocked_set = self.mock_handler.set_displayname
         mocked_set.side_effect = AuthError(400, "message")
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                 "/profile/%s/displayname" % ("@4567:test"), '"Frank Jr."')
 
         self.assertTrue(400 <= code < 499,
@@ -99,7 +100,7 @@ class ProfileTestCase(unittest.TestCase):
         mocked_get = self.mock_handler.get_displayname
         mocked_get.return_value = defer.succeed("Bob")
 
-        (code, response) = yield self.mock_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/profile/%s/displayname" % ("@opaque:elsewhere"), None)
 
         self.assertEquals(200, code)
@@ -110,7 +111,7 @@ class ProfileTestCase(unittest.TestCase):
         mocked_set = self.mock_handler.set_displayname
         mocked_set.side_effect = SynapseError(400, "message")
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                 "/profile/%s/displayname" % ("@opaque:elsewhere"), None)
 
         self.assertTrue(400 <= code <= 499,
@@ -121,7 +122,7 @@ class ProfileTestCase(unittest.TestCase):
         mocked_get = self.mock_handler.get_avatar_url
         mocked_get.return_value = defer.succeed("http://my.server/me.png")
 
-        (code, response) = yield self.mock_server.trigger("GET",
+        (code, response) = yield self.mock_resource.trigger("GET",
                 "/profile/%s/avatar_url" % (myid), None)
 
         self.assertEquals(200, code)
@@ -133,7 +134,7 @@ class ProfileTestCase(unittest.TestCase):
         mocked_set = self.mock_handler.set_avatar_url
         mocked_set.return_value = defer.succeed(())
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                 "/profile/%s/avatar_url" % (myid),
                 '{"avatar_url": "http://my.server/pic.gif"}')
 

tests/rest/test_rooms.py

@@ -27,7 +27,7 @@ from synapse.server import HomeServer
 import json
 import urllib
 
-from ..utils import MockHttpServer, MemoryDataStore
+from ..utils import MockHttpResource, MemoryDataStore
 from .utils import RestTestCase
 
 from mock import Mock
@@ -42,7 +42,7 @@ class RoomPermissionsTestCase(RestTestCase):
 
     @defer.inlineCallbacks
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
 
         state_handler = Mock(spec=["handle_new_event"])
         state_handler.handle_new_event.return_value = True
@@ -67,7 +67,7 @@ class RoomPermissionsTestCase(RestTestCase):
 
         self.auth_user_id = self.rmcreator_id
 
-        synapse.rest.room.register_servlets(hs, self.mock_server)
+        synapse.rest.room.register_servlets(hs, self.mock_resource)
 
         self.auth = hs.get_auth()
 
@@ -85,14 +85,14 @@ class RoomPermissionsTestCase(RestTestCase):
         # send a message in one of the rooms
         self.created_rmid_msg_path = ("/rooms/%s/messages/%s/midaaa1" %
                                 (self.created_rmid, self.rmcreator_id))
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT",
                            self.created_rmid_msg_path,
                            '{"msgtype":"m.text","body":"test msg"}')
         self.assertEquals(200, code, msg=str(response))
 
         # set topic for public room
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT",
                            "/rooms/%s/topic" % self.created_public_rmid,
                            '{"topic":"Public Room Topic"}')
@@ -104,36 +104,36 @@ class RoomPermissionsTestCase(RestTestCase):
     def tearDown(self):
         pass
 
-    @defer.inlineCallbacks
-    def test_get_message(self):
-        # get message in uncreated room, expect 403
-        (code, response) = yield self.mock_server.trigger_get(
-                           "/rooms/noroom/messages/someid/m1")
-        self.assertEquals(403, code, msg=str(response))
-
-        # get message in created room not joined (no state), expect 403
-        (code, response) = yield self.mock_server.trigger_get(
-                           self.created_rmid_msg_path)
-        self.assertEquals(403, code, msg=str(response))
-
-        # get message in created room and invited, expect 403
-        yield self.invite(room=self.created_rmid, src=self.rmcreator_id,
-                          targ=self.user_id)
-        (code, response) = yield self.mock_server.trigger_get(
-                           self.created_rmid_msg_path)
-        self.assertEquals(403, code, msg=str(response))
-
-        # get message in created room and joined, expect 200
-        yield self.join(room=self.created_rmid, user=self.user_id)
-        (code, response) = yield self.mock_server.trigger_get(
-                           self.created_rmid_msg_path)
-        self.assertEquals(200, code, msg=str(response))
-
-        # get message in created room and left, expect 403
-        yield self.leave(room=self.created_rmid, user=self.user_id)
-        (code, response) = yield self.mock_server.trigger_get(
-                           self.created_rmid_msg_path)
-        self.assertEquals(403, code, msg=str(response))
+#    @defer.inlineCallbacks
+#    def test_get_message(self):
+#        # get message in uncreated room, expect 403
+#        (code, response) = yield self.mock_resource.trigger_get(
+#                           "/rooms/noroom/messages/someid/m1")
+#        self.assertEquals(403, code, msg=str(response))
+#
+#        # get message in created room not joined (no state), expect 403
+#        (code, response) = yield self.mock_resource.trigger_get(
+#                           self.created_rmid_msg_path)
+#        self.assertEquals(403, code, msg=str(response))
+#
+#        # get message in created room and invited, expect 403
+#        yield self.invite(room=self.created_rmid, src=self.rmcreator_id,
+#                          targ=self.user_id)
+#        (code, response) = yield self.mock_resource.trigger_get(
+#                           self.created_rmid_msg_path)
+#        self.assertEquals(403, code, msg=str(response))
+#
+#        # get message in created room and joined, expect 200
+#        yield self.join(room=self.created_rmid, user=self.user_id)
+#        (code, response) = yield self.mock_resource.trigger_get(
+#                           self.created_rmid_msg_path)
+#        self.assertEquals(200, code, msg=str(response))
+#
+#        # get message in created room and left, expect 403
+#        yield self.leave(room=self.created_rmid, user=self.user_id)
+#        (code, response) = yield self.mock_resource.trigger_get(
+#                           self.created_rmid_msg_path)
+#        self.assertEquals(403, code, msg=str(response))
 
     @defer.inlineCallbacks
     def test_send_message(self):
@@ -142,33 +142,33 @@ class RoomPermissionsTestCase(RestTestCase):
                         (self.created_rmid, self.user_id))
 
         # send message in uncreated room, expect 403
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT",
                            "/rooms/%s/messages/%s/mid1" %
                            (self.uncreated_rmid, self.user_id), msg_content)
         self.assertEquals(403, code, msg=str(response))
 
         # send message in created room not joined (no state), expect 403
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT", send_msg_path, msg_content)
         self.assertEquals(403, code, msg=str(response))
 
         # send message in created room and invited, expect 403
         yield self.invite(room=self.created_rmid, src=self.rmcreator_id,
                           targ=self.user_id)
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT", send_msg_path, msg_content)
         self.assertEquals(403, code, msg=str(response))
 
         # send message in created room and joined, expect 200
         yield self.join(room=self.created_rmid, user=self.user_id)
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT", send_msg_path, msg_content)
         self.assertEquals(200, code, msg=str(response))
 
         # send message in created room and left, expect 403
         yield self.leave(room=self.created_rmid, user=self.user_id)
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT", send_msg_path, msg_content)
         self.assertEquals(403, code, msg=str(response))
 
@@ -178,56 +178,56 @@ class RoomPermissionsTestCase(RestTestCase):
         topic_path = "/rooms/%s/topic" % self.created_rmid
 
         # set/get topic in uncreated room, expect 403
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT", "/rooms/%s/topic" % self.uncreated_rmid,
                            topic_content)
         self.assertEquals(403, code, msg=str(response))
-        (code, response) = yield self.mock_server.trigger_get(
+        (code, response) = yield self.mock_resource.trigger_get(
                            "/rooms/%s/topic" % self.uncreated_rmid)
         self.assertEquals(403, code, msg=str(response))
 
         # set/get topic in created PRIVATE room not joined, expect 403
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT", topic_path, topic_content)
         self.assertEquals(403, code, msg=str(response))
-        (code, response) = yield self.mock_server.trigger_get(topic_path)
+        (code, response) = yield self.mock_resource.trigger_get(topic_path)
         self.assertEquals(403, code, msg=str(response))
 
         # set topic in created PRIVATE room and invited, expect 403
         yield self.invite(room=self.created_rmid, src=self.rmcreator_id,
                           targ=self.user_id)
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT", topic_path, topic_content)
         self.assertEquals(403, code, msg=str(response))
 
         # get topic in created PRIVATE room and invited, expect 200 (or 404)
-        (code, response) = yield self.mock_server.trigger_get(topic_path)
+        (code, response) = yield self.mock_resource.trigger_get(topic_path)
         self.assertEquals(404, code, msg=str(response))
 
         # set/get topic in created PRIVATE room and joined, expect 200
         yield self.join(room=self.created_rmid, user=self.user_id)
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT", topic_path, topic_content)
         self.assertEquals(200, code, msg=str(response))
-        (code, response) = yield self.mock_server.trigger_get(topic_path)
+        (code, response) = yield self.mock_resource.trigger_get(topic_path)
         self.assertEquals(200, code, msg=str(response))
         self.assert_dict(json.loads(topic_content), response)
 
         # set/get topic in created PRIVATE room and left, expect 403
         yield self.leave(room=self.created_rmid, user=self.user_id)
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT", topic_path, topic_content)
         self.assertEquals(403, code, msg=str(response))
-        (code, response) = yield self.mock_server.trigger_get(topic_path)
+        (code, response) = yield self.mock_resource.trigger_get(topic_path)
         self.assertEquals(403, code, msg=str(response))
 
         # get topic in PUBLIC room, not joined, expect 200 (or 404)
-        (code, response) = yield self.mock_server.trigger_get(
+        (code, response) = yield self.mock_resource.trigger_get(
                            "/rooms/%s/topic" % self.created_public_rmid)
         self.assertEquals(200, code, msg=str(response))
 
         # set topic in PUBLIC room, not joined, expect 403
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
                            "PUT",
                            "/rooms/%s/topic" % self.created_public_rmid,
                            topic_content)
@@ -237,7 +237,7 @@ class RoomPermissionsTestCase(RestTestCase):
     def _test_get_membership(self, room=None, members=[], expect_code=None):
         path = "/rooms/%s/members/%s/state"
         for member in members:
-            (code, response) = yield self.mock_server.trigger_get(
+            (code, response) = yield self.mock_resource.trigger_get(
                                path %
                                (room, member))
             self.assertEquals(expect_code, code)
@@ -391,7 +391,7 @@ class RoomsMemberListTestCase(RestTestCase):
     user_id = "@sid1:red"
 
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
 
         state_handler = Mock(spec=["handle_new_event"])
         state_handler.handle_new_event.return_value = True
@@ -416,7 +416,7 @@ class RoomsMemberListTestCase(RestTestCase):
             return hs.parse_userid(self.auth_user_id)
         hs.get_auth().get_user_by_token = _get_user_by_token
 
-        synapse.rest.room.register_servlets(hs, self.mock_server)
+        synapse.rest.room.register_servlets(hs, self.mock_resource)
 
     def tearDown(self):
         pass
@@ -425,13 +425,13 @@ class RoomsMemberListTestCase(RestTestCase):
     def test_get_member_list(self):
         room_id = "!aa:test"
         yield self.create_room_as(room_id, self.user_id)
-        (code, response) = yield self.mock_server.trigger_get(
+        (code, response) = yield self.mock_resource.trigger_get(
                            "/rooms/%s/members/list" % room_id)
         self.assertEquals(200, code, msg=str(response))
 
     @defer.inlineCallbacks
     def test_get_member_list_no_room(self):
-        (code, response) = yield self.mock_server.trigger_get(
+        (code, response) = yield self.mock_resource.trigger_get(
                            "/rooms/roomdoesnotexist/members/list")
         self.assertEquals(403, code, msg=str(response))
 
@@ -439,7 +439,7 @@ class RoomsMemberListTestCase(RestTestCase):
     def test_get_member_list_no_permission(self):
         room_id = "!bb:test"
         yield self.create_room_as(room_id, "@some_other_guy:red")
-        (code, response) = yield self.mock_server.trigger_get(
+        (code, response) = yield self.mock_resource.trigger_get(
                            "/rooms/%s/members/list" % room_id)
         self.assertEquals(403, code, msg=str(response))
 
@@ -452,17 +452,17 @@ class RoomsMemberListTestCase(RestTestCase):
         yield self.invite(room=room_id, src=room_creator,
                           targ=self.user_id)
         # can't see list if you're just invited.
-        (code, response) = yield self.mock_server.trigger_get(room_path)
+        (code, response) = yield self.mock_resource.trigger_get(room_path)
         self.assertEquals(403, code, msg=str(response))
 
         yield self.join(room=room_id, user=self.user_id)
         # can see list now joined
-        (code, response) = yield self.mock_server.trigger_get(room_path)
+        (code, response) = yield self.mock_resource.trigger_get(room_path)
         self.assertEquals(200, code, msg=str(response))
 
         yield self.leave(room=room_id, user=self.user_id)
         # can no longer see list, you've left.
-        (code, response) = yield self.mock_server.trigger_get(room_path)
+        (code, response) = yield self.mock_resource.trigger_get(room_path)
         self.assertEquals(403, code, msg=str(response))
 
 
@@ -471,7 +471,7 @@ class RoomsCreateTestCase(RestTestCase):
     user_id = "@sid1:red"
 
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
         self.auth_user_id = self.user_id
 
         state_handler = Mock(spec=["handle_new_event"])
@@ -495,7 +495,7 @@ class RoomsCreateTestCase(RestTestCase):
             return hs.parse_userid(self.auth_user_id)
         hs.get_auth().get_user_by_token = _get_user_by_token
 
-        synapse.rest.room.register_servlets(hs, self.mock_server)
+        synapse.rest.room.register_servlets(hs, self.mock_resource)
 
     def tearDown(self):
         pass
@@ -503,7 +503,7 @@ class RoomsCreateTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_post_room_no_keys(self):
         # POST with no config keys, expect new room id
-        (code, response) = yield self.mock_server.trigger("POST", "/rooms",
+        (code, response) = yield self.mock_resource.trigger("POST", "/rooms",
                                                           "{}")
         self.assertEquals(200, code, response)
         self.assertTrue("room_id" in response)
@@ -511,7 +511,7 @@ class RoomsCreateTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_post_room_visibility_key(self):
         # POST with visibility config key, expect new room id
-        (code, response) = yield self.mock_server.trigger("POST", "/rooms",
+        (code, response) = yield self.mock_resource.trigger("POST", "/rooms",
                                                 '{"visibility":"private"}')
         self.assertEquals(200, code)
         self.assertTrue("room_id" in response)
@@ -519,7 +519,7 @@ class RoomsCreateTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_post_room_custom_key(self):
         # POST with custom config keys, expect new room id
-        (code, response) = yield self.mock_server.trigger("POST", "/rooms",
+        (code, response) = yield self.mock_resource.trigger("POST", "/rooms",
                                                 '{"custom":"stuff"}')
         self.assertEquals(200, code)
         self.assertTrue("room_id" in response)
@@ -527,7 +527,7 @@ class RoomsCreateTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_post_room_known_and_unknown_keys(self):
         # POST with custom + known config keys, expect new room id
-        (code, response) = yield self.mock_server.trigger("POST", "/rooms",
+        (code, response) = yield self.mock_resource.trigger("POST", "/rooms",
                                  '{"visibility":"private","custom":"things"}')
         self.assertEquals(200, code)
         self.assertTrue("room_id" in response)
@@ -535,18 +535,18 @@ class RoomsCreateTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_post_room_invalid_content(self):
         # POST with invalid content / paths, expect 400
-        (code, response) = yield self.mock_server.trigger("POST", "/rooms",
+        (code, response) = yield self.mock_resource.trigger("POST", "/rooms",
                                                           '{"visibili')
         self.assertEquals(400, code)
 
-        (code, response) = yield self.mock_server.trigger("POST", "/rooms",
+        (code, response) = yield self.mock_resource.trigger("POST", "/rooms",
                                                           '["hello"]')
         self.assertEquals(400, code)
 
     @defer.inlineCallbacks
     def test_put_room_no_keys(self):
         # PUT with no config keys, expect new room id
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
             "PUT", "/rooms/%21aa%3Atest", "{}"
         )
         self.assertEquals(200, code)
@@ -555,7 +555,7 @@ class RoomsCreateTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_put_room_visibility_key(self):
         # PUT with known config keys, expect new room id
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
             "PUT", "/rooms/%21bb%3Atest", '{"visibility":"private"}'
         )
         self.assertEquals(200, code)
@@ -564,7 +564,7 @@ class RoomsCreateTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_put_room_custom_key(self):
         # PUT with custom config keys, expect new room id
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
             "PUT", "/rooms/%21cc%3Atest", '{"custom":"stuff"}'
         )
         self.assertEquals(200, code)
@@ -573,7 +573,7 @@ class RoomsCreateTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_put_room_known_and_unknown_keys(self):
         # PUT with custom + known config keys, expect new room id
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
             "PUT", "/rooms/%21dd%3Atest",
             '{"visibility":"private","custom":"things"}'
         )
@@ -584,12 +584,12 @@ class RoomsCreateTestCase(RestTestCase):
     def test_put_room_invalid_content(self):
         # PUT with invalid content / room names, expect 400
 
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
             "PUT", "/rooms/ee", '{"sdf"'
         )
         self.assertEquals(400, code)
 
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
             "PUT", "/rooms/ee", '["hello"]'
         )
         self.assertEquals(400, code)
@@ -599,7 +599,7 @@ class RoomsCreateTestCase(RestTestCase):
         yield self.create_room_as("!aa:test", self.user_id)
 
         # PUT with conflicting room ID, expect 409
-        (code, response) = yield self.mock_server.trigger(
+        (code, response) = yield self.mock_resource.trigger(
             "PUT", "/rooms/%21aa%3Atest", "{}"
         )
         self.assertEquals(409, code)
@@ -611,7 +611,7 @@ class RoomTopicTestCase(RestTestCase):
 
     @defer.inlineCallbacks
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
         self.auth_user_id = self.user_id
         self.room_id = "!rid1:test"
         self.path = "/rooms/%s/topic" % self.room_id
@@ -637,7 +637,7 @@ class RoomTopicTestCase(RestTestCase):
             return hs.parse_userid(self.auth_user_id)
         hs.get_auth().get_user_by_token = _get_user_by_token
 
-        synapse.rest.room.register_servlets(hs, self.mock_server)
+        synapse.rest.room.register_servlets(hs, self.mock_resource)
 
         # create the room
         yield self.create_room_as(self.room_id, self.user_id)
@@ -648,50 +648,50 @@ class RoomTopicTestCase(RestTestCase):
     @defer.inlineCallbacks
     def test_invalid_puts(self):
         # missing keys or invalid json
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            self.path, '{}')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            self.path, '{"_name":"bob"}')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            self.path, '{"nao')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            self.path, '[{"_name":"bob"},{"_name":"jill"}]')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            self.path, 'text only')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            self.path, '')
         self.assertEquals(400, code, msg=str(response))
 
         # valid key, wrong type
         content = '{"topic":["Topic name"]}'
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            self.path, content)
         self.assertEquals(400, code, msg=str(response))
 
     @defer.inlineCallbacks
     def test_rooms_topic(self):
         # nothing should be there
-        (code, response) = yield self.mock_server.trigger_get(self.path)
+        (code, response) = yield self.mock_resource.trigger_get(self.path)
         self.assertEquals(404, code, msg=str(response))
 
         # valid put
         content = '{"topic":"Topic name"}'
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            self.path, content)
         self.assertEquals(200, code, msg=str(response))
 
         # valid get
-        (code, response) = yield self.mock_server.trigger_get(self.path)
+        (code, response) = yield self.mock_resource.trigger_get(self.path)
         self.assertEquals(200, code, msg=str(response))
         self.assert_dict(json.loads(content), response)
 
@@ -699,12 +699,12 @@ class RoomTopicTestCase(RestTestCase):
     def test_rooms_topic_with_extra_keys(self):
         # valid put with extra keys
         content = '{"topic":"Seasons","subtopic":"Summer"}'
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            self.path, content)
         self.assertEquals(200, code, msg=str(response))
 
         # valid get
-        (code, response) = yield self.mock_server.trigger_get(self.path)
+        (code, response) = yield self.mock_resource.trigger_get(self.path)
         self.assertEquals(200, code, msg=str(response))
         self.assert_dict(json.loads(content), response)
 
@@ -715,7 +715,7 @@ class RoomMemberStateTestCase(RestTestCase):
 
     @defer.inlineCallbacks
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
         self.auth_user_id = self.user_id
         self.room_id = "!rid1:test"
 
@@ -740,7 +740,7 @@ class RoomMemberStateTestCase(RestTestCase):
             return hs.parse_userid(self.auth_user_id)
         hs.get_auth().get_user_by_token = _get_user_by_token
 
-        synapse.rest.room.register_servlets(hs, self.mock_server)
+        synapse.rest.room.register_servlets(hs, self.mock_resource)
 
         yield self.create_room_as(self.room_id, self.user_id)
 
@@ -751,34 +751,34 @@ class RoomMemberStateTestCase(RestTestCase):
     def test_invalid_puts(self):
         path = "/rooms/%s/members/%s/state" % (self.room_id, self.user_id)
         # missing keys or invalid json
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '{}')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '{"_name":"bob"}')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '{"nao')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '[{"_name":"bob"},{"_name":"jill"}]')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, 'text only')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '')
         self.assertEquals(400, code, msg=str(response))
 
         # valid keys, wrong types
         content = ('{"membership":["%s","%s","%s"]}' %
                   (Membership.INVITE, Membership.JOIN, Membership.LEAVE))
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(400, code, msg=str(response))
 
     @defer.inlineCallbacks
@@ -789,12 +789,17 @@ class RoomMemberStateTestCase(RestTestCase):
 
         # valid join message (NOOP since we made the room)
         content = '{"membership":"%s"}' % Membership.JOIN
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(200, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("GET", path, None)
+        (code, response) = yield self.mock_resource.trigger("GET", path, None)
         self.assertEquals(200, code, msg=str(response))
-        self.assertEquals(json.loads(content), response)
+
+        expected_response = {
+            "membership": Membership.JOIN,
+            "prev": Membership.JOIN,
+        }
+        self.assertEquals(expected_response, response)
 
     @defer.inlineCallbacks
     def test_rooms_members_other(self):
@@ -805,10 +810,10 @@ class RoomMemberStateTestCase(RestTestCase):
 
         # valid invite message
         content = '{"membership":"%s"}' % Membership.INVITE
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(200, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("GET", path, None)
+        (code, response) = yield self.mock_resource.trigger("GET", path, None)
         self.assertEquals(200, code, msg=str(response))
         self.assertEquals(json.loads(content), response)
 
@@ -822,10 +827,10 @@ class RoomMemberStateTestCase(RestTestCase):
         # valid invite message with custom key
         content = ('{"membership":"%s","invite_text":"%s"}' %
                     (Membership.INVITE, "Join us!"))
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(200, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("GET", path, None)
+        (code, response) = yield self.mock_resource.trigger("GET", path, None)
         self.assertEquals(200, code, msg=str(response))
         self.assertEquals(json.loads(content), response)
 
@@ -836,7 +841,7 @@ class RoomMessagesTestCase(RestTestCase):
 
     @defer.inlineCallbacks
     def setUp(self):
-        self.mock_server = MockHttpServer(prefix=PATH_PREFIX)
+        self.mock_resource = MockHttpResource(prefix=PATH_PREFIX)
         self.auth_user_id = self.user_id
         self.room_id = "!rid1:test"
 
@@ -861,7 +866,7 @@ class RoomMessagesTestCase(RestTestCase):
             return hs.parse_userid(self.auth_user_id)
         hs.get_auth().get_user_by_token = _get_user_by_token
 
-        synapse.rest.room.register_servlets(hs, self.mock_server)
+        synapse.rest.room.register_servlets(hs, self.mock_resource)
 
         yield self.create_room_as(self.room_id, self.user_id)
 
@@ -874,27 +879,27 @@ class RoomMessagesTestCase(RestTestCase):
             urllib.quote(self.room_id), self.user_id
         )
         # missing keys or invalid json
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '{}')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '{"_name":"bob"}')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '{"nao')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '[{"_name":"bob"},{"_name":"jill"}]')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, 'text only')
         self.assertEquals(400, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("PUT",
+        (code, response) = yield self.mock_resource.trigger("PUT",
                            path, '')
         self.assertEquals(400, code, msg=str(response))
 
@@ -905,34 +910,34 @@ class RoomMessagesTestCase(RestTestCase):
         )
 
         content = '{"body":"test","msgtype":{"type":"a"}}'
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(400, code, msg=str(response))
 
         # custom message types
         content = '{"body":"test","msgtype":"test.custom.text"}'
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(200, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("GET", path, None)
-        self.assertEquals(200, code, msg=str(response))
-        self.assert_dict(json.loads(content), response)
+#        (code, response) = yield self.mock_resource.trigger("GET", path, None)
+#        self.assertEquals(200, code, msg=str(response))
+#        self.assert_dict(json.loads(content), response)
 
         # m.text message type
         path = "/rooms/%s/messages/%s/mid2" % (
             urllib.quote(self.room_id), self.user_id
         )
         content = '{"body":"test2","msgtype":"m.text"}'
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(200, code, msg=str(response))
 
-        (code, response) = yield self.mock_server.trigger("GET", path, None)
-        self.assertEquals(200, code, msg=str(response))
-        self.assert_dict(json.loads(content), response)
+#        (code, response) = yield self.mock_resource.trigger("GET", path, None)
+#        self.assertEquals(200, code, msg=str(response))
+#        self.assert_dict(json.loads(content), response)
 
         # trying to send message in different user path
         path = "/rooms/%s/messages/%s/mid2" % (
             urllib.quote(self.room_id), "invalid" + self.user_id
         )
         content = '{"body":"test2","msgtype":"m.text"}'
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(403, code, msg=str(response))

tests/rest/utils.py

@@ -27,12 +27,12 @@ class RestTestCase(unittest.TestCase):
     """Contains extra helper functions to quickly and clearly perform a given
     REST action, which isn't the focus of the test.
 
-    This subclass assumes there are mock_server and auth_user_id attributes.
+    This subclass assumes there are mock_resource and auth_user_id attributes.
     """
 
     def __init__(self, *args, **kwargs):
         super(RestTestCase, self).__init__(*args, **kwargs)
-        self.mock_server = None
+        self.mock_resource = None
         self.auth_user_id = None
 
     def mock_get_user_by_token(self, token=None):
@@ -48,7 +48,7 @@ class RestTestCase(unittest.TestCase):
             content = '{"visibility":"private"}'
         if tok:
             path = path + "?access_token=%s" % tok
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(200, code, msg=str(response))
         self.auth_user_id = temp_id
 
@@ -81,11 +81,11 @@ class RestTestCase(unittest.TestCase):
             path = path + "?access_token=%s" % tok
 
         if membership == Membership.LEAVE:
-            (code, response) = yield self.mock_server.trigger("DELETE", path,
+            (code, response) = yield self.mock_resource.trigger("DELETE", path,
                                     None)
             self.assertEquals(expect_code, code, msg=str(response))
         else:
-            (code, response) = yield self.mock_server.trigger("PUT", path,
+            (code, response) = yield self.mock_resource.trigger("PUT", path,
                                     '{"membership":"%s"}' % membership)
             self.assertEquals(expect_code, code, msg=str(response))
 
@@ -93,7 +93,7 @@ class RestTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def register(self, user_id):
-        (code, response) = yield self.mock_server.trigger("POST", "/register",
+        (code, response) = yield self.mock_resource.trigger("POST", "/register",
                                 '{"user_id":"%s"}' % user_id)
         self.assertEquals(200, code)
         defer.returnValue(response)
@@ -111,7 +111,7 @@ class RestTestCase(unittest.TestCase):
         if tok:
             path = path + "?access_token=%s" % tok
 
-        (code, response) = yield self.mock_server.trigger("PUT", path, content)
+        (code, response) = yield self.mock_resource.trigger("PUT", path, content)
         self.assertEquals(expect_code, code, msg=str(response))
 
     def assert_dict(self, required, actual):

tests/test_state.py

@@ -36,7 +36,8 @@ class StateTestCase(unittest.TestCase):
             "get_unresolved_state_tree",
             "update_current_state",
             "get_latest_pdus_in_context",
-            "get_current_state",
+            "get_current_state_pdu",
+            "get_pdu",
         ])
         self.replication = Mock(spec=["get_pdu"])
 
@@ -220,6 +221,8 @@ class StateTestCase(unittest.TestCase):
 
         self.replication.get_pdu.side_effect = set_return_tree
 
+        self.persistence.get_pdu.return_value = None
+
         is_new = yield self.state.handle_new_state(new_pdu)
 
         self.assertTrue(is_new)
@@ -244,7 +247,7 @@ class StateTestCase(unittest.TestCase):
         pdus = [tup]
 
         self.persistence.get_latest_pdus_in_context.return_value = pdus
-        self.persistence.get_current_state.return_value = state_pdu
+        self.persistence.get_current_state_pdu.return_value = state_pdu
 
         yield self.state.handle_new_event(event)
 

tests/utils.py

@@ -29,7 +29,8 @@ import json
 import urlparse
 
 
-class MockHttpServer(HttpServer):
+# This is a mock /resource/ not an entire server
+class MockHttpResource(HttpServer):
 
     def __init__(self, prefix=""):
         self.callbacks = []  # 3-tuple of method/pattern/function
@@ -111,35 +112,20 @@ class MockClock(object):
 
 class MemoryDataStore(object):
 
-    class RoomMember(namedtuple(
-        "RoomMember",
-        ["room_id", "user_id", "sender", "membership", "content"]
-    )):
-        def as_event(self, event_factory):
-            return event_factory.create_event(
-                etype=RoomMemberEvent.TYPE,
-                room_id=self.room_id,
-                target_user_id=self.user_id,
-                user_id=self.sender,
-                content=json.loads(self.content),
-            )
-
-    PathData = namedtuple("PathData",
-                          ["room_id", "path", "content"])
-
-    Message = namedtuple("Message",
-                         ["room_id", "msg_id", "user_id", "content"])
-
-    Room = namedtuple("Room",
-                      ["room_id", "is_public", "creator"])
+    Room = namedtuple(
+        "Room",
+        ["room_id", "is_public", "creator"]
+    )
 
     def __init__(self):
         self.tokens_to_users = {}
         self.paths_to_content = {}
+
         self.members = {}
-        self.messages = {}
         self.rooms = {}
-        self.room_members = {}
+
+        self.current_state = {}
+        self.events = []
 
     def register(self, user_id, token, password_hash):
         if user_id in self.tokens_to_users.values():
@@ -162,120 +148,106 @@ class MemoryDataStore(object):
         if room_id in self.rooms:
             raise StoreError(409, "Conflicting room!")
 
-        room = MemoryDataStore.Room(room_id=room_id, is_public=is_public,
-                    creator=room_creator_user_id)
+        room = MemoryDataStore.Room(
+            room_id=room_id,
+            is_public=is_public,
+            creator=room_creator_user_id
+        )
         self.rooms[room_id] = room
-        #self.store_room_member(user_id=room_creator_user_id, room_id=room_id,
-                               #membership=Membership.JOIN,
-                               #content={"membership": Membership.JOIN})
 
-    def get_message(self, user_id=None, room_id=None, msg_id=None):
-        try:
-            return self.messages[user_id + room_id + msg_id]
-        except:
-            return None
+    def get_room_member(self, user_id, room_id):
+        return self.members.get(room_id, {}).get(user_id)
 
-    def store_message(self, user_id=None, room_id=None, msg_id=None,
-                      content=None):
-        msg = MemoryDataStore.Message(room_id=room_id, msg_id=msg_id,
-                    user_id=user_id, content=content)
-        self.messages[user_id + room_id + msg_id] = msg
-
-    def get_room_member(self, user_id=None, room_id=None):
-        try:
-            return self.members[user_id + room_id]
-        except:
-            return None
-
-    def get_room_members(self, room_id=None, membership=None):
-        try:
-            return self.room_members[room_id]
-        except:
-            return None
+    def get_room_members(self, room_id, membership=None):
+        if membership:
+            return [
+                v for k, v in self.members.get(room_id, {}).items()
+                if v.membership == membership
+            ]
+        else:
+            return self.members.get(room_id, {}).values()
 
     def get_rooms_for_user_where_membership_is(self, user_id, membership_list):
-        return [r for r in self.room_members
-                if user_id in self.room_members[r]]
+        return [
+            r for r in self.members
+            if self.members[r].get(user_id).membership in membership_list
+        ]
 
-    def store_room_member(self, user_id=None, sender=None, room_id=None,
-                          membership=None, content=None):
-        member = MemoryDataStore.RoomMember(room_id=room_id, user_id=user_id,
-            sender=sender, membership=membership, content=json.dumps(content))
-        self.members[user_id + room_id] = member
-
-        # TODO should be latest state
-        if room_id not in self.room_members:
-            self.room_members[room_id] = []
-        self.room_members[room_id].append(member)
-
-    def get_room_data(self, room_id, etype, state_key=""):
-        path = "%s-%s-%s" % (room_id, etype, state_key)
-        try:
-            return self.paths_to_content[path]
-        except:
-            return None
-
-    def store_room_data(self, room_id, etype, state_key="", content=None):
-        path = "%s-%s-%s" % (room_id, etype, state_key)
-        data = MemoryDataStore.PathData(path=path, room_id=room_id,
-                    content=content)
-        self.paths_to_content[path] = data
-
-    def get_message_stream(self, user_id=None, from_key=None, to_key=None,
+    def get_room_events_stream(self, user_id=None, from_key=None, to_key=None,
                             room_id=None, limit=0, with_feedback=False):
         return ([], from_key)  # TODO
 
-    def get_room_member_stream(self, user_id=None, from_key=None, to_key=None):
-        return ([], from_key)  # TODO
-
-    def get_feedback_stream(self, user_id=None, from_key=None, to_key=None,
-                            room_id=None, limit=0):
-        return ([], from_key)  # TODO
-
-    def get_room_data_stream(self, user_id=None, from_key=None, to_key=None,
-                            room_id=None, limit=0):
-        return ([], from_key)  # TODO
-
-    def to_events(self, data_store_list):
-        return data_store_list  # TODO
-
-    def get_max_message_id(self):
-        return 0  # TODO
-
-    def get_max_feedback_id(self):
-        return 0  # TODO
-
-    def get_max_room_member_id(self):
-        return 0  # TODO
-
-    def get_max_room_data_id(self):
-        return 0  # TODO
-
     def get_joined_hosts_for_room(self, room_id):
         return defer.succeed([])
 
     def persist_event(self, event):
-        if event.type == MessageEvent.TYPE:
-            return self.store_message(
-                user_id=event.user_id,
-                room_id=event.room_id,
-                msg_id=event.msg_id,
-                content=json.dumps(event.content)
-            )
-        elif event.type == RoomMemberEvent.TYPE:
-            return self.store_room_member(
-                user_id=event.target_user_id,
-                room_id=event.room_id,
-                content=event.content,
-                membership=event.content["membership"]
-            )
+        if event.type == RoomMemberEvent.TYPE:
+            room_id = event.room_id
+            user = event.target_user_id
+            membership = event.membership
+            self.members.setdefault(room_id, {})[user] = event
+
+        if hasattr(event, "state_key"):
+            key = (event.room_id, event.type, event.state_key)
+            self.current_state[key] = event
+
+        self.events.append(event)
+
+    def get_current_state(self, room_id, event_type=None, state_key=""):
+        if event_type:
+            key = (room_id, event_type, state_key)
+            return self.current_state.get(key)
         else:
-            raise NotImplementedError(
-                "Don't know how to persist type=%s" % event.type
-            )
+            return [
+                e for e in self.current_state
+                if e[0] == room_id
+            ]
 
     def set_presence_state(self, user_localpart, state):
         return defer.succeed({"state": 0})
 
     def get_presence_list(self, user_localpart, accepted):
         return []
+
+    def get_room_events_max_id(self):
+        return 0  # TODO (erikj)
+
+def _format_call(args, kwargs):
+    return ", ".join(
+        ["%r" % (a) for a in args] + 
+        ["%s=%r" % (k, v) for k, v in kwargs.items()]
+    )
+
+
+class DeferredMockCallable(object):
+    """A callable instance that stores a set of pending call expectations and
+    return values for them. It allows a unit test to assert that the given set
+    of function calls are eventually made, by awaiting on them to be called.
+    """
+
+    def __init__(self):
+        self.expectations = []
+
+    def __call__(self, *args, **kwargs):
+        if not self.expectations:
+            raise ValueError("%r has no pending calls to handle call(%s)" % (
+                self, _format_call(args, kwargs))
+            )
+
+        for (call, result, d) in self.expectations:
+            if args == call[1] and kwargs == call[2]:
+                d.callback(None)
+                return result
+
+        raise AssertionError("Was not expecting call(%s)" %
+            _format_call(args, kwargs)
+        )
+
+    def expect_call_and_return(self, call, result):
+        self.expectations.append((call, result, defer.Deferred()))
+
+    @defer.inlineCallbacks
+    def await_calls(self):
+        while self.expectations:
+            (_, _, d) = self.expectations.pop(0)
+            yield d

webclient/app-controller.js

@@ -21,8 +21,8 @@ limitations under the License.
 'use strict';
 
 angular.module('MatrixWebClientController', ['matrixService'])
-.controller('MatrixWebClientController', ['$scope', '$location', '$rootScope', 'matrixService',
-                               function($scope, $location, $rootScope, matrixService) {
+.controller('MatrixWebClientController', ['$scope', '$location', '$rootScope', 'matrixService', 'eventStreamService',
+                               function($scope, $location, $rootScope, matrixService, eventStreamService) {
          
     // Check current URL to avoid to display the logout button on the login page
     $scope.location = $location.path();
@@ -44,19 +44,46 @@ angular.module('MatrixWebClientController', ['matrixService'])
         else {
             $scope.config = matrixService.config();        
         }
-    };    
+    };
     
+    $scope.closeConfig = function() {
+        if ($scope.config) {
+            $scope.config = undefined;
+        }
+    };
+
+    if (matrixService.isUserLoggedIn()) {
+        // eventStreamService.resume();
+    }
     
     // Logs the user out 
     $scope.logout = function() {
+        // kill the event stream
+        eventStreamService.stop();
+    
         // Clean permanent data
         matrixService.setConfig({});
         matrixService.saveConfig();
         
         // And go to the login page
-        $location.path("login");
-    };    
-                          
+        $location.url("login");
+    };
+
+    // Listen to the event indicating that the access token is no longer valid.
+    // In this case, the user needs to log in again.
+    $scope.$on("M_UNKNOWN_TOKEN", function() {
+        console.log("Invalid access token -> log user out");
+        $scope.logout();
+    });
+    
+    $scope.requestNotifications = function() {
+        if (window.Notification) {
+            console.log("Notification.permission: " + window.Notification.permission);
+            window.Notification.requestPermission(function(){});
+        }
+    };
+    
+    
 }]);
 
-   
+   

webclient/app-directive.js

@@ -0,0 +1,38 @@
+/*
+ Copyright 2014 matrix.org
+ 
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ 
+ http://www.apache.org/licenses/LICENSE-2.0
+ 
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+
+'use strict';
+
+angular.module('matrixWebClient')
+.directive('ngEnter', function () {
+    return function (scope, element, attrs) {
+        element.bind("keydown keypress", function (event) {
+            if(event.which === 13) {
+                scope.$apply(function () {
+                    scope.$eval(attrs.ngEnter);
+                });
+                event.preventDefault();
+            }
+        });
+    };
+})
+.directive('ngFocus', ['$timeout', function($timeout) {
+    return {
+        link: function(scope, element, attr) {
+            $timeout(function() { element[0].focus(); }, 0);
+        }
+    };
+}]);

webclient/app-filter.js

@@ -0,0 +1,82 @@
+/*
+ Copyright 2014 matrix.org
+ 
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ 
+ http://www.apache.org/licenses/LICENSE-2.0
+ 
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ */
+
+'use strict';
+
+angular.module('matrixWebClient')
+.filter('duration', function() {
+    return function(time) {
+        if (!time) return;
+        var t = parseInt(time / 1000);
+        var s = t % 60;
+        var m = parseInt(t / 60) % 60;
+        var h = parseInt(t / (60 * 60)) % 24;
+        var d = parseInt(t / (60 * 60 * 24));
+        if (t < 60) {
+            return s + "s";
+        }
+        if (t < 60 * 60) {
+            return m + "m "; //  + s + "s";
+        }
+        if (t < 24 * 60 * 60) {
+            return h + "h "; // + m + "m";
+        }
+        return d + "d "; // + h + "h";
+    };
+})
+.filter('orderMembersList', function($sce) {
+    return function(members) {
+        var filtered = [];
+
+        var displayNames = {};
+        angular.forEach(members, function(value, key) {
+            value["id"] = key;
+            filtered.push( value );
+            if (value["displayname"]) {
+                if (!displayNames[value["displayname"]]) {
+                    displayNames[value["displayname"]] = [];
+                }
+                displayNames[value["displayname"]].push(key);
+            }
+        });
+
+        // FIXME: we shouldn't disambiguate displayNames on every orderMembersList
+        // invocation but keep track of duplicates incrementally somewhere
+        angular.forEach(displayNames, function(value, key) {
+            if (value.length > 1) {
+                // console.log(key + ": " + value);
+                for (var i=0; i < value.length; i++) {
+                    var v = value[i];
+                    // FIXME: this permenantly rewrites the displayname for a given
+                    // room member. which means we can't reset their name if it is
+                    // no longer ambiguous!
+                    members[v].displayname += " (" + v + ")";
+                    // console.log(v + " " + members[v]);
+                };
+            }
+        });
+
+        filtered.sort(function (a, b) {
+            return ((a["mtime_age"] || 10e10) > (b["mtime_age"] || 10e10) ? 1 : -1);
+        });
+        return filtered;
+    };
+})
+.filter('unsafe', ['$sce', function($sce) {
+    return function(text) {
+        return $sce.trustAsHtml(text);
+    };
+}]);

webclient/app.css

@@ -10,7 +10,7 @@ h1 {
 
 /*** Overall page layout ***/
 
-.page {
+#page {
     position: absolute;
     top: 80px;
     bottom: 100px;
@@ -20,13 +20,13 @@ h1 {
     margin: 20px;
 }
 
-.wrapper {
+#wrapper {
     margin: auto;
     max-width: 1280px;
     height: 100%;
 }
 
-.roomName {
+#roomName {
     max-width: 1280px;
     width: 100%;
     text-align: right;
@@ -36,7 +36,7 @@ h1 {
     margin-bottom: 10px;
 }
 
-.controlPanel {
+#controlPanel {
     position: absolute;
     bottom: 0px;
     width: 100%;
@@ -44,39 +44,47 @@ h1 {
     border-top: #aaa 1px solid;
 }
 
-.controls {
+#controls {
     max-width: 1280px;
     padding: 12px;
     margin: auto;
 }
 
-.inputBarTable {
+#inputBarTable {
     width: 100%;
 }
 
-.inputBarTable tr td {
+#inputBarTable tr td {
     padding: 1px 4px;
 }
 
-.mainInput {
+#mainInput {
     width: 100%;
 }
 
+.blink {
+    background-color: #faa;
+}
+
+.mouse-pointer {
+    cursor: pointer;
+}
+
 /*** Participant list ***/
 
-.usersTableWrapper {
+#usersTableWrapper {
     float: right;
     width: 120px;
     height: 100%;
     overflow-y: auto;
 }
 
-.usersTable {
+#usersTable {
     width: 100%;
     border-collapse: collapse;
 }
 
-.usersTable td {
+#usersTable td {
     padding: 0px;
 }
 
@@ -90,6 +98,7 @@ h1 {
 .userAvatar .userAvatarImage {
     position: absolute;
     top: 0px;
+    object-fit: cover;    
 }
 
 .userAvatar .userAvatarGradient {
@@ -103,7 +112,7 @@ h1 {
     margin: 2px;
     bottom: 0px;
     font-size: 8pt;
-    word-wrap: break-word;
+    word-break: break-all;
 }
 
 .userPresence {
@@ -124,26 +133,28 @@ h1 {
 
 /*** Message table ***/
 
-.messageTableWrapper {
+#messageTableWrapper {
     height: 100%;
     margin-right: 140px;
     overflow-y: auto;
     width: auto;
 }
 
-.messageTable {
+#messageTable {
     margin: auto;
     max-width: 1280px;
     width: 100%;
     border-collapse: collapse;
+    table-layout: fixed;
 }
         
-.messageTable td {
+#messageTable td {
     padding: 0px;
 }
 
 .leftBlock {
-    width: 1px;
+    width: 14em;
+    word-wrap: break-word;
     vertical-align: top;
     background-color: #fff;
     color: #888;
@@ -176,19 +187,18 @@ h1 {
     vertical-align: top;
     line-height: 0;
 }
+
+.avatarImage {
+    object-fit: cover;
+}
         
-.text {
-    background-color: #eee;
-    border: 1px solid #d8d8d8;
-    height: 32px;
-    display: inline-table;
-    max-width: 90%;
-    word-wrap: break-word;
-    word-break: break-all;
+.emote {
+    background-color: transparent ! important;
+    border: 0px ! important;
 }
 
-.emote {
-    background-color: #fff ! important;
+.membership {
+    background-color: transparent ! important;
     border: 0px ! important;
 }
 
@@ -200,45 +210,111 @@ h1 {
     height: auto;
 }
 
+.text {
+    vertical-align: top;
+}
+
 .bubble {
-    padding: 6px;
+    background-color: #eee;
+    border: 1px solid #d8d8d8;
+    display: inline-block;
+    margin-bottom: -1px;
+    max-width: 90%;
+    font-size: 16px;
+    word-wrap: break-word;
+    padding-top: 7px;
+    padding-bottom: 5px;
     padding-left: 1em;
     padding-right: 1em;
+    vertical-align: middle;
+    -webkit-text-size-adjust:100%
+}
+
+.differentUser td {
+    padding-bottom: 5px ! important;
 }
 
 .mine {
     text-align: right;
 }
 
+.text.emote .bubble,
+.text.membership .bubble,
+.mine .text.emote .bubble,
+.mine .text.membership .bubble
+  {
+    background-color: transparent ! important;    
+    border: 0px ! important;
+}
+
 .mine .text .bubble {
+    background-color: #f8f8ff ! important;    
     text-align: left ! important;
-    background-color: #d8d8e8 ! important;
 }
 
-.mine .emote .bubble {
-    background-color: #fff ! important;
+#room-fullscreen-image {
+    position: absolute;
+    top: 0px;
+    height: 0px;
+    width: 100%;
+    height: 100%;
 }
 
+#room-fullscreen-image img {
+    max-width: 100%;
+    max-height: 100%;
+    bottom: 0;
+    left: 0;
+    margin: auto;
+    overflow: auto;
+    position: fixed;
+    right: 0;
+    top: 0;
+}
+
+/*** Profile ***/
+
+.profile-avatar {
+    width: 160px;
+    height: 160px;
+    display:table-cell;
+    vertical-align: middle;
+    text-align: center;
+}
+
+.profile-avatar img {
+    max-width: 100%;
+    max-height: 100%;
+}
+
+/*** User profile page ***/
+#user-ids {
+    padding-left: 1em;
+}
+
+#user-displayname {
+    font-size: 16pt;
+}
 /******************************/
 
-.header {
+#header {
     padding-left: 20px;
     padding-right: 20px;
     max-width: 1280px;
     margin: auto;
 }
 
-.header-buttons {
+#header-buttons {
     float: right;
 }
 
-.config {
+#config {
     position: absolute;
     z-index: 100;
     top: 100px;
     left: 50%;
-    width: 400px;
-    margin-left: -200px;
+    width: 500px;
+    margin-left: -250px;
     text-align: center;
     padding: 20px;
     background-color: #aaa;

webclient/app.js

@@ -20,17 +20,27 @@ var matrixWebClient = angular.module('matrixWebClient', [
     'LoginController',
     'RoomController',
     'RoomsController',
-    'matrixService'
+    'UserController',
+    'matrixService',
+    'eventStreamService',
+    'eventHandlerService',
+    'infinite-scroll'
 ]);
 
-matrixWebClient.config(['$routeProvider',
-    function($routeProvider) {
+matrixWebClient.config(['$routeProvider', '$provide', '$httpProvider',
+    function($routeProvider, $provide, $httpProvider) {
         $routeProvider.
             when('/login', {
                 templateUrl: 'login/login.html',
                 controller: 'LoginController'
             }).
-            when('/room/:room_id', {
+            when('/room/:room_id_or_alias', {
+                templateUrl: 'room/room.html',
+                controller: 'RoomController'
+            }).
+            when('/room/', {    // room URL with room alias in it (ex: http://127.0.0.1:8000/#/room/#public:localhost:8080) will come here.
+                                // The reason is that 2nd hash key breaks routeProvider parameters cutting so that the URL will not match with 
+                                // the previous '/room/:room_id_or_alias' URL rule
                 templateUrl: 'room/room.html',
                 controller: 'RoomController'
             }).
@@ -38,41 +48,38 @@ matrixWebClient.config(['$routeProvider',
                 templateUrl: 'rooms/rooms.html',
                 controller: 'RoomsController'
             }).
+            when('/user/:user_matrix_id', {
+                templateUrl: 'user/user.html',
+                controller: 'UserController'
+            }).
             otherwise({
                 redirectTo: '/rooms'
             });
+            
+        $provide.factory('AccessTokenInterceptor', ['$q', '$rootScope', 
+            function ($q, $rootScope) {
+            return {
+                responseError: function(rejection) {
+                    if (rejection.status === 403 && "data" in rejection && 
+                            "errcode" in rejection.data && 
+                            rejection.data.errcode === "M_UNKNOWN_TOKEN") {
+                        console.log("Got a 403 with an unknown token. Logging out.")
+                        $rootScope.$broadcast("M_UNKNOWN_TOKEN");
+                    }
+                    return $q.reject(rejection);
+                }
+            };
+        }]);
+        $httpProvider.interceptors.push('AccessTokenInterceptor');
     }]);
 
-matrixWebClient.run(['$location', 'matrixService' , function($location, matrixService) {
-    // If we have no persistent login information, go to the login page
-    var config = matrixService.config();
-    if (!config || !config.access_token) {
+matrixWebClient.run(['$location', 'matrixService', 'eventStreamService', function($location, matrixService, eventStreamService) {
+    // If user auth details are not in cache, go to the login page
+    if (!matrixService.isUserLoggedIn()) {
+        eventStreamService.stop();
         $location.path("login");
     }
+    else {
+        // eventStreamService.resume();
+    }
 }]);
-
-matrixWebClient
-    .directive('ngEnter', function () {
-        return function (scope, element, attrs) {
-            element.bind("keydown keypress", function (event) {
-                if(event.which === 13) {
-                    scope.$apply(function () {
-                        scope.$eval(attrs.ngEnter);
-                    });
-                    event.preventDefault();
-                }
-            });
-        };
-    })
-    .directive('ngFocus', ['$timeout', function($timeout) {
-        return {
-            link: function(scope, element, attr) {
-                $timeout(function() { element[0].focus() }, 0);
-            }
-        };
-    }])
-    .filter('to_trusted', ['$sce', function($sce){
-        return function(text) {
-            return $sce.trustAsHtml(text);
-        };
-    }]);