Bump versions and changelog

This uses the ignoreSynchronization flag because of the longpoll on the event
stream. It would be better to use $interval, but couldn't get that to
*reliably* work when testing. I suspect that $interval won't help us here,
since there is genuinely an open $http connection, as we're doing a long
poll. https://github.com/angular/protractor/issues/49 for more info.

.gitignore

@@ -25,5 +25,6 @@ graph/*.png
 graph/*.dot
 
 webclient/config.js
+webclient/test/environment-protractor.js
 
 uploads

CHANGES.rst

@@ -1,3 +1,76 @@
+Changes in synapse 0.3.3 (2014-09-22)
+=====================================
+
+Homeserver:
+ * Fix bug where you continued to get events for rooms you had left.
+
+Webclient:
+ * Add support for video calls with basic UI.
+ * Fix bug where one to one chats were named after your display name rather
+   than the other person's.
+ * Fix bug which caused lag when typing in the textarea.
+ * Refuse to run on browsers we know won't work.
+ * Trigger pagination when joining new rooms.
+ * Fix bug where we sometimes didn't display invitations in recents.
+ * Automatically join room when accepting a VoIP call.
+ * Disable outgoing and reject incoming calls on browsers we don't support
+   VoIP in.
+ * Don't display desktop notifications for messages in the room you are
+   non-idle and speaking in.
+
+Changes in synapse 0.3.2 (2014-09-18)
+=====================================
+
+Webclient:
+ * Fix bug where an empty "bing words" list in old accounts didn't send
+   notifications when it should have done.
+
+Changes in synapse 0.3.1 (2014-09-18)
+=====================================
+This is a release to hotfix v0.3.0 to fix two regressions.
+
+Webclient:
+ * Fix a regression where we sometimes displayed duplicate events.
+ * Fix a regression where we didn't immediately remove rooms you were
+   banned in from the recents list.
+
+Changes in synapse 0.3.0 (2014-09-18)
+=====================================
+See UPGRADE for information about changes to the client server API, including
+breaking backwards compatibility with VoIP calls and registration API.
+
+Homeserver:
+ * When a user changes their displayname or avatar the server will now update 
+   all their join states to reflect this.
+ * The server now adds "age" key to events to indicate how old they are. This
+   is clock independent, so at no point does any server or webclient have to
+   assume their clock is in sync with everyone else.
+ * Fix bug where we didn't correctly pull in missing PDUs.
+ * Fix bug where prev_content key wasn't always returned.
+ * Add support for password resets.
+
+Webclient:
+ * Improve page content loading.
+ * Join/parts now trigger desktop notifications.
+ * Always show room aliases in the UI if one is present.
+ * No longer show user-count in the recents side panel.
+ * Add up & down arrow support to the text box for message sending to step
+   through your sent history.
+ * Don't display notifications for our own messages.
+ * Emotes are now formatted correctly in desktop notifications.
+ * The recents list now differentiates between public & private rooms.
+ * Fix bug where when switching between rooms the pagination flickered before
+   the view jumped to the bottom of the screen.
+ * Add bing word support.
+
+Registration API:
+ * The registration API has been overhauled to function like the login API. In
+   practice, this means registration requests must now include the following:
+   'type':'m.login.password'. See UPGRADE for more information on this.
+ * The 'user_id' key has been renamed to 'user' to better match the login API.
+ * There is an additional login type: 'm.login.email.identity'.
+ * The command client and web client have been updated to reflect these changes.
+
 Changes in synapse 0.2.3 (2014-09-12)
 =====================================
 

README.rst

@@ -102,7 +102,7 @@ service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc.
 
 Synapse ships with two basic demo Matrix clients: webclient (a basic group chat
 web client demo implemented in AngularJS) and cmdclient (a basic Python
-commandline utility which lets you easily see what the JSON APIs are up to).
+command line utility which lets you easily see what the JSON APIs are up to).
 
 We'd like to invite you to take a look at the Matrix spec, try to run a
 homeserver, and join the existing Matrix chatrooms already out there, experiment
@@ -122,7 +122,7 @@ Homeserver Installation
 First, the dependencies need to be installed.  Start by installing 
 'python2.7-dev' and the various tools of the compiler toolchain.
 
-  Installing prerequisites on ubuntu::
+  Installing prerequisites on Ubuntu::
 
     $ sudo apt-get install build-essential python2.7-dev libffi-dev
 
@@ -151,8 +151,8 @@ you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and
 installing it. Installing PyNaCl using pip may also work (remember to remove any
 other versions installed by setuputils in, for example, ~/.local/lib).
 
-On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'`` you will
-need to ``export CFLAGS=-Qunused-arguments``.
+On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'`` 
+you will need to ``export CFLAGS=-Qunused-arguments``.
 
 This will run a process of downloading and installing into your
 user's .local/lib directory all of the required dependencies that are
@@ -203,9 +203,10 @@ For the first form, simply pass the required hostname (of the machine) as the
         --generate-config
     $ python synapse/app/homeserver.py --config-path homeserver.config
     
-Alternatively, you can run synapse via synctl - running ``synctl start`` to generate a
-homeserver.yaml config file, where you can then edit server-name to specify
-machine.my.domain.name, and then set the actual server running again with synctl start.
+Alternatively, you can run synapse via synctl - running ``synctl start`` to 
+generate a homeserver.yaml config file, where you can then edit server-name to 
+specify machine.my.domain.name, and then set the actual server running again 
+with synctl start.
 
 For the second form, first create your SRV record and publish it in DNS. This
 needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname
@@ -293,7 +294,8 @@ track 3PID logins and publish end-user public keys.
 
 It's currently early days for identity servers as Matrix is not yet using 3PIDs
 as the primary means of identity and E2E encryption is not complete. As such,
-we are running a single identity server (http://matrix.org:8090) at the current time.
+we are running a single identity server (http://matrix.org:8090) at the current 
+time.
 
 
 Where's the spec?!

UPGRADE.rst

@@ -1,3 +1,34 @@
+Upgrading to v0.3.0
+===================
+
+This registration API now closely matches the login API. This introduces a bit
+more backwards and forwards between the HS and the client, but this improves
+the overall flexibility of the API. You can now GET on /register to retrieve a list
+of valid registration flows. Upon choosing one, they are submitted in the same
+way as login, e.g::
+
+  {
+    type: m.login.password,
+    user: foo,
+    password: bar
+  }
+
+The default HS supports 2 flows, with and without Identity Server email
+authentication. Enabling captcha on the HS will add in an extra step to all
+flows: ``m.login.recaptcha`` which must be completed before you can transition
+to the next stage. There is a new login type: ``m.login.email.identity`` which
+contains the ``threepidCreds`` key which were previously sent in the original
+register request. For more information on this, see the specification.
+
+Web Client
+----------
+
+The VoIP specification has changed between v0.2.0 and v0.3.0. Users should
+refresh any browser tabs to get the latest web client code. Users on
+v0.2.0 of the web client will not be able to call those on v0.3.0 and
+vice versa.
+
+
 Upgrading to v0.2.0
 ===================
 

VERSION

@@ -1 +1 @@
-0.2.3
+0.3.3

WISHLIST.rst

@@ -5,3 +5,5 @@ Broad-sweeping stuff which would be nice to have
  - homeserver implementation in go
  - homeserver implementation in node.js
  - client SDKs
+ - libpurple library
+ - irssi plugin?

cmdclient/console.py

@@ -145,35 +145,50 @@ class SynapseCmd(cmd.Cmd):
         <noupdate> : Do not automatically clobber config values.
         """
         args = self._parse(line, ["userid", "noupdate"])
-        path = "/register"
 
         password = None
         pwd = None
         pwd2 = "_"
         while pwd != pwd2:
-            pwd = getpass.getpass("(Optional) Type a password for this user: ")
-            if len(pwd) == 0:
-                print "Not using a password for this user."
-                break
+            pwd = getpass.getpass("Type a password for this user: ")
             pwd2 = getpass.getpass("Retype the password: ")
-            if pwd != pwd2:
+            if pwd != pwd2 or len(pwd) == 0:
                 print "Password mismatch."
+                pwd = None
             else:
                 password = pwd
 
-        body = {}
+        body = {
+            "type": "m.login.password"
+        }
         if "userid" in args:
-            body["user_id"] = args["userid"]
+            body["user"] = args["userid"]
         if password:
             body["password"] = password
 
-        reactor.callFromThread(self._do_register, "POST", path, body,
+        reactor.callFromThread(self._do_register, body,
                                "noupdate" not in args)
 
     @defer.inlineCallbacks
-    def _do_register(self, method, path, data, update_config):
-        url = self._url() + path
-        json_res = yield self.http_client.do_request(method, url, data=data)
+    def _do_register(self, data, update_config):
+        # check the registration flows
+        url = self._url() + "/register"
+        json_res = yield self.http_client.do_request("GET", url)
+        print json.dumps(json_res, indent=4)
+
+        passwordFlow = None
+        for flow in json_res["flows"]:
+            if flow["type"] == "m.login.recaptcha" or ("stages" in flow and "m.login.recaptcha" in flow["stages"]):
+                print "Unable to register: Home server requires captcha."
+                return
+            if flow["type"] == "m.login.password" and "stages" not in flow:
+                passwordFlow = flow
+                break
+
+        if not passwordFlow:
+            return
+
+        json_res = yield self.http_client.do_request("POST", url, data=data)
         print json.dumps(json_res, indent=4)
         if update_config and "user_id" in json_res:
             self.config["user"] = json_res["user_id"]

docs/client-server/swagger_matrix/api-docs-registration

@@ -3,35 +3,38 @@
   "apis": [
     {
       "operations": [
+        {
+          "method": "GET", 
+          "nickname": "get_registration_info", 
+          "notes": "All login stages MUST be mentioned if there is >1 login type.", 
+          "summary": "Get the login mechanism to use when registering.", 
+          "type": "RegistrationFlows"
+        }, 
         {
           "method": "POST", 
-          "nickname": "register", 
-          "notes": "Volatile: This API is likely to change.", 
+          "nickname": "submit_registration", 
+          "notes": "If this is part of a multi-stage registration, there MUST be a 'session' key.", 
           "parameters": [
             {
-              "description": "A registration request", 
+              "description": "A registration submission", 
               "name": "body", 
               "paramType": "body", 
               "required": true, 
-              "type": "RegistrationRequest"
+              "type": "RegistrationSubmission"
             }
           ], 
           "responseMessages": [
             {
               "code": 400, 
-              "message": "No JSON object."
+              "message": "Bad login type"
             }, 
             {
               "code": 400, 
-              "message": "User ID must only contain characters which do not require url encoding."
-            },
-            {
-              "code": 400, 
-              "message": "User ID already taken."
+              "message": "Missing JSON keys"
             }
           ], 
-          "summary": "Register with the home server.", 
-          "type": "RegistrationResponse"
+          "summary": "Submit a registration action.", 
+          "type": "RegistrationResult"
         }
       ], 
       "path": "/register"
@@ -42,30 +45,68 @@
     "application/json"
   ], 
   "models": {
-    "RegistrationResponse": {
-      "id": "RegistrationResponse", 
+    "RegistrationFlows": {
+      "id": "RegistrationFlows",
       "properties": {
-        "access_token": {
-          "description": "The access token for this user.", 
-          "type": "string"
+        "flows": {
+          "description": "A list of valid registration flows.",
+          "type": "array",
+          "items": {
+            "$ref": "RegistrationInfo"
+          }
+        }
+      }
+    },
+    "RegistrationInfo": {
+      "id": "RegistrationInfo", 
+      "properties": {
+        "stages": {
+          "description": "Multi-stage registration only: An array of all the login types required to registration.", 
+          "items": {
+            "$ref": "string"
+          }, 
+          "type": "array"
         }, 
-        "user_id": {
-          "description": "The fully-qualified user ID.", 
-          "type": "string"
-        },
-        "home_server": {
-          "description": "The name of the home server.",
+        "type": {
+          "description": "The first login type that must be used when logging in.", 
           "type": "string"
         }
       }
     }, 
-    "RegistrationRequest": {
-      "id": "RegistrationRequest", 
+    "RegistrationResult": {
+      "id": "RegistrationResult", 
       "properties": {
+        "access_token": {
+          "description": "The access token for this user's registration if this is the final stage of the registration process.", 
+          "type": "string"
+        },
         "user_id": {
-          "description": "The desired user ID. If not specified, a random user ID will be allocated.", 
-          "type": "string",
-          "required": false
+          "description": "The user's fully-qualified user ID.",
+          "type": "string"
+        }, 
+        "next": {
+          "description": "Multi-stage registration only: The next registration type to submit.", 
+          "type": "string"
+        },
+        "session": {
+          "description": "Multi-stage registration only: The session token to send when submitting the next registration type.",
+          "type": "string"
+        }
+      }
+    }, 
+    "RegistrationSubmission": {
+      "id": "RegistrationSubmission", 
+      "properties": {
+        "type": {
+          "description": "The type of registration being submitted.", 
+          "type": "string"
+        },
+        "session": {
+          "description": "Multi-stage registration only: The session token from an earlier registration stage.",
+          "type": "string"
+        },
+        "_registration_type_defined_keys_": {
+          "description": "Keys as defined by the specified registration type, e.g. \"user\", \"password\""
         }
       }
     }

docs/freenode.txt

@@ -0,0 +1 @@
+NCjcRSEG

docs/specification.rst

@@ -1169,8 +1169,14 @@ This event is sent by the caller when they wish to establish a call.
   Required keys:
     - ``call_id`` : "string" - A unique identifier for the call
     - ``offer`` : "offer object" - The session description
-    - ``version`` : "integer" - The version of the VoIP specification this message
-                                adheres to. This specification is version 0.
+    - ``version`` : "integer" - The version of the VoIP specification this
+                                message adheres to. This specification is
+                                version 0.
+    - ``lifetime`` : "integer" - The time in milliseconds that the invite is
+                                 valid for. Once the invite age exceeds this
+                                 value, clients should discard it. They
+                                 should also no longer show the call as
+                                 awaiting an answer in the UI.
       
   Optional keys:
     None.
@@ -1305,12 +1311,6 @@ display name other than it being a valid unicode string.
 
 Registration and login
 ======================
-.. WARNING::
-  The registration API is likely to change.
-
-.. TODO
-  - TODO Kegan : Make registration like login (just omit the "user" key on the 
-    initial request?)
 
 Clients must register with a home server in order to use Matrix. After 
 registering, the client will be given an access token which must be used in ALL
@@ -1323,9 +1323,11 @@ a token sent to their email address, etc. This specification does not define how
 home servers should authorise their users who want to login to their existing 
 accounts, but instead defines the standard interface which implementations 
 should follow so that ANY client can login to ANY home server. Clients login
-using the |login|_ API.
+using the |login|_ API. Clients register using the |register|_ API. Registration
+follows the same procedure as login, but the path requests are sent to are
+different.
 
-The login process breaks down into the following:
+The registration/login process breaks down into the following:
   1. Determine the requirements for logging in.
   2. Submit the login stage credentials.
   3. Get credentials or be told the next stage in the login process and repeat 
@@ -1383,7 +1385,7 @@ This specification defines the following login types:
  - ``m.login.oauth2``
  - ``m.login.email.code``
  - ``m.login.email.url``
-
+ - ``m.login.email.identity``
 
 Password-based
 --------------
@@ -1531,6 +1533,31 @@ If the link has not been visited yet, a standard error response with an errcode
 ``M_LOGIN_EMAIL_URL_NOT_YET`` should be returned.
 
 
+Email-based (identity server)
+-----------------------------
+:Type:
+  ``m.login.email.identity``
+:Description:
+  Login is supported by authorising an email address with an identity server.
+
+Prior to submitting this, the client should authenticate with an identity server.
+After authenticating, the session information should be submitted to the home server.
+
+To respond to this type, reply with::
+
+  {
+    "type": "m.login.email.identity",
+    "threepidCreds": [
+      {
+        "sid": "<identity server session id>",
+        "clientSecret": "<identity server client secret>",
+        "idServer": "<url of identity server authed with, e.g. 'matrix.org:8090'>"
+      }
+    ]
+  }
+
+
+
 N-Factor Authentication
 -----------------------
 Multiple login stages can be combined to create N-factor authentication during login.
@@ -2242,6 +2269,9 @@ Transaction:
 .. |login| replace:: ``/login``
 .. _login: /docs/api/client-server/#!/-login
 
+.. |register| replace:: ``/register``
+.. _register: /docs/api/client-server/#!/-registration
+
 .. |/rooms/<room_id>/messages| replace:: ``/rooms/<room_id>/messages``
 .. _/rooms/<room_id>/messages: /docs/api/client-server/#!/-rooms/get_messages
 

synapse/__init__.py

@@ -16,4 +16,4 @@
 """ This is a reference implementation of a synapse home server.
 """
 
-__version__ = "0.2.3"
+__version__ = "0.3.3"

synapse/api/constants.py

@@ -50,3 +50,12 @@ class JoinRules(object):
     KNOCK = u"knock"
     INVITE = u"invite"
     PRIVATE = u"private"
+
+
+class LoginType(object):
+    PASSWORD = u"m.login.password"
+    OAUTH = u"m.login.oauth2"
+    EMAIL_CODE = u"m.login.email.code"
+    EMAIL_URL = u"m.login.email.url"
+    EMAIL_IDENTITY = u"m.login.email.identity"
+    RECAPTCHA = u"m.login.recaptcha"

synapse/api/events/__init__.py

@@ -17,6 +17,19 @@ from synapse.api.errors import SynapseError, Codes
 from synapse.util.jsonobject import JsonEncodedObject
 
 
+def serialize_event(hs, e):
+    # FIXME(erikj): To handle the case of presence events and the like
+    if not isinstance(e, SynapseEvent):
+        return e
+
+    d = e.get_dict()
+    if "age_ts" in d:
+        d["age"] = int(hs.get_clock().time_msec()) - d["age_ts"]
+        del d["age_ts"]
+
+    return d
+
+
 class SynapseEvent(JsonEncodedObject):
 
     """Base class for Synapse events. These are JSON objects which must abide
@@ -43,6 +56,8 @@ class SynapseEvent(JsonEncodedObject):
         "content",  # HTTP body, JSON
         "state_key",
         "required_power_level",
+        "age_ts",
+        "prev_content",
     ]
 
     internal_keys = [
@@ -141,7 +156,8 @@ class SynapseEvent(JsonEncodedObject):
                     return "Missing %s key" % key
 
                 if type(content[key]) != type(template[key]):
-                    return "Key %s is of the wrong type." % key
+                    return "Key %s is of the wrong type (got %s, want %s)" % (
+                        key, type(content[key]), type(template[key]))
 
                 if type(content[key]) == dict:
                     # we must go deeper
@@ -158,10 +174,6 @@ class SynapseEvent(JsonEncodedObject):
 
 class SynapseStateEvent(SynapseEvent):
 
-    valid_keys = SynapseEvent.valid_keys + [
-        "prev_content",
-    ]
-
     def __init__(self, **kwargs):
         if "state_key" not in kwargs:
             kwargs["state_key"] = ""

synapse/api/events/factory.py

@@ -59,6 +59,14 @@ class EventFactory(object):
         if "ts" not in kwargs:
             kwargs["ts"] = int(self.clock.time_msec())
 
+        # The "age" key is a delta timestamp that should be converted into an
+        # absolute timestamp the minute we see it.
+        if "age" in kwargs:
+            kwargs["age_ts"] = int(self.clock.time_msec()) - int(kwargs["age"])
+            del kwargs["age"]
+        elif "age_ts" not in kwargs:
+            kwargs["age_ts"] = int(self.clock.time_msec())
+
         if etype in self._event_list:
             handler = self._event_list[etype]
         else:

synapse/app/homeserver.py

@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.storage import read_schema
+from synapse.storage import prepare_database
 
 from synapse.server import HomeServer
 
@@ -36,30 +36,14 @@ from daemonize import Daemonize
 import twisted.manhole.telnet
 
 import logging
-import sqlite3
 import os
 import re
 import sys
+import sqlite3
 
 logger = logging.getLogger(__name__)
 
 
-SCHEMAS = [
-    "transactions",
-    "pdu",
-    "users",
-    "profiles",
-    "presence",
-    "im",
-    "room_aliases",
-]
-
-
-# Remember to update this number every time an incompatible change is made to
-# database schema files, so the users will be informed on server restarts.
-SCHEMA_VERSION = 3
-
-
 class SynapseHomeServer(HomeServer):
 
     def build_http_client(self):
@@ -80,52 +64,12 @@ class SynapseHomeServer(HomeServer):
         )
 
     def build_db_pool(self):
-        """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we
-        don't have to worry about overwriting existing content.
-        """
-        logging.info("Preparing database: %s...", self.db_name)
-
-        with sqlite3.connect(self.db_name) as db_conn:
-            c = db_conn.cursor()
-            c.execute("PRAGMA user_version")
-            row = c.fetchone()
-
-            if row and row[0]:
-                user_version = row[0]
-
-                if user_version > SCHEMA_VERSION:
-                    raise ValueError("Cannot use this database as it is too " +
-                        "new for the server to understand"
-                    )
-                elif user_version < SCHEMA_VERSION:
-                    logging.info("Upgrading database from version %d",
-                        user_version
-                    )
-
-                    # Run every version since after the current version.
-                    for v in range(user_version + 1, SCHEMA_VERSION + 1):
-                        sql_script = read_schema("delta/v%d" % (v))
-                        c.executescript(sql_script)
-
-                    db_conn.commit()
-
-            else:
-                for sql_loc in SCHEMAS:
-                    sql_script = read_schema(sql_loc)
-
-                    c.executescript(sql_script)
-                db_conn.commit()
-                c.execute("PRAGMA user_version = %d" % SCHEMA_VERSION)
-
-            c.close()
-
-        logging.info("Database prepared in %s.", self.db_name)
-
-        pool = adbapi.ConnectionPool(
-            'sqlite3', self.db_name, check_same_thread=False,
-            cp_min=1, cp_max=1)
-
-        return pool
+        return adbapi.ConnectionPool(
+            "sqlite3", self.get_db_name(),
+            check_same_thread=False,
+            cp_min=1,
+            cp_max=1
+        )
 
     def create_resource_tree(self, web_client, redirect_root_to_web_client):
         """Create the resource tree for this Home Server.
@@ -230,10 +174,6 @@ class SynapseHomeServer(HomeServer):
             logger.info("Synapse now listening on port %d", unsecure_port)
 
 
-def run():
-    reactor.run()
-
-
 def setup():
     config = HomeServerConfig.load_config(
         "Synapse Homeserver",
@@ -268,7 +208,15 @@ def setup():
         web_client=config.webclient,
         redirect_root_to_web_client=True,
     )
-    hs.start_listening(config.bind_port, config.unsecure_port)
+
+    db_name = hs.get_db_name()
+
+    logging.info("Preparing database: %s...", db_name)
+
+    with sqlite3.connect(db_name) as db_conn:
+        prepare_database(db_conn)
+
+    logging.info("Database prepared in %s.", db_name)
 
     hs.get_db_pool()
 
@@ -279,12 +227,14 @@ def setup():
         f.namespace['hs'] = hs
         reactor.listenTCP(config.manhole, f, interface='127.0.0.1')
 
+    hs.start_listening(config.bind_port, config.unsecure_port)
+
     if config.daemonize:
         print config.pid_file
         daemon = Daemonize(
             app="synapse-homeserver",
             pid=config.pid_file,
-            action=run,
+            action=reactor.run,
             auto_close_fds=False,
             verbose=True,
             logger=logger,
@@ -292,7 +242,7 @@ def setup():
 
         daemon.start()
     else:
-        run()
+        reactor.run()
 
 
 if __name__ == '__main__':

synapse/config/captcha.py

@@ -14,13 +14,16 @@
 
 from ._base import Config
 
+
 class CaptchaConfig(Config):
 
     def __init__(self, args):
         super(CaptchaConfig, self).__init__(args)
         self.recaptcha_private_key = args.recaptcha_private_key
         self.enable_registration_captcha = args.enable_registration_captcha
-        self.captcha_ip_origin_is_x_forwarded = args.captcha_ip_origin_is_x_forwarded
+        self.captcha_ip_origin_is_x_forwarded = (
+            args.captcha_ip_origin_is_x_forwarded
+        )
 
     @classmethod
     def add_arguments(cls, parser):
@@ -32,11 +35,12 @@ class CaptchaConfig(Config):
         )
         group.add_argument(
             "--enable-registration-captcha", type=bool, default=False,
-            help="Enables ReCaptcha checks when registering, preventing signup "+
-            "unless a captcha is answered. Requires a valid ReCaptcha public/private key."
+            help="Enables ReCaptcha checks when registering, preventing signup"
+            + " unless a captcha is answered. Requires a valid ReCaptcha "
+            + "public/private key."
         )
         group.add_argument(
             "--captcha_ip_origin_is_x_forwarded", type=bool, default=False,
-            help="When checking captchas, use the X-Forwarded-For (XFF) header as the client IP "+
-            "and not the actual client IP."
+            help="When checking captchas, use the X-Forwarded-For (XFF) header"
+            + " as the client IP and not the actual client IP."
         )

synapse/config/email.py

@@ -0,0 +1,39 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ._base import Config
+
+
+class EmailConfig(Config):
+
+    def __init__(self, args):
+        super(EmailConfig, self).__init__(args)
+        self.email_from_address = args.email_from_address
+        self.email_smtp_server = args.email_smtp_server
+
+    @classmethod
+    def add_arguments(cls, parser):
+        super(EmailConfig, cls).add_arguments(parser)
+        email_group = parser.add_argument_group("email")
+        email_group.add_argument(
+            "--email-from-address",
+            default="FROM@EXAMPLE.COM",
+            help="The address to send emails from (e.g. for password resets)."
+        )
+        email_group.add_argument(
+            "--email-smtp-server",
+            default="",
+            help="The SMTP server to send emails from (e.g. for password resets)."
+        )

synapse/config/homeserver.py

@@ -20,11 +20,15 @@ from .database import DatabaseConfig
 from .ratelimiting import RatelimitConfig
 from .repository import ContentRepositoryConfig
 from .captcha import CaptchaConfig
+from .email import EmailConfig
+
 
 class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,
-                       RatelimitConfig, ContentRepositoryConfig, CaptchaConfig):
+                       RatelimitConfig, ContentRepositoryConfig, CaptchaConfig,
+                       EmailConfig):
     pass
 
-if __name__=='__main__':
+
+if __name__ == '__main__':
     import sys
     HomeServerConfig.load_config("Generate config", sys.argv[1:], "HomeServer")

synapse/federation/replication.py

@@ -291,6 +291,13 @@ class ReplicationLayer(object):
     def on_incoming_transaction(self, transaction_data):
         transaction = Transaction(**transaction_data)
 
+        for p in transaction.pdus:
+            if "age" in p:
+                p["age_ts"] = int(self._clock.time_msec()) - int(p["age"])
+                del p["age"]
+
+        pdu_list = [Pdu(**p) for p in transaction.pdus]
+
         logger.debug("[%s] Got transaction", transaction.transaction_id)
 
         response = yield self.transaction_actions.have_responded(transaction)
@@ -303,8 +310,6 @@ class ReplicationLayer(object):
 
         logger.debug("[%s] Transacition is new", transaction.transaction_id)
 
-        pdu_list = [Pdu(**p) for p in transaction.pdus]
-
         dl = []
         for pdu in pdu_list:
             dl.append(self._handle_new_pdu(pdu))
@@ -405,9 +410,14 @@ class ReplicationLayer(object):
         """Returns a new Transaction containing the given PDUs suitable for
         transmission.
         """
+        pdus = [p.get_dict() for p in pdu_list]
+        for p in pdus:
+            if "age_ts" in pdus:
+                p["age"] = int(self.clock.time_msec()) - p["age_ts"]
+
         return Transaction(
-            pdus=[p.get_dict() for p in pdu_list],
             origin=self.server_name,
+            pdus=pdus,
             ts=int(self._clock.time_msec()),
             destination=None,
         )
@@ -593,8 +603,21 @@ class _TransactionQueue(object):
             logger.debug("TX [%s] Sending transaction...", destination)
 
             # Actually send the transaction
+
+            # FIXME (erikj): This is a bit of a hack to make the Pdu age
+            # keys work
+            def cb(transaction):
+                now = int(self._clock.time_msec())
+                if "pdus" in transaction:
+                    for p in transaction["pdus"]:
+                        if "age_ts" in p:
+                            p["age"] = now - int(p["age_ts"])
+
+                return transaction
+
             code, response = yield self.transport_layer.send_transaction(
-                transaction
+                transaction,
+                on_send_callback=cb,
             )
 
             logger.debug("TX [%s] Sent transaction", destination)

synapse/federation/transport.py

@@ -144,7 +144,7 @@ class TransportLayer(object):
 
     @defer.inlineCallbacks
     @log_function
-    def send_transaction(self, transaction):
+    def send_transaction(self, transaction, on_send_callback=None):
         """ Sends the given Transaction to it's destination
 
         Args:
@@ -165,10 +165,23 @@ class TransportLayer(object):
 
         data = transaction.get_dict()
 
+        # FIXME (erikj): This is a bit of a hack to make the Pdu age
+        # keys work
+        def cb(destination, method, path_bytes, producer):
+            if not on_send_callback:
+                return
+
+            transaction = json.loads(producer.body)
+
+            new_transaction = on_send_callback(transaction)
+
+            producer.reset(new_transaction)
+
         code, response = yield self.client.put_json(
             transaction.destination,
             path=PREFIX + "/send/%s/" % transaction.transaction_id,
-            data=data
+            data=data,
+            on_send_callback=cb,
         )
 
         logger.debug(

synapse/handlers/events.py

@@ -15,7 +15,6 @@
 
 from twisted.internet import defer
 
-from synapse.api.events import SynapseEvent
 from synapse.util.logutils import log_function
 
 from ._base import BaseHandler
@@ -71,10 +70,7 @@ class EventStreamHandler(BaseHandler):
                 auth_user, room_ids, pagin_config, timeout
             )
 
-            chunks = [
-                e.get_dict() if isinstance(e, SynapseEvent) else e
-                for e in events
-            ]
+            chunks = [self.hs.serialize_event(e) for e in events]
 
             chunk = {
                 "chunk": chunks,
@@ -92,7 +88,9 @@ class EventStreamHandler(BaseHandler):
                 # 10 seconds of grace to allow the client to reconnect again
                 #   before we think they're gone
                 def _later():
-                    logger.debug("_later stopped_user_eventstream %s", auth_user)
+                    logger.debug(
+                        "_later stopped_user_eventstream %s", auth_user
+                    )
                     self.distributor.fire(
                         "stopped_user_eventstream", auth_user
                     )

synapse/handlers/federation.py

@@ -93,22 +93,18 @@ class FederationHandler(BaseHandler):
         """
         event = self.pdu_codec.event_from_pdu(pdu)
 
+        logger.debug("Got event: %s", event.event_id)
+
         with (yield self.lock_manager.lock(pdu.context)):
             if event.is_state and not backfilled:
                 is_new_state = yield self.state_handler.handle_new_state(
                     pdu
                 )
-                if not is_new_state:
-                    return
             else:
                 is_new_state = False
         # TODO: Implement something in federation that allows us to
         # respond to PDU.
 
-        if hasattr(event, "state_key") and not is_new_state:
-            logger.debug("Ignoring old state.")
-            return
-
         target_is_mine = False
         if hasattr(event, "target_host"):
             target_is_mine = event.target_host == self.hs.hostname
@@ -139,7 +135,11 @@ class FederationHandler(BaseHandler):
 
         else:
             with (yield self.room_lock.lock(event.room_id)):
-                yield self.store.persist_event(event, backfilled)
+                yield self.store.persist_event(
+                    event,
+                    backfilled,
+                    is_new_state=is_new_state
+                )
 
             room = yield self.store.get_room(event.room_id)
 

synapse/handlers/login.py

@@ -17,9 +17,13 @@ from twisted.internet import defer
 
 from ._base import BaseHandler
 from synapse.api.errors import LoginError, Codes
+from synapse.http.client import PlainHttpClient
+from synapse.util.emailutils import EmailException
+import synapse.util.emailutils as emailutils
 
 import bcrypt
 import logging
+import urllib
 
 logger = logging.getLogger(__name__)
 
@@ -62,4 +66,41 @@ class LoginHandler(BaseHandler):
             defer.returnValue(token)
         else:
             logger.warn("Failed password login for user %s", user)
-            raise LoginError(403, "", errcode=Codes.FORBIDDEN)
+            raise LoginError(403, "", errcode=Codes.FORBIDDEN)
+
+    @defer.inlineCallbacks
+    def reset_password(self, user_id, email):
+        is_valid = yield self._check_valid_association(user_id, email)
+        logger.info("reset_password user=%s email=%s valid=%s", user_id, email,
+                    is_valid)
+        if is_valid:
+            try:
+                # send an email out
+                emailutils.send_email(
+                    smtp_server=self.hs.config.email_smtp_server,
+                    from_addr=self.hs.config.email_from_address,
+                    to_addr=email,
+                    subject="Password Reset",
+                    body="TODO."
+                )
+            except EmailException as e:
+                logger.exception(e)
+
+    @defer.inlineCallbacks
+    def _check_valid_association(self, user_id, email):
+        identity = yield self._query_email(email)
+        if identity and "mxid" in identity:
+            if identity["mxid"] == user_id:
+                defer.returnValue(True)
+                return
+        defer.returnValue(False)
+
+    @defer.inlineCallbacks
+    def _query_email(self, email):
+        httpCli = PlainHttpClient(self.hs)
+        data = yield httpCli.get_json(
+            'matrix.org:8090',  # TODO FIXME This should be configurable.
+            "/_matrix/identity/api/v1/lookup?medium=email&address=" +
+            "%s" % urllib.quote(email)
+        )
+        defer.returnValue(data)

synapse/handlers/message.py

@@ -124,7 +124,7 @@ class MessageHandler(BaseHandler):
         )
 
         chunk = {
-            "chunk": [e.get_dict() for e in events],
+            "chunk": [self.hs.serialize_event(e) for e in events],
             "start": pagin_config.from_token.to_string(),
             "end": next_token.to_string(),
         }
@@ -268,6 +268,9 @@ class MessageHandler(BaseHandler):
             user, pagination_config, None
         )
 
+        public_rooms = yield self.store.get_rooms(is_public=True)
+        public_room_ids = [r["room_id"] for r in public_rooms]
+
         limit = pagin_config.limit
         if not limit:
             limit = 10
@@ -276,6 +279,8 @@ class MessageHandler(BaseHandler):
             d = {
                 "room_id": event.room_id,
                 "membership": event.membership,
+                "visibility": ("public" if event.room_id in
+                              public_room_ids else "private"),
             }
 
             if event.membership == Membership.INVITE:
@@ -296,7 +301,7 @@ class MessageHandler(BaseHandler):
                 end_token = now_token.copy_and_replace("room_key", token[1])
 
                 d["messages"] = {
-                    "chunk": [m.get_dict() for m in messages],
+                    "chunk": [self.hs.serialize_event(m) for m in messages],
                     "start": start_token.to_string(),
                     "end": end_token.to_string(),
                 }
@@ -304,7 +309,7 @@ class MessageHandler(BaseHandler):
                 current_state = yield self.store.get_current_state(
                     event.room_id
                 )
-                d["state"] = [c.get_dict() for c in current_state]
+                d["state"] = [self.hs.serialize_event(c) for c in current_state]
             except:
                 logger.exception("Failed to get snapshot")
 

synapse/handlers/profile.py

@@ -15,9 +15,9 @@
 
 from twisted.internet import defer
 
-from synapse.api.errors import SynapseError, AuthError
-
-from synapse.api.errors import CodeMessageException
+from synapse.api.errors import SynapseError, AuthError, CodeMessageException
+from synapse.api.constants import Membership
+from synapse.api.events.room import RoomMemberEvent
 
 from ._base import BaseHandler
 
@@ -97,6 +97,8 @@ class ProfileHandler(BaseHandler):
             }
         )
 
+        yield self._update_join_states(target_user)
+
     @defer.inlineCallbacks
     def get_avatar_url(self, target_user):
         if target_user.is_mine:
@@ -144,6 +146,8 @@ class ProfileHandler(BaseHandler):
             }
         )
 
+        yield self._update_join_states(target_user)
+
     @defer.inlineCallbacks
     def collect_presencelike_data(self, user, state):
         if not user.is_mine:
@@ -180,3 +184,39 @@ class ProfileHandler(BaseHandler):
             )
 
         defer.returnValue(response)
+
+    @defer.inlineCallbacks
+    def _update_join_states(self, user):
+        if not user.is_mine:
+            return
+
+        joins = yield self.store.get_rooms_for_user_where_membership_is(
+            user.to_string(),
+            [Membership.JOIN],
+        )
+
+        for j in joins:
+            snapshot = yield self.store.snapshot_room(
+                j.room_id, j.state_key, RoomMemberEvent.TYPE,
+                j.state_key
+            )
+
+            content = {
+                "membership": j.content["membership"],
+                "prev": j.content["membership"],
+            }
+
+            yield self.distributor.fire(
+                "collect_presencelike_data", user, content
+            )
+
+            new_event = self.event_factory.create_event(
+                etype=j.type,
+                room_id=j.room_id,
+                state_key=j.state_key,
+                content=content,
+                user_id=j.state_key,
+            )
+
+            yield self.state_handler.handle_new_event(new_event, snapshot)
+            yield self._on_new_room_event(new_event, snapshot)

synapse/handlers/register.py

@@ -40,8 +40,7 @@ class RegistrationHandler(BaseHandler):
         self.distributor.declare("registered_user")
 
     @defer.inlineCallbacks
-    def register(self, localpart=None, password=None, threepidCreds=None, 
-                 captcha_info={}):
+    def register(self, localpart=None, password=None):
         """Registers a new client on the server.
 
         Args:
@@ -54,37 +53,6 @@ class RegistrationHandler(BaseHandler):
         Raises:
             RegistrationError if there was a problem registering.
         """
-        if captcha_info:
-            captcha_response = yield self._validate_captcha(
-                captcha_info["ip"], 
-                captcha_info["private_key"],
-                captcha_info["challenge"],
-                captcha_info["response"]
-            )
-            if not captcha_response["valid"]:
-                logger.info("Invalid captcha entered from %s. Error: %s", 
-                            captcha_info["ip"], captcha_response["error_url"])
-                raise InvalidCaptchaError(
-                    error_url=captcha_response["error_url"]
-                )
-            else:
-                logger.info("Valid captcha entered from %s", captcha_info["ip"])
-
-        if threepidCreds:
-            for c in threepidCreds:
-                logger.info("validating theeepidcred sid %s on id server %s",
-                            c['sid'], c['idServer'])
-                try:
-                    threepid = yield self._threepid_from_creds(c)
-                except:
-                    logger.err()
-                    raise RegistrationError(400, "Couldn't validate 3pid")
-                    
-                if not threepid:
-                    raise RegistrationError(400, "Couldn't validate 3pid")
-                logger.info("got threepid medium %s address %s", 
-                            threepid['medium'], threepid['address'])
-
         password_hash = None
         if password:
             password_hash = bcrypt.hashpw(password, bcrypt.gensalt())
@@ -126,15 +94,54 @@ class RegistrationHandler(BaseHandler):
                         raise RegistrationError(
                             500, "Cannot generate user ID.")
 
-        # Now we have a matrix ID, bind it to the threepids we were given
-        if threepidCreds:
-            for c in threepidCreds:
-                # XXX: This should be a deferred list, shouldn't it?
-                yield self._bind_threepid(c, user_id)
-                
-
         defer.returnValue((user_id, token))
 
+    @defer.inlineCallbacks
+    def check_recaptcha(self, ip, private_key, challenge, response):
+        """Checks a recaptcha is correct."""
+
+        captcha_response = yield self._validate_captcha(
+            ip,
+            private_key,
+            challenge,
+            response
+        )
+        if not captcha_response["valid"]:
+            logger.info("Invalid captcha entered from %s. Error: %s",
+                        ip, captcha_response["error_url"])
+            raise InvalidCaptchaError(
+                error_url=captcha_response["error_url"]
+            )
+        else:
+            logger.info("Valid captcha entered from %s", ip)
+
+    @defer.inlineCallbacks
+    def register_email(self, threepidCreds):
+        """Registers emails with an identity server."""
+
+        for c in threepidCreds:
+            logger.info("validating theeepidcred sid %s on id server %s",
+                        c['sid'], c['idServer'])
+            try:
+                threepid = yield self._threepid_from_creds(c)
+            except:
+                logger.err()
+                raise RegistrationError(400, "Couldn't validate 3pid")
+
+            if not threepid:
+                raise RegistrationError(400, "Couldn't validate 3pid")
+            logger.info("got threepid medium %s address %s",
+                        threepid['medium'], threepid['address'])
+
+    @defer.inlineCallbacks
+    def bind_emails(self, user_id, threepidCreds):
+        """Links emails with a user ID and informs an identity server."""
+
+        # Now we have a matrix ID, bind it to the threepids we were given
+        for c in threepidCreds:
+            # XXX: This should be a deferred list, shouldn't it?
+            yield self._bind_threepid(c, user_id)
+
     def _generate_token(self, user_id):
         # urlsafe variant uses _ and - so use . as the separator and replace
         # all =s with .s so http clients don't quote =s when it is used as
@@ -149,17 +156,17 @@ class RegistrationHandler(BaseHandler):
     def _threepid_from_creds(self, creds):
         httpCli = PlainHttpClient(self.hs)
         # XXX: make this configurable!
-        trustedIdServers = [ 'matrix.org:8090' ]
+        trustedIdServers = ['matrix.org:8090']
         if not creds['idServer'] in trustedIdServers:
-            logger.warn('%s is not a trusted ID server: rejecting 3pid '+
+            logger.warn('%s is not a trusted ID server: rejecting 3pid ' +
                         'credentials', creds['idServer'])
             defer.returnValue(None)
         data = yield httpCli.get_json(
             creds['idServer'],
             "/_matrix/identity/api/v1/3pid/getValidated3pid",
-            { 'sid': creds['sid'], 'clientSecret': creds['clientSecret'] }
+            {'sid': creds['sid'], 'clientSecret': creds['clientSecret']}
         )
-        
+
         if 'medium' in data:
             defer.returnValue(data)
         defer.returnValue(None)
@@ -170,44 +177,45 @@ class RegistrationHandler(BaseHandler):
         data = yield httpCli.post_urlencoded_get_json(
             creds['idServer'],
             "/_matrix/identity/api/v1/3pid/bind",
-            { 'sid': creds['sid'], 'clientSecret': creds['clientSecret'], 
-            'mxid':mxid }
+            {'sid': creds['sid'], 'clientSecret': creds['clientSecret'],
+            'mxid': mxid}
         )
         defer.returnValue(data)
-        
+
     @defer.inlineCallbacks
     def _validate_captcha(self, ip_addr, private_key, challenge, response):
         """Validates the captcha provided.
-        
+
         Returns:
             dict: Containing 'valid'(bool) and 'error_url'(str) if invalid.
-        
+
         """
-        response = yield self._submit_captcha(ip_addr, private_key, challenge, 
+        response = yield self._submit_captcha(ip_addr, private_key, challenge,
                                               response)
         # parse Google's response. Lovely format..
         lines = response.split('\n')
         json = {
             "valid": lines[0] == 'true',
-            "error_url": "http://www.google.com/recaptcha/api/challenge?"+
+            "error_url": "http://www.google.com/recaptcha/api/challenge?" +
                          "error=%s" % lines[1]
         }
         defer.returnValue(json)
-        
+
     @defer.inlineCallbacks
     def _submit_captcha(self, ip_addr, private_key, challenge, response):
         client = PlainHttpClient(self.hs)
         data = yield client.post_urlencoded_get_raw(
             "www.google.com:80",
             "/recaptcha/api/verify",
-            accept_partial=True,  # twisted dislikes google's response, no content length.
-            args={ 
-                'privatekey': private_key, 
+            # twisted dislikes google's response, no content length.
+            accept_partial=True,
+            args={
+                'privatekey': private_key,
                 'remoteip': ip_addr,
                 'challenge': challenge,
                 'response': response
             }
         )
         defer.returnValue(data)
-        
+
 

synapse/handlers/room.py

@@ -335,7 +335,7 @@ class RoomMemberHandler(BaseHandler):
 
         member_list = yield self.store.get_room_members(room_id=room_id)
         event_list = [
-            entry.get_dict()
+            self.hs.serialize_event(entry)
             for entry in member_list
         ]
         chunk_data = {

synapse/http/client.py

@@ -122,7 +122,7 @@ class TwistedHttpClient(HttpClient):
         self.hs = hs
 
     @defer.inlineCallbacks
-    def put_json(self, destination, path, data):
+    def put_json(self, destination, path, data, on_send_callback=None):
         if destination in _destination_mappings:
             destination = _destination_mappings[destination]
 
@@ -131,7 +131,8 @@ class TwistedHttpClient(HttpClient):
             "PUT",
             path.encode("ascii"),
             producer=_JsonProducer(data),
-            headers_dict={"Content-Type": ["application/json"]}
+            headers_dict={"Content-Type": ["application/json"]},
+            on_send_callback=on_send_callback,
         )
 
         logger.debug("Getting resp body")
@@ -218,7 +219,7 @@ class TwistedHttpClient(HttpClient):
     @defer.inlineCallbacks
     def _create_request(self, destination, method, path_bytes, param_bytes=b"",
                         query_bytes=b"", producer=None, headers_dict={},
-                        retry_on_dns_fail=True):
+                        retry_on_dns_fail=True, on_send_callback=None):
         """ Creates and sends a request to the given url
         """
         headers_dict[b"User-Agent"] = [b"Synapse"]
@@ -242,6 +243,9 @@ class TwistedHttpClient(HttpClient):
         endpoint = self._getEndpoint(reactor, destination);
 
         while True:
+            if on_send_callback:
+                on_send_callback(destination, method, path_bytes, producer)
+
             try:
                 response = yield self.agent.request(
                     destination,
@@ -310,6 +314,9 @@ class _JsonProducer(object):
     """ Used by the twisted http client to create the HTTP body from json
     """
     def __init__(self, jsn):
+        self.reset(jsn)
+
+    def reset(self, jsn):
         self.body = encode_canonical_json(jsn)
         self.length = len(self.body)
 

synapse/rest/events.py

@@ -59,7 +59,7 @@ class EventRestServlet(RestServlet):
         event = yield handler.get_event(auth_user, event_id)
 
         if event:
-            defer.returnValue((200, event.get_dict()))
+            defer.returnValue((200, self.hs.serialize_event(event)))
         else:
             defer.returnValue((404, "Event not found."))
 

synapse/rest/login.py

@@ -73,6 +73,27 @@ class LoginFallbackRestServlet(RestServlet):
         return (200, {})
 
 
+class PasswordResetRestServlet(RestServlet):
+    PATTERN = client_path_pattern("/login/reset")
+
+    @defer.inlineCallbacks
+    def on_POST(self, request):
+        reset_info = _parse_json(request)
+        try:
+            email = reset_info["email"]
+            user_id = reset_info["user_id"]
+            handler = self.handlers.login_handler
+            yield handler.reset_password(user_id, email)
+            # purposefully give no feedback to avoid people hammering different
+            # combinations.
+            defer.returnValue((200, {}))
+        except KeyError:
+            raise SynapseError(
+                400,
+                "Missing keys. Requires 'email' and 'user_id'."
+            )
+
+
 def _parse_json(request):
     try:
         content = json.loads(request.content.read())
@@ -85,3 +106,4 @@ def _parse_json(request):
 
 def register_servlets(hs, http_server):
     LoginRestServlet(hs).register(http_server)
+    # TODO PasswordResetRestServlet(hs).register(http_server)

synapse/rest/register.py

@@ -17,89 +17,218 @@
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError, Codes
+from synapse.api.constants import LoginType
 from base import RestServlet, client_path_pattern
+import synapse.util.stringutils as stringutils
 
 import json
+import logging
 import urllib
 
+logger = logging.getLogger(__name__)
+
 
 class RegisterRestServlet(RestServlet):
+    """Handles registration with the home server.
+
+    This servlet is in control of the registration flow; the registration
+    handler doesn't have a concept of multi-stages or sessions.
+    """
+
     PATTERN = client_path_pattern("/register$")
 
+    def __init__(self, hs):
+        super(RegisterRestServlet, self).__init__(hs)
+        # sessions are stored as:
+        # self.sessions = {
+        #   "session_id" : { __session_dict__ }
+        # }
+        # TODO: persistent storage
+        self.sessions = {}
+
+    def on_GET(self, request):
+        if self.hs.config.enable_registration_captcha:
+            return (200, {
+                "flows": [
+                    {
+                        "type": LoginType.RECAPTCHA,
+                        "stages": ([LoginType.RECAPTCHA,
+                                    LoginType.EMAIL_IDENTITY,
+                                    LoginType.PASSWORD])
+                    },
+                    {
+                        "type": LoginType.RECAPTCHA,
+                        "stages": [LoginType.RECAPTCHA, LoginType.PASSWORD]
+                    }
+                ]
+            })
+        else:
+            return (200, {
+                "flows": [
+                    {
+                        "type": LoginType.EMAIL_IDENTITY,
+                        "stages": ([LoginType.EMAIL_IDENTITY,
+                                    LoginType.PASSWORD])
+                    },
+                    {
+                        "type": LoginType.PASSWORD
+                    }
+                ]
+            })
+
     @defer.inlineCallbacks
     def on_POST(self, request):
-        desired_user_id = None
-        password = None
+        register_json = _parse_json(request)
+
+        session = (register_json["session"] if "session" in register_json
+                  else None)
+        login_type = None
+        if "type" not in register_json:
+            raise SynapseError(400, "Missing 'type' key.")
+
         try:
-            register_json = json.loads(request.content.read())
-            if "password" in register_json:
-                password = register_json["password"].encode("utf-8")
-
-            if type(register_json["user_id"]) == unicode:
-                desired_user_id = register_json["user_id"].encode("utf-8")
-                if urllib.quote(desired_user_id) != desired_user_id:
-                    raise SynapseError(
-                        400,
-                        "User ID must only contain characters which do not " +
-                        "require URL encoding.")
-        except ValueError:
-            defer.returnValue((400, "No JSON object."))
-        except KeyError:
-            pass  # user_id is optional
-
-        threepidCreds = None
-        if 'threepidCreds' in register_json:
-            threepidCreds = register_json['threepidCreds']
-            
-        captcha = {}
-        if self.hs.config.enable_registration_captcha:
-            challenge = None
-            user_response = None
-            try:
-                captcha_type = register_json["captcha"]["type"]
-                if captcha_type != "m.login.recaptcha":
-                    raise SynapseError(400, "Sorry, only m.login.recaptcha " +
-                                       "requests are supported.")
-                challenge = register_json["captcha"]["challenge"]
-                user_response = register_json["captcha"]["response"]
-            except KeyError:
-                raise SynapseError(400, "Captcha response is required",
-                                   errcode=Codes.CAPTCHA_NEEDED)
-            
-            # TODO determine the source IP : May be an X-Forwarding-For header depending on config
-            ip_addr = request.getClientIP()
-            if self.hs.config.captcha_ip_origin_is_x_forwarded:
-                # use the header
-                if request.requestHeaders.hasHeader("X-Forwarded-For"):
-                    ip_addr = request.requestHeaders.getRawHeaders(
-                        "X-Forwarded-For")[0]
-            
-            captcha = {
-                "ip": ip_addr,
-                "private_key": self.hs.config.recaptcha_private_key,
-                "challenge": challenge,
-                "response": user_response
+            login_type = register_json["type"]
+            stages = {
+                LoginType.RECAPTCHA: self._do_recaptcha,
+                LoginType.PASSWORD: self._do_password,
+                LoginType.EMAIL_IDENTITY: self._do_email_identity
             }
-            
 
+            session_info = self._get_session_info(request, session)
+            logger.debug("%s : session info %s   request info %s",
+                         login_type, session_info, register_json)
+            response = yield stages[login_type](
+                request,
+                register_json,
+                session_info
+            )
+
+            if "access_token" not in response:
+                # isn't a final response
+                response["session"] = session_info["id"]
+
+            defer.returnValue((200, response))
+        except KeyError as e:
+            logger.exception(e)
+            raise SynapseError(400, "Missing JSON keys for login type %s." % login_type)
+
+    def on_OPTIONS(self, request):
+        return (200, {})
+
+    def _get_session_info(self, request, session_id):
+        if not session_id:
+            # create a new session
+            while session_id is None or session_id in self.sessions:
+                session_id = stringutils.random_string(24)
+            self.sessions[session_id] = {
+                "id": session_id,
+                LoginType.EMAIL_IDENTITY: False,
+                LoginType.RECAPTCHA: False
+            }
+
+        return self.sessions[session_id]
+
+    def _save_session(self, session):
+        # TODO: Persistent storage
+        logger.debug("Saving session %s", session)
+        self.sessions[session["id"]] = session
+
+    def _remove_session(self, session):
+        logger.debug("Removing session %s", session)
+        self.sessions.pop(session["id"])
+
+    @defer.inlineCallbacks
+    def _do_recaptcha(self, request, register_json, session):
+        if not self.hs.config.enable_registration_captcha:
+            raise SynapseError(400, "Captcha not required.")
+
+        challenge = None
+        user_response = None
+        try:
+            challenge = register_json["challenge"]
+            user_response = register_json["response"]
+        except KeyError:
+            raise SynapseError(400, "Captcha response is required",
+                               errcode=Codes.CAPTCHA_NEEDED)
+
+        # May be an X-Forwarding-For header depending on config
+        ip_addr = request.getClientIP()
+        if self.hs.config.captcha_ip_origin_is_x_forwarded:
+            # use the header
+            if request.requestHeaders.hasHeader("X-Forwarded-For"):
+                ip_addr = request.requestHeaders.getRawHeaders(
+                    "X-Forwarded-For")[0]
+
+        handler = self.handlers.registration_handler
+        yield handler.check_recaptcha(
+            ip_addr,
+            self.hs.config.recaptcha_private_key,
+            challenge,
+            user_response
+        )
+        session[LoginType.RECAPTCHA] = True  # mark captcha as done
+        self._save_session(session)
+        defer.returnValue({
+            "next": [LoginType.PASSWORD, LoginType.EMAIL_IDENTITY]
+        })
+
+    @defer.inlineCallbacks
+    def _do_email_identity(self, request, register_json, session):
+        if (self.hs.config.enable_registration_captcha and
+                not session[LoginType.RECAPTCHA]):
+            raise SynapseError(400, "Captcha is required.")
+
+        threepidCreds = register_json['threepidCreds']
+        handler = self.handlers.registration_handler
+        yield handler.register_email(threepidCreds)
+        session["threepidCreds"] = threepidCreds  # store creds for next stage
+        session[LoginType.EMAIL_IDENTITY] = True  # mark email as done
+        self._save_session(session)
+        defer.returnValue({
+            "next": LoginType.PASSWORD
+        })
+
+    @defer.inlineCallbacks
+    def _do_password(self, request, register_json, session):
+        if (self.hs.config.enable_registration_captcha and
+                not session[LoginType.RECAPTCHA]):
+            # captcha should've been done by this stage!
+            raise SynapseError(400, "Captcha is required.")
+
+        password = register_json["password"].encode("utf-8")
+        desired_user_id = (register_json["user"].encode("utf-8") if "user"
+                          in register_json else None)
+        if desired_user_id and urllib.quote(desired_user_id) != desired_user_id:
+            raise SynapseError(
+                400,
+                "User ID must only contain characters which do not " +
+                "require URL encoding.")
         handler = self.handlers.registration_handler
         (user_id, token) = yield handler.register(
             localpart=desired_user_id,
-            password=password,
-            threepidCreds=threepidCreds,
-            captcha_info=captcha)
+            password=password
+        )
+
+        if session[LoginType.EMAIL_IDENTITY]:
+            yield handler.bind_emails(user_id, session["threepidCreds"])
 
         result = {
             "user_id": user_id,
             "access_token": token,
             "home_server": self.hs.hostname,
         }
-        defer.returnValue(
-            (200, result)
-        )
+        self._remove_session(session)
+        defer.returnValue(result)
 
-    def on_OPTIONS(self, request):
-        return (200, {})
+
+def _parse_json(request):
+    try:
+        content = json.loads(request.content.read())
+        if type(content) != dict:
+            raise SynapseError(400, "Content must be a JSON object.")
+        return content
+    except ValueError:
+        raise SynapseError(400, "Content not JSON.")
 
 
 def register_servlets(hs, http_server):

synapse/rest/room.py

@@ -378,7 +378,7 @@ class RoomTriggerBackfill(RestServlet):
         handler = self.handlers.federation_handler
         events = yield handler.backfill(remote_server, room_id, limit)
 
-        res = [event.get_dict() for event in events]
+        res = [self.hs.serialize_event(event) for event in events]
         defer.returnValue((200, res))
 
 

synapse/server.py

@@ -20,6 +20,7 @@
 
 # Imports required for the default HomeServer() implementation
 from synapse.federation import initialize_http_replication
+from synapse.api.events import serialize_event
 from synapse.api.events.factory import EventFactory
 from synapse.notifier import Notifier
 from synapse.api.auth import Auth
@@ -57,6 +58,7 @@ class BaseHomeServer(object):
     DEPENDENCIES = [
         'clock',
         'http_client',
+        'db_name',
         'db_pool',
         'persistence_service',
         'replication_layer',
@@ -138,6 +140,9 @@ class BaseHomeServer(object):
         object."""
         return RoomID.from_string(s, hs=self)
 
+    def serialize_event(self, e):
+        return serialize_event(self, e)
+
 # Build magic accessors for every dependency
 for depname in BaseHomeServer.DEPENDENCIES:
     BaseHomeServer._make_dependency_method(depname)

synapse/storage/__init__.py

@@ -47,6 +47,23 @@ import os
 
 logger = logging.getLogger(__name__)
 
+
+SCHEMAS = [
+    "transactions",
+    "pdu",
+    "users",
+    "profiles",
+    "presence",
+    "im",
+    "room_aliases",
+]
+
+
+# Remember to update this number every time an incompatible change is made to
+# database schema files, so the users will be informed on server restarts.
+SCHEMA_VERSION = 3
+
+
 class _RollbackButIsFineException(Exception):
     """ This exception is used to rollback a transaction without implying
     something went wrong.
@@ -68,7 +85,8 @@ class DataStore(RoomMemberStore, RoomStore,
 
     @defer.inlineCallbacks
     @log_function
-    def persist_event(self, event=None, backfilled=False, pdu=None):
+    def persist_event(self, event=None, backfilled=False, pdu=None,
+                      is_new_state=True):
         stream_ordering = None
         if backfilled:
             if not self.min_token_deferred.called:
@@ -77,12 +95,13 @@ class DataStore(RoomMemberStore, RoomStore,
             stream_ordering = self.min_token
 
         try:
-            yield self._db_pool.runInteraction(
+            yield self.runInteraction(
                 self._persist_pdu_event_txn,
                 pdu=pdu,
                 event=event,
                 backfilled=backfilled,
                 stream_ordering=stream_ordering,
+                is_new_state=is_new_state,
             )
         except _RollbackButIsFineException as e:
             pass
@@ -109,12 +128,14 @@ class DataStore(RoomMemberStore, RoomStore,
         defer.returnValue(event)
 
     def _persist_pdu_event_txn(self, txn, pdu=None, event=None,
-                               backfilled=False, stream_ordering=None):
+                               backfilled=False, stream_ordering=None,
+                               is_new_state=True):
         if pdu is not None:
             self._persist_event_pdu_txn(txn, pdu)
         if event is not None:
             return self._persist_event_txn(
-                txn, event, backfilled, stream_ordering
+                txn, event, backfilled, stream_ordering,
+                is_new_state=is_new_state,
             )
 
     def _persist_event_pdu_txn(self, txn, pdu):
@@ -141,7 +162,8 @@ class DataStore(RoomMemberStore, RoomStore,
         self._update_min_depth_for_context_txn(txn, pdu.context, pdu.depth)
 
     @log_function
-    def _persist_event_txn(self, txn, event, backfilled, stream_ordering=None):
+    def _persist_event_txn(self, txn, event, backfilled, stream_ordering=None,
+                           is_new_state=True):
         if event.type == RoomMemberEvent.TYPE:
             self._store_room_member_txn(txn, event)
         elif event.type == FeedbackEvent.TYPE:
@@ -195,7 +217,7 @@ class DataStore(RoomMemberStore, RoomStore,
             )
             raise _RollbackButIsFineException("_persist_event")
 
-        if not backfilled and hasattr(event, "state_key"):
+        if is_new_state and hasattr(event, "state_key"):
             vals = {
                 "event_id": event.event_id,
                 "room_id": event.room_id,
@@ -286,7 +308,7 @@ class DataStore(RoomMemberStore, RoomStore,
                 prev_state_pdu=prev_state_pdu,
             )
 
-        return self._db_pool.runInteraction(_snapshot)
+        return self.runInteraction(_snapshot)
 
 
 class Snapshot(object):
@@ -356,3 +378,42 @@ def read_schema(schema):
     """
     with open(schema_path(schema)) as schema_file:
         return schema_file.read()
+
+
+def prepare_database(db_conn):
+    """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we
+    don't have to worry about overwriting existing content.
+    """
+    c = db_conn.cursor()
+    c.execute("PRAGMA user_version")
+    row = c.fetchone()
+
+    if row and row[0]:
+        user_version = row[0]
+
+        if user_version > SCHEMA_VERSION:
+            raise ValueError("Cannot use this database as it is too " +
+                "new for the server to understand"
+            )
+        elif user_version < SCHEMA_VERSION:
+            logging.info("Upgrading database from version %d",
+                user_version
+            )
+
+            # Run every version since after the current version.
+            for v in range(user_version + 1, SCHEMA_VERSION + 1):
+                sql_script = read_schema("delta/v%d" % (v))
+                c.executescript(sql_script)
+
+            db_conn.commit()
+
+    else:
+        for sql_loc in SCHEMAS:
+            sql_script = read_schema(sql_loc)
+
+            c.executescript(sql_script)
+        db_conn.commit()
+        c.execute("PRAGMA user_version = %d" % SCHEMA_VERSION)
+
+    c.close()
+

synapse/storage/_base.py

@@ -26,6 +26,44 @@ import json
 
 logger = logging.getLogger(__name__)
 
+sql_logger = logging.getLogger("synapse.storage.SQL")
+
+
+class LoggingTransaction(object):
+    """An object that almost-transparently proxies for the 'txn' object
+    passed to the constructor. Adds logging to the .execute() method."""
+    __slots__ = ["txn"]
+
+    def __init__(self, txn):
+        object.__setattr__(self, "txn", txn)
+
+    def __getattribute__(self, name):
+        if name == "execute":
+            return object.__getattribute__(self, "execute")
+
+        return getattr(object.__getattribute__(self, "txn"), name)
+
+    def __setattr__(self, name, value):
+        setattr(object.__getattribute__(self, "txn"), name, value)
+
+    def execute(self, sql, *args, **kwargs):
+        # TODO(paul): Maybe use 'info' and 'debug' for values?
+        sql_logger.debug("[SQL] %s", sql)
+        try:
+            if args and args[0]:
+                values = args[0]
+                sql_logger.debug("[SQL values] " +
+                    ", ".join(("<%s>",) * len(values)), *values)
+        except:
+            # Don't let logging failures stop SQL from working
+            pass
+
+        # TODO(paul): Here would be an excellent place to put some timing
+        #   measurements, and log (warning?) slow queries.
+        return object.__getattribute__(self, "txn").execute(
+            sql, *args, **kwargs
+        )
+
 
 class SQLBaseStore(object):
 
@@ -35,6 +73,13 @@ class SQLBaseStore(object):
         self.event_factory = hs.get_event_factory()
         self._clock = hs.get_clock()
 
+    def runInteraction(self, func, *args, **kwargs):
+        """Wraps the .runInteraction() method on the underlying db_pool."""
+        def inner_func(txn, *args, **kwargs):
+            return func(LoggingTransaction(txn), *args, **kwargs)
+
+        return self._db_pool.runInteraction(inner_func, *args, **kwargs)
+
     def cursor_to_dict(self, cursor):
         """Converts a SQL cursor into an list of dicts.
 
@@ -60,11 +105,6 @@ class SQLBaseStore(object):
         Returns:
             The result of decoder(results)
         """
-        logger.debug(
-            "[SQL] %s  Args=%s Func=%s",
-            query, args, decoder.__name__ if decoder else None
-        )
-
         def interaction(txn):
             cursor = txn.execute(query, args)
             if decoder:
@@ -72,7 +112,7 @@ class SQLBaseStore(object):
             else:
                 return cursor.fetchall()
 
-        return self._db_pool.runInteraction(interaction)
+        return self.runInteraction(interaction)
 
     def _execute_and_decode(self, query, *args):
         return self._execute(self.cursor_to_dict, query, *args)
@@ -88,7 +128,7 @@ class SQLBaseStore(object):
             values : dict of new column names and values for them
             or_replace : bool; if True performs an INSERT OR REPLACE
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._simple_insert_txn, table, values, or_replace=or_replace
         )
 
@@ -172,7 +212,7 @@ class SQLBaseStore(object):
             txn.execute(sql, keyvalues.values())
             return txn.fetchall()
 
-        res = yield self._db_pool.runInteraction(func)
+        res = yield self.runInteraction(func)
 
         defer.returnValue([r[0] for r in res])
 
@@ -195,7 +235,7 @@ class SQLBaseStore(object):
             txn.execute(sql, keyvalues.values())
             return self.cursor_to_dict(txn)
 
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _simple_update_one(self, table, keyvalues, updatevalues,
                            retcols=None):
@@ -263,7 +303,7 @@ class SQLBaseStore(object):
                     raise StoreError(500, "More than one row matched")
 
             return ret
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _simple_delete_one(self, table, keyvalues):
         """Executes a DELETE query on the named table, expecting to delete a
@@ -284,7 +324,7 @@ class SQLBaseStore(object):
                 raise StoreError(404, "No row found")
             if txn.rowcount > 1:
                 raise StoreError(500, "more than one row matched")
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _simple_max_id(self, table):
         """Executes a SELECT query on the named table, expecting to return the
@@ -302,7 +342,7 @@ class SQLBaseStore(object):
                 return 0
             return max_id
 
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _parse_event_from_row(self, row_dict):
         d = copy.deepcopy({k: v for k, v in row_dict.items() if v})
@@ -315,13 +355,17 @@ class SQLBaseStore(object):
         d["content"] = json.loads(d["content"])
         del d["unrecognized_keys"]
 
+        if "age_ts" not in d:
+            # For compatibility
+            d["age_ts"] = d["ts"] if "ts" in d else 0
+
         return self.event_factory.create_event(
             etype=d["type"],
             **d
         )
 
     def _parse_events(self, rows):
-        return self._db_pool.runInteraction(self._parse_events_txn, rows)
+        return self.runInteraction(self._parse_events_txn, rows)
 
     def _parse_events_txn(self, txn, rows):
         events = [self._parse_event_from_row(r) for r in rows]

synapse/storage/pdu.py

@@ -43,7 +43,7 @@ class PduStore(SQLBaseStore):
             PduTuple: If the pdu does not exist in the database, returns None
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_pdu_tuple, pdu_id, origin
         )
 
@@ -95,7 +95,7 @@ class PduStore(SQLBaseStore):
             list: A list of PduTuples
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_current_state_for_context,
             context
         )
@@ -143,7 +143,7 @@ class PduStore(SQLBaseStore):
             pdu_origin (str)
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._mark_as_processed, pdu_id, pdu_origin
         )
 
@@ -152,7 +152,7 @@ class PduStore(SQLBaseStore):
 
     def get_all_pdus_from_context(self, context):
         """Get a list of all PDUs for a given context."""
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_all_pdus_from_context, context,
         )
 
@@ -179,7 +179,7 @@ class PduStore(SQLBaseStore):
         Return:
             list: A list of PduTuples
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_backfill, context, pdu_list, limit
         )
 
@@ -240,7 +240,7 @@ class PduStore(SQLBaseStore):
             txn
             context (str)
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_min_depth_for_context, context
         )
 
@@ -346,7 +346,7 @@ class PduStore(SQLBaseStore):
             bool
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._is_pdu_new,
             pdu_id=pdu_id,
             origin=origin,
@@ -499,7 +499,7 @@ class StatePduStore(SQLBaseStore):
         )
 
     def get_unresolved_state_tree(self, new_state_pdu):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_unresolved_state_tree, new_state_pdu
         )
 
@@ -538,7 +538,7 @@ class StatePduStore(SQLBaseStore):
 
     def update_current_state(self, pdu_id, origin, context, pdu_type,
                              state_key):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._update_current_state,
             pdu_id, origin, context, pdu_type, state_key
         )
@@ -577,7 +577,7 @@ class StatePduStore(SQLBaseStore):
             PduEntry
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_current_state_pdu, context, pdu_type, state_key
         )
 
@@ -636,7 +636,7 @@ class StatePduStore(SQLBaseStore):
         Returns:
             bool: True if the new_pdu clobbered the current state, False if not
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._handle_new_state, new_pdu
         )
 

synapse/storage/registration.py

@@ -62,7 +62,7 @@ class RegistrationStore(SQLBaseStore):
         Raises:
             StoreError if the user_id could not be registered.
         """
-        yield self._db_pool.runInteraction(self._register, user_id, token,
+        yield self.runInteraction(self._register, user_id, token,
                                            password_hash)
 
     def _register(self, txn, user_id, token, password_hash):
@@ -99,7 +99,7 @@ class RegistrationStore(SQLBaseStore):
         Raises:
             StoreError if no user was found.
         """
-        user_id = yield self._db_pool.runInteraction(self._query_for_auth,
+        user_id = yield self.runInteraction(self._query_for_auth,
                                                      token)
         defer.returnValue(user_id)
 

synapse/storage/room.py

@@ -149,7 +149,7 @@ class RoomStore(SQLBaseStore):
             defer.returnValue(None)
 
     def get_power_level(self, room_id, user_id):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_power_level,
             room_id, user_id,
         )
@@ -182,7 +182,7 @@ class RoomStore(SQLBaseStore):
             return None
 
     def get_ops_levels(self, room_id):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_ops_levels,
             room_id,
         )

synapse/storage/roommember.py

@@ -149,7 +149,7 @@ class RoomMemberStore(SQLBaseStore):
             membership_list (list): A list of synapse.api.constants.Membership
             values which the user must be in.
         Returns:
-            A list of dicts with "room_id" and "membership" keys.
+            A list of RoomMemberEvent objects
         """
         if not membership_list:
             return defer.succeed(None)
@@ -198,10 +198,11 @@ class RoomMemberStore(SQLBaseStore):
         return results
 
     @defer.inlineCallbacks
-    def user_rooms_intersect(self, user_list):
-        """ Checks whether a list of users share a room.
+    def user_rooms_intersect(self, user_id_list):
+        """ Checks whether all the users whose IDs are given in a list share a
+        room.
         """
-        user_list_clause = " OR ".join(["m.user_id = ?"] * len(user_list))
+        user_list_clause = " OR ".join(["m.user_id = ?"] * len(user_id_list))
         sql = (
             "SELECT m.room_id FROM room_memberships as m "
             "INNER JOIN current_state_events as c "
@@ -211,8 +212,8 @@ class RoomMemberStore(SQLBaseStore):
             "GROUP BY m.room_id HAVING COUNT(m.room_id) = ?"
         ) % {"clause": user_list_clause}
 
-        args = user_list
-        args.append(len(user_list))
+        args = list(user_id_list)
+        args.append(len(user_id_list))
 
         rows = yield self._execute(None, sql, *args)
 

synapse/storage/stream.py

@@ -146,7 +146,7 @@ class StreamStore(SQLBaseStore):
         current_room_membership_sql = (
             "SELECT m.room_id FROM room_memberships as m "
             "INNER JOIN current_state_events as c ON m.event_id = c.event_id "
-            "WHERE m.user_id = ?"
+            "WHERE m.user_id = ? AND m.membership = 'join'"
         )
 
         # We also want to get any membership events about that user, e.g.
@@ -286,7 +286,7 @@ class StreamStore(SQLBaseStore):
         defer.returnValue(ret)
 
     def get_room_events_max_id(self):
-        return self._db_pool.runInteraction(self._get_room_events_max_id_txn)
+        return self.runInteraction(self._get_room_events_max_id_txn)
 
     def _get_room_events_max_id_txn(self, txn):
         txn.execute(

synapse/storage/transactions.py

@@ -41,7 +41,7 @@ class TransactionStore(SQLBaseStore):
             this transaction or a 2-tuple of (int, dict)
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_received_txn_response, transaction_id, origin
         )
 
@@ -72,7 +72,7 @@ class TransactionStore(SQLBaseStore):
             response_json (str)
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._set_received_txn_response,
             transaction_id, origin, code, response_dict
         )
@@ -104,7 +104,7 @@ class TransactionStore(SQLBaseStore):
             list: A list of previous transaction ids.
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._prep_send_transaction,
             transaction_id, destination, ts, pdu_list
         )
@@ -159,7 +159,7 @@ class TransactionStore(SQLBaseStore):
             code (int)
             response_json (str)
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._delivered_txn,
             transaction_id, destination, code, response_dict
         )
@@ -184,7 +184,7 @@ class TransactionStore(SQLBaseStore):
         Returns:
             list: A list of `ReceivedTransactionsTable.EntryType`
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_transactions_after, transaction_id, destination
         )
 
@@ -214,7 +214,7 @@ class TransactionStore(SQLBaseStore):
         Returns
             list: A list of PduTuple
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_pdus_after_transaction,
             transaction_id, destination
         )

synapse/util/emailutils.py

@@ -0,0 +1,71 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" This module allows you to send out emails.
+"""
+import email.utils
+import smtplib
+import twisted.python.log
+from email.mime.text import MIMEText
+from email.mime.multipart import MIMEMultipart
+
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+class EmailException(Exception):
+    pass
+
+
+def send_email(smtp_server, from_addr, to_addr, subject, body):
+    """Sends an email.
+
+    Args:
+        smtp_server(str): The SMTP server to use.
+        from_addr(str): The address to send from.
+        to_addr(str): The address to send to.
+        subject(str): The subject of the email.
+        body(str): The plain text body of the email.
+    Raises:
+        EmailException if there was a problem sending the mail.
+    """
+    if not smtp_server or not from_addr or not to_addr:
+        raise EmailException("Need SMTP server, from and to addresses. Check " +
+                             "the config to set these.")
+
+    msg = MIMEMultipart('alternative')
+    msg['Subject'] = subject
+    msg['From'] = from_addr
+    msg['To'] = to_addr
+    plain_part = MIMEText(body)
+    msg.attach(plain_part)
+
+    raw_from = email.utils.parseaddr(from_addr)[1]
+    raw_to = email.utils.parseaddr(to_addr)[1]
+    if not raw_from or not raw_to:
+        raise EmailException("Couldn't parse from/to address.")
+
+    logger.info("Sending email to %s on server %s with subject %s",
+                to_addr, smtp_server, subject)
+
+    try:
+        smtp = smtplib.SMTP(smtp_server)
+        smtp.sendmail(raw_from, raw_to, msg.as_string())
+        smtp.quit()
+    except Exception as origException:
+        twisted.python.log.err()
+        ese = EmailException()
+        ese.cause = origException
+        raise ese

tests/api/test_ratelimiting.py

@@ -1,6 +1,6 @@
 from synapse.api.ratelimiting import Ratelimiter
 
-import unittest
+from tests import unittest
 
 class TestRatelimiter(unittest.TestCase):
 

tests/events/test_events.py

@@ -15,7 +15,7 @@
 
 from synapse.api.events import SynapseEvent
 
-import unittest
+from tests import unittest
 
 
 class SynapseTemplateCheckTestCase(unittest.TestCase):

tests/federation/test_federation.py

@@ -14,11 +14,10 @@
 
 # trial imports
 from twisted.internet import defer
-from twisted.trial import unittest
+from tests import unittest
 
 # python imports
-from mock import Mock
-import logging
+from mock import Mock, ANY
 
 from ..utils import MockHttpResource, MockClock
 
@@ -28,9 +27,6 @@ from synapse.federation.units import Pdu
 from synapse.storage.pdu import PduTuple, PduEntry
 
 
-logging.getLogger().addHandler(logging.NullHandler())
-
-
 def make_pdu(prev_pdus=[], **kwargs):
     """Provide some default fields for making a PduTuple."""
     pdu_fields = {
@@ -185,7 +181,8 @@ class FederationTestCase(unittest.TestCase):
                             "depth": 1,
                         },
                     ]
-                }
+                },
+                on_send_callback=ANY,
         )
 
     @defer.inlineCallbacks
@@ -216,7 +213,9 @@ class FederationTestCase(unittest.TestCase):
                             "content": {"testing": "content here"},
                         }
                     ],
-                })
+                },
+                on_send_callback=ANY,
+        )
 
     @defer.inlineCallbacks
     def test_recv_edu(self):

tests/federation/test_pdu_codec.py

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from twisted.trial import unittest
+from tests import unittest
 
 from synapse.federation.pdu_codec import (
     PduCodec, encode_event_id, decode_event_id

tests/handlers/test_directory.py

@@ -14,19 +14,17 @@
 # limitations under the License.
 
 
-from twisted.trial import unittest
+from tests import unittest
 from twisted.internet import defer
 
 from mock import Mock
-import logging
 
 from synapse.server import HomeServer
 from synapse.http.client import HttpClient
 from synapse.handlers.directory import DirectoryHandler
 from synapse.storage.directory import RoomAliasMapping
 
-
-logging.getLogger().addHandler(logging.NullHandler())
+from tests.utils import SQLiteMemoryDbPool
 
 
 class DirectoryHandlers(object):
@@ -37,6 +35,7 @@ class DirectoryHandlers(object):
 class DirectoryTestCase(unittest.TestCase):
     """ Tests the directory service. """
 
+    @defer.inlineCallbacks
     def setUp(self):
         self.mock_federation = Mock(spec=[
             "make_query",
@@ -47,11 +46,11 @@ class DirectoryTestCase(unittest.TestCase):
             self.query_handlers[query_type] = handler
         self.mock_federation.register_query_handler = register_query_handler
 
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
         hs = HomeServer("test",
-            datastore=Mock(spec=[
-                "get_association_from_room_alias",
-                "get_joined_hosts_for_room",
-            ]),
+            db_pool=db_pool,
             http_client=None,
             resource_for_federation=Mock(),
             replication_layer=self.mock_federation,
@@ -60,20 +59,16 @@ class DirectoryTestCase(unittest.TestCase):
 
         self.handler = hs.get_handlers().directory_handler
 
-        self.datastore = hs.get_datastore()
-
-        def hosts(room_id):
-            return defer.succeed([])
-        self.datastore.get_joined_hosts_for_room.side_effect = hosts
+        self.store = hs.get_datastore()
 
         self.my_room = hs.parse_roomalias("#my-room:test")
+        self.your_room = hs.parse_roomalias("#your-room:test")
         self.remote_room = hs.parse_roomalias("#another:remote")
 
     @defer.inlineCallbacks
     def test_get_local_association(self):
-        mocked_get = self.datastore.get_association_from_room_alias
-        mocked_get.return_value = defer.succeed(
-            RoomAliasMapping("!8765qwer:test", "#my-room:test", ["test"])
+        yield self.store.create_room_alias_association(
+            self.my_room, "!8765qwer:test", ["test"]
         )
 
         result = yield self.handler.get_association(self.my_room)
@@ -106,9 +101,8 @@ class DirectoryTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_incoming_fed_query(self):
-        mocked_get = self.datastore.get_association_from_room_alias
-        mocked_get.return_value = defer.succeed(
-            RoomAliasMapping("!8765asdf:test", "#your-room:test", ["test"])
+        yield self.store.create_room_alias_association(
+            self.your_room, "!8765asdf:test", ["test"]
         )
 
         response = yield self.query_handlers["directory"](

tests/handlers/test_federation.py

@@ -14,7 +14,7 @@
 
 
 from twisted.internet import defer
-from twisted.trial import unittest
+from tests import unittest
 
 from synapse.api.events.room import (
     InviteJoinEvent, MessageEvent, RoomMemberEvent
@@ -26,12 +26,8 @@ from synapse.federation.units import Pdu
 
 from mock import NonCallableMock, ANY
 
-import logging
-
 from ..utils import get_mock_call_args
 
-logging.getLogger().addHandler(logging.NullHandler())
-
 
 class FederationTestCase(unittest.TestCase):
 
@@ -78,7 +74,9 @@ class FederationTestCase(unittest.TestCase):
 
         yield self.handlers.federation_handler.on_receive_pdu(pdu, False)
 
-        self.datastore.persist_event.assert_called_once_with(ANY, False)
+        self.datastore.persist_event.assert_called_once_with(
+            ANY, False, is_new_state=False
+        )
         self.notifier.on_new_room_event.assert_called_once_with(ANY)
 
     @defer.inlineCallbacks

tests/handlers/test_presence.py

@@ -14,14 +14,15 @@
 # limitations under the License.
 
 
-from twisted.trial import unittest
+from tests import unittest
 from twisted.internet import defer, reactor
 
 from mock import Mock, call, ANY
-import logging
 import json
 
-from ..utils import MockHttpResource, MockClock, DeferredMockCallable
+from tests.utils import (
+    MockHttpResource, MockClock, DeferredMockCallable, SQLiteMemoryDbPool
+)
 
 from synapse.server import HomeServer
 from synapse.api.constants import PresenceState
@@ -34,9 +35,6 @@ UNAVAILABLE = PresenceState.UNAVAILABLE
 ONLINE = PresenceState.ONLINE
 
 
-logging.getLogger().addHandler(logging.NullHandler())
-
-
 def _expect_edu(destination, edu_type, content, origin="test"):
     return {
         "origin": origin,
@@ -64,41 +62,36 @@ class JustPresenceHandlers(object):
 class PresenceStateTestCase(unittest.TestCase):
     """ Tests presence management. """
 
+    @defer.inlineCallbacks
     def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
         hs = HomeServer("test",
-                clock=MockClock(),
-                db_pool=None,
-                datastore=Mock(spec=[
-                    "get_presence_state",
-                    "set_presence_state",
-                    "add_presence_list_pending",
-                    "set_presence_list_accepted",
-                ]),
-                handlers=None,
-                resource_for_federation=Mock(),
-                http_client=None,
-            )
+            clock=MockClock(),
+            db_pool=db_pool,
+            handlers=None,
+            resource_for_federation=Mock(),
+            http_client=None,
+        )
         hs.handlers = JustPresenceHandlers(hs)
 
-        self.datastore = hs.get_datastore()
-
-        def is_presence_visible(observed_localpart, observer_userid):
-            allow = (observed_localpart == "apple" and
-                observer_userid == "@banana:test"
-            )
-            return defer.succeed(allow)
-        self.datastore.is_presence_visible = is_presence_visible
+        self.store = hs.get_datastore()
 
         # Mock the RoomMemberHandler
         room_member_handler = Mock(spec=[])
         hs.handlers.room_member_handler = room_member_handler
-        logging.getLogger().debug("Mocking room_member_handler=%r", room_member_handler)
 
         # Some local users to test with
         self.u_apple = hs.parse_userid("@apple:test")
         self.u_banana = hs.parse_userid("@banana:test")
         self.u_clementine = hs.parse_userid("@clementine:test")
 
+        yield self.store.create_presence(self.u_apple.localpart)
+        yield self.store.set_presence_state(
+            self.u_apple.localpart, {"state": ONLINE, "status_msg": "Online"}
+        )
+
         self.handler = hs.get_handlers().presence_handler
 
         self.room_members = []
@@ -122,7 +115,7 @@ class PresenceStateTestCase(unittest.TestCase):
 
             shared = all(map(lambda i: i in room_member_ids, userlist))
             return defer.succeed(shared)
-        self.datastore.user_rooms_intersect = user_rooms_intersect
+        self.store.user_rooms_intersect = user_rooms_intersect
 
         self.mock_start = Mock()
         self.mock_stop = Mock()
@@ -132,11 +125,6 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_get_my_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
-        )
-
         state = yield self.handler.get_state(
             target_user=self.u_apple, auth_user=self.u_apple
         )
@@ -145,13 +133,12 @@ class PresenceStateTestCase(unittest.TestCase):
             {"presence": ONLINE, "status_msg": "Online"},
             state
         )
-        mocked_get.assert_called_with("apple")
 
     @defer.inlineCallbacks
     def test_get_allowed_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
+        yield self.store.allow_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
         )
 
         state = yield self.handler.get_state(
@@ -162,15 +149,9 @@ class PresenceStateTestCase(unittest.TestCase):
             {"presence": ONLINE, "status_msg": "Online"},
             state
         )
-        mocked_get.assert_called_with("apple")
 
     @defer.inlineCallbacks
     def test_get_same_room_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
-        )
-
         self.room_members = [self.u_apple, self.u_clementine]
 
         state = yield self.handler.get_state(
@@ -184,11 +165,6 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_get_disallowed_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
-        )
-
         self.room_members = []
 
         yield self.assertFailure(
@@ -200,16 +176,17 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_set_my_state(self):
-        mocked_set = self.datastore.set_presence_state
-        mocked_set.return_value = defer.succeed({"state": OFFLINE})
-
         yield self.handler.set_state(
                 target_user=self.u_apple, auth_user=self.u_apple,
                 state={"presence": UNAVAILABLE, "status_msg": "Away"})
 
-        mocked_set.assert_called_with("apple",
-            {"state": UNAVAILABLE, "status_msg": "Away"}
+        self.assertEquals(
+            {"state": UNAVAILABLE,
+             "status_msg": "Away",
+             "mtime": 1000000},
+            (yield self.store.get_presence_state(self.u_apple.localpart))
         )
+
         self.mock_start.assert_called_with(self.u_apple,
                 state={
                     "presence": UNAVAILABLE,
@@ -227,50 +204,34 @@ class PresenceStateTestCase(unittest.TestCase):
 class PresenceInvitesTestCase(unittest.TestCase):
     """ Tests presence management. """
 
+    @defer.inlineCallbacks
     def setUp(self):
         self.mock_http_client = Mock(spec=[])
         self.mock_http_client.put_json = DeferredMockCallable()
 
         self.mock_federation_resource = MockHttpResource()
 
-        hs = HomeServer("test",
-                clock=MockClock(),
-                db_pool=None,
-                datastore=Mock(spec=[
-                    "has_presence_state",
-                    "allow_presence_visible",
-                    "add_presence_list_pending",
-                    "set_presence_list_accepted",
-                    "get_presence_list",
-                    "del_presence_list",
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
 
-                    # Bits that Federation needs
-                    "prep_send_transaction",
-                    "delivered_txn",
-                    "get_received_txn_response",
-                    "set_received_txn_response",
-                ]),
-                handlers=None,
-                resource_for_client=Mock(),
-                resource_for_federation=self.mock_federation_resource,
-                http_client=self.mock_http_client,
-            )
+        hs = HomeServer("test",
+            clock=MockClock(),
+            db_pool=db_pool,
+            handlers=None,
+            resource_for_client=Mock(),
+            resource_for_federation=self.mock_federation_resource,
+            http_client=self.mock_http_client,
+        )
         hs.handlers = JustPresenceHandlers(hs)
 
-        self.datastore = hs.get_datastore()
-
-        def has_presence_state(user_localpart):
-            return defer.succeed(
-                user_localpart in ("apple", "banana"))
-        self.datastore.has_presence_state = has_presence_state
-
-        def get_received_txn_response(*args):
-            return defer.succeed(None)
-        self.datastore.get_received_txn_response = get_received_txn_response
+        self.store = hs.get_datastore()
 
         # Some local users to test with
         self.u_apple = hs.parse_userid("@apple:test")
         self.u_banana = hs.parse_userid("@banana:test")
+        yield self.store.create_presence(self.u_apple.localpart)
+        yield self.store.create_presence(self.u_banana.localpart)
+
         # ID of a local user that does not exist
         self.u_durian = hs.parse_userid("@durian:test")
 
@@ -293,12 +254,16 @@ class PresenceInvitesTestCase(unittest.TestCase):
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_banana)
 
-        self.datastore.add_presence_list_pending.assert_called_with(
-                "apple", "@banana:test")
-        self.datastore.allow_presence_visible.assert_called_with(
-                "banana", "@apple:test")
-        self.datastore.set_presence_list_accepted.assert_called_with(
-                "apple", "@banana:test")
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 1}],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
+        self.assertTrue(
+            (yield self.store.is_presence_visible(
+                observed_localpart=self.u_banana.localpart,
+                observer_userid=self.u_apple.to_string(),
+            ))
+        )
 
         self.mock_start.assert_called_with(
                 self.u_apple, target_user=self.u_banana)
@@ -308,10 +273,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_durian)
 
-        self.datastore.add_presence_list_pending.assert_called_with(
-                "apple", "@durian:test")
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@durian:test")
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
     @defer.inlineCallbacks
     def test_invite_remote(self):
@@ -324,7 +289,8 @@ class PresenceInvitesTestCase(unittest.TestCase):
                         "observer_user": "@apple:test",
                         "observed_user": "@cabbage:elsewhere",
                     }
-                )
+                ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -332,8 +298,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_cabbage)
 
-        self.datastore.add_presence_list_pending.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        self.assertEquals(
+            [{"observed_user_id": "@cabbage:elsewhere", "accepted": 0}],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
         yield put_json.await_calls()
 
@@ -350,7 +318,8 @@ class PresenceInvitesTestCase(unittest.TestCase):
                         "observer_user": "@cabbage:elsewhere",
                         "observed_user": "@apple:test",
                     }
-                )
+                ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -365,8 +334,12 @@ class PresenceInvitesTestCase(unittest.TestCase):
             )
         )
 
-        self.datastore.allow_presence_visible.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        self.assertTrue(
+            (yield self.store.is_presence_visible(
+                observed_localpart=self.u_apple.localpart,
+                observer_userid=self.u_cabbage.to_string(),
+            ))
+        )
 
         yield put_json.await_calls()
 
@@ -381,7 +354,8 @@ class PresenceInvitesTestCase(unittest.TestCase):
                         "observer_user": "@cabbage:elsewhere",
                         "observed_user": "@durian:test",
                     }
-                )
+                ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -400,6 +374,11 @@ class PresenceInvitesTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_accepted_remote(self):
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_cabbage.to_string(),
+        )
+
         yield self.mock_federation_resource.trigger("PUT",
             "/_matrix/federation/v1/send/1000000/",
             _make_edu_json("elsewhere", "m.presence_accept",
@@ -410,14 +389,21 @@ class PresenceInvitesTestCase(unittest.TestCase):
             )
         )
 
-        self.datastore.set_presence_list_accepted.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        self.assertEquals(
+            [{"observed_user_id": "@cabbage:elsewhere", "accepted": 1}],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
         self.mock_start.assert_called_with(
                 self.u_apple, target_user=self.u_cabbage)
 
     @defer.inlineCallbacks
     def test_denied_remote(self):
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid="@eggplant:elsewhere",
+        )
+
         yield self.mock_federation_resource.trigger("PUT",
             "/_matrix/federation/v1/send/1000000/",
             _make_edu_json("elsewhere", "m.presence_deny",
@@ -428,32 +414,65 @@ class PresenceInvitesTestCase(unittest.TestCase):
             )
         )
 
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@eggplant:elsewhere")
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
     @defer.inlineCallbacks
     def test_drop_local(self):
-        yield self.handler.drop(
-                observer_user=self.u_apple, observed_user=self.u_banana)
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
 
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@banana:test")
+        yield self.handler.drop(
+            observer_user=self.u_apple,
+            observed_user=self.u_banana,
+        )
+
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
         self.mock_stop.assert_called_with(
                 self.u_apple, target_user=self.u_banana)
 
     @defer.inlineCallbacks
     def test_drop_remote(self):
-        yield self.handler.drop(
-                observer_user=self.u_apple, observed_user=self.u_cabbage)
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_cabbage.to_string(),
+        )
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_cabbage.to_string(),
+        )
 
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        yield self.handler.drop(
+            observer_user=self.u_apple,
+            observed_user=self.u_cabbage,
+        )
+
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
     @defer.inlineCallbacks
     def test_get_presence_list(self):
-        self.datastore.get_presence_list.return_value = defer.succeed(
-            [{"observed_user_id": "@banana:test"}]
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
         )
 
         presence = yield self.handler.get_presence_list(
@@ -461,29 +480,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
 
         self.assertEquals([
             {"observed_user": self.u_banana,
-             "presence": OFFLINE},
+             "presence": OFFLINE,
+             "accepted": 1},
         ], presence)
 
-        self.datastore.get_presence_list.assert_called_with("apple",
-            accepted=None
-        )
-
-        self.datastore.get_presence_list.return_value = defer.succeed(
-            [{"observed_user_id": "@banana:test"}]
-        )
-
-        presence = yield self.handler.get_presence_list(
-            observer_user=self.u_apple, accepted=True
-        )
-
-        self.assertEquals([
-            {"observed_user": self.u_banana,
-             "presence": OFFLINE},
-        ], presence)
-
-        self.datastore.get_presence_list.assert_called_with("apple",
-                accepted=True)
-
 
 class PresencePushTestCase(unittest.TestCase):
     """ Tests steady-state presence status updates.
@@ -770,7 +770,8 @@ class PresencePushTestCase(unittest.TestCase):
                              "last_active_ago": 0},
                         ],
                     }
-                )
+                ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -785,7 +786,8 @@ class PresencePushTestCase(unittest.TestCase):
                              "last_active_ago": 0},
                         ],
                     }
-                )
+                ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -911,6 +913,7 @@ class PresencePushTestCase(unittest.TestCase):
                         ],
                     }
                 ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -925,6 +928,7 @@ class PresencePushTestCase(unittest.TestCase):
                         ],
                     }
                 ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -954,6 +958,7 @@ class PresencePushTestCase(unittest.TestCase):
                         ],
                     }
                 ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -1150,6 +1155,7 @@ class PresencePollingTestCase(unittest.TestCase):
                         "poll": [ "@potato:remote" ],
                     },
                 ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -1162,6 +1168,7 @@ class PresencePollingTestCase(unittest.TestCase):
                         "push": [ {"user_id": "@clementine:test" }],
                     },
                 ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -1190,6 +1197,7 @@ class PresencePollingTestCase(unittest.TestCase):
                         "push": [ {"user_id": "@fig:test" }],
                     },
                 ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -1222,6 +1230,7 @@ class PresencePollingTestCase(unittest.TestCase):
                         "unpoll": [ "@potato:remote" ],
                     },
                 ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -1253,6 +1262,7 @@ class PresencePollingTestCase(unittest.TestCase):
                         ],
                     },
                 ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )

tests/handlers/test_presencelike.py

@@ -16,11 +16,10 @@
 """This file contains tests of the "presence-like" data that is shared between
 presence and profiles; namely, the displayname and avatar_url."""
 
-from twisted.trial import unittest
+from tests import unittest
 from twisted.internet import defer
 
 from mock import Mock, call, ANY
-import logging
 
 from ..utils import MockClock
 
@@ -35,9 +34,6 @@ UNAVAILABLE = PresenceState.UNAVAILABLE
 ONLINE = PresenceState.ONLINE
 
 
-logging.getLogger().addHandler(logging.NullHandler())
-
-
 class MockReplication(object):
     def __init__(self):
         self.edu_handlers = {}
@@ -69,6 +65,8 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
                     "is_presence_visible",
 
                     "set_profile_displayname",
+
+                    "get_rooms_for_user_where_membership_is",
                 ]),
                 handlers=None,
                 resource_for_federation=Mock(),
@@ -136,6 +134,10 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
         # Remote user
         self.u_potato = hs.parse_userid("@potato:remote")
 
+        self.mock_get_joined = (
+            self.datastore.get_rooms_for_user_where_membership_is
+        )
+
     @defer.inlineCallbacks
     def test_set_my_state(self):
         self.presence_list = [
@@ -156,6 +158,11 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_push_local(self):
+        def get_joined(*args):
+            return defer.succeed([])
+
+        self.mock_get_joined.side_effect = get_joined
+
         self.presence_list = [
             {"observed_user_id": "@banana:test"},
             {"observed_user_id": "@clementine:test"},

tests/handlers/test_profile.py

@@ -14,18 +14,17 @@
 # limitations under the License.
 
 
-from twisted.trial import unittest
+from tests import unittest
 from twisted.internet import defer
 
 from mock import Mock
-import logging
 
 from synapse.api.errors import AuthError
 from synapse.server import HomeServer
 from synapse.handlers.profile import ProfileHandler
+from synapse.api.constants import Membership
 
-
-logging.getLogger().addHandler(logging.NullHandler())
+from tests.utils import SQLiteMemoryDbPool
 
 
 class ProfileHandlers(object):
@@ -36,6 +35,7 @@ class ProfileHandlers(object):
 class ProfileTestCase(unittest.TestCase):
     """ Tests profile management. """
 
+    @defer.inlineCallbacks
     def setUp(self):
         self.mock_federation = Mock(spec=[
             "make_query",
@@ -46,27 +46,26 @@ class ProfileTestCase(unittest.TestCase):
             self.query_handlers[query_type] = handler
         self.mock_federation.register_query_handler = register_query_handler
 
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
         hs = HomeServer("test",
-                db_pool=None,
+                db_pool=db_pool,
                 http_client=None,
-                datastore=Mock(spec=[
-                    "get_profile_displayname",
-                    "set_profile_displayname",
-                    "get_profile_avatar_url",
-                    "set_profile_avatar_url",
-                ]),
                 handlers=None,
                 resource_for_federation=Mock(),
                 replication_layer=self.mock_federation,
             )
         hs.handlers = ProfileHandlers(hs)
 
-        self.datastore = hs.get_datastore()
+        self.store = hs.get_datastore()
 
         self.frank = hs.parse_userid("@1234ABCD:test")
         self.bob   = hs.parse_userid("@4567:test")
         self.alice = hs.parse_userid("@alice:remote")
 
+        yield self.store.create_profile(self.frank.localpart)
+
         self.handler = hs.get_handlers().profile_handler
 
         # TODO(paul): Icky signal declarings.. booo
@@ -74,22 +73,22 @@ class ProfileTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_get_my_name(self):
-        mocked_get = self.datastore.get_profile_displayname
-        mocked_get.return_value = defer.succeed("Frank")
+        yield self.store.set_profile_displayname(
+            self.frank.localpart, "Frank"
+        )
 
         displayname = yield self.handler.get_displayname(self.frank)
 
         self.assertEquals("Frank", displayname)
-        mocked_get.assert_called_with("1234ABCD")
 
     @defer.inlineCallbacks
     def test_set_my_name(self):
-        mocked_set = self.datastore.set_profile_displayname
-        mocked_set.return_value = defer.succeed(())
-
         yield self.handler.set_displayname(self.frank, self.frank, "Frank Jr.")
 
-        mocked_set.assert_called_with("1234ABCD", "Frank Jr.")
+        self.assertEquals(
+            (yield self.store.get_profile_displayname(self.frank.localpart)),
+            "Frank Jr."
+        )
 
     @defer.inlineCallbacks
     def test_set_my_name_noauth(self):
@@ -114,32 +113,31 @@ class ProfileTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_incoming_fed_query(self):
-        mocked_get = self.datastore.get_profile_displayname
-        mocked_get.return_value = defer.succeed("Caroline")
+        yield self.store.create_profile("caroline")
+        yield self.store.set_profile_displayname("caroline", "Caroline")
 
         response = yield self.query_handlers["profile"](
             {"user_id": "@caroline:test", "field": "displayname"}
         )
 
         self.assertEquals({"displayname": "Caroline"}, response)
-        mocked_get.assert_called_with("caroline")
 
     @defer.inlineCallbacks
     def test_get_my_avatar(self):
-        mocked_get = self.datastore.get_profile_avatar_url
-        mocked_get.return_value = defer.succeed("http://my.server/me.png")
+        yield self.store.set_profile_avatar_url(
+            self.frank.localpart, "http://my.server/me.png"
+        )
 
         avatar_url = yield self.handler.get_avatar_url(self.frank)
 
         self.assertEquals("http://my.server/me.png", avatar_url)
-        mocked_get.assert_called_with("1234ABCD")
 
     @defer.inlineCallbacks
     def test_set_my_avatar(self):
-        mocked_set = self.datastore.set_profile_avatar_url
-        mocked_set.return_value = defer.succeed(())
-
         yield self.handler.set_avatar_url(self.frank, self.frank,
                 "http://my.server/pic.gif")
 
-        mocked_set.assert_called_with("1234ABCD", "http://my.server/pic.gif")
+        self.assertEquals(
+            (yield self.store.get_profile_avatar_url(self.frank.localpart)),
+            "http://my.server/pic.gif"
+        )

tests/handlers/test_room.py

@@ -15,7 +15,7 @@
 
 
 from twisted.internet import defer
-from twisted.trial import unittest
+from tests import unittest
 
 from synapse.api.events.room import (
     InviteJoinEvent, RoomMemberEvent, RoomConfigEvent
@@ -27,10 +27,6 @@ from synapse.server import HomeServer
 
 from mock import Mock, NonCallableMock
 
-import logging
-
-logging.getLogger().addHandler(logging.NullHandler())
-
 
 class RoomMemberHandlerTestCase(unittest.TestCase):
 

tests/handlers/test_typing.py

@@ -14,12 +14,11 @@
 # limitations under the License.
 
 
-from twisted.trial import unittest
+from tests import unittest
 from twisted.internet import defer
 
 from mock import Mock, call, ANY
 import json
-import logging
 
 from ..utils import MockHttpResource, MockClock, DeferredMockCallable
 
@@ -27,9 +26,6 @@ from synapse.server import HomeServer
 from synapse.handlers.typing import TypingNotificationHandler
 
 
-logging.getLogger().addHandler(logging.NullHandler())
-
-
 def _expect_edu(destination, edu_type, content, origin="test"):
     return {
         "origin": origin,
@@ -173,7 +169,8 @@ class TypingNotificationsTestCase(unittest.TestCase):
                         "user_id": self.u_apple.to_string(),
                         "typing": True,
                     }
-                )
+                ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )
@@ -223,7 +220,8 @@ class TypingNotificationsTestCase(unittest.TestCase):
                         "user_id": self.u_apple.to_string(),
                         "typing": False,
                     }
-                )
+                ),
+                on_send_callback=ANY,
             ),
             defer.succeed((200, "OK"))
         )

tests/rest/test_events.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 """ Tests REST events for /events paths."""
-from twisted.trial import unittest
+from tests import unittest
 
 # twisted imports
 from twisted.internet import defer
@@ -27,14 +27,12 @@ from synapse.server import HomeServer
 
 # python imports
 import json
-import logging
 
 from ..utils import MockHttpResource, MemoryDataStore
 from .utils import RestTestCase
 
 from mock import Mock, NonCallableMock
 
-logging.getLogger().addHandler(logging.NullHandler())
 
 PATH_PREFIX = "/_matrix/client/api/v1"
 

tests/rest/test_presence.py

@@ -15,11 +15,10 @@
 
 """Tests REST events for /presence paths."""
 
-from twisted.trial import unittest
+from tests import unittest
 from twisted.internet import defer
 
 from mock import Mock
-import logging
 
 from ..utils import MockHttpResource
 
@@ -28,9 +27,6 @@ from synapse.handlers.presence import PresenceHandler
 from synapse.server import HomeServer
 
 
-logging.getLogger().addHandler(logging.NullHandler())
-
-
 OFFLINE = PresenceState.OFFLINE
 UNAVAILABLE = PresenceState.UNAVAILABLE
 ONLINE = PresenceState.ONLINE

tests/rest/test_profile.py

@@ -15,7 +15,7 @@
 
 """Tests REST events for /profile paths."""
 
-from twisted.trial import unittest
+from tests import unittest
 from twisted.internet import defer
 
 from mock import Mock
@@ -28,6 +28,7 @@ from synapse.server import HomeServer
 myid = "@1234ABCD:test"
 PATH_PREFIX = "/_matrix/client/api/v1"
 
+
 class ProfileTestCase(unittest.TestCase):
     """ Tests profile management. """
 

tests/rest/utils.py

@@ -17,7 +17,7 @@
 from twisted.internet import defer
 
 # trial imports
-from twisted.trial import unittest
+from tests import unittest
 
 from synapse.api.constants import Membership
 
@@ -95,8 +95,14 @@ class RestTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def register(self, user_id):
-        (code, response) = yield self.mock_resource.trigger("POST", "/register",
-                                '{"user_id":"%s"}' % user_id)
+        (code, response) = yield self.mock_resource.trigger(
+            "POST",
+            "/register",
+            json.dumps({
+                "user": user_id,
+                "password": "test",
+                "type": "m.login.password"
+            }))
         self.assertEquals(200, code)
         defer.returnValue(response)
 

tests/storage/TESTS_NEEDED_FOR

@@ -0,0 +1,5 @@
+synapse/storage/feedback.py
+synapse/storage/keys.py
+synapse/storage/pdu.py
+synapse/storage/stream.py
+synapse/storage/transactions.py

tests/storage/test_base.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 
-from twisted.trial import unittest
+from tests import unittest
 from twisted.internet import defer
 
 from mock import Mock, call

tests/storage/test_directory.py

@@ -0,0 +1,68 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.directory import DirectoryStore
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class DirectoryStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        self.store = DirectoryStore(hs)
+
+        self.room = hs.parse_roomid("!abcde:test")
+        self.alias = hs.parse_roomalias("#my-room:test")
+
+    @defer.inlineCallbacks
+    def test_room_to_alias(self):
+        yield self.store.create_room_alias_association(
+            room_alias=self.alias,
+            room_id=self.room.to_string(),
+            servers=["test"],
+        )
+
+        self.assertEquals(
+            ["#my-room:test"],
+            (yield self.store.get_aliases_for_room(self.room.to_string()))
+        )
+
+    @defer.inlineCallbacks
+    def test_alias_to_room(self):
+        yield self.store.create_room_alias_association(
+            room_alias=self.alias,
+            room_id=self.room.to_string(),
+            servers=["test"],
+        )
+
+
+        self.assertObjectHasAttributes(
+            {"room_id": self.room.to_string(),
+             "servers": ["test"]},
+            (yield self.store.get_association_from_room_alias(self.alias))
+        )

tests/storage/test_presence.py

@@ -0,0 +1,167 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.presence import PresenceStore
+
+from tests.utils import SQLiteMemoryDbPool, MockClock
+
+
+class PresenceStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            clock=MockClock(),
+            db_pool=db_pool,
+        )
+
+        self.store = PresenceStore(hs)
+
+        self.u_apple = hs.parse_userid("@apple:test")
+        self.u_banana = hs.parse_userid("@banana:test")
+
+    @defer.inlineCallbacks
+    def test_state(self):
+        yield self.store.create_presence(
+            self.u_apple.localpart
+        )
+
+        state = yield self.store.get_presence_state(
+            self.u_apple.localpart
+        )
+
+        self.assertEquals(
+            {"state": None, "status_msg": None, "mtime": None}, state
+        )
+
+        yield self.store.set_presence_state(
+            self.u_apple.localpart, {"state": "online", "status_msg": "Here"}
+        )
+
+        state = yield self.store.get_presence_state(
+            self.u_apple.localpart
+        )
+
+        self.assertEquals(
+            {"state": "online", "status_msg": "Here", "mtime": 1000000}, state
+        )
+
+    @defer.inlineCallbacks
+    def test_visibility(self):
+        self.assertFalse((yield self.store.is_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )))
+
+        yield self.store.allow_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )
+
+        self.assertTrue((yield self.store.is_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )))
+
+        yield self.store.disallow_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )
+
+        self.assertFalse((yield self.store.is_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )))
+
+    @defer.inlineCallbacks
+    def test_presence_list(self):
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )
+
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 0}],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )
+
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 1}],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 1}],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )
+
+        yield self.store.del_presence_list(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )

tests/storage/test_profile.py

@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.profile import ProfileStore
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class ProfileStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        self.store = ProfileStore(hs)
+
+        self.u_frank = hs.parse_userid("@frank:test")
+
+    @defer.inlineCallbacks
+    def test_displayname(self):
+        yield self.store.create_profile(
+            self.u_frank.localpart
+        )
+
+        yield self.store.set_profile_displayname(
+            self.u_frank.localpart, "Frank"
+        )
+
+        self.assertEquals(
+            "Frank",
+            (yield self.store.get_profile_displayname(self.u_frank.localpart))
+        )
+
+    @defer.inlineCallbacks
+    def test_avatar_url(self):
+        yield self.store.create_profile(
+            self.u_frank.localpart
+        )
+
+        yield self.store.set_profile_avatar_url(
+                self.u_frank.localpart, "http://my.site/here"
+        )
+
+        self.assertEquals(
+            "http://my.site/here",
+            (yield self.store.get_profile_avatar_url(self.u_frank.localpart))
+        )

tests/storage/test_registration.py

@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.registration import RegistrationStore
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RegistrationStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        self.store = RegistrationStore(hs)
+
+        self.user_id = "@my-user:test"
+        self.tokens = ["AbCdEfGhIjKlMnOpQrStUvWxYz",
+                        "BcDeFgHiJkLmNoPqRsTuVwXyZa"]
+        self.pwhash = "{xx1}123456789"
+
+    @defer.inlineCallbacks
+    def test_register(self):
+        yield self.store.register(self.user_id, self.tokens[0], self.pwhash)
+
+        self.assertEquals(
+            # TODO(paul): Surely this field should be 'user_id', not 'name'
+            #  Additionally surely it shouldn't come in a 1-element list
+            [{"name": self.user_id, "password_hash": self.pwhash}],
+            (yield self.store.get_user_by_id(self.user_id))
+        )
+
+        self.assertEquals(
+            self.user_id,
+            (yield self.store.get_user_by_token(self.tokens[0]))
+        )
+
+    @defer.inlineCallbacks
+    def test_add_tokens(self):
+        yield self.store.register(self.user_id, self.tokens[0], self.pwhash)
+        yield self.store.add_access_token_to_user(self.user_id, self.tokens[1])
+
+        self.assertEquals(
+            self.user_id,
+            (yield self.store.get_user_by_token(self.tokens[1]))
+        )
+

tests/storage/test_room.py

@@ -0,0 +1,176 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.events.room import (
+    RoomNameEvent, RoomTopicEvent
+)
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RoomStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        # We can't test RoomStore on its own without the DirectoryStore, for
+        # management of the 'room_aliases' table
+        self.store = hs.get_datastore()
+
+        self.room = hs.parse_roomid("!abcde:test")
+        self.alias = hs.parse_roomalias("#a-room-name:test")
+        self.u_creator = hs.parse_userid("@creator:test")
+
+        yield self.store.store_room(self.room.to_string(),
+            room_creator_user_id=self.u_creator.to_string(),
+            is_public=True
+        )
+
+    @defer.inlineCallbacks
+    def test_get_room(self):
+        self.assertObjectHasAttributes(
+            {"room_id": self.room.to_string(),
+             "creator": self.u_creator.to_string(),
+             "is_public": True},
+            (yield self.store.get_room(self.room.to_string()))
+        )
+
+    @defer.inlineCallbacks
+    def test_store_room_config(self):
+        yield self.store.store_room_config(self.room.to_string(),
+            visibility=False
+        )
+
+        self.assertObjectHasAttributes(
+            {"is_public": False},
+            (yield self.store.get_room(self.room.to_string()))
+        )
+
+    @defer.inlineCallbacks
+    def test_get_rooms(self):
+        # get_rooms does an INNER JOIN on the room_aliases table :(
+
+        rooms = yield self.store.get_rooms(is_public=True)
+        # Should be empty before we add the alias
+        self.assertEquals([], rooms)
+
+        yield self.store.create_room_alias_association(
+            room_alias=self.alias,
+            room_id=self.room.to_string(),
+            servers=["test"]
+        )
+
+        rooms = yield self.store.get_rooms(is_public=True)
+
+        self.assertEquals(1, len(rooms))
+        self.assertEquals({
+            "name": None,
+            "room_id": self.room.to_string(),
+            "topic": None,
+            "aliases": [self.alias.to_string()],
+        }, rooms[0])
+
+
+class RoomEventsStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        # Room events need the full datastore, for persist_event() and
+        # get_room_state()
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory();
+
+        self.room = hs.parse_roomid("!abcde:test")
+
+        yield self.store.store_room(self.room.to_string(),
+            room_creator_user_id="@creator:text",
+            is_public=True
+        )
+
+    @defer.inlineCallbacks
+    def inject_room_event(self, **kwargs):
+        yield self.store.persist_event(
+            self.event_factory.create_event(
+                room_id=self.room.to_string(),
+                **kwargs
+            )
+        )
+
+    @defer.inlineCallbacks
+    def test_room_name(self):
+        name = u"A-Room-Name"
+
+        yield self.inject_room_event(
+            etype=RoomNameEvent.TYPE,
+            name=name,
+            content={"name": name},
+            depth=1,
+        )
+
+        state = yield self.store.get_current_state(
+            room_id=self.room.to_string()
+        )
+
+        self.assertEquals(1, len(state))
+        self.assertObjectHasAttributes(
+            {"type": "m.room.name",
+             "room_id": self.room.to_string(),
+             "name": name},
+            state[0]
+        )
+
+    @defer.inlineCallbacks
+    def test_room_name(self):
+        topic = u"A place for things"
+
+        yield self.inject_room_event(
+            etype=RoomTopicEvent.TYPE,
+            topic=topic,
+            content={"topic": topic},
+            depth=1,
+        )
+
+        state = yield self.store.get_current_state(
+            room_id=self.room.to_string()
+        )
+
+        self.assertEquals(1, len(state))
+        self.assertObjectHasAttributes(
+            {"type": "m.room.topic",
+             "room_id": self.room.to_string(),
+             "topic": topic},
+            state[0]
+        )
+
+    # Not testing the various 'level' methods for now because there's lots
+    # of them and need coalescing; see JIRA SPEC-11

tests/storage/test_roommember.py

@@ -0,0 +1,157 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.constants import Membership
+from synapse.api.events.room import RoomMemberEvent
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RoomMemberStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        # We can't test the RoomMemberStore on its own without the other event
+        # storage logic
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory()
+
+        self.u_alice = hs.parse_userid("@alice:test")
+        self.u_bob = hs.parse_userid("@bob:test")
+
+        # User elsewhere on another host
+        self.u_charlie = hs.parse_userid("@charlie:elsewhere")
+
+        self.room = hs.parse_roomid("!abc123:test")
+
+    @defer.inlineCallbacks
+    def inject_room_member(self, room, user, membership):
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            self.event_factory.create_event(
+                etype=RoomMemberEvent.TYPE,
+                user_id=user.to_string(),
+                state_key=user.to_string(),
+                room_id=room.to_string(),
+                membership=membership,
+                content={"membership": membership},
+                depth=1,
+            )
+        )
+
+    @defer.inlineCallbacks
+    def test_one_member(self):
+        yield self.inject_room_member(self.room, self.u_alice, Membership.JOIN)
+
+        self.assertEquals(
+            Membership.JOIN,
+            (yield self.store.get_room_member(
+                user_id=self.u_alice.to_string(),
+                room_id=self.room.to_string(),
+            )).membership
+        )
+        self.assertEquals(
+            [self.u_alice.to_string()],
+            [m.user_id for m in (
+                yield self.store.get_room_members(self.room.to_string())
+            )]
+        )
+        self.assertEquals(
+            [self.room.to_string()],
+            [m.room_id for m in (
+                yield self.store.get_rooms_for_user_where_membership_is(
+                    self.u_alice.to_string(), [Membership.JOIN]
+                ))
+            ]
+        )
+        self.assertFalse(
+            (yield self.store.user_rooms_intersect(
+                [self.u_alice.to_string(), self.u_bob.to_string()]
+            ))
+        )
+
+    @defer.inlineCallbacks
+    def test_two_members(self):
+        yield self.inject_room_member(self.room, self.u_alice, Membership.JOIN)
+        yield self.inject_room_member(self.room, self.u_bob, Membership.JOIN)
+
+        self.assertEquals(
+            {self.u_alice.to_string(), self.u_bob.to_string()},
+            {m.user_id for m in (
+                yield self.store.get_room_members(self.room.to_string())
+            )}
+        )
+        self.assertTrue(
+            (yield self.store.user_rooms_intersect(
+                [self.u_alice.to_string(), self.u_bob.to_string()]
+            ))
+        )
+
+    @defer.inlineCallbacks
+    def test_room_hosts(self):
+        yield self.inject_room_member(self.room, self.u_alice, Membership.JOIN)
+
+        self.assertEquals(
+            ["test"],
+            (yield self.store.get_joined_hosts_for_room(self.room.to_string()))
+        )
+
+        # Should still have just one host after second join from it
+        yield self.inject_room_member(self.room, self.u_bob, Membership.JOIN)
+
+        self.assertEquals(
+            ["test"],
+            (yield self.store.get_joined_hosts_for_room(self.room.to_string()))
+        )
+
+        # Should now have two hosts after join from other host
+        yield self.inject_room_member(self.room, self.u_charlie, Membership.JOIN)
+
+        self.assertEquals(
+            {"test", "elsewhere"},
+            set((yield
+                self.store.get_joined_hosts_for_room(self.room.to_string())
+            ))
+        )
+
+        # Should still have both hosts
+        yield self.inject_room_member(self.room, self.u_alice, Membership.LEAVE)
+
+        self.assertEquals(
+            {"test", "elsewhere"},
+            set((yield
+                self.store.get_joined_hosts_for_room(self.room.to_string())
+            ))
+        )
+
+        # Should have only one host after other leaves
+        yield self.inject_room_member(self.room, self.u_charlie, Membership.LEAVE)
+
+        self.assertEquals(
+            ["test"],
+            (yield self.store.get_joined_hosts_for_room(self.room.to_string()))
+        )

tests/storage/test_stream.py

@@ -0,0 +1,226 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.constants import Membership
+from synapse.api.events.room import RoomMemberEvent, MessageEvent
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class StreamStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer(
+            "test",
+            db_pool=db_pool,
+        )
+
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory()
+
+        self.u_alice = hs.parse_userid("@alice:test")
+        self.u_bob = hs.parse_userid("@bob:test")
+
+        self.room1 = hs.parse_roomid("!abc123:test")
+        self.room2 = hs.parse_roomid("!xyx987:test")
+
+        self.depth = 1
+
+    @defer.inlineCallbacks
+    def inject_room_member(self, room, user, membership, prev_state=None):
+        self.depth += 1
+
+        event = self.event_factory.create_event(
+            etype=RoomMemberEvent.TYPE,
+            user_id=user.to_string(),
+            state_key=user.to_string(),
+            room_id=room.to_string(),
+            membership=membership,
+            content={"membership": membership},
+            depth=self.depth,
+        )
+
+        if prev_state:
+            event.prev_state = prev_state
+
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            event
+        )
+
+        defer.returnValue(event)
+
+    @defer.inlineCallbacks
+    def inject_message(self, room, user, body):
+        self.depth += 1
+
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            self.event_factory.create_event(
+                etype=MessageEvent.TYPE,
+                user_id=user.to_string(),
+                room_id=room.to_string(),
+                content={"body": body, "msgtype": u"message"},
+                depth=self.depth,
+            )
+        )
+
+    @defer.inlineCallbacks
+    def test_event_stream_get_other(self):
+        # Both bob and alice joins the room
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        # Initial stream key:
+        start = yield self.store.get_room_events_max_id()
+
+        yield self.inject_message(self.room1, self.u_alice, u"test")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_bob.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": MessageEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {"body": "test", "msgtype": "message"},
+            },
+            event,
+        )
+
+    @defer.inlineCallbacks
+    def test_event_stream_get_own(self):
+        # Both bob and alice joins the room
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        # Initial stream key:
+        start = yield self.store.get_room_events_max_id()
+
+        yield self.inject_message(self.room1, self.u_alice, u"test")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_alice.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": MessageEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {"body": "test", "msgtype": "message"},
+            },
+            event,
+        )
+
+    @defer.inlineCallbacks
+    def test_event_stream_join_leave(self):
+        # Both bob and alice joins the room
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        # Then bob leaves again.
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.LEAVE
+        )
+
+        # Initial stream key:
+        start = yield self.store.get_room_events_max_id()
+
+        yield self.inject_message(self.room1, self.u_alice, u"test")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_bob.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        # We should not get the message, as it happened *after* bob left.
+        self.assertEqual(0, len(results))
+
+    @defer.inlineCallbacks
+    def test_event_stream_prev_content(self):
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        event1 = yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+
+        start = yield self.store.get_room_events_max_id()
+
+        event2 = yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN,
+            prev_state=event1.event_id,
+        )
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_bob.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        # We should not get the message, as it happened *after* bob left.
+        self.assertEqual(1, len(results))
+
+        event = results[0]
+
+        self.assertTrue(hasattr(event, "prev_content"), msg="No prev_content key")

tests/test_distributor.py

@@ -13,8 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from tests import unittest
 from twisted.internet import defer
-from twisted.trial import unittest
 
 from mock import Mock, patch
 

tests/test_state.py

@@ -13,8 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from tests import unittest
 from twisted.internet import defer
-from twisted.trial import unittest
 from twisted.python.log import PythonLoggingObserver
 
 from synapse.state import StateHandler
@@ -26,7 +26,6 @@ from collections import namedtuple
 
 from mock import Mock
 
-import logging
 import mock
 
 

tests/test_types.py

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import unittest
+from tests import unittest
 
 from synapse.server import BaseHomeServer
 from synapse.types import UserID, RoomAlias

tests/unittest.py

@@ -0,0 +1,90 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from twisted.trial import unittest
+
+import logging
+
+
+# logging doesn't have a "don't log anything at all EVARRRR setting,
+# but since the highest value is 50, 1000000 should do ;)
+NEVER = 1000000
+
+logging.getLogger().addHandler(logging.StreamHandler())
+logging.getLogger().setLevel(NEVER)
+
+
+def around(target):
+    """A CLOS-style 'around' modifier, which wraps the original method of the
+    given instance with another piece of code.
+
+    @around(self)
+    def method_name(orig, *args, **kwargs):
+        return orig(*args, **kwargs)
+    """
+    def _around(code):
+        name = code.__name__
+        orig = getattr(target, name)
+        def new(*args, **kwargs):
+            return code(orig, *args, **kwargs)
+        setattr(target, name, new)
+    return _around
+
+
+class TestCase(unittest.TestCase):
+    """A subclass of twisted.trial's TestCase which looks for 'loglevel'
+    attributes on both itself and its individual test methods, to override the
+    root logger's logging level while that test (case|method) runs."""
+
+    def __init__(self, methodName, *args, **kwargs):
+        super(TestCase, self).__init__(methodName, *args, **kwargs)
+
+        method = getattr(self, methodName)
+
+        level = getattr(method, "loglevel",
+                    getattr(self, "loglevel",
+                        NEVER))
+
+        @around(self)
+        def setUp(orig):
+            old_level = logging.getLogger().level
+
+            if old_level != level:
+                @around(self)
+                def tearDown(orig):
+                    ret = orig()
+                    logging.getLogger().setLevel(old_level)
+                    return ret
+
+            logging.getLogger().setLevel(level)
+            return orig()
+
+    def assertObjectHasAttributes(self, attrs, obj):
+        """Asserts that the given object has each of the attributes given, and
+        that the value of each matches according to assertEquals."""
+        for (key, value) in attrs.items():
+            if not hasattr(obj, key):
+                raise AssertionError("Expected obj to have a '.%s'" % key)
+            try:
+                self.assertEquals(attrs[key], getattr(obj, key))
+            except AssertionError as e:
+                raise (type(e))(e.message + " for '.%s'" % key)
+
+
+def DEBUG(target):
+    """A decorator to set the .loglevel attribute to logging.DEBUG.
+    Can apply to either a TestCase or an individual test method."""
+    target.loglevel = logging.DEBUG
+    return target

tests/util/test_lock.py

@@ -15,7 +15,7 @@
 
 
 from twisted.internet import defer
-from twisted.trial import unittest
+from tests import unittest
 
 from synapse.util.lockutils import LockManager
 
@@ -105,4 +105,4 @@ class LockManagerTestCase(unittest.TestCase):
             pass
 
         with (yield self.lock_manager.lock(key)):
-            pass
+            pass

tests/utils.py

@@ -16,12 +16,14 @@
 from synapse.http.server import HttpServer
 from synapse.api.errors import cs_error, CodeMessageException, StoreError
 from synapse.api.constants import Membership
+from synapse.storage import prepare_database
 
 from synapse.api.events.room import (
     RoomMemberEvent, MessageEvent
 )
 
 from twisted.internet import defer, reactor
+from twisted.enterprise.adbapi import ConnectionPool
 
 from collections import namedtuple
 from mock import patch, Mock
@@ -120,6 +122,18 @@ class MockClock(object):
         self.now += secs
 
 
+class SQLiteMemoryDbPool(ConnectionPool, object):
+    def __init__(self):
+        super(SQLiteMemoryDbPool, self).__init__(
+            "sqlite3", ":memory:",
+            cp_min=1,
+            cp_max=1,
+        )
+
+    def prepare(self):
+        return self.runWithConnection(prepare_database)
+
+
 class MemoryDataStore(object):
 
     Room = namedtuple(

webclient/README

@@ -1,8 +1,8 @@
 Basic Usage
 -----------
 
-The web client should automatically run when running the home server. Alternatively, you can run
-it stand-alone:
+The web client should automatically run when running the home server. 
+Alternatively, you can run it stand-alone:
 
     $ python -m SimpleHTTPServer
 

webclient/app-controller.js

@@ -26,6 +26,12 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
          
     // Check current URL to avoid to display the logout button on the login page
     $scope.location = $location.path();
+
+    // disable nganimate for the local and remote video elements because ngAnimate appears
+    // to be buggy and leaves animation classes on the video elements causing them to show
+    // when they should not (their animations are pure CSS3)
+    $animate.enabled(false, angular.element('#localVideo'));
+    $animate.enabled(false, angular.element('#remoteVideo'));
     
     // Update the location state when the ng location changed
     $rootScope.$on('$routeChangeSuccess', function (event, current, previous) {
@@ -93,7 +99,13 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
     };
 
     $rootScope.$watch('currentCall', function(newVal, oldVal) {
-        if (!$rootScope.currentCall) return;
+        if (!$rootScope.currentCall) {
+            // This causes the still frame to be flushed out of the video elements,
+            // avoiding a flash of the last frame of the previous call when starting the next
+            angular.element('#localVideo')[0].load();
+            angular.element('#remoteVideo')[0].load();
+            return;
+        }
 
         var roomMembers = angular.copy($rootScope.events.rooms[$rootScope.currentCall.room_id].members);
         delete roomMembers[matrixService.config().user_id];
@@ -126,14 +138,33 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
             angular.element('#ringAudio')[0].pause();
             angular.element('#ringbackAudio')[0].pause();
             angular.element('#callendAudio')[0].play();
+            $scope.videoMode = undefined;
         } else if (newVal == 'ended' && oldVal == 'invite_sent' && $rootScope.currentCall.hangupParty == 'remote') {
             angular.element('#ringAudio')[0].pause();
             angular.element('#ringbackAudio')[0].pause();
             angular.element('#busyAudio')[0].play();
+        } else if (newVal == 'ended' && oldVal == 'invite_sent' && $rootScope.currentCall.hangupParty == 'local' && $rootScope.currentCall.hangupReason == 'invite_timeout') {
+            angular.element('#ringAudio')[0].pause();
+            angular.element('#ringbackAudio')[0].pause();
+            angular.element('#busyAudio')[0].play();
         } else if (oldVal == 'invite_sent') {
             angular.element('#ringbackAudio')[0].pause();
         } else if (oldVal == 'ringing') {
             angular.element('#ringAudio')[0].pause();
+        } else if (newVal == 'connected') {
+            $timeout(function() {
+                if ($scope.currentCall.type == 'video') $scope.videoMode = 'large';
+            }, 500);
+        }
+
+        if ($rootScope.currentCall && $rootScope.currentCall.type == 'video' && $rootScope.currentCall.state != 'connected') {
+            $scope.videoMode = 'mini';
+        }
+    });
+    $rootScope.$watch('currentCall.type', function(newVal, oldVal) {
+        // need to listen for this too as the type of the call won't be know when it's created
+        if ($rootScope.currentCall && $rootScope.currentCall.type == 'video' && $rootScope.currentCall.state != 'connected') {
+            $scope.videoMode = 'mini';
         }
     });
 
@@ -146,6 +177,8 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
         }
         call.onError = $scope.onCallError;
         call.onHangup = $scope.onCallHangup;
+        call.localVideoElement = angular.element('#localVideo')[0];
+        call.remoteVideoElement = angular.element('#remoteVideo')[0];
         $rootScope.currentCall = call;
     });
 
@@ -166,7 +199,7 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
     
     $rootScope.onCallError = function(errStr) {
         $scope.feedback = errStr;
-    }
+    };
 
     $rootScope.onCallHangup = function(call) {
         if (call == $rootScope.currentCall) {
@@ -174,5 +207,5 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
                 if (call == $rootScope.currentCall) $rootScope.currentCall = undefined;
             }, 4070);
         }
-    }
+    };
 }]);

webclient/app.css

@@ -20,7 +20,12 @@ a:visited { color: #666; }
 a:hover   { color: #000; }
 a:active  { color: #000; }
 
-#page {
+textarea, input {
+   font-family: inherit;
+   font-size: inherit;
+}
+
+.page {
     min-height: 100%;
     margin-bottom: -32px; /* to make room for the footer */
 }
@@ -34,9 +39,15 @@ a:active  { color: #000; }
     padding-right: 20px;
 }
 
+#unsupportedBrowser {
+    padding-top: 240px;
+    text-align: center;
+}
+
 #header
 {
     position: absolute;
+    z-index: 2;
     top: 0px;
     width: 100%;
     background-color: #333;
@@ -89,6 +100,80 @@ a:active  { color: #000; }
     font-size: 80%;
 }
 
+#videoBackground {
+    position: absolute;
+    height: 100%;
+    width: 100%;
+    top: 0px;
+    left: 0px;
+    z-index: 1;
+    background-color: rgba(0,0,0,0.0);
+    pointer-events: none;
+    transition: background-color linear 500ms;
+}
+
+#videoBackground.large {
+    background-color: rgba(0,0,0,0.85);
+    pointer-events: auto;
+}
+
+#videoContainer {
+    position: relative;
+    top: 32px;
+    max-width: 1280px;
+    margin: auto;
+}
+
+#videoContainerPadding {
+    width: 1280px;
+}
+
+#localVideo {
+    position: absolute;
+    width: 128px;
+    height: 72px;
+    z-index: 1;
+    transition: left linear 500ms, top linear 500ms, width linear 500ms, height linear 500ms;
+}
+
+#localVideo.mini {
+    top: 0px;
+    left: 130px;
+}
+
+#localVideo.large {
+    top: 70px;
+    left: 20px;
+}
+
+#localVideo.ended {
+    -webkit-filter: grayscale(1);
+    filter: grayscale(1);
+}
+
+#remoteVideo {
+    position: relative;
+    height: auto;
+    transition: left linear 500ms, top linear 500ms, width linear 500ms, height linear 500ms;
+}
+
+#remoteVideo.mini {
+    left: 260px;
+    top: 0px;
+    width: 128px;
+}
+
+#remoteVideo.large {
+    left: 0px;
+    top: 50px;
+    width: 100%;
+}
+
+#remoteVideo.ended {
+    -webkit-filter: grayscale(1);
+    filter: grayscale(1);
+}
+
 #headerContent {
     color: #ccc;
     max-width: 1280px;
@@ -96,6 +181,7 @@ a:active  { color: #000; }
     text-align: right;
     height: 32px;
     line-height: 32px;
+    position: relative;
 }
 
 #headerContent a:link,
@@ -528,8 +614,8 @@ a:active  { color: #000; }
 }
 
 .bubble .message {
-    /* Break lines when encountering CR+LF */
-    white-space: pre;
+    /* Wrap words and break lines on CR+LF */
+    white-space: pre-wrap;
 }
 .bubble .messagePending {
     opacity: 0.3
@@ -538,6 +624,10 @@ a:active  { color: #000; }
     color: #F00;
 }
 
+.messageBing {
+    color: #00F;
+}
+
 #room-fullscreen-image {
     position: absolute;
     top: 0px;
@@ -599,7 +689,11 @@ a:active  { color: #000; }
     width: auto;
 }
 
-.recentsRoomSummaryTS {
+.recentsPublicRoom {
+    font-weight: bold;
+}
+
+.recentsRoomSummaryUsersCount, .recentsRoomSummaryTS {
     color: #888;
     font-size: 12px;
     width: 7em;
@@ -612,6 +706,11 @@ a:active  { color: #000; }
     padding-bottom: 5px;
 }
 
+/* Do not show users count in the recents fragment displayed on the room page */
+#roomPage .recentsRoomSummaryUsersCount {
+    width: 0em;
+}
+
 /*** Recents in the room page ***/
 
 #roomRecentsTableWrapper {

webclient/app.js

@@ -80,7 +80,24 @@ matrixWebClient.config(['$routeProvider', '$provide', '$httpProvider',
         $httpProvider.interceptors.push('AccessTokenInterceptor');
     }]);
 
-matrixWebClient.run(['$location', 'matrixService', function($location, matrixService) {
+matrixWebClient.run(['$location', '$rootScope', 'matrixService', function($location, $rootScope, matrixService) {
+
+    // Check browser support
+    // Support IE from 9.0. AngularJS needs some tricks to run on IE8 and below
+    var version = parseFloat($.browser.version);
+    if ($.browser.msie && version < 9.0) {
+        $rootScope.unsupportedBrowser = {
+            browser: navigator.userAgent,
+            reason: "Internet Explorer is supported from version 9"
+        };
+    }
+    // The app requires localStorage
+    if(typeof(Storage) === "undefined") {
+        $rootScope.unsupportedBrowser = {
+            browser: navigator.userAgent,
+            reason: "It does not support HTML local storage"
+        };
+    }
 
     // If user auth details are not in cache, go to the login page
     if (!matrixService.isUserLoggedIn() &&

webclient/components/fileInput/file-input-directive.js

@@ -31,13 +31,23 @@ angular.module('mFileInput', [])
         },
 
         link: function(scope, element, attrs, ctrl) {
-            element.bind("click", function() {
-                element.find("input")[0].click();
-                element.find("input").bind("change", function(e) {
-                    scope.selectedFile = this.files[0];
-                    scope.$apply();
+            
+            // Check if HTML5 file selection is supported
+            if (window.FileList) {
+                element.bind("click", function() {
+                    element.find("input")[0].click();
+                    element.find("input").bind("change", function(e) {
+                        scope.selectedFile = this.files[0];
+                        scope.$apply();
+                    });
                 });
-            });
+            }
+            else {
+                setTimeout(function() {
+                    element.attr("disabled", true);
+                    element.attr("title", "The app uses the HTML5 File API to send files. Your browser does not support it.");
+                }, 1);
+            }
 
             // Change the mouse icon on mouseover on this element
             element.css("cursor", "pointer");

webclient/components/matrix/event-handler-service.js

@@ -27,7 +27,8 @@ Typically, this service will store events or broadcast them to any listeners
 if typically all the $on method would do is update its own $scope.
 */
 angular.module('eventHandlerService', [])
-.factory('eventHandlerService', ['matrixService', '$rootScope', '$q', function(matrixService, $rootScope, $q) {
+.factory('eventHandlerService', ['matrixService', '$rootScope', '$q', '$timeout', 'mPresence', 
+function(matrixService, $rootScope, $q, $timeout, mPresence) {
     var ROOM_CREATE_EVENT = "ROOM_CREATE_EVENT";
     var MSG_EVENT = "MSG_EVENT";
     var MEMBER_EVENT = "MEMBER_EVENT";
@@ -38,6 +39,51 @@ angular.module('eventHandlerService', [])
     var TOPIC_EVENT = "TOPIC_EVENT";
     var RESET_EVENT = "RESET_EVENT";    // eventHandlerService has been resetted
 
+    // used for dedupping events - could be expanded in future...
+    // FIXME: means that we leak memory over time (along with lots of the rest
+    // of the app, given we never try to reap memory yet)
+    var eventMap = {};
+
+    $rootScope.presence = {};
+    
+    // TODO: This is attached to the rootScope so .html can just go containsBingWord
+    // for determining classes so it is easy to highlight bing messages. It seems a
+    // bit strange to put the impl in this service though, but I can't think of a better
+    // file to put it in.
+    $rootScope.containsBingWord = function(content) {
+        if (!content || $.type(content) != "string") {
+            return false;
+        }
+        var bingWords = matrixService.config().bingWords;
+        var shouldBing = false;
+        
+        // case-insensitive name check for user_id OR display_name if they exist
+        var myUserId = matrixService.config().user_id;
+        if (myUserId) {
+            myUserId = myUserId.toLocaleLowerCase();
+        }
+        var myDisplayName = matrixService.config().display_name;
+        if (myDisplayName) {
+            myDisplayName = myDisplayName.toLocaleLowerCase();
+        }
+        if ( (myDisplayName && content.toLocaleLowerCase().indexOf(myDisplayName) != -1) ||
+             (myUserId && content.toLocaleLowerCase().indexOf(myUserId) != -1) ) {
+            shouldBing = true;
+        }
+        
+        // bing word list check
+        if (bingWords && !shouldBing) {
+            for (var i=0; i<bingWords.length; i++) {
+                var re = RegExp(bingWords[i]);
+                if (content.search(re) != -1) {
+                    shouldBing = true;
+                    break;
+                }
+            }
+        }
+        return shouldBing;
+    };
+
     var initialSyncDeferred;
 
     var reset = function() {
@@ -46,28 +92,36 @@ angular.module('eventHandlerService', [])
         $rootScope.events = {
             rooms: {} // will contain roomId: { messages:[], members:{userid1: event} }
         };
-    }
+
+        $rootScope.presence = {};
+
+        eventMap = {};
+    };
     reset();
 
-    // used for dedupping events - could be expanded in future...
-    // FIXME: means that we leak memory over time (along with lots of the rest
-    // of the app, given we never try to reap memory yet)
-    var eventMap = {};
-
-    $rootScope.presence = {};
-    
-    var initRoom = function(room_id) {
+    var initRoom = function(room_id, room) {
         if (!(room_id in $rootScope.events.rooms)) {
             console.log("Creating new handler entry for " + room_id);
-            $rootScope.events.rooms[room_id] = {};
-            $rootScope.events.rooms[room_id].messages = [];
-            $rootScope.events.rooms[room_id].members = {};
-
-            // Pagination information
-            $rootScope.events.rooms[room_id].pagination = {
-                earliest_token: "END"   // how far back we've paginated
+            $rootScope.events.rooms[room_id] = {
+                room_id: room_id,
+                messages: [],
+                members: {},
+                // Pagination information
+                pagination: {
+                    earliest_token: "END"   // how far back we've paginated
+                }
             };
         }
+
+        if (room) {
+            // Report all other metadata of the room object (membership, inviter, visibility, ...)
+            for (var field in room) {
+                if (-1 === ["room_id", "messages", "state"].indexOf(field)) {
+                    $rootScope.events.rooms[room_id][field] = room[field];
+                }
+            }
+            $rootScope.events.rooms[room_id].membership = room.membership;
+        }
     };
 
     var resetRoomMessages = function(room_id) {
@@ -132,6 +186,54 @@ angular.module('eventHandlerService', [])
             else {
                 $rootScope.events.rooms[event.room_id].messages.push(event);
             }
+            
+            if (window.Notification && event.user_id != matrixService.config().user_id) {
+                var shouldBing = $rootScope.containsBingWord(event.content.body);
+
+                // Ideally we would notify only when the window is hidden (i.e. document.hidden = true).
+                //
+                // However, Chrome on Linux and OSX currently returns document.hidden = false unless the window is
+                // explicitly showing a different tab.  So we need another metric to determine hiddenness - we
+                // simply use idle time.  If the user has been idle enough that their presence goes to idle, then
+                // we also display notifs when things happen.
+                //
+                // This is far far better than notifying whenever anything happens anyway, otherwise you get spammed
+                // to death with notifications when the window is in the foreground, which is horrible UX (especially
+                // if you have not defined any bingers and so get notified for everything).
+                var isIdle = (document.hidden || matrixService.presence.unavailable === mPresence.getState());
+                
+                // We need a way to let people get notifications for everything, if they so desire.  The way to do this
+                // is to specify zero bingwords.
+                var bingWords = matrixService.config().bingWords;
+                if (bingWords === undefined || bingWords.length === 0) {
+                    shouldBing = true;
+                }
+                
+                if (shouldBing && isIdle) {
+                    console.log("Displaying notification for "+JSON.stringify(event));
+                    var member = $rootScope.events.rooms[event.room_id].members[event.user_id];
+                    var displayname = undefined;
+                    if (member) {
+                        displayname = member.displayname;
+                    }
+
+                    var message = event.content.body;
+                    if (event.content.msgtype === "m.emote") {
+                        message = "* " + displayname + " " + message;
+                    }
+
+                    var notification = new window.Notification(
+                        (displayname || event.user_id) +
+                        " (" + (matrixService.getRoomIdToAliasMapping(event.room_id) || event.room_id) + ")", // FIXME: don't leak room_ids here
+                    {
+                        "body": message,
+                        "icon": member ? member.avatar_url : undefined
+                    });
+                    $timeout(function() {
+                        notification.close();
+                    }, 5 * 1000);
+                }
+            }
         }
         else {
             $rootScope.events.rooms[event.room_id].messages.unshift(event);
@@ -146,13 +248,6 @@ angular.module('eventHandlerService', [])
     };
     
     var handleRoomMember = function(event, isLiveEvent, isStateEvent) {
-        // if the server is stupidly re-relaying a no-op join, discard it.
-        if (event.prev_content && 
-            event.content.membership === "join" &&
-            event.content.membership === event.prev_content.membership)
-        {
-            return;
-        }
         
         // add membership changes as if they were a room message if something interesting changed
         // Exception: Do not do this if the event is a room state event because such events already come
@@ -204,7 +299,7 @@ angular.module('eventHandlerService', [])
 
     var handleCallEvent = function(event, isLiveEvent) {
         $rootScope.$broadcast(CALL_EVENT, event, isLiveEvent);
-        if (event.type == 'm.call.invite') {
+        if (event.type === 'm.call.invite') {
             $rootScope.events.rooms[event.room_id].messages.push(event);
         }
     };
@@ -231,7 +326,7 @@ angular.module('eventHandlerService', [])
             }
         }
         return index;
-    }
+    };
     
     return {
         ROOM_CREATE_EVENT: ROOM_CREATE_EVENT,
@@ -248,12 +343,18 @@ angular.module('eventHandlerService', [])
             reset();
             $rootScope.$broadcast(RESET_EVENT);
         },
+        
+        initRoom: function(room) {
+            initRoom(room.room_id, room);
+        },
     
         handleEvent: function(event, isLiveEvent, isStateEvent) {
 
             // FIXME: /initialSync on a particular room is not yet available
             // So initRoom on a new room is not called. Make sure the room data is initialised here
-            initRoom(event.room_id);
+            if (event.room_id) {
+                initRoom(event.room_id);
+            }
 
             // Avoid duplicated events
             // Needed for rooms where initialSync has not been done. 
@@ -322,13 +423,29 @@ angular.module('eventHandlerService', [])
         },
 
         // Handle messages from /initialSync or /messages
-        handleRoomMessages: function(room_id, messages, isLiveEvents) {
+        handleRoomMessages: function(room_id, messages, isLiveEvents, dir) {
             initRoom(room_id);
-            this.handleEvents(messages.chunk, isLiveEvents);
 
-            // Store how far back we've paginated
-            // This assumes the paginations requests are contiguous and in reverse chronological order
-            $rootScope.events.rooms[room_id].pagination.earliest_token = messages.end;
+            var events = messages.chunk;
+
+            // Handles messages according to their time order
+            if (dir && 'b' === dir) {
+                // paginateBackMessages requests messages to be in reverse chronological order
+                for (var i=0; i<events.length; i++) {
+                    this.handleEvent(events[i], isLiveEvents, isLiveEvents);
+                }
+                
+                // Store how far back we've paginated
+                $rootScope.events.rooms[room_id].pagination.earliest_token = messages.end;
+            }
+            else {
+                // InitialSync returns messages in chronological order
+                for (var i=events.length - 1; i>=0; i--) {
+                    this.handleEvent(events[i], isLiveEvents, isLiveEvents);
+                }
+                // Store where to start pagination
+                $rootScope.events.rooms[room_id].pagination.earliest_token = messages.start;
+            }
         },
 
         handleInitialSyncDone: function(initialSyncData) {
@@ -343,6 +460,82 @@ angular.module('eventHandlerService', [])
 
         resetRoomMessages: function(room_id) {
             resetRoomMessages(room_id);
+        },
+        
+        /**
+         * Return the last message event of a room
+         * @param {String} room_id the room id
+         * @param {Boolean} filterFake true to not take into account fake messages
+         * @returns {undefined | Event} the last message event if available
+         */
+        getLastMessage: function(room_id, filterEcho) {
+            var lastMessage;
+
+            var room = $rootScope.events.rooms[room_id];
+            if (room) {
+                for (var i = room.messages.length - 1; i >= 0; i--) {
+                    var message = room.messages[i];
+
+                    if (!filterEcho || undefined === message.echo_msg_state) {
+                        lastMessage = message;
+                        break;
+                    }
+                }
+            }
+
+            return lastMessage;
+        },
+        
+        /**
+         * Compute the room users number, ie the number of members who has joined the room.
+         * @param {String} room_id the room id
+         * @returns {undefined | Number} the room users number if available
+         */
+        getUsersCountInRoom: function(room_id) {
+            var memberCount;
+
+            var room = $rootScope.events.rooms[room_id];
+            if (room) {
+                memberCount = 0;
+
+                for (var i in room.members) {
+                    var member = room.members[i];
+
+                    if ("join" === member.membership) {
+                        memberCount = memberCount + 1;
+                    }
+                }
+            }
+
+            return memberCount;
+        },
+        
+        /**
+         * Get the member object of a room member
+         * @param {String} room_id the room id
+         * @param {String} user_id the id of the user
+         * @returns {undefined | Object} the member object of this user in this room if he is part of the room
+         */
+        getMember: function(room_id, user_id) {
+            var member;
+            
+            var room = $rootScope.events.rooms[room_id];
+            if (room) {
+                member = room.members[user_id];
+            }
+            return member;
+        },
+        
+        setRoomVisibility: function(room_id, visible) {
+            if (!visible) {
+                return;
+            }
+            initRoom(room_id);
+            
+            var room = $rootScope.events.rooms[room_id];
+            if (room) {
+                room.visibility = visible;
+            }
         }
     };
 }]);

webclient/components/matrix/event-stream-service.js

@@ -104,12 +104,16 @@ angular.module('eventStreamService', [])
         settings.isActive = true;
         var deferred = $q.defer();
 
-        // Initial sync: get all information and the last message of all rooms of the user
-        matrixService.initialSync(1, false).then(
+        // Initial sync: get all information and the last 30 messages of all rooms of the user
+        // 30 messages should be enough to display a full page of messages in a room
+        // without requiring to make an additional request
+        matrixService.initialSync(30, false).then(
             function(response) {
                 var rooms = response.data.rooms;
                 for (var i = 0; i < rooms.length; ++i) {
                     var room = rooms[i];
+                    
+                    eventHandlerService.initRoom(room);
 
                     if ("messages" in room) {
                         eventHandlerService.handleRoomMessages(room.room_id, room.messages, false);

webclient/components/matrix/matrix-call.js

@@ -40,8 +40,15 @@ window.RTCPeerConnection = window.RTCPeerConnection || window.webkitRTCPeerConne
 window.RTCSessionDescription = window.RTCSessionDescription || window.webkitRTCSessionDescription || window.mozRTCSessionDescription;
 window.RTCIceCandidate = window.RTCIceCandidate || window.webkitRTCIceCandidate || window.mozRTCIceCandidate;
 
+// Returns true if the browser supports all required features to make WebRTC call
+var isWebRTCSupported = function () {
+    return !!(navigator.getUserMedia || window.RTCPeerConnection || window.RTCSessionDescription || window.RTCIceCandidate);
+};
+
 angular.module('MatrixCall', [])
 .factory('MatrixCall', ['matrixService', 'matrixPhoneService', '$rootScope', '$timeout', function MatrixCallFactory(matrixService, matrixPhoneService, $rootScope, $timeout) {
+    $rootScope.isWebRTCSupported = isWebRTCSupported();
+
     var MatrixCall = function(room_id) {
         this.room_id = room_id;
         this.call_id = "c" + new Date().getTime();
@@ -51,13 +58,21 @@ angular.module('MatrixCall', [])
         // a queue for candidates waiting to go out. We try to amalgamate candidates into a single candidate message where possible
         this.candidateSendQueue = [];
         this.candidateSendTries = 0;
+
+        var self = this;
+        $rootScope.$watch(this.remoteVideoElement, function (oldValue, newValue) {
+            self.tryPlayRemoteStream();
+        });
+
     }
 
+    MatrixCall.CALL_TIMEOUT = 60000;
+
     MatrixCall.prototype.createPeerConnection = function() {
         var stunServer = 'stun:stun.l.google.com:19302';
         var pc;
         if (window.mozRTCPeerConnection) {
-            pc = window.mozRTCPeerConnection({'url': stunServer});
+            pc = new window.mozRTCPeerConnection({'url': stunServer});
         } else {
             pc = new window.RTCPeerConnection({"iceServers":[{"urls":"stun:stun.l.google.com:19302"}]});
         }
@@ -69,28 +84,98 @@ angular.module('MatrixCall', [])
         return pc;
     }
 
-    MatrixCall.prototype.placeCall = function(config) {
-        var self = this;
-        matrixPhoneService.callPlaced(this);
-        navigator.getUserMedia({audio: config.audio, video: config.video}, function(s) { self.gotUserMediaForInvite(s); }, function(e) { self.getUserMediaFailed(e); });
-        this.state = 'wait_local_media';
-        this.direction = 'outbound';
-        this.config = config;
+    MatrixCall.prototype.getUserMediaVideoContraints = function(callType) {
+        switch (callType) {
+            case 'voice':
+                return ({audio: true, video: false});
+            case 'video':
+                return ({audio: true, video: {
+                    mandatory: {
+                        minWidth: 640,
+                        maxWidth: 640,
+                        minHeight: 360,
+                        maxHeight: 360,
+                    }
+                }});
+        }
     };
 
-    MatrixCall.prototype.initWithInvite = function(msg) {
-        this.msg = msg;
+    MatrixCall.prototype.placeVoiceCall = function() {
+        this.placeCallWithConstraints(this.getUserMediaVideoContraints('voice'));
+        this.type = 'voice';
+    };
+
+    MatrixCall.prototype.placeVideoCall = function(config) {
+        this.placeCallWithConstraints(this.getUserMediaVideoContraints('video'));
+        this.type = 'video';
+    };
+
+    MatrixCall.prototype.placeCallWithConstraints = function(constraints) {
+        var self = this;
+        matrixPhoneService.callPlaced(this);
+        navigator.getUserMedia(constraints, function(s) { self.gotUserMediaForInvite(s); }, function(e) { self.getUserMediaFailed(e); });
+        this.state = 'wait_local_media';
+        this.direction = 'outbound';
+        this.config = constraints;
+    };
+
+    MatrixCall.prototype.initWithInvite = function(event) {
+        this.msg = event.content;
         this.peerConn = this.createPeerConnection();
         this.peerConn.setRemoteDescription(new RTCSessionDescription(this.msg.offer), this.onSetRemoteDescriptionSuccess, this.onSetRemoteDescriptionError);
         this.state = 'ringing';
         this.direction = 'inbound';
+
+        if (window.mozRTCPeerConnection) {
+            // firefox's RTCPeerConnection doesn't add streams until it starts getting media on them
+            // so we need to figure out whether a video channel has been offered by ourselves.
+            if (this.msg.offer.sdp.indexOf('m=video') > -1) {
+                this.type = 'video';
+            } else {
+                this.type = 'voice';
+            }
+        }
+
+        var self = this;
+        $timeout(function() {
+            if (self.state == 'ringing') {
+                self.state = 'ended';
+                self.hangupParty = 'remote'; // effectively
+                self.stopAllMedia();
+                if (self.peerConn.signalingState != 'closed') self.peerConn.close();
+                if (self.onHangup) self.onHangup(self);
+            }
+        }, this.msg.lifetime - event.age);
+    };
+
+    // perverse as it may seem, sometimes we want to instantiate a call with a hangup message
+    // (because when getting the state of the room on load, events come in reverse order and
+    // we want to remember that a call has been hung up)
+    MatrixCall.prototype.initWithHangup = function(event) {
+        this.msg = event.content;
+        this.state = 'ended';
     };
 
     MatrixCall.prototype.answer = function() {
         console.log("Answering call "+this.call_id);
+
         var self = this;
+
+        var roomMembers = $rootScope.events.rooms[this.room_id].members;
+        if (roomMembers[matrixService.config().user_id].membership != 'join') {
+            console.log("We need to join the room before we can accept this call");
+            matrixService.join(this.room_id).then(function() {
+                self.answer();
+            }, function() {
+                console.log("Failed to join room: can't answer call!");
+                self.onError("Unable to join room to answer call!");
+                self.hangup();
+            });
+            return;
+        }
+
         if (!this.localAVStream && !this.waitForLocalAVStream) {
-            navigator.getUserMedia({audio: true, video: false}, function(s) { self.gotUserMediaForAnswer(s); }, function(e) { self.getUserMediaFailed(e); });
+            navigator.getUserMedia(this.getUserMediaVideoContraints(this.type), function(s) { self.gotUserMediaForAnswer(s); }, function(e) { self.getUserMediaFailed(e); });
             this.state = 'wait_local_media';
         } else if (this.localAVStream) {
             this.gotUserMediaForAnswer(this.localAVStream);
@@ -112,17 +197,24 @@ angular.module('MatrixCall', [])
         }
     };
 
-    MatrixCall.prototype.hangup = function(suppressEvent) {
+    MatrixCall.prototype.hangup = function(reason, suppressEvent) {
         console.log("Ending call "+this.call_id);
 
+        // pausing now keeps the last frame (ish) of the video call in the video element
+        // rather than it just turning black straight away
+        if (this.remoteVideoElement) this.remoteVideoElement.pause();
+        if (this.localVideoElement) this.localVideoElement.pause();
+
         this.stopAllMedia();
         if (this.peerConn) this.peerConn.close();
 
         this.hangupParty = 'local';
+        this.hangupReason = reason;
 
         var content = {
             version: 0,
             call_id: this.call_id,
+            reason: reason
         };
         this.sendEventWithRetry('m.call.hangup', content);
         this.state = 'ended';
@@ -136,6 +228,13 @@ angular.module('MatrixCall', [])
         }
         if (this.state == 'ended') return;
 
+        if (this.localVideoElement && this.type == 'video') {
+            var vidTrack = stream.getVideoTracks()[0];
+            this.localVideoElement.src = URL.createObjectURL(stream);
+            this.localVideoElement.muted = true;
+            this.localVideoElement.play();
+        }
+
         this.localAVStream = stream;
         var audioTracks = stream.getAudioTracks();
         for (var i = 0; i < audioTracks.length; i++) {
@@ -157,6 +256,13 @@ angular.module('MatrixCall', [])
     MatrixCall.prototype.gotUserMediaForAnswer = function(stream) {
         if (this.state == 'ended') return;
 
+        if (this.localVideoElement && this.type == 'video') {
+            var vidTrack = stream.getVideoTracks()[0];
+            this.localVideoElement.src = URL.createObjectURL(stream);
+            this.localVideoElement.muted = true;
+            this.localVideoElement.play();
+        }
+
         this.localAVStream = stream;
         var audioTracks = stream.getAudioTracks();
         for (var i = 0; i < audioTracks.length; i++) {
@@ -167,7 +273,7 @@ angular.module('MatrixCall', [])
         var constraints = {
             'mandatory': {
                 'OfferToReceiveAudio': true,
-                'OfferToReceiveVideo': false
+                'OfferToReceiveVideo': this.type == 'video'
             },
         };
         this.peerConn.createAnswer(function(d) { self.createdAnswer(d); }, function(e) {}, constraints);
@@ -176,30 +282,29 @@ angular.module('MatrixCall', [])
     };
 
     MatrixCall.prototype.gotLocalIceCandidate = function(event) {
-        console.log(event);
         if (event.candidate) {
+            console.log("Got local ICE "+event.candidate.sdpMid+" candidate: "+event.candidate.candidate);
             this.sendCandidate(event.candidate);
         }
     }
 
     MatrixCall.prototype.gotRemoteIceCandidate = function(cand) {
-        console.log("Got ICE candidate from remote: "+cand);
+        console.log("Got remote ICE "+cand.sdpMid+" candidate: "+cand.candidate);
         if (this.state == 'ended') {
             console.log("Ignoring remote ICE candidate because call has ended");
             return;
         }
-        var candidateObject = new RTCIceCandidate({
-            sdpMLineIndex: cand.label,
-            candidate: cand.candidate
-        });
-        this.peerConn.addIceCandidate(candidateObject, function() {}, function(e) {});
+        this.peerConn.addIceCandidate(new RTCIceCandidate(cand), function() {}, function(e) {});
     };
 
     MatrixCall.prototype.receivedAnswer = function(msg) {
+        if (this.state == 'ended') return;
+
         this.peerConn.setRemoteDescription(new RTCSessionDescription(msg.answer), this.onSetRemoteDescriptionSuccess, this.onSetRemoteDescriptionError);
         this.state = 'connecting';
     };
 
+
     MatrixCall.prototype.gotLocalOffer = function(description) {
         console.log("Created offer: "+description);
 
@@ -213,11 +318,18 @@ angular.module('MatrixCall', [])
         var content = {
             version: 0,
             call_id: this.call_id,
-            offer: description
+            offer: description,
+            lifetime: MatrixCall.CALL_TIMEOUT
         };
         this.sendEventWithRetry('m.call.invite', content);
 
         var self = this;
+        $timeout(function() {
+            if (self.state == 'invite_sent') {
+                self.hangup('invite_timeout');
+            }
+        }, MatrixCall.CALL_TIMEOUT);
+
         $rootScope.$apply(function() {
             self.state = 'invite_sent';
         });
@@ -243,7 +355,7 @@ angular.module('MatrixCall', [])
     };
 
     MatrixCall.prototype.getUserMediaFailed = function() {
-        this.onError("Couldn't start capturing audio! Is your microphone set up?");
+        this.onError("Couldn't start capturing! Is your microphone set up?");
         this.hangup();
     };
 
@@ -257,6 +369,8 @@ angular.module('MatrixCall', [])
                 self.state = 'connected';
                 self.didConnect = true;
             });
+        } else if (this.peerConn.iceConnectionState == 'failed') {
+            this.hangup('ice_failed');
         }
     };
 
@@ -279,6 +393,14 @@ angular.module('MatrixCall', [])
 
         this.remoteAVStream = s;
 
+        if (this.direction == 'inbound') {
+            if (s.getVideoTracks().length > 0) {
+                this.type = 'video';
+            } else {
+                this.type = 'voice';
+            }
+        }
+
         var self = this;
         forAllTracksOnStream(s, function(t) {
             // not currently implemented in chrome
@@ -288,9 +410,16 @@ angular.module('MatrixCall', [])
         event.stream.onended = function(e) { self.onRemoteStreamEnded(e); }; 
         // not currently implemented in chrome
         event.stream.onstarted = function(e) { self.onRemoteStreamStarted(e); };
-        var player = new Audio();
-        player.src = URL.createObjectURL(s);
-        player.play();
+
+        this.tryPlayRemoteStream();
+    };
+
+    MatrixCall.prototype.tryPlayRemoteStream = function(event) {
+        if (this.remoteVideoElement && this.remoteAVStream) {
+            var player = this.remoteVideoElement;
+            player.src = URL.createObjectURL(this.remoteAVStream);
+            player.play();
+        }
     };
 
     MatrixCall.prototype.onRemoteStreamStarted = function(event) {
@@ -319,12 +448,15 @@ angular.module('MatrixCall', [])
         });
     };
 
-    MatrixCall.prototype.onHangupReceived = function() {
+    MatrixCall.prototype.onHangupReceived = function(msg) {
         console.log("Hangup received");
+        if (this.remoteVideoElement) this.remoteVideoElement.pause();
+        if (this.localVideoElement) this.localVideoElement.pause();
         this.state = 'ended';
         this.hangupParty = 'remote';
+        this.hangupReason = msg.reason;
         this.stopAllMedia();
-        if (this.peerConn.signalingState != 'closed') this.peerConn.close();
+        if (this.peerConn && this.peerConn.signalingState != 'closed') this.peerConn.close();
         if (this.onHangup) this.onHangup(this);
     };
 
@@ -335,13 +467,15 @@ angular.module('MatrixCall', [])
             newCall.waitForLocalAVStream = true;
         } else if (this.state == 'create_offer') {
             console.log("Handing local stream to new call");
-            newCall.localAVStream = this.localAVStream;
+            newCall.gotUserMediaForAnswer(this.localAVStream);
             delete(this.localAVStream);
         } else if (this.state == 'invite_sent') {
             console.log("Handing local stream to new call");
-            newCall.localAVStream = this.localAVStream;
+            newCall.gotUserMediaForAnswer(this.localAVStream);
             delete(this.localAVStream);
         }
+        newCall.localVideoElement = this.localVideoElement;
+        newCall.remoteVideoElement = this.remoteVideoElement;
         this.successor = newCall;
         this.hangup(true);
     };
@@ -403,7 +537,7 @@ angular.module('MatrixCall', [])
         }
 
         if (this.candidateSendTries > 5) {
-            console.log("Failed to send candidates on attempt "+ev.tries+". Giving up for now.");
+            console.log("Failed to send candidates on attempt "+this.candidateSendTries+". Giving up for now.");
             this.candidateSendTries = 0;
             return;
         }

webclient/components/matrix/matrix-filter.js

@@ -26,75 +26,101 @@ angular.module('matrixFilter', [])
         // If there is an alias, use it
         // TODO: only one alias is managed for now
         var alias = matrixService.getRoomIdToAliasMapping(room_id);
-        if (alias) {
-            roomName = alias;
-        }
 
-        if (undefined === roomName) {
+        var room = $rootScope.events.rooms[room_id];
+        if (room) {
+            // Get name from room state date
+            var room_name_event = room["m.room.name"];
+            if (room_name_event) {
+                roomName = room_name_event.content.name;
+            }
+            else if (alias) {
+                roomName = alias;
+            }
+            else if (room.members) {
 
-            var room = $rootScope.events.rooms[room_id];
-            if (room) {
-                // Get name from room state date
-                var room_name_event = room["m.room.name"];
-                if (room_name_event) {
-                    roomName = room_name_event.content.name;
-                }
-                else if (room.members) {
-                    // Else, build the name from its users
-                    // FIXME: Is it still required?
-                    // Limit the room renaming to 1:1 room
-                    if (2 === Object.keys(room.members).length) {
-                        for (var i in room.members) {
-                            var member = room.members[i];
-                            if (member.state_key !== matrixService.config().user_id) {
+                var user_id = matrixService.config().user_id;
 
-                                if (member.state_key in $rootScope.presence) {
-                                    // If the user is available in presence, use the displayname there
-                                    // as it is the most uptodate
-                                    roomName = $rootScope.presence[member.state_key].content.displayname;
-                                }
-                                else if (member.content.displayname) {
-                                    roomName = member.content.displayname;
-                                }
-                                else {
-                                    roomName = member.state_key;
-                                }
+                // Else, build the name from its users
+                // Limit the room renaming to 1:1 room
+                if (2 === Object.keys(room.members).length) {
+                    for (var i in room.members) {
+                        var member = room.members[i];
+                        if (member.state_key !== user_id) {
+
+                            if (member.state_key in $rootScope.presence) {
+                                // If the user is available in presence, use the displayname there
+                                // as it is the most uptodate
+                                roomName = $rootScope.presence[member.state_key].content.displayname;
+                            }
+                            else if (member.content.displayname) {
+                                roomName = member.content.displayname;
+                            }
+                            else {
+                                roomName = member.state_key;
                             }
                         }
                     }
-                    else if (1 === Object.keys(room.members).length) {
+                }
+                else if (1 === Object.keys(room.members).length) {
+                    var otherUserId;
+
+                    if (Object.keys(room.members)[0] !== user_id) {
+                        otherUserId = Object.keys(room.members)[0];
+                    }
+                    else {
                         // The other member may be in the invite list, get all invited users
                         var invitedUserIDs = [];
                         for (var i in room.messages) {
                             var message = room.messages[i];
                             if ("m.room.member" === message.type && "invite" === message.membership) {
-                                // Make sure there is no duplicate user
-                                if (-1 === invitedUserIDs.indexOf(message.state_key)) {
-                                    invitedUserIDs.push(message.state_key);
+                                // Filter out the current user
+                                var member_id = message.state_key;
+                                if (member_id === user_id) {
+                                    member_id = message.user_id;
+                                }
+                                if (member_id !== user_id) {
+                                    // Make sure there is no duplicate user
+                                    if (-1 === invitedUserIDs.indexOf(member_id)) {
+                                        invitedUserIDs.push(member_id);
+                                    }
                                 }
                             } 
                         }
-                        
+
                         // For now, only 1:1 room needs to be renamed. It means only 1 invited user
                         if (1 === invitedUserIDs.length) {
-                            var userID = invitedUserIDs[0];
-
-                            // Try to resolve his displayname in presence global data
-                            if (userID in $rootScope.presence) {
-                                roomName = $rootScope.presence[userID].content.displayname;
-                            }
-                            else {
-                                roomName = userID;
-                            }
+                            otherUserId = invitedUserIDs[0];
                         }
                     }
+
+                    // Try to resolve his displayname in presence global data
+                    if (otherUserId in $rootScope.presence) {
+                        roomName = $rootScope.presence[otherUserId].content.displayname;
+                    }
+                    else {
+                        roomName = otherUserId;
+                    }
                 }
             }
         }
 
+        // Always show the alias in the room displayed name
+        if (roomName && alias && alias !== roomName) {
+            roomName += " (" + alias + ")";
+        }
+
         if (undefined === roomName) {
             // By default, use the room ID
             roomName = room_id;
+
+            // XXX: this is *INCREDIBLY* heavy logging for a function that calls every single
+            // time any kind of digest runs which refreshes a room name...
+            // commenting it out for now.
+
+            // Log some information that lead to this leak
+            // console.log("Room ID leak for " + room_id);
+            // console.log("room object: " + JSON.stringify(room, undefined, 4));   
         }
 
         return roomName;

webclient/components/matrix/matrix-phone-service.js

@@ -24,22 +24,62 @@ angular.module('matrixPhoneService', [])
     matrixPhoneService.INCOMING_CALL_EVENT = "INCOMING_CALL_EVENT";
     matrixPhoneService.REPLACED_CALL_EVENT = "REPLACED_CALL_EVENT";
     matrixPhoneService.allCalls = {};
+    // a place to save candidates that come in for calls we haven't got invites for yet (when paginating backwards)
+    matrixPhoneService.candidatesByCall = {};
 
     matrixPhoneService.callPlaced = function(call) {
         matrixPhoneService.allCalls[call.call_id] = call;
     };
 
     $rootScope.$on(eventHandlerService.CALL_EVENT, function(ngEvent, event, isLive) {
-        if (!isLive) return; // until matrix supports expiring messages
         if (event.user_id == matrixService.config().user_id) return;
+
         var msg = event.content;
+
         if (event.type == 'm.call.invite') {
+            if (event.age == undefined || msg.lifetime == undefined) {
+                // if the event doesn't have either an age (the HS is too old) or a lifetime
+                // (the sending client was too old when it sent it) then fall back to old behaviour
+                if (!isLive) return; // until matrix supports expiring messages
+            }
+
+            if (event.age > msg.lifetime) {
+                console.log("Ignoring expired call event of type "+event.type);
+                return;
+            }
+
+            var call = undefined;
+            if (!isLive) {
+                // if this event wasn't live then this call may already be over
+                call = matrixPhoneService.allCalls[msg.call_id];
+                if (call && call.state == 'ended') {
+                    return;
+                }
+            }
+
             var MatrixCall = $injector.get('MatrixCall');
             var call = new MatrixCall(event.room_id);
+
+            if (!isWebRTCSupported()) {
+                console.log("Incoming call ID "+msg.call_id+" but this browser doesn't support WebRTC");
+                // don't hang up the call: there could be other clients connected that do support WebRTC and declining the
+                // the call on their behalf would be really annoying.
+                // instead, we broadcast a fake call event with a non-functional call object
+                $rootScope.$broadcast(matrixPhoneService.INCOMING_CALL_EVENT, call);
+                return;
+            }
+
             call.call_id = msg.call_id;
-            call.initWithInvite(msg);
+            call.initWithInvite(event);
             matrixPhoneService.allCalls[call.call_id] = call;
 
+            // if we stashed candidate events for that call ID, play them back now
+            if (!isLive && matrixPhoneService.candidatesByCall[call.call_id] != undefined) {
+                for (var i = 0; i < matrixPhoneService.candidatesByCall[call.call_id].length; ++i) {
+                    call.gotRemoteIceCandidate(matrixPhoneService.candidatesByCall[call.call_id][i]);
+                }
+            }
+
             // Were we trying to call that user (room)?
             var existingCall;
             var callIds = Object.keys(matrixPhoneService.allCalls);
@@ -79,21 +119,35 @@ angular.module('matrixPhoneService', [])
             call.receivedAnswer(msg);
         } else if (event.type == 'm.call.candidates') {
             var call = matrixPhoneService.allCalls[msg.call_id];
-            if (!call) {
+            if (!call && isLive) {
                 console.log("Got candidates for unknown call ID "+msg.call_id);
                 return;
-            }
-            for (var i = 0; i < msg.candidates.length; ++i) {
-                call.gotRemoteIceCandidate(msg.candidates[i]);
+            } else if (!call) {
+                if (matrixPhoneService.candidatesByCall[msg.call_id] == undefined) {
+                    matrixPhoneService.candidatesByCall[msg.call_id] = [];
+                }
+                matrixPhoneService.candidatesByCall[msg.call_id] = matrixPhoneService.candidatesByCall[msg.call_id].concat(msg.candidates);
+            } else {
+                for (var i = 0; i < msg.candidates.length; ++i) {
+                    call.gotRemoteIceCandidate(msg.candidates[i]);
+                }
             }
         } else if (event.type == 'm.call.hangup') {
             var call = matrixPhoneService.allCalls[msg.call_id];
-            if (!call) {
+            if (!call && isLive) {
                 console.log("Got hangup for unknown call ID "+msg.call_id);
-                return;
+            } else if (!call) {
+                // if not live, store the fact that the call has ended because we're probably getting events backwards so
+                // the hangup will come before the invite
+                var MatrixCall = $injector.get('MatrixCall');
+                var call = new MatrixCall(event.room_id);
+                call.call_id = msg.call_id;
+                call.initWithHangup(event);
+                matrixPhoneService.allCalls[msg.call_id] = call;
+            } else {
+                call.onHangupReceived(msg);
+                delete(matrixPhoneService.allCalls[msg.call_id]);
             }
-            call.onHangupReceived();
-            delete(matrixPhoneService.allCalls[msg.call_id]);
         }
     });
     

webclient/components/matrix/matrix-service.js

@@ -81,38 +81,155 @@ angular.module('matrixService', [])
 
         return $http(request);
     };
+    
+    var doRegisterLogin = function(path, loginType, sessionId, userName, password, threepidCreds) {
+        var data = {};
+        if (loginType === "m.login.recaptcha") {
+            var challengeToken = Recaptcha.get_challenge();
+            var captchaEntry = Recaptcha.get_response();
+            data = {
+                type: "m.login.recaptcha",
+                challenge: challengeToken,
+                response: captchaEntry
+            };
+        }
+        else if (loginType === "m.login.email.identity") {
+            data = {
+                threepidCreds: threepidCreds
+            };
+        }
+        else if (loginType === "m.login.password") {
+            data = {
+                user: userName,
+                password: password
+            };
+        }
+        
+        if (sessionId) {
+            data.session = sessionId;
+        }
+        data.type = loginType;
+        console.log("doRegisterLogin >>> " + loginType);
+        return doRequest("POST", path, undefined, data);
+    };
 
     return {
         /****** Home server API ******/
         prefix: prefixPath,
 
         // Register an user
-        register: function(user_name, password, threepidCreds, useCaptcha) {         
-            // The REST path spec
+        register: function(user_name, password, threepidCreds, useCaptcha) {
+            // registration is composed of multiple requests, to check you can
+            // register, then to actually register. This deferred will fire when
+            // all the requests are done, along with the final response.
+            var deferred = $q.defer();
             var path = "/register";
             
-            var data = {
-                 user_id: user_name,
-                 password: password,
-                 threepidCreds: threepidCreds
-            };
+            // check we can actually register with this HS.
+            doRequest("GET", path, undefined, undefined).then(
+                function(response) {
+                    console.log("/register [1] : "+JSON.stringify(response));
+                    var flows = response.data.flows;
+                    var knownTypes = [
+                        "m.login.password",
+                        "m.login.recaptcha",
+                        "m.login.email.identity"
+                    ];
+                    // if they entered 3pid creds, we want to use a flow which uses it.
+                    var useThreePidFlow = threepidCreds != undefined;
+                    var flowIndex = 0;
+                    var firstRegType = undefined;
+                    
+                    for (var i=0; i<flows.length; i++) {
+                        var isThreePidFlow = false;
+                        if (flows[i].stages) {
+                            for (var j=0; j<flows[i].stages.length; j++) {
+                                var regType = flows[i].stages[j];
+                                if (knownTypes.indexOf(regType) === -1) {
+                                    deferred.reject("Unknown type: "+regType);
+                                    return;
+                                }
+                                if (regType == "m.login.email.identity") {
+                                    isThreePidFlow = true;
+                                }
+                                if (!useCaptcha && regType == "m.login.recaptcha") {
+                                    console.error("Web client setup to not use captcha, but HS demands a captcha.");
+                                    deferred.reject({
+                                        data: {
+                                            errcode: "M_CAPTCHA_NEEDED",
+                                            error: "Home server requires a captcha."
+                                        }
+                                    });
+                                    return;
+                                }
+                            }
+                        }
+                        
+                        if ( (isThreePidFlow && useThreePidFlow) || (!isThreePidFlow && !useThreePidFlow) ) {
+                            flowIndex = i;
+                        }
+                        
+                        if (knownTypes.indexOf(flows[i].type) == -1) {
+                            deferred.reject("Unknown type: "+flows[i].type);
+                            return;
+                        }
+                    }
+                    
+                    // looks like we can register fine, go ahead and do it.
+                    console.log("Using flow " + JSON.stringify(flows[flowIndex]));
+                    firstRegType = flows[flowIndex].type;
+                    var sessionId = undefined;
+                    
+                    // generic response processor so it can loop as many times as required
+                    var loginResponseFunc = function(response) {
+                        if (response.data.session) {
+                            sessionId = response.data.session;
+                        }
+                        console.log("login response: " + JSON.stringify(response.data));
+                        if (response.data.access_token) {
+                            deferred.resolve(response);
+                        }
+                        else if (response.data.next) {
+                            var nextType = response.data.next;
+                            if (response.data.next instanceof Array) {
+                                for (var i=0; i<response.data.next.length; i++) {
+                                    if (useThreePidFlow && response.data.next[i] == "m.login.email.identity") {
+                                        nextType = response.data.next[i];
+                                        break;
+                                    }
+                                    else if (!useThreePidFlow && response.data.next[i] != "m.login.email.identity") {
+                                        nextType = response.data.next[i];
+                                        break;
+                                    }
+                                }
+                            }
+                            return doRegisterLogin(path, nextType, sessionId, user_name, password, threepidCreds).then(
+                                loginResponseFunc,
+                                function(err) {
+                                    deferred.reject(err);
+                                }
+                            );
+                        }
+                        else {
+                            deferred.reject("Unknown continuation: "+JSON.stringify(response));
+                        }
+                    };
+                    
+                    // set the ball rolling
+                    doRegisterLogin(path, firstRegType, undefined, user_name, password, threepidCreds).then(
+                        loginResponseFunc,
+                        function(err) {
+                            deferred.reject(err);
+                        }
+                    );
+                    
+                },
+                function(err) {
+                    deferred.reject(err);
+                }
+            );
             
-            if (useCaptcha) {
-                // Not all home servers will require captcha on signup, but if this flag is checked,
-                // send captcha information.
-                // TODO: Might be nice to make this a bit more flexible..
-                var challengeToken = Recaptcha.get_challenge();
-                var captchaEntry = Recaptcha.get_response();
-                var captchaType = "m.login.recaptcha";
-                
-                data.captcha = {
-                    type: captchaType,
-                    challenge: challengeToken,
-                    response: captchaEntry
-                };
-            }   
-
-            return doRequest("POST", path, undefined, data);
+            return deferred.promise;
         },
 
         // Create a room

webclient/home/home-controller.js

@@ -53,6 +53,8 @@ angular.module('HomeController', ['matrixService', 'eventHandlerService', 'Recen
 
                     // Add room_alias & room_display_name members
                     angular.extend(room, matrixService.getRoomAliasAndDisplayName(room));
+                    
+                    eventHandlerService.setRoomVisibility(room.room_id, "public");
                 }
             }
         );
@@ -117,6 +119,10 @@ angular.module('HomeController', ['matrixService', 'eventHandlerService', 'Recen
         matrixService.getDisplayName($scope.config.user_id).then(
             function(response) {
                 $scope.profile.displayName = response.data.displayname;
+                var config = matrixService.config();
+                config.display_name = response.data.displayname;
+                matrixService.setConfig(config);
+                matrixService.saveConfig();
             },
             function(error) {
                 $scope.feedback = "Can't load display name";

webclient/index.html

@@ -45,6 +45,13 @@
 </head>
 
 <body>
+    <div id="videoBackground" ng-class="videoMode">
+        <div id="videoContainer" ng-class="videoMode">
+            <div id="videoContainerPadding"></div>
+            <video id="localVideo" ng-class="[videoMode, currentCall.state]" ng-show="currentCall && currentCall.type == 'video' && (currentCall.state == 'connected' || currentCall.state == 'connecting' || currentCall.state == 'invite_sent' || currentCall.state == 'ended')"></video>
+            <video id="remoteVideo" ng-class="[videoMode, currentCall.state]" ng-show="currentCall && currentCall.type == 'video' && (currentCall.state == 'connected' || (currentCall.state == 'ended' && currentCall.didConnect))"></video>
+        </div>
+    </div>
 
     <div id="header">
         <!-- Do not show buttons on the login page -->
@@ -58,19 +65,22 @@
                     <br />
                     <span id="callState">
                         <span ng-show="currentCall.state == 'invite_sent'">Calling...</span>
-                        <span ng-show="currentCall.state == 'ringing'">Incoming Call</span>
+                        <span ng-show="currentCall.state == 'ringing' && currentCall && currentCall.type == 'video'">Incoming Video Call</span>
+                        <span ng-show="currentCall.state == 'ringing' && currentCall && currentCall.type == 'voice'">Incoming Voice Call</span>
                         <span ng-show="currentCall.state == 'connecting'">Call Connecting...</span>
                         <span ng-show="currentCall.state == 'connected'">Call Connected</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'remote'">Call Rejected</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local'">Call Canceled</span>
-                        <span ng-show="currentCall.state == 'ended' && currentCall.didConnect && currentCall.direction == 'outbound'">Call Ended</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'inbound'">Call Canceled</span>
-                        <span ng-show="currentCall.state == 'ended' && currentCall.didConnect && currentCall.direction == 'inbound'">Call Ended</span>
+                        <span ng-show="currentCall.state == 'ended' && currentCall.hangupReason == 'ice_failed'">Media Connection Failed</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'remote'">Call Rejected</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local'">Call Canceled</span>
+                        <span ng-show="currentCall.state == 'ended' && currentCall.hangupReason == 'invite_timeout' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local'">User Not Responding</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && currentCall.didConnect && currentCall.direction == 'outbound'">Call Ended</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && !currentCall.didConnect && currentCall.direction == 'inbound'">Call Canceled</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && currentCall.didConnect && currentCall.direction == 'inbound'">Call Ended</span>
                         <span ng-show="currentCall.state == 'wait_local_media'">Waiting for media permission...</span>
                     </span>
                 </div>
                 <span ng-show="currentCall.state == 'ringing'">
-                    <button ng-click="answerCall()">Answer</button>
+                    <button ng-click="answerCall()" ng-disabled="!isWebRTCSupported" title="{{isWebRTCSupported ? '' : 'Your browser does not support VoIP' }}">Answer {{ currentCall.type }} call</button>
                     <button ng-click="hangupCall()">Reject</button>
                 </span>
                 <button ng-click="hangupCall()" ng-show="currentCall && currentCall.state != 'ringing' && currentCall.state != 'ended' && currentCall.state != 'fledgling'">Hang up</button>
@@ -91,6 +101,7 @@
                     <source src="media/busy.mp3" type="audio/mpeg" />
                 </audio>
             </div>
+
             <a href id="headerUserId" ng-click='goToUserPage(user_id)'>{{ user_id }}</a>
             &nbsp;
             <button ng-click='goToPage("/")'>Home</button>
@@ -99,9 +110,20 @@
         </div>
     </div>
 
-    <div id="page" ng-view></div>
+    <div class="page" ng-hide="unsupportedBrowser" ng-view></div>
 
-    <div id="footer" ng-hide="location.indexOf('/room') == 0">
+    <div class="page" ng-show="unsupportedBrowser">
+        <div id="unsupportedBrowser" ng-show="unsupportedBrowser">
+            Sorry, your browser is not supported. <br/>
+                Reason: {{ unsupportedBrowser.reason }}
+
+            <br/><br/>
+            Your browser: <br/>
+            {{ unsupportedBrowser.browser }}
+        </div>
+    </div>
+
+    <div id="footer" ng-hide="location.indexOf('/room') === 0">
         <div id="footerContent">
             &copy; 2014 Matrix.org
         </div>

webclient/login/login.html

@@ -23,7 +23,7 @@
                     <br/>
                     <input id="password" size="32" type="password" ng-model="account.password" placeholder="Password"/>
                     <br/><br/>
-                    <button ng-click="login()" ng-disabled="!account.user_id || !account.password || !account.homeserver">Login</button>
+                    <button id="login" ng-click="login()" ng-disabled="!account.user_id || !account.password || !account.homeserver">Login</button>
                     <br/><br/>
                 </div>
 

webclient/recents/recents-controller.js

@@ -16,134 +16,16 @@
 
 'use strict';
 
-angular.module('RecentsController', ['matrixService', 'matrixFilter', 'eventHandlerService'])
-.controller('RecentsController', ['$rootScope', '$scope', 'matrixService', 'eventHandlerService', 
-                               function($rootScope, $scope, matrixService, eventHandlerService) {
-                                   
-    // FIXME: Angularjs reloads the controller (and resets its $scope) each time
-    // the page URL changes, use $rootScope to avoid to have to reload data
-    $rootScope.rooms;
+angular.module('RecentsController', ['matrixService', 'matrixFilter'])
+.controller('RecentsController', ['$rootScope', '$scope', 'eventHandlerService', 
+                               function($rootScope, $scope, eventHandlerService) {
+
+    // Expose the service to the view
+    $scope.eventHandlerService = eventHandlerService;
 
     // $rootScope of the parent where the recents component is included can override this value
     // in order to highlight a specific room in the list
     $rootScope.recentsSelectedRoomID;
-    
-    var listenToEventStream = function() {
-        // Refresh the list on matrix invitation and message event
-        $rootScope.$on(eventHandlerService.MEMBER_EVENT, function(ngEvent, event, isLive) {
-            if (isLive) {
-                if (!$rootScope.rooms[event.room_id]) {
-                    // The user has joined a new room, which we do not have data yet. The reason is that
-                    // the room has appeared in the scope of the user rooms after the global initialSync
-                    // FIXME: an initialSync on this specific room should be done
-                    $rootScope.rooms[event.room_id] = {
-                        room_id:event.room_id
-                    };
-                }
-                else if (event.state_key === matrixService.config().user_id && "invite" !== event.membership && "join" !== event.membership) {
-                    // The user has been kicked or banned from the room, remove this room from the recents
-                    delete $rootScope.rooms[event.room_id];
-                }
-                
-                if ($rootScope.rooms[event.room_id]) {
-                    $rootScope.rooms[event.room_id].lastMsg = event;
-                }
-                
-                // Update room users count
-                $rootScope.rooms[event.room_id].numUsersInRoom = getUsersCountInRoom(event.room_id);
-            }
-        });
-        $rootScope.$on(eventHandlerService.MSG_EVENT, function(ngEvent, event, isLive) {
-            if (isLive) {
-                $rootScope.rooms[event.room_id].lastMsg = event;              
-            }
-        });
-        $rootScope.$on(eventHandlerService.CALL_EVENT, function(ngEvent, event, isLive) {
-            if (isLive) {
-                $rootScope.rooms[event.room_id].lastMsg = event;
-            }
-        });
-        $rootScope.$on(eventHandlerService.ROOM_CREATE_EVENT, function(ngEvent, event, isLive) {
-            if (isLive) {
-                $rootScope.rooms[event.room_id] = event;
-            }
-        });
-        $rootScope.$on(eventHandlerService.NAME_EVENT, function(ngEvent, event, isLive) {
-            if (isLive) {
-                $rootScope.rooms[event.room_id].lastMsg = event;
-            }
-        });
-        $rootScope.$on(eventHandlerService.TOPIC_EVENT, function(ngEvent, event, isLive) {
-            if (isLive) {
-                $rootScope.rooms[event.room_id].lastMsg = event;
-            }
-        });
-    };
-    
-    /**
-     * Compute the room users number, ie the number of members who has joined the room.
-     * @param {String} room_id the room id
-     * @returns {undefined | Number} the room users number if available
-     */
-    var getUsersCountInRoom = function(room_id) {
-        var memberCount;
-        
-        var room = $rootScope.events.rooms[room_id];
-        if (room) {
-            memberCount = 0;
-            
-            for (var i in room.members) {
-                var member = room.members[i];
-                
-                if ("join" === member.membership) {
-                    memberCount = memberCount + 1;
-                }
-            }
-        }
-        
-        return memberCount;
-    };
 
-    $scope.onInit = function() {
-        // Init recents list only once
-        if ($rootScope.rooms) {
-            return;
-        }
-        
-        $rootScope.rooms = {};
-        
-        // Use initialSync data to init the recents list
-        eventHandlerService.waitForInitialSyncCompletion().then(
-            function(initialSyncData) {
-            
-                var rooms = initialSyncData.data.rooms;
-                for (var i=0; i<rooms.length; i++) {
-                    var room = rooms[i];
-                    
-                    // Add room_alias & room_display_name members
-                    $rootScope.rooms[room.room_id] = angular.extend(room, matrixService.getRoomAliasAndDisplayName(room));
-
-                    // Create a shortcut for the last message of this room
-                    if (room.messages && room.messages.chunk && room.messages.chunk[0]) {
-                        $rootScope.rooms[room.room_id].lastMsg = room.messages.chunk[0];
-                    }
-                    
-                    $rootScope.rooms[room.room_id].numUsersInRoom = getUsersCountInRoom(room.room_id);
-                }
-
-                // From now, update recents from the stream
-                listenToEventStream();
-            },
-            function(error) {
-                $rootScope.feedback = "Failure: " + error.data;
-            }
-        );
-    };
-
-    // Clean data when user logs out
-    $scope.$on(eventHandlerService.RESET_EVENT, function() {
-
-        delete $rootScope.rooms;
-    });
 }]);
 

webclient/recents/recents-filter.js

@@ -17,31 +17,51 @@
 'use strict';
 
 angular.module('RecentsController')
-.filter('orderRecents', function() {
+.filter('orderRecents', ["matrixService", "eventHandlerService", function(matrixService, eventHandlerService) {
     return function(rooms) {
+        var user_id = matrixService.config().user_id;
 
         // Transform the dict into an array
         // The key, room_id, is already in value objects
         var filtered = [];
-        angular.forEach(rooms, function(value, key) {
-            filtered.push( value );
+        angular.forEach(rooms, function(room, room_id) {
+            
+            // Show the room only if the user has joined it or has been invited
+            // (ie, do not show it if he has been banned)
+            var member = eventHandlerService.getMember(room_id, user_id);
+            if (member && ("invite" === member.membership || "join" === member.membership)) {
+            
+                // Count users here
+                // TODO: Compute it directly in eventHandlerService
+                room.numUsersInRoom = eventHandlerService.getUsersCountInRoom(room_id);
+
+                filtered.push(room);
+            }
+            else if ("invite" === room.membership) {
+                // The only information we have about the room is that the user has been invited
+                filtered.push(room);
+            }
         });
 
         // And time sort them
         // The room with the lastest message at first
-        filtered.sort(function (a, b) {
+        filtered.sort(function (roomA, roomB) {
+
+            var lastMsgRoomA = eventHandlerService.getLastMessage(roomA.room_id, true);
+            var lastMsgRoomB = eventHandlerService.getLastMessage(roomB.room_id, true);
+
             // Invite message does not have a body message nor ts
             // Puth them at the top of the list
-            if (undefined === a.lastMsg) {
+            if (undefined === lastMsgRoomA) {
                 return -1;
             }
-            else if (undefined === b.lastMsg) {
+            else if (undefined === lastMsgRoomB) {
                 return 1;
             }
             else {
-                return b.lastMsg.ts - a.lastMsg.ts;
+                return lastMsgRoomB.ts - lastMsgRoomA.ts;
             }
         });
         return filtered;
     };
-});
+}]);

webclient/recents/recents.html

@@ -1,20 +1,24 @@
-<div ng-controller="RecentsController" data-ng-init="onInit()">
+<div ng-controller="RecentsController">
     <table class="recentsTable">
-        <tbody ng-repeat="(rm_id, room) in rooms | orderRecents" 
+        <tbody ng-repeat="(index, room) in events.rooms | orderRecents" 
                ng-click="goToPage('room/' + (room.room_alias ? room.room_alias : room.room_id) )" 
                class ="recentsRoom" 
-               ng-class="{'recentsRoomSelected': (room.room_id === recentsSelectedRoomID)}">
+               ng-class="{'recentsRoomSelected': (room.room_id === recentsSelectedRoomID)}">                                           
             <tr>
-                <td class="recentsRoomName">
+                <td ng-class="room['m.room.join_rules'].content.join_rule == 'public' ? 'recentsRoomName recentsPublicRoom' : 'recentsRoomName'">
                     {{ room.room_id | mRoomName }}
                 </td>
-                <td class="recentsRoomSummaryTS">
+                <td class="recentsRoomSummaryUsersCount">
                     <span ng-show="undefined !== room.numUsersInRoom">
                         {{ room.numUsersInRoom || '1' }} {{ room.numUsersInRoom == 1 ? 'user' : 'users' }}                     
                     </span>
                 </td>
                 <td class="recentsRoomSummaryTS">
-                    {{ (room.lastMsg.ts) | date:'MMM d HH:mm' }}
+                    <!-- Use a temp var as alias to the last room message.
+                         Declaring it in this way ensures the data-binding -->
+                    {{ lastMsg = eventHandlerService.getLastMessage(room.room_id, true);"" }}
+
+                    {{ (lastMsg.ts) | date:'MMM d HH:mm' }}
                 </td>
             </tr>
 
@@ -22,70 +26,70 @@
                 <td colspan="3" class="recentsRoomSummary">
 
                     <div ng-show="room.membership === 'invite'">
-                        {{ room.lastMsg.inviter | mUserDisplayName: room.room_id }} invited you
+                        {{ room.inviter | mUserDisplayName: room.room_id }} invited you
                     </div>
                     
-                    <div ng-hide="room.membership === 'invite'" ng-switch="room.lastMsg.type">
+                    <div ng-hide="room.membership === 'invite'" ng-switch="lastMsg.type">
                         <div ng-switch-when="m.room.member">
-                            <span ng-if="'join' === room.lastMsg.content.membership">
-                                {{ room.lastMsg.state_key | mUserDisplayName: room.room_id}} joined
+                            <span ng-if="'join' === lastMsg.content.membership">
+                                {{ lastMsg.state_key | mUserDisplayName: room.room_id}} joined
                             </span>
-                            <span ng-if="'leave' === room.lastMsg.content.membership">
-                                <span ng-if="room.lastMsg.user_id === room.lastMsg.state_key">
-                                    {{room.lastMsg.state_key | mUserDisplayName: room.room_id }} left
+                            <span ng-if="'leave' === lastMsg.content.membership">
+                                <span ng-if="lastMsg.user_id === lastMsg.state_key">
+                                    {{lastMsg.state_key | mUserDisplayName: room.room_id }} left
                                 </span>
-                                <span ng-if="room.lastMsg.user_id !== room.lastMsg.state_key">
-                                    {{ room.lastMsg.user_id | mUserDisplayName: room.room_id }}
-                                    {{ {"join": "kicked", "ban": "unbanned"}[room.lastMsg.content.prev] }}
-                                    {{ room.lastMsg.state_key | mUserDisplayName: room.room_id }}
+                                <span ng-if="lastMsg.user_id !== lastMsg.state_key">
+                                    {{ lastMsg.user_id | mUserDisplayName: room.room_id }}
+                                    {{ {"join": "kicked", "ban": "unbanned"}[lastMsg.content.prev] }}
+                                    {{ lastMsg.state_key | mUserDisplayName: room.room_id }}
                                 </span>
-                                <span ng-if="'join' === room.lastMsg.content.prev && room.lastMsg.content.reason">
-                                    : {{ room.lastMsg.content.reason }}
+                                <span ng-if="'join' === lastMsg.content.prev && lastMsg.content.reason">
+                                    : {{ lastMsg.content.reason }}
                                 </span>
                             </span>
-                            <span ng-if="'invite' === room.lastMsg.content.membership || 'ban' === room.lastMsg.content.membership">
-                                {{ room.lastMsg.user_id | mUserDisplayName: room.room_id }}
-                                {{ {"invite": "invited", "ban": "banned"}[room.lastMsg.content.membership] }}
-                                {{ room.lastMsg.state_key | mUserDisplayName: room.room_id }}
-                                <span ng-if="'ban' === room.lastMsg.content.prev && room.lastMsg.content.reason">
-                                    : {{ room.lastMsg.content.reason }}
+                            <span ng-if="'invite' === lastMsg.content.membership || 'ban' === lastMsg.content.membership">
+                                {{ lastMsg.user_id | mUserDisplayName: room.room_id }}
+                                {{ {"invite": "invited", "ban": "banned"}[lastMsg.content.membership] }}
+                                {{ lastMsg.state_key | mUserDisplayName: room.room_id }}
+                                <span ng-if="'ban' === lastMsg.content.prev && lastMsg.content.reason">
+                                    : {{ lastMsg.content.reason }}
                                 </span>
                             </span>
                         </div>
 
                         <div ng-switch-when="m.room.message">
-                            <div ng-switch="room.lastMsg.content.msgtype">
+                            <div ng-switch="lastMsg.content.msgtype">
                                 <div ng-switch-when="m.text">
-                                    {{ room.lastMsg.user_id | mUserDisplayName: room.room_id }} :
-                                    <span ng-bind-html="(room.lastMsg.content.body) | linky:'_blank'">
+                                    {{ lastMsg.user_id | mUserDisplayName: room.room_id }} :
+                                    <span ng-bind-html="(lastMsg.content.body) | linky:'_blank'">
                                     </span>
                                 </div>
 
                                 <div ng-switch-when="m.image">
-                                    {{ room.lastMsg.user_id | mUserDisplayName: room.room_id }} sent an image
+                                    {{ lastMsg.user_id | mUserDisplayName: room.room_id }} sent an image
                                 </div>
 
                                 <div ng-switch-when="m.emote">
-                                    <span ng-bind-html="'* ' + (room.lastMsg.user_id | mUserDisplayName: room.room_id) + ' ' + room.lastMsg.content.body | linky:'_blank'">
+                                    <span ng-bind-html="'* ' + (lastMsg.user_id | mUserDisplayName: room.room_id) + ' ' + lastMsg.content.body | linky:'_blank'">
                                     </span>
                                 </div>
 
                                 <div ng-switch-default>
-                                    {{ room.lastMsg.content }}
+                                    {{ lastMsg.content }}
                                 </div>
                             </div>
                         </div>
 
                         <div ng-switch-when="m.room.topic">
-                            {{ room.lastMsg.user_id | mUserDisplayName: room.room_id }} changed the topic to: {{ room.lastMsg.content.topic }}
+                            {{ lastMsg.user_id | mUserDisplayName: room.room_id }} changed the topic to: {{ lastMsg.content.topic }}
                         </div>
 
                         <div ng-switch-when="m.room.name">
-                            {{ room.lastMsg.user_id | mUserDisplayName: room.room_id }} changed the room name to: {{ room.lastMsg.content.name }}
+                            {{ lastMsg.user_id | mUserDisplayName: room.room_id }} changed the room name to: {{ lastMsg.content.name }}
                         </div>
 
                         <div ng-switch-default>
-                            <div ng-if="room.lastMsg.type.indexOf('m.call.') === 0">
+                            <div ng-if="lastMsg.type.indexOf('m.call.') === 0">
                                 Call
                             </div>
                         </div>

webclient/room/room-controller.js

@@ -15,8 +15,8 @@ limitations under the License.
 */
 
 angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
-.controller('RoomController', ['$filter', '$scope', '$timeout', '$routeParams', '$location', '$rootScope', 'matrixService', 'eventHandlerService', 'mFileUpload', 'mPresence', 'matrixPhoneService', 'MatrixCall',
-                               function($filter, $scope, $timeout, $routeParams, $location, $rootScope, matrixService, eventHandlerService, mFileUpload, mPresence, matrixPhoneService, MatrixCall) {
+.controller('RoomController', ['$filter', '$scope', '$timeout', '$routeParams', '$location', '$rootScope', 'matrixService', 'mPresence', 'eventHandlerService', 'mFileUpload', 'matrixPhoneService', 'MatrixCall',
+                               function($filter, $scope, $timeout, $routeParams, $location, $rootScope, matrixService, mPresence, eventHandlerService, mFileUpload, matrixPhoneService, MatrixCall) {
    'use strict';
     var MESSAGES_PER_PAGINATION = 30;
     var THUMBNAIL_SIZE = 320;
@@ -32,7 +32,8 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         can_paginate: false, // this is toggled off when we are not ready yet to paginate or when we run out of items
         paginating: false, // used to avoid concurrent pagination requests pulling in dup contents
         stream_failure: undefined, // the response when the stream fails
-        waiting_for_joined_event: false  // true when the join request is pending. Back to false once the corresponding m.room.member event is received
+        waiting_for_joined_event: false,  // true when the join request is pending. Back to false once the corresponding m.room.member event is received
+        messages_visibility: "hidden", // In order to avoid flickering when scrolling down the message table at the page opening, delay the message table display
     };
     $scope.members = {};
     $scope.autoCompleting = false;
@@ -53,8 +54,13 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 return;
             };
 
-            // Use the filter applied in html to set the input value
-            $scope.name.newNameText = $filter('mRoomName')($scope.room_id);
+            var nameEvent = $rootScope.events.rooms[$scope.room_id]['m.room.name'];
+            if (nameEvent) {
+                $scope.name.newNameText = nameEvent.content.name;
+            }
+            else {
+                $scope.name.newNameText = "";
+            }
 
             // Force focus to the input
             $timeout(function() {
@@ -131,6 +137,13 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
             
             $timeout(function() {
                 objDiv.scrollTop = objDiv.scrollHeight;
+
+                // Show the message table once the first scrolldown is done 
+                if ("visible" !== $scope.state.messages_visibility) {
+                    $timeout(function() {
+                        $scope.state.messages_visibility = "visible";
+                    }, 0);
+                }
             }, 0);
         }
     };
@@ -139,27 +152,11 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         if (isLive && event.room_id === $scope.room_id) {
             
             scrollToBottom();
-
-            if (window.Notification) {
-                // Show notification when the window is hidden, or the user is idle
-                if (document.hidden || matrixService.presence.unavailable === mPresence.getState()) {
-                    var notification = new window.Notification(
-                        ($scope.members[event.user_id].displayname || event.user_id) +
-                        " (" + ($scope.room_alias || $scope.room_id) + ")", // FIXME: don't leak room_ids here
-                    {
-                        "body": event.content.body,
-                        "icon": $scope.members[event.user_id].avatar_url
-                    });
-                    $timeout(function() {
-                        notification.close();
-                    }, 5 * 1000);
-                }
-            }
         }
     });
     
     $scope.$on(eventHandlerService.MEMBER_EVENT, function(ngEvent, event, isLive) {
-        if (isLive) {
+        if (isLive && event.room_id === $scope.room_id) {
             if ($scope.state.waiting_for_joined_event) {
                 // The user has successfully joined the room, we can getting data for this room
                 $scope.state.waiting_for_joined_event = false;
@@ -177,19 +174,32 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 else {
                     user = event.user_id;
                 }
-
                  
                 if ("ban" === event.membership) {
                     $scope.state.permission_denied = "You have been banned by " + user;
                 }
                 else {
                     $scope.state.permission_denied = "You have been kicked by " + user;
-                }
-                
+                }  
             }
             else {
                 scrollToBottom();
                 updateMemberList(event); 
+
+                // Notify when a user joins
+                if ((document.hidden  || matrixService.presence.unavailable === mPresence.getState())
+                        && event.state_key !== $scope.state.user_id  && "join" === event.membership) {
+                    var notification = new window.Notification(
+                        event.content.displayname +
+                        " (" + (matrixService.getRoomIdToAliasMapping(event.room_id) || event.room_id) + ")", // FIXME: don't leak room_ids here
+                    {
+                        "body": event.content.displayname + " joined",
+                        "icon": event.content.avatar_url ? event.content.avatar_url : undefined
+                    });
+                    $timeout(function() {
+                        notification.close();
+                    }, 5 * 1000);
+                }
             }
         }
     });
@@ -220,7 +230,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
     };
 
     var paginate = function(numItems) {
-        // console.log("paginate " + numItems);
+        //console.log("paginate " + numItems + " and first_pagination is " + $scope.state.first_pagination);
         if ($scope.state.paginating || !$scope.room_id) {
             return;
         }
@@ -235,7 +245,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         matrixService.paginateBackMessages($scope.room_id, $rootScope.events.rooms[$scope.room_id].pagination.earliest_token, numItems).then(
             function(response) {
 
-                eventHandlerService.handleRoomMessages($scope.room_id, response.data, false);
+                eventHandlerService.handleRoomMessages($scope.room_id, response.data, false, 'b');
                 if (response.data.chunk.length < MESSAGES_PER_PAGINATION) {
                     // no more messages to paginate. this currently never gets turned true again, as we never
                     // expire paginated contents in the current implementation.
@@ -260,7 +270,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 }
                 
                 if ($scope.state.first_pagination) {
-                    scrollToBottom();
+                    scrollToBottom(true);
                     $scope.state.first_pagination = false;
                 }
                 else {
@@ -406,25 +416,28 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
     };
 
     $scope.send = function() {
-        if ($scope.textInput === "") {
+        var input = $('#mainInput').val();
+        
+        if (undefined === input || input === "") {
             return;
         }
         
         scrollToBottom(true);
-        
+
+        // Store the command in the history
+        history.push(input);
+
         var promise;
         var cmd;
         var args;
         var echo = false;
         
         // Check for IRC style commands first
-        var line = $scope.textInput;
-        
         // trim any trailing whitespace, as it can confuse the parser for IRC-style commands
-        line = line.replace(/\s+$/, "");
+        input = input.replace(/\s+$/, "");
         
-        if (line[0] === "/" && line[1] !== "/") {
-            var bits = line.match(/^(\S+?)( +(.*))?$/);
+        if (input[0] === "/" && input[1] !== "/") {
+            var bits = input.match(/^(\S+?)( +(.*))?$/);
             cmd = bits[1];
             args = bits[3];
             
@@ -567,7 +580,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         // By default send this as a message unless it's an IRC-style command
         if (!promise && !cmd) {
             // Make the request
-            promise = matrixService.sendTextMessage($scope.room_id, line);
+            promise = matrixService.sendTextMessage($scope.room_id, input);
             echo = true;
         }
         
@@ -576,7 +589,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
             // To do so, create a minimalist fake text message event and add it to the in-memory list of room messages
             var echoMessage = {
                 content: {
-                    body: (cmd === "/me" ? args : line),
+                    body: (cmd === "/me" ? args : input),
                     hsob_ts: new Date().getTime(), // fake a timestamp
                     msgtype: (cmd === "/me" ? "m.emote" : "m.text"),
                 },
@@ -586,7 +599,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 echo_msg_state: "messagePending"     // Add custom field to indicate the state of this fake message to HTML
             };
 
-            $scope.textInput = "";
+            $('#mainInput').val('');
             $rootScope.events.rooms[$scope.room_id].messages.push(echoMessage);
             scrollToBottom();
         }
@@ -606,7 +619,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                         echoMessage.event_id = response.data.event_id;
                     }
                     else {
-                        $scope.textInput = "";
+                        $('#mainInput').val('');
                     }         
                 },
                 function(error) {
@@ -676,6 +689,10 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
     var onInit2 = function() {
         console.log("onInit2");
         
+        // Scroll down as soon as possible so that we point to the last message
+        // if it already exists in memory
+        scrollToBottom(true);
+
         // Make sure the initialSync has been before going further
         eventHandlerService.waitForInitialSyncCompletion().then(
             function() {
@@ -684,6 +701,16 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 
                 // The room members is available in the data fetched by initialSync
                 if ($rootScope.events.rooms[$scope.room_id]) {
+
+                    // There is no need to do a 1st pagination (initialSync provided enough to fill a page)
+                    if ($rootScope.events.rooms[$scope.room_id].messages.length) {
+                        $scope.state.first_pagination = false;
+                    }
+                    else {
+                        // except if we just joined a room, we won't have this history from initial sync, so we should try to paginate it anyway                        
+                        $scope.state.first_pagination = true;
+                    }
+
                     var members = $rootScope.events.rooms[$scope.room_id].members;
 
                     // Update the member list
@@ -729,7 +756,10 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         // Make recents highlight the current room
         $scope.recentsSelectedRoomID = $scope.room_id;
 
-		// Get the up-to-date the current member list
+        // Init the history for this room
+        history.init();
+
+        // Get the up-to-date the current member list
         matrixService.getMemberList($scope.room_id).then(
             function(response) {
                 for (var i = 0; i < response.data.chunk.length; i++) {
@@ -743,9 +773,18 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 // Arm list timing update timer
                 updateMemberListPresenceAge();
 
-                // Start pagination
+                // Allow pagination
                 $scope.state.can_paginate = true;
-                paginate(MESSAGES_PER_PAGINATION);
+
+                // Do a first pagination only if it is required
+                // FIXME: Should be no more require when initialSync/{room_id} will be available
+                if ($scope.state.first_pagination) {
+                    paginate(MESSAGES_PER_PAGINATION);
+                }
+                else {
+                    // There are already messages, go to the last message
+                    scrollToBottom(true);
+                }
             },
             function(error) {
                 $scope.feedback = "Failed get member list: " + error.data.error;
@@ -820,7 +859,9 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         var call = new MatrixCall($scope.room_id);
         call.onError = $rootScope.onCallError;
         call.onHangup = $rootScope.onCallHangup;
-        call.placeCall({audio: true, video: false});
+        // remote video element is used for playing audio in voice calls
+        call.remoteVideoElement = angular.element('#remoteVideo')[0];
+        call.placeVoiceCall();
         $rootScope.currentCall = call;
     };
 
@@ -828,8 +869,88 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         var call = new MatrixCall($scope.room_id);
         call.onError = $rootScope.onCallError;
         call.onHangup = $rootScope.onCallHangup;
-        call.placeCall({audio: true, video: true});
+        call.localVideoElement = angular.element('#localVideo')[0];
+        call.remoteVideoElement = angular.element('#remoteVideo')[0];
+        call.placeVideoCall();
         $rootScope.currentCall = call;
     };
 
+    // Manage history of typed messages
+    // History is saved in sessionStoratge so that it survives when the user
+    // navigates through the rooms and when it refreshes the page
+    var history = {
+        // The list of typed messages. Index 0 is the more recents
+        data: [],
+
+        // The position in the history currently displayed
+        position: -1,
+
+        // The message the user has started to type before going into the history
+        typingMessage: undefined,
+
+        // Init/load data for the current room
+        init: function() {
+            var data = sessionStorage.getItem("history_" + $scope.room_id);
+            if (data) {
+                this.data = JSON.parse(data);
+            }
+        },
+
+        // Store a message in the history
+        push: function(message) {
+            this.data.unshift(message);
+
+            // Update the session storage
+            sessionStorage.setItem("history_" + $scope.room_id, JSON.stringify(this.data));
+
+            // Reset history position
+            this.position = -1;
+            this.typingMessage = undefined;
+        },
+
+        // Move in the history
+        go: function(offset) {
+
+            if (-1 === this.position) {
+                // User starts to go to into the history, save the current line
+                this.typingMessage = $('#mainInput').val();
+            }
+            else {
+                // If the user modified this line in history, keep the change
+                this.data[this.position] = $('#mainInput').val();
+            }
+
+            // Bounds the new position to valid data
+            var newPosition = this.position + offset;
+            newPosition = Math.max(-1, newPosition);
+            newPosition = Math.min(newPosition, this.data.length - 1);
+            this.position = newPosition;
+
+            if (-1 !== this.position) {
+                // Show the message from the history
+                $('#mainInput').val(this.data[this.position]);
+            }
+            else if (undefined !== this.typingMessage) {
+                // Go back to the message the user started to type
+                $('#mainInput').val(this.typingMessage);
+            }
+        }
+    };
+
+    // Make history singleton methods available from HTML
+    $scope.history = {
+        goUp: function($event) {
+            if ($scope.room_id) {
+                history.go(1);
+            }
+            $event.preventDefault();
+        },
+        goDown: function($event) {
+            if ($scope.room_id) {
+                history.go(-1);
+            }
+            $event.preventDefault();
+        }
+    };
+
 }]);

webclient/room/room.html

@@ -9,7 +9,7 @@
                     {{ room_id | mRoomName }}
                 </div>
                 <form ng-submit="name.updateName()" ng-show="name.isEditing" class="roomNameForm">
-                    <input ng-model="name.newNameText" ng-blur="name.cancelEdit()" class="roomNameInput" />
+                    <input ng-model="name.newNameText" ng-blur="name.cancelEdit()" class="roomNameInput" placeholder="Room name"/>
                 </form>
             </div>
 
@@ -23,7 +23,7 @@
                         {{ events.rooms[room_id]['m.room.topic'].content.topic | limitTo: 200}}
                     </div>
                     <form ng-submit="topic.updateTopic()" ng-show="topic.isEditing" class="roomTopicForm">
-                        <input ng-model="topic.newTopicText" ng-blur="topic.cancelEdit()" class="roomTopicInput" />
+                        <input ng-model="topic.newTopicText" ng-blur="topic.cancelEdit()" class="roomTopicInput"  placeholder="Topic"/>
                     </form>
                 </div>
             </div>
@@ -56,7 +56,10 @@
         </table>
     </div>
     
-    <div id="messageTableWrapper" ng-hide="state.permission_denied" keep-scroll>
+    <div id="messageTableWrapper" 
+         ng-hide="state.permission_denied" 
+         ng-style="{ 'visibility': state.messages_visibility }"
+         keep-scroll>
         <!-- FIXME: need to have better timestamp semantics than the (msg.content.hsob_ts || msg.ts) hack below -->
         <table id="messageTable" infinite-scroll="paginateMore()">
             <tr ng-repeat="msg in events.rooms[room_id].messages"
@@ -105,11 +108,11 @@
                         
                         <span ng-show='msg.content.msgtype === "m.text"' 
                               class="message"
-                              ng-class="msg.echo_msg_state"
+                              ng-class="containsBingWord(msg.content.body) && msg.user_id != state.user_id ? msg.echo_msg_state + ' messageBing' : msg.echo_msg_state"
                               ng-bind-html="((msg.content.msgtype === 'm.text') ? msg.content.body : '') | linky:'_blank'"/>
 
-                        <span ng-show='msg.type === "m.call.invite" && msg.user_id == state.user_id'>Outgoing Call</span>
-                        <span ng-show='msg.type === "m.call.invite" && msg.user_id != state.user_id'>Incoming Call</span>
+                        <span ng-show='msg.type === "m.call.invite" && msg.user_id == state.user_id'>Outgoing Call{{ isWebRTCSupported ? '' : ' (But your browser does not support VoIP)' }}</span>
+                        <span ng-show='msg.type === "m.call.invite" && msg.user_id != state.user_id'>Incoming Call{{ isWebRTCSupported ? '' : ' (But your browser does not support VoIP)' }}</span>
 
                         <div ng-show='msg.content.msgtype === "m.image"'>
                             <div ng-hide='msg.content.thumbnail_url' ng-style="msg.content.body.h && { 'height' : (msg.content.body.h < 320) ? msg.content.body.h : 320}">
@@ -154,9 +157,10 @@
                         {{ state.user_id }} 
                     </td>
                     <td width="*">
-                        <textarea id="mainInput" rows="1" ng-model="textInput" ng-enter="send()"
+                        <textarea id="mainInput" rows="1" ng-enter="send()"
                                   ng-disabled="state.permission_denied"
-                                  ng-focus="true"  autocomplete="off" tab-complete/>
+                                  ng-keydown="(38 === $event.which) ? history.goUp($event) : ((40 === $event.which) ? history.goDown($event) : 0)"
+                                  ng-focus="true" autocomplete="off" tab-complete/>
                     </td>
                     <td id="buttonsCell">
                         <button ng-click="send()" ng-disabled="state.permission_denied">Send</button>
@@ -172,7 +176,20 @@
                         <button ng-click="inviteUser()" ng-disabled="state.permission_denied">Invite</button>
                 </span>
                 <button ng-click="leaveRoom()" ng-disabled="state.permission_denied">Leave</button>
-                <button ng-click="startVoiceCall()" ng-show="(currentCall == undefined || currentCall.state == 'ended') && memberCount() == 2" ng-disabled="state.permission_denied">Voice Call</button>
+                <button ng-click="startVoiceCall()"
+                        ng-show="(currentCall == undefined || currentCall.state == 'ended')"
+                        ng-disabled="state.permission_denied || !isWebRTCSupported || memberCount() != 2"
+                        title ="{{ !isWebRTCSupported ? 'VoIP requires webRTC but your browser does not support it' : (memberCount() == 2 ? '' : 'VoIP calls can only be made in rooms with two participants') }}"
+                        >
+                    Voice Call
+                </button>
+                <button ng-click="startVideoCall()"
+                        ng-show="(currentCall == undefined || currentCall.state == 'ended')"
+                        ng-disabled="state.permission_denied || !isWebRTCSupported || memberCount() != 2"
+                        title ="{{ !isWebRTCSupported ? 'VoIP requires webRTC but your browser does not support it' : (memberCount() == 2 ? '' : 'VoIP calls can only be made in rooms with two participants') }}"
+                        >
+                    Video Call
+                </button>
             </div>
         
             {{ feedback }}

webclient/settings/settings-controller.js

@@ -194,7 +194,16 @@ angular.module('SettingsController', ['matrixService', 'mFileUpload', 'mFileInpu
     
     /*** Desktop notifications section ***/
     $scope.settings = {
-        notifications: undefined
+        notifications: undefined,
+        bingWords: matrixService.config().bingWords
+    };
+    
+    $scope.saveBingWords = function() {
+        console.log("Saving words: "+JSON.stringify($scope.settings.bingWords));
+        var config = matrixService.config();
+        config.bingWords = $scope.settings.bingWords;
+        matrixService.setConfig(config);
+        matrixService.saveConfig();
     };
 
     // If the browser supports it, check the desktop notification state

webclient/settings/settings.html

@@ -52,6 +52,15 @@
         <div class="section" ng-switch="settings.notifications">
             <div ng-switch-when="granted">
                 Notifications are enabled.
+                <div class="section">
+                    <h4>Specific words to alert on:</h4>
+                    <p>Leave blank to alert on all messages. Your username & display name always alerts.</p>
+                    <input size=40 name="bingWords" ng-model="settings.bingWords" ng-list placeholder="Enter words separated with , (supports regex)"
+                    ng-blur="saveBingWords()"/>
+                    <ul>
+                        <li ng-repeat="word in settings.bingWords">{{word}}</li>
+                    </ul>
+                </div>
             </div>
             <div ng-switch-when="denied">
                 You have denied permission for notifications.<br/>

webclient/test/README

@@ -0,0 +1,33 @@
+Requires:
+ - nodejs/npm
+ - npm install karma
+ - npm install jasmine
+ - npm install protractor (e2e testing)
+
+Setting up continuous integration / run the unit tests (make sure you're in
+this directory so it can find the config file):
+  karma start
+
+
+Setting up e2e tests (only if you don't have a selenium server to run the tests
+on. If you do, edit the config to point to that url):
+
+  webdriver-manager update
+  webdriver-manager start
+
+  Create a file "environment-protractor.js" in this directory and type:
+    module.exports = {
+        seleniumAddress: 'http://localhost:4444/wd/hub',
+        baseUrl: "http://localhost:8008",
+        username: "YOUR_TEST_USERNAME",
+        password: "YOUR_TEST_PASSWORD"
+    }
+
+Running e2e tests:
+  protractor protractor.conf.js
+
+NOTE: This will create a public room on the target home server.
+
+
+
+

webclient/test/e2e/home.spec.js

@@ -0,0 +1,16 @@
+var env = require("../environment-protractor.js");
+
+describe("home page", function() {
+
+    beforeEach(function() {
+        ptor = protractor.getInstance();
+        // FIXME we use longpoll on the event stream, and I can't get $interval
+        // playing nicely with it. Patches welcome to fix this.
+        ptor.ignoreSynchronization = true;
+    }); 
+
+    it("should have a title", function() {
+        browser.get(env.baseUrl);
+        expect(browser.getTitle()).toEqual("[matrix]");
+    });
+});

webclient/test/karma.conf.js

@@ -0,0 +1,82 @@
+// Karma configuration
+// Generated on Thu Sep 18 2014 14:25:57 GMT+0100 (BST)
+
+module.exports = function(config) {
+  config.set({
+
+    // base path that will be used to resolve all patterns (eg. files, exclude)
+    basePath: '',
+
+
+    // frameworks to use
+    // available frameworks: https://npmjs.org/browse/keyword/karma-adapter
+    frameworks: ['jasmine'],
+
+
+    // list of files / patterns to load in the browser
+    // XXX: Order is important, and doing /js/angular* makes the tests not run :/
+    files: [
+      '../js/jquery*',
+      '../js/angular.js',
+      '../js/angular-mocks.js',
+      '../js/angular-route.js',
+      '../js/angular-animate.js',
+      '../js/angular-sanitize.js',
+      '../js/ng-infinite-scroll-matrix.js',
+      '../login/**/*.*',
+      '../room/**/*.*',
+      '../components/**/*.*',
+      '../user/**/*.*',
+      '../home/**/*.*',
+      '../recents/**/*.*',
+      '../settings/**/*.*',
+      '../app.js',
+      '../app*',
+      './unit/**/*.js'
+    ],
+
+
+    // list of files to exclude
+    exclude: [
+    ],
+
+
+    // preprocess matching files before serving them to the browser
+    // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
+    preprocessors: {
+    },
+
+
+    // test results reporter to use
+    // possible values: 'dots', 'progress'
+    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
+    reporters: ['progress'],
+
+
+    // web server port
+    port: 9876,
+
+
+    // enable / disable colors in the output (reporters and logs)
+    colors: true,
+
+
+    // level of logging
+    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
+    logLevel: config.LOG_DEBUG,
+
+
+    // enable / disable watching file and executing tests whenever any file changes
+    autoWatch: true,
+
+
+    // start these browsers
+    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
+    browsers: ['Chrome'],
+
+
+    // Continuous Integration mode
+    // if true, Karma captures browsers, runs the tests and exits
+    singleRun: false
+  });
+};

webclient/test/protractor.conf.js

@@ -0,0 +1,18 @@
+var env = require("./environment-protractor.js");
+exports.config = {
+    seleniumAddress: env.seleniumAddress,
+    specs: ['e2e/*.spec.js'],
+    onPrepare: function() {
+        browser.driver.get(env.baseUrl);
+        browser.driver.findElement(by.id("user_id")).sendKeys(env.username);
+        browser.driver.findElement(by.id("password")).sendKeys(env.password);
+        browser.driver.findElement(by.id("login")).click();
+
+        // wait till the login is done, detect via url change
+        browser.driver.wait(function() {
+            return browser.driver.getCurrentUrl().then(function(url) {
+                return !(/login/.test(url))
+            });
+        });
+    }
+}

webclient/test/unit/user-controller.spec.js

@@ -0,0 +1,57 @@
+describe("UserCtrl", function() {
+    var scope, ctrl, matrixService, routeParams, $q, $timeout;
+    var userId = "@foo:bar";
+    var displayName = "Foo";
+    var avatarUrl = "avatar.url";
+    
+    beforeEach(module('matrixWebClient'));
+
+    beforeEach(function() {
+
+        inject(function($rootScope, $injector, $controller, _$q_, _$timeout_) {
+            $q = _$q_;
+            $timeout = _$timeout_;
+
+            matrixService = {
+                config: function() {
+                    return {
+                        user_id: userId
+                    };
+                },
+
+                getDisplayName: function(uid) {
+                    var d = $q.defer();
+                    d.resolve({
+                        data: {
+                            displayname: displayName
+                        }
+                    });
+                    return d.promise;
+                },
+
+                getProfilePictureUrl: function(uid) {
+                    var d = $q.defer();
+                    d.resolve({
+                        data: {
+                            avatar_url: avatarUrl
+                        }
+                    });
+                    return d.promise;
+                }
+            };
+            scope = $rootScope.$new();
+            routeParams = {
+                user_matrix_id: userId
+            };
+            ctrl = $controller('UserController', {
+                '$scope': scope, 
+                '$routeParams': routeParams, 
+                'matrixService': matrixService
+            });
+        });
+    });
+
+    it('should display your user id', function() {
+        expect(scope.user_id).toEqual(userId);
+    });
+});