Merge branch 'release-v0.3.4' of github.com:matrix-org/synapse

Displayname are then disambiguate where it is necessary

.gitignore

@@ -25,5 +25,6 @@ graph/*.png
 graph/*.dot
 
 webclient/config.js
+webclient/test/environment-protractor.js
 
 uploads

CHANGES.rst

@@ -1,3 +1,48 @@
+Changes in synapse 0.3.4 (2014-09-25)
+=====================================
+This version adds support for using a TURN server. See docs/turn-howto.rst on
+how to set one up.
+
+Homeserver:
+ * Add support for redaction of messages.
+ * Fix bug where inviting a user on a remote home server could take up to
+   20-30s.
+ * Implement a get current room state API.
+ * Add support specifying and retrieving turn server configuration.
+
+Webclient:
+ * Add button to send messages to users from the home page.
+ * Add support for using TURN for VoIP calls.
+ * Show display name change messages.
+ * Fix bug where the client didn't get the state of a newly joined room
+   until after it has been refreshed.
+ * Fix bugs with tab complete.
+ * Fix bug where holding down the down arrow caused chrome to chew 100% CPU.
+ * Fix bug where desktop notifications occasionally used "Undefined" as the
+   display name.
+ * Fix more places where we sometimes saw room IDs incorrectly.
+ * Fix bug which caused lag when entering text in the text box.
+
+Changes in synapse 0.3.3 (2014-09-22)
+=====================================
+
+Homeserver:
+ * Fix bug where you continued to get events for rooms you had left.
+
+Webclient:
+ * Add support for video calls with basic UI.
+ * Fix bug where one to one chats were named after your display name rather
+   than the other person's.
+ * Fix bug which caused lag when typing in the textarea.
+ * Refuse to run on browsers we know won't work.
+ * Trigger pagination when joining new rooms.
+ * Fix bug where we sometimes didn't display invitations in recents.
+ * Automatically join room when accepting a VoIP call.
+ * Disable outgoing and reject incoming calls on browsers we don't support
+   VoIP in.
+ * Don't display desktop notifications for messages in the room you are
+   non-idle and speaking in.
+
 Changes in synapse 0.3.2 (2014-09-18)
 =====================================
 
@@ -41,7 +86,6 @@ Webclient:
  * The recents list now differentiates between public & private rooms.
  * Fix bug where when switching between rooms the pagination flickered before
    the view jumped to the bottom of the screen.
- * Add support for password resets.
  * Add bing word support.
 
 Registration API:

README.rst

@@ -102,7 +102,7 @@ service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc.
 
 Synapse ships with two basic demo Matrix clients: webclient (a basic group chat
 web client demo implemented in AngularJS) and cmdclient (a basic Python
-commandline utility which lets you easily see what the JSON APIs are up to).
+command line utility which lets you easily see what the JSON APIs are up to).
 
 We'd like to invite you to take a look at the Matrix spec, try to run a
 homeserver, and join the existing Matrix chatrooms already out there, experiment
@@ -122,7 +122,7 @@ Homeserver Installation
 First, the dependencies need to be installed.  Start by installing 
 'python2.7-dev' and the various tools of the compiler toolchain.
 
-  Installing prerequisites on ubuntu::
+  Installing prerequisites on Ubuntu::
 
     $ sudo apt-get install build-essential python2.7-dev libffi-dev
 
@@ -151,8 +151,8 @@ you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and
 installing it. Installing PyNaCl using pip may also work (remember to remove any
 other versions installed by setuputils in, for example, ~/.local/lib).
 
-On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'`` you will
-need to ``export CFLAGS=-Qunused-arguments``.
+On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'`` 
+you will need to ``export CFLAGS=-Qunused-arguments``.
 
 This will run a process of downloading and installing into your
 user's .local/lib directory all of the required dependencies that are
@@ -203,9 +203,10 @@ For the first form, simply pass the required hostname (of the machine) as the
         --generate-config
     $ python synapse/app/homeserver.py --config-path homeserver.config
     
-Alternatively, you can run synapse via synctl - running ``synctl start`` to generate a
-homeserver.yaml config file, where you can then edit server-name to specify
-machine.my.domain.name, and then set the actual server running again with synctl start.
+Alternatively, you can run synapse via synctl - running ``synctl start`` to 
+generate a homeserver.yaml config file, where you can then edit server-name to 
+specify machine.my.domain.name, and then set the actual server running again 
+with synctl start.
 
 For the second form, first create your SRV record and publish it in DNS. This
 needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname
@@ -293,7 +294,8 @@ track 3PID logins and publish end-user public keys.
 
 It's currently early days for identity servers as Matrix is not yet using 3PIDs
 as the primary means of identity and E2E encryption is not complete. As such,
-we are running a single identity server (http://matrix.org:8090) at the current time.
+we are running a single identity server (http://matrix.org:8090) at the current 
+time.
 
 
 Where's the spec?!

VERSION

@@ -1 +1 @@
-0.3.2
+0.3.4

docs/client-server/swagger_matrix/api-docs-rooms

@@ -639,7 +639,7 @@
         {
           "method": "GET",
           "summary": "Get a list of all the current state events for this room.",
-          "notes": "NOT YET IMPLEMENTED.",
+          "notes": "This is equivalent to the events returned under the 'state' key for this room in /initialSync.",
           "type": "array",
           "items": {
             "$ref": "Event"

docs/turn-howto.rst

@@ -0,0 +1,93 @@
+How to enable VoIP relaying on your Home Server with TURN
+
+Overview
+--------
+The synapse Matrix Home Server supports integration with TURN server via the
+TURN server REST API
+(http://tools.ietf.org/html/draft-uberti-behave-turn-rest-00). This allows
+the Home Server to generate credentials that are valid for use on the TURN
+server through the use of a secret shared between the Home Server and the
+TURN server.
+
+This document described how to install coturn
+(https://code.google.com/p/coturn/) which also supports the TURN REST API,
+and integrate it with synapse.
+
+coturn Setup
+============
+
+ 1. Check out coturn::
+      svn checkout http://coturn.googlecode.com/svn/trunk/ coturn
+      cd coturn
+
+ 2. Configure it::
+      ./configure
+
+    You may need to install libevent2: if so, you should do so
+    in the way recommended by your operating system.
+    You can ignore warnings about lack of database support: a
+    database is unnecessary for this purpose.
+
+ 3. Build and install it::
+      make
+      make install
+
+ 4. Make a config file in /etc/turnserver.conf. You can customise
+    a config file from turnserver.conf.default. The relevant
+    lines, with example values, are::
+
+      lt-cred-mech
+      use-auth-secret
+      static-auth-secret=[your secret key here]
+      realm=turn.myserver.org
+
+    See turnserver.conf.default for explanations of the options.
+    One way to generate the static-auth-secret is with pwgen::
+
+       pwgen -s 64 1
+
+ 5. Ensure youe firewall allows traffic into the TURN server on
+    the ports you've configured it to listen on (remember to allow
+    both TCP and UDP if you've enabled both).
+
+ 6. If you've configured coturn to support TLS/DTLS, generate or
+    import your private key and certificate.
+
+ 7. Start the turn server::
+       bin/turnserver -o
+
+
+synapse Setup
+=============
+
+Your home server configuration file needs the following extra keys:
+
+ 1. "turn_uris": This needs to be a yaml list
+    of public-facing URIs for your TURN server to be given out 
+    to your clients. Add separate entries for each transport your
+    TURN server supports.
+
+ 2. "turn_shared_secret": This is the secret shared between your Home
+    server and your TURN server, so you should set it to the same
+    string you used in turnserver.conf.
+
+ 3. "turn_user_lifetime": This is the amount of time credentials
+    generated by your Home Server are valid for (in milliseconds).
+    Shorter times offer less potential for abuse at the expense
+    of increased traffic between web clients and your home server
+    to refresh credentials. The TURN REST API specification recommends
+    one day (86400000).
+
+As an example, here is the relevant section of the config file for
+matrix.org::
+
+    turn_uris: turn:turn.matrix.org:3478?transport=udp,turn:turn.matrix.org:3478?transport=tcp
+    turn_shared_secret: n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons
+    turn_user_lifetime: 86400000
+
+Now, restart synapse::
+
+    cd /where/you/run/synapse
+    ./synctl restart
+
+...and your Home Server now supports VoIP relaying!

synapse/__init__.py

@@ -16,4 +16,4 @@
 """ This is a reference implementation of a synapse home server.
 """
 
-__version__ = "0.3.2"
+__version__ = "0.3.4"

synapse/api/auth.py

@@ -19,7 +19,9 @@ from twisted.internet import defer
 
 from synapse.api.constants import Membership, JoinRules
 from synapse.api.errors import AuthError, StoreError, Codes, SynapseError
-from synapse.api.events.room import RoomMemberEvent, RoomPowerLevelsEvent
+from synapse.api.events.room import (
+    RoomMemberEvent, RoomPowerLevelsEvent, RoomRedactionEvent,
+)
 from synapse.util.logutils import log_function
 
 import logging
@@ -70,6 +72,9 @@ class Auth(object):
                 if event.type == RoomPowerLevelsEvent.TYPE:
                     yield self._check_power_levels(event)
 
+                if event.type == RoomRedactionEvent.TYPE:
+                    yield self._check_redaction(event)
+
                 defer.returnValue(True)
             else:
                 raise AuthError(500, "Unknown event: %s" % event)
@@ -170,7 +175,7 @@ class Auth(object):
                     event.room_id,
                     event.user_id,
                 )
-                _, kick_level = yield self.store.get_ops_levels(event.room_id)
+                _, kick_level, _ = yield self.store.get_ops_levels(event.room_id)
 
                 if kick_level:
                     kick_level = int(kick_level)
@@ -187,7 +192,7 @@ class Auth(object):
                 event.user_id,
             )
 
-            ban_level, _ = yield self.store.get_ops_levels(event.room_id)
+            ban_level, _, _  = yield self.store.get_ops_levels(event.room_id)
 
             if ban_level:
                 ban_level = int(ban_level)
@@ -321,6 +326,29 @@ class Auth(object):
                     "You don't have permission to change that state"
                 )
 
+    @defer.inlineCallbacks
+    def _check_redaction(self, event):
+        user_level = yield self.store.get_power_level(
+            event.room_id,
+            event.user_id,
+        )
+
+        if user_level:
+            user_level = int(user_level)
+        else:
+            user_level = 0
+
+        _, _, redact_level  = yield self.store.get_ops_levels(event.room_id)
+
+        if not redact_level:
+            redact_level = 50
+
+        if user_level < redact_level:
+            raise AuthError(
+                403,
+                "You don't have permission to redact events"
+            )
+
     @defer.inlineCallbacks
     def _check_power_levels(self, event):
         for k, v in event.content.items():
@@ -372,11 +400,11 @@ class Auth(object):
         }
 
         removed = set(old_people.keys()) - set(new_people.keys())
-        added = set(old_people.keys()) - set(new_people.keys())
+        added = set(new_people.keys()) - set(old_people.keys())
         same = set(old_people.keys()) & set(new_people.keys())
 
         for r in removed:
-            if int(old_list.content[r]) > user_level:
+            if int(old_list[r]) > user_level:
                 raise AuthError(
                     403,
                     "You don't have permission to remove user: %s" % (r, )

synapse/api/events/__init__.py

@@ -22,7 +22,8 @@ def serialize_event(hs, e):
     if not isinstance(e, SynapseEvent):
         return e
 
-    d = e.get_dict()
+    # Should this strip out None's?
+    d = {k: v for k, v in e.get_dict().items()}
     if "age_ts" in d:
         d["age"] = int(hs.get_clock().time_msec()) - d["age_ts"]
         del d["age_ts"]
@@ -58,17 +59,19 @@ class SynapseEvent(JsonEncodedObject):
         "required_power_level",
         "age_ts",
         "prev_content",
+        "prev_state",
+        "redacted_because",
     ]
 
     internal_keys = [
         "is_state",
         "prev_events",
-        "prev_state",
         "depth",
         "destinations",
         "origin",
         "outlier",
         "power_level",
+        "redacted",
     ]
 
     required_keys = [
@@ -156,7 +159,8 @@ class SynapseEvent(JsonEncodedObject):
                     return "Missing %s key" % key
 
                 if type(content[key]) != type(template[key]):
-                    return "Key %s is of the wrong type." % key
+                    return "Key %s is of the wrong type (got %s, want %s)" % (
+                        key, type(content[key]), type(template[key]))
 
                 if type(content[key]) == dict:
                     # we must go deeper

synapse/api/events/factory.py

@@ -17,7 +17,8 @@ from synapse.api.events.room import (
     RoomTopicEvent, MessageEvent, RoomMemberEvent, FeedbackEvent,
     InviteJoinEvent, RoomConfigEvent, RoomNameEvent, GenericEvent,
     RoomPowerLevelsEvent, RoomJoinRulesEvent, RoomOpsPowerLevelsEvent,
-    RoomCreateEvent, RoomAddStateLevelEvent, RoomSendEventLevelEvent
+    RoomCreateEvent, RoomAddStateLevelEvent, RoomSendEventLevelEvent,
+    RoomRedactionEvent,
 )
 
 from synapse.util.stringutils import random_string
@@ -39,6 +40,7 @@ class EventFactory(object):
         RoomAddStateLevelEvent,
         RoomSendEventLevelEvent,
         RoomOpsPowerLevelsEvent,
+        RoomRedactionEvent,
     ]
 
     def __init__(self, hs):

synapse/api/events/room.py

@@ -180,3 +180,12 @@ class RoomAliasesEvent(SynapseStateEvent):
 
     def get_content_template(self):
         return {}
+
+
+class RoomRedactionEvent(SynapseEvent):
+    TYPE = "m.room.redaction"
+
+    valid_keys = SynapseEvent.valid_keys + ["redacts"]
+
+    def get_content_template(self):
+        return {}

synapse/api/events/utils.py

@@ -0,0 +1,64 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .room import (
+    RoomMemberEvent, RoomJoinRulesEvent, RoomPowerLevelsEvent,
+    RoomAddStateLevelEvent, RoomSendEventLevelEvent, RoomOpsPowerLevelsEvent,
+    RoomAliasesEvent, RoomCreateEvent,
+)
+
+def prune_event(event):
+    """ Prunes the given event of all keys we don't know about or think could
+    potentially be dodgy.
+
+    This is used when we "redact" an event. We want to remove all fields that
+    the user has specified, but we do want to keep necessary information like
+    type, state_key etc.
+    """
+
+    # Remove all extraneous fields.
+    event.unrecognized_keys = {}
+
+    new_content = {}
+
+    def add_fields(*fields):
+        for field in fields:
+            if field in event.content:
+                new_content[field] = event.content[field]
+
+    if event.type == RoomMemberEvent.TYPE:
+        add_fields("membership")
+    elif event.type == RoomCreateEvent.TYPE:
+        add_fields("creator")
+    elif event.type == RoomJoinRulesEvent.TYPE:
+        add_fields("join_rule")
+    elif event.type == RoomPowerLevelsEvent.TYPE:
+        # TODO: Actually check these are valid user_ids etc.
+        add_fields("default")
+        for k, v in event.content.items():
+            if k.startswith("@") and isinstance(v, (int, long)):
+                new_content[k] = v
+    elif event.type == RoomAddStateLevelEvent.TYPE:
+        add_fields("level")
+    elif event.type == RoomSendEventLevelEvent.TYPE:
+        add_fields("level")
+    elif event.type == RoomOpsPowerLevelsEvent.TYPE:
+        add_fields("kick_level", "ban_level", "redact_level")
+    elif event.type == RoomAliasesEvent.TYPE:
+        add_fields("aliases")
+
+    event.content = new_content
+
+    return event

synapse/app/homeserver.py

@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.storage import read_schema
+from synapse.storage import prepare_database
 
 from synapse.server import HomeServer
 
@@ -36,30 +36,14 @@ from daemonize import Daemonize
 import twisted.manhole.telnet
 
 import logging
-import sqlite3
 import os
 import re
 import sys
+import sqlite3
 
 logger = logging.getLogger(__name__)
 
 
-SCHEMAS = [
-    "transactions",
-    "pdu",
-    "users",
-    "profiles",
-    "presence",
-    "im",
-    "room_aliases",
-]
-
-
-# Remember to update this number every time an incompatible change is made to
-# database schema files, so the users will be informed on server restarts.
-SCHEMA_VERSION = 3
-
-
 class SynapseHomeServer(HomeServer):
 
     def build_http_client(self):
@@ -80,52 +64,12 @@ class SynapseHomeServer(HomeServer):
         )
 
     def build_db_pool(self):
-        """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we
-        don't have to worry about overwriting existing content.
-        """
-        logging.info("Preparing database: %s...", self.db_name)
-
-        with sqlite3.connect(self.db_name) as db_conn:
-            c = db_conn.cursor()
-            c.execute("PRAGMA user_version")
-            row = c.fetchone()
-
-            if row and row[0]:
-                user_version = row[0]
-
-                if user_version > SCHEMA_VERSION:
-                    raise ValueError("Cannot use this database as it is too " +
-                        "new for the server to understand"
-                    )
-                elif user_version < SCHEMA_VERSION:
-                    logging.info("Upgrading database from version %d",
-                        user_version
-                    )
-
-                    # Run every version since after the current version.
-                    for v in range(user_version + 1, SCHEMA_VERSION + 1):
-                        sql_script = read_schema("delta/v%d" % (v))
-                        c.executescript(sql_script)
-
-                    db_conn.commit()
-
-            else:
-                for sql_loc in SCHEMAS:
-                    sql_script = read_schema(sql_loc)
-
-                    c.executescript(sql_script)
-                db_conn.commit()
-                c.execute("PRAGMA user_version = %d" % SCHEMA_VERSION)
-
-            c.close()
-
-        logging.info("Database prepared in %s.", self.db_name)
-
-        pool = adbapi.ConnectionPool(
-            'sqlite3', self.db_name, check_same_thread=False,
-            cp_min=1, cp_max=1)
-
-        return pool
+        return adbapi.ConnectionPool(
+            "sqlite3", self.get_db_name(),
+            check_same_thread=False,
+            cp_min=1,
+            cp_max=1
+        )
 
     def create_resource_tree(self, web_client, redirect_root_to_web_client):
         """Create the resource tree for this Home Server.
@@ -230,10 +174,6 @@ class SynapseHomeServer(HomeServer):
             logger.info("Synapse now listening on port %d", unsecure_port)
 
 
-def run():
-    reactor.run()
-
-
 def setup():
     config = HomeServerConfig.load_config(
         "Synapse Homeserver",
@@ -268,7 +208,15 @@ def setup():
         web_client=config.webclient,
         redirect_root_to_web_client=True,
     )
-    hs.start_listening(config.bind_port, config.unsecure_port)
+
+    db_name = hs.get_db_name()
+
+    logging.info("Preparing database: %s...", db_name)
+
+    with sqlite3.connect(db_name) as db_conn:
+        prepare_database(db_conn)
+
+    logging.info("Database prepared in %s.", db_name)
 
     hs.get_db_pool()
 
@@ -279,12 +227,14 @@ def setup():
         f.namespace['hs'] = hs
         reactor.listenTCP(config.manhole, f, interface='127.0.0.1')
 
+    hs.start_listening(config.bind_port, config.unsecure_port)
+
     if config.daemonize:
         print config.pid_file
         daemon = Daemonize(
             app="synapse-homeserver",
             pid=config.pid_file,
-            action=run,
+            action=reactor.run,
             auto_close_fds=False,
             verbose=True,
             logger=logger,
@@ -292,7 +242,7 @@ def setup():
 
         daemon.start()
     else:
-        run()
+        reactor.run()
 
 
 if __name__ == '__main__':

synapse/config/captcha.py

@@ -14,13 +14,17 @@
 
 from ._base import Config
 
+
 class CaptchaConfig(Config):
 
     def __init__(self, args):
         super(CaptchaConfig, self).__init__(args)
         self.recaptcha_private_key = args.recaptcha_private_key
         self.enable_registration_captcha = args.enable_registration_captcha
-        self.captcha_ip_origin_is_x_forwarded = args.captcha_ip_origin_is_x_forwarded
+        self.captcha_ip_origin_is_x_forwarded = (
+            args.captcha_ip_origin_is_x_forwarded
+        )
+        self.captcha_bypass_secret = args.captcha_bypass_secret
 
     @classmethod
     def add_arguments(cls, parser):
@@ -32,11 +36,16 @@ class CaptchaConfig(Config):
         )
         group.add_argument(
             "--enable-registration-captcha", type=bool, default=False,
-            help="Enables ReCaptcha checks when registering, preventing signup "+
-            "unless a captcha is answered. Requires a valid ReCaptcha public/private key."
+            help="Enables ReCaptcha checks when registering, preventing signup"
+            + " unless a captcha is answered. Requires a valid ReCaptcha "
+            + "public/private key."
         )
         group.add_argument(
             "--captcha_ip_origin_is_x_forwarded", type=bool, default=False,
-            help="When checking captchas, use the X-Forwarded-For (XFF) header as the client IP "+
-            "and not the actual client IP."
-        )
+            help="When checking captchas, use the X-Forwarded-For (XFF) header"
+            + " as the client IP and not the actual client IP."
+        )
+        group.add_argument(
+            "--captcha_bypass_secret", type=str,
+            help="A secret key used to bypass the captcha test entirely."
+        )

synapse/config/homeserver.py

@@ -21,11 +21,12 @@ from .ratelimiting import RatelimitConfig
 from .repository import ContentRepositoryConfig
 from .captcha import CaptchaConfig
 from .email import EmailConfig
+from .voip import VoipConfig
 
 
 class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,
                        RatelimitConfig, ContentRepositoryConfig, CaptchaConfig,
-                       EmailConfig):
+                       EmailConfig, VoipConfig):
     pass
 
 

synapse/config/voip.py

@@ -0,0 +1,41 @@
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ._base import Config
+
+
+class VoipConfig(Config):
+
+    def __init__(self, args):
+        super(VoipConfig, self).__init__(args)
+        self.turn_uris = args.turn_uris
+        self.turn_shared_secret = args.turn_shared_secret
+        self.turn_user_lifetime = args.turn_user_lifetime
+
+    @classmethod
+    def add_arguments(cls, parser):
+        super(VoipConfig, cls).add_arguments(parser)
+        group = parser.add_argument_group("voip")
+        group.add_argument(
+            "--turn-uris", type=str, default=None,
+            help="The public URIs of the TURN server to give to clients"
+        )
+        group.add_argument(
+            "--turn-shared-secret", type=str, default=None,
+            help="The shared secret used to compute passwords for the TURN server"
+        )
+        group.add_argument(
+            "--turn-user-lifetime", type=int, default=(1000 * 60 * 60),
+            help="How long generated TURN credentials last, in ms"
+        )

synapse/handlers/federation.py

@@ -169,7 +169,15 @@ class FederationHandler(BaseHandler):
                     )
 
             if not backfilled:
-                yield self.notifier.on_new_room_event(event)
+                extra_users = []
+                if event.type == RoomMemberEvent.TYPE:
+                    target_user_id = event.state_key
+                    target_user = self.hs.parse_userid(target_user_id)
+                    extra_users.append(target_user)
+
+                yield self.notifier.on_new_room_event(
+                    event, extra_users=extra_users
+                )
 
         if event.type == RoomMemberEvent.TYPE:
             if event.membership == Membership.JOIN:

synapse/handlers/message.py

@@ -232,6 +232,22 @@ class MessageHandler(BaseHandler):
         # store message in db
         yield self._on_new_room_event(event, snapshot)
 
+    @defer.inlineCallbacks
+    def get_state_events(self, user_id, room_id):
+        """Retrieve all state events for a given room.
+
+        Args:
+            user_id(str): The user requesting state events.
+            room_id(str): The room ID to get all state events from.
+        Returns:
+            A list of dicts representing state events. [{}, {}, {}]
+        """
+        yield self.auth.check_joined_room(room_id, user_id)
+
+        # TODO: This is duplicating logic from snapshot_all_rooms
+        current_state = yield self.store.get_current_state(room_id)
+        defer.returnValue([self.hs.serialize_event(c) for c in current_state])
+
     @defer.inlineCallbacks
     def snapshot_all_rooms(self, user_id=None, pagin_config=None,
                            feedback=False):

synapse/handlers/room.py

@@ -145,17 +145,6 @@ class RoomCreationHandler(BaseHandler):
                 content={"name": name},
             )
 
-            yield handle_event(name_event)
-        elif room_alias:
-            name = room_alias.to_string()
-            name_event = self.event_factory.create_event(
-                etype=RoomNameEvent.TYPE,
-                room_id=room_id,
-                user_id=user_id,
-                required_power_level=50,
-                content={"name": name},
-            )
-
             yield handle_event(name_event)
 
         if "topic" in config:
@@ -255,6 +244,7 @@ class RoomCreationHandler(BaseHandler):
             etype=RoomOpsPowerLevelsEvent.TYPE,
             ban_level=50,
             kick_level=50,
+            redact_level=50,
         )
 
         return [

synapse/rest/__init__.py

@@ -15,7 +15,7 @@
 
 
 from . import (
-    room, events, register, login, profile, presence, initial_sync, directory
+    room, events, register, login, profile, presence, initial_sync, directory, voip
 )
 
 
@@ -42,3 +42,4 @@ class RestServletFactory(object):
         presence.register_servlets(hs, client_resource)
         initial_sync.register_servlets(hs, client_resource)
         directory.register_servlets(hs, client_resource)
+        voip.register_servlets(hs, client_resource)

synapse/rest/register.py

@@ -21,6 +21,8 @@ from synapse.api.constants import LoginType
 from base import RestServlet, client_path_pattern
 import synapse.util.stringutils as stringutils
 
+from hashlib import sha1
+import hmac
 import json
 import logging
 import urllib
@@ -28,6 +30,16 @@ import urllib
 logger = logging.getLogger(__name__)
 
 
+# We ought to be using hmac.compare_digest() but on older pythons it doesn't
+# exist. It's a _really minor_ security flaw to use plain string comparison
+# because the timing attack is so obscured by all the other code here it's
+# unlikely to make much difference
+if hasattr(hmac, "compare_digest"):
+    compare_digest = hmac.compare_digest
+else:
+    compare_digest = lambda a, b: a == b
+
+
 class RegisterRestServlet(RestServlet):
     """Handles registration with the home server.
 
@@ -142,6 +154,38 @@ class RegisterRestServlet(RestServlet):
         if not self.hs.config.enable_registration_captcha:
             raise SynapseError(400, "Captcha not required.")
 
+        yield self._check_recaptcha(request, register_json, session)
+
+        session[LoginType.RECAPTCHA] = True  # mark captcha as done
+        self._save_session(session)
+        defer.returnValue({
+            "next": [LoginType.PASSWORD, LoginType.EMAIL_IDENTITY]
+        })
+
+    @defer.inlineCallbacks
+    def _check_recaptcha(self, request, register_json, session):
+        if ("captcha_bypass_hmac" in register_json and
+                self.hs.config.captcha_bypass_secret):
+            if "user" not in register_json:
+                raise SynapseError(400, "Captcha bypass needs 'user'")
+
+            want = hmac.new(
+                key=self.hs.config.captcha_bypass_secret,
+                msg=register_json["user"],
+                digestmod=sha1,
+            ).hexdigest()
+
+            # str() because otherwise hmac complains that 'unicode' does not
+            # have the buffer interface
+            got = str(register_json["captcha_bypass_hmac"])
+
+            if compare_digest(want, got):
+                session["user"] = register_json["user"]
+                defer.returnValue(None)
+            else:
+                raise SynapseError(400, "Captcha bypass HMAC incorrect",
+                    errcode=Codes.CAPTCHA_NEEDED)
+
         challenge = None
         user_response = None
         try:
@@ -166,11 +210,6 @@ class RegisterRestServlet(RestServlet):
             challenge,
             user_response
         )
-        session[LoginType.RECAPTCHA] = True  # mark captcha as done
-        self._save_session(session)
-        defer.returnValue({
-            "next": [LoginType.PASSWORD, LoginType.EMAIL_IDENTITY]
-        })
 
     @defer.inlineCallbacks
     def _do_email_identity(self, request, register_json, session):
@@ -195,6 +234,10 @@ class RegisterRestServlet(RestServlet):
             # captcha should've been done by this stage!
             raise SynapseError(400, "Captcha is required.")
 
+        if ("user" in session and "user" in register_json and
+                session["user"] != register_json["user"]):
+            raise SynapseError(400, "Cannot change user ID during registration")
+
         password = register_json["password"].encode("utf-8")
         desired_user_id = (register_json["user"].encode("utf-8") if "user"
                           in register_json else None)

synapse/rest/room.py

@@ -19,7 +19,7 @@ from twisted.internet import defer
 from base import RestServlet, client_path_pattern
 from synapse.api.errors import SynapseError, Codes
 from synapse.streams.config import PaginationConfig
-from synapse.api.events.room import RoomMemberEvent
+from synapse.api.events.room import RoomMemberEvent, RoomRedactionEvent
 from synapse.api.constants import Membership
 
 import json
@@ -329,12 +329,13 @@ class RoomStateRestServlet(RestServlet):
     @defer.inlineCallbacks
     def on_GET(self, request, room_id):
         user = yield self.auth.get_user_by_req(request)
-        # TODO: Get all the current state for this room and return in the same
-        # format as initial sync, that is:
-        # [
-        #   { state event }, { state event }
-        # ]
-        defer.returnValue((200, []))
+        handler = self.handlers.message_handler
+        # Get all the current state for this room
+        events = yield handler.get_state_events(
+            room_id=urllib.unquote(room_id),
+            user_id=user.to_string(),
+        )
+        defer.returnValue((200, events))
 
 
 # TODO: Needs unit testing
@@ -430,6 +431,41 @@ class RoomMembershipRestServlet(RestServlet):
         self.txns.store_client_transaction(request, txn_id, response)
         defer.returnValue(response)
 
+class RoomRedactEventRestServlet(RestServlet):
+    def register(self, http_server):
+        PATTERN = ("/rooms/(?P<room_id>[^/]*)/redact/(?P<event_id>[^/]*)")
+        register_txn_path(self, PATTERN, http_server)
+
+    @defer.inlineCallbacks
+    def on_POST(self, request, room_id, event_id):
+        user = yield self.auth.get_user_by_req(request)
+        content = _parse_json(request)
+
+        event = self.event_factory.create_event(
+            etype=RoomRedactionEvent.TYPE,
+            room_id=urllib.unquote(room_id),
+            user_id=user.to_string(),
+            content=content,
+            redacts=event_id,
+        )
+
+        msg_handler = self.handlers.message_handler
+        yield msg_handler.send_message(event)
+
+        defer.returnValue((200, {"event_id": event.event_id}))
+
+    @defer.inlineCallbacks
+    def on_PUT(self, request, room_id, event_id, txn_id):
+        try:
+            defer.returnValue(self.txns.get_client_transaction(request, txn_id))
+        except KeyError:
+            pass
+
+        response = yield self.on_POST(request, room_id, event_id)
+
+        self.txns.store_client_transaction(request, txn_id, response)
+        defer.returnValue(response)
+
 
 def _parse_json(request):
     try:
@@ -485,3 +521,4 @@ def register_servlets(hs, http_server):
     PublicRoomListRestServlet(hs).register(http_server)
     RoomStateRestServlet(hs).register(http_server)
     RoomInitialSyncRestServlet(hs).register(http_server)
+    RoomRedactEventRestServlet(hs).register(http_server)

synapse/rest/voip.py

@@ -0,0 +1,60 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from twisted.internet import defer
+
+from base import RestServlet, client_path_pattern
+
+
+import hmac
+import hashlib
+import base64
+
+
+class VoipRestServlet(RestServlet):
+    PATTERN = client_path_pattern("/voip/turnServer$")
+
+    @defer.inlineCallbacks
+    def on_GET(self, request):
+        auth_user = yield self.auth.get_user_by_req(request)
+
+        turnUris = self.hs.config.turn_uris
+        turnSecret = self.hs.config.turn_shared_secret
+        userLifetime = self.hs.config.turn_user_lifetime
+        if not turnUris or not turnSecret or not userLifetime:
+            defer.returnValue( (200, {}) )
+
+        expiry = self.hs.get_clock().time_msec() + userLifetime
+        username = "%d:%s" % (expiry, auth_user.to_string())
+         
+        mac = hmac.new(turnSecret, msg=username, digestmod=hashlib.sha1)
+        # We need to use standard base64 encoding here, *not* syutil's encode_base64
+        # because we need to add the standard padding to get the same result as the
+        # TURN server.
+        password = base64.b64encode(mac.digest())
+
+        defer.returnValue( (200, {
+            'username': username,
+            'password': password,
+            'ttl': userLifetime / 1000,
+            'uris': turnUris,
+        }) )
+
+    def on_OPTIONS(self, request):
+        return (200, {})
+
+
+def register_servlets(hs, http_server):
+    VoipRestServlet(hs).register(http_server)

synapse/server.py

@@ -58,6 +58,7 @@ class BaseHomeServer(object):
     DEPENDENCIES = [
         'clock',
         'http_client',
+        'db_name',
         'db_pool',
         'persistence_service',
         'replication_layer',

synapse/storage/__init__.py

@@ -24,6 +24,7 @@ from synapse.api.events.room import (
     RoomAddStateLevelEvent,
     RoomSendEventLevelEvent,
     RoomOpsPowerLevelsEvent,
+    RoomRedactionEvent,
 )
 
 from synapse.util.logutils import log_function
@@ -47,6 +48,24 @@ import os
 
 logger = logging.getLogger(__name__)
 
+
+SCHEMAS = [
+    "transactions",
+    "pdu",
+    "users",
+    "profiles",
+    "presence",
+    "im",
+    "room_aliases",
+    "redactions",
+]
+
+
+# Remember to update this number every time an incompatible change is made to
+# database schema files, so the users will be informed on server restarts.
+SCHEMA_VERSION = 4
+
+
 class _RollbackButIsFineException(Exception):
     """ This exception is used to rollback a transaction without implying
     something went wrong.
@@ -78,7 +97,7 @@ class DataStore(RoomMemberStore, RoomStore,
             stream_ordering = self.min_token
 
         try:
-            yield self._db_pool.runInteraction(
+            yield self.runInteraction(
                 self._persist_pdu_event_txn,
                 pdu=pdu,
                 event=event,
@@ -165,6 +184,8 @@ class DataStore(RoomMemberStore, RoomStore,
             self._store_send_event_level(txn, event)
         elif event.type == RoomOpsPowerLevelsEvent.TYPE:
             self._store_ops_level(txn, event)
+        elif event.type == RoomRedactionEvent.TYPE:
+            self._store_redaction(txn, event)
 
         vals = {
             "topological_ordering": event.depth,
@@ -186,7 +207,7 @@ class DataStore(RoomMemberStore, RoomStore,
         unrec = {
             k: v
             for k, v in event.get_full_dict().items()
-            if k not in vals.keys()
+            if k not in vals.keys() and k not in ["redacted", "redacted_because"]
         }
         vals["unrecognized_keys"] = json.dumps(unrec)
 
@@ -200,7 +221,8 @@ class DataStore(RoomMemberStore, RoomStore,
             )
             raise _RollbackButIsFineException("_persist_event")
 
-        if is_new_state and hasattr(event, "state_key"):
+        is_state = hasattr(event, "state_key") and event.state_key is not None
+        if is_new_state and is_state:
             vals = {
                 "event_id": event.event_id,
                 "room_id": event.room_id,
@@ -224,14 +246,28 @@ class DataStore(RoomMemberStore, RoomStore,
                 }
             )
 
+    def _store_redaction(self, txn, event):
+        txn.execute(
+            "INSERT OR IGNORE INTO redactions "
+            "(event_id, redacts) VALUES (?,?)",
+            (event.event_id, event.redacts)
+        )
+
     @defer.inlineCallbacks
     def get_current_state(self, room_id, event_type=None, state_key=""):
+        del_sql = (
+            "SELECT event_id FROM redactions WHERE redacts = e.event_id "
+            "LIMIT 1"
+        )
+
         sql = (
-            "SELECT e.* FROM events as e "
+            "SELECT e.*, (%(redacted)s) AS redacted FROM events as e "
             "INNER JOIN current_state_events as c ON e.event_id = c.event_id "
             "INNER JOIN state_events as s ON e.event_id = s.event_id "
             "WHERE c.room_id = ? "
-        )
+        ) % {
+            "redacted": del_sql,
+        }
 
         if event_type:
             sql += " AND s.type = ? AND s.state_key = ? "
@@ -291,7 +327,7 @@ class DataStore(RoomMemberStore, RoomStore,
                 prev_state_pdu=prev_state_pdu,
             )
 
-        return self._db_pool.runInteraction(_snapshot)
+        return self.runInteraction(_snapshot)
 
 
 class Snapshot(object):
@@ -361,3 +397,42 @@ def read_schema(schema):
     """
     with open(schema_path(schema)) as schema_file:
         return schema_file.read()
+
+
+def prepare_database(db_conn):
+    """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we
+    don't have to worry about overwriting existing content.
+    """
+    c = db_conn.cursor()
+    c.execute("PRAGMA user_version")
+    row = c.fetchone()
+
+    if row and row[0]:
+        user_version = row[0]
+
+        if user_version > SCHEMA_VERSION:
+            raise ValueError("Cannot use this database as it is too " +
+                "new for the server to understand"
+            )
+        elif user_version < SCHEMA_VERSION:
+            logging.info("Upgrading database from version %d",
+                user_version
+            )
+
+            # Run every version since after the current version.
+            for v in range(user_version + 1, SCHEMA_VERSION + 1):
+                sql_script = read_schema("delta/v%d" % (v))
+                c.executescript(sql_script)
+
+            db_conn.commit()
+
+    else:
+        for sql_loc in SCHEMAS:
+            sql_script = read_schema(sql_loc)
+
+            c.executescript(sql_script)
+        db_conn.commit()
+        c.execute("PRAGMA user_version = %d" % SCHEMA_VERSION)
+
+    c.close()
+

synapse/storage/_base.py

@@ -17,6 +17,7 @@ import logging
 from twisted.internet import defer
 
 from synapse.api.errors import StoreError
+from synapse.api.events.utils import prune_event
 from synapse.util.logutils import log_function
 
 import collections
@@ -26,6 +27,44 @@ import json
 
 logger = logging.getLogger(__name__)
 
+sql_logger = logging.getLogger("synapse.storage.SQL")
+
+
+class LoggingTransaction(object):
+    """An object that almost-transparently proxies for the 'txn' object
+    passed to the constructor. Adds logging to the .execute() method."""
+    __slots__ = ["txn"]
+
+    def __init__(self, txn):
+        object.__setattr__(self, "txn", txn)
+
+    def __getattribute__(self, name):
+        if name == "execute":
+            return object.__getattribute__(self, "execute")
+
+        return getattr(object.__getattribute__(self, "txn"), name)
+
+    def __setattr__(self, name, value):
+        setattr(object.__getattribute__(self, "txn"), name, value)
+
+    def execute(self, sql, *args, **kwargs):
+        # TODO(paul): Maybe use 'info' and 'debug' for values?
+        sql_logger.debug("[SQL] %s", sql)
+        try:
+            if args and args[0]:
+                values = args[0]
+                sql_logger.debug("[SQL values] " +
+                    ", ".join(("<%s>",) * len(values)), *values)
+        except:
+            # Don't let logging failures stop SQL from working
+            pass
+
+        # TODO(paul): Here would be an excellent place to put some timing
+        #   measurements, and log (warning?) slow queries.
+        return object.__getattribute__(self, "txn").execute(
+            sql, *args, **kwargs
+        )
+
 
 class SQLBaseStore(object):
 
@@ -35,6 +74,13 @@ class SQLBaseStore(object):
         self.event_factory = hs.get_event_factory()
         self._clock = hs.get_clock()
 
+    def runInteraction(self, func, *args, **kwargs):
+        """Wraps the .runInteraction() method on the underlying db_pool."""
+        def inner_func(txn, *args, **kwargs):
+            return func(LoggingTransaction(txn), *args, **kwargs)
+
+        return self._db_pool.runInteraction(inner_func, *args, **kwargs)
+
     def cursor_to_dict(self, cursor):
         """Converts a SQL cursor into an list of dicts.
 
@@ -60,11 +106,6 @@ class SQLBaseStore(object):
         Returns:
             The result of decoder(results)
         """
-        logger.debug(
-            "[SQL] %s  Args=%s Func=%s",
-            query, args, decoder.__name__ if decoder else None
-        )
-
         def interaction(txn):
             cursor = txn.execute(query, args)
             if decoder:
@@ -72,7 +113,7 @@ class SQLBaseStore(object):
             else:
                 return cursor.fetchall()
 
-        return self._db_pool.runInteraction(interaction)
+        return self.runInteraction(interaction)
 
     def _execute_and_decode(self, query, *args):
         return self._execute(self.cursor_to_dict, query, *args)
@@ -88,7 +129,7 @@ class SQLBaseStore(object):
             values : dict of new column names and values for them
             or_replace : bool; if True performs an INSERT OR REPLACE
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._simple_insert_txn, table, values, or_replace=or_replace
         )
 
@@ -172,7 +213,7 @@ class SQLBaseStore(object):
             txn.execute(sql, keyvalues.values())
             return txn.fetchall()
 
-        res = yield self._db_pool.runInteraction(func)
+        res = yield self.runInteraction(func)
 
         defer.returnValue([r[0] for r in res])
 
@@ -195,7 +236,7 @@ class SQLBaseStore(object):
             txn.execute(sql, keyvalues.values())
             return self.cursor_to_dict(txn)
 
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _simple_update_one(self, table, keyvalues, updatevalues,
                            retcols=None):
@@ -263,7 +304,7 @@ class SQLBaseStore(object):
                     raise StoreError(500, "More than one row matched")
 
             return ret
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _simple_delete_one(self, table, keyvalues):
         """Executes a DELETE query on the named table, expecting to delete a
@@ -284,7 +325,7 @@ class SQLBaseStore(object):
                 raise StoreError(404, "No row found")
             if txn.rowcount > 1:
                 raise StoreError(500, "more than one row matched")
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _simple_max_id(self, table):
         """Executes a SELECT query on the named table, expecting to return the
@@ -302,10 +343,10 @@ class SQLBaseStore(object):
                 return 0
             return max_id
 
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _parse_event_from_row(self, row_dict):
-        d = copy.deepcopy({k: v for k, v in row_dict.items() if v})
+        d = copy.deepcopy({k: v for k, v in row_dict.items()})
 
         d.pop("stream_ordering", None)
         d.pop("topological_ordering", None)
@@ -325,7 +366,7 @@ class SQLBaseStore(object):
         )
 
     def _parse_events(self, rows):
-        return self._db_pool.runInteraction(self._parse_events_txn, rows)
+        return self.runInteraction(self._parse_events_txn, rows)
 
     def _parse_events_txn(self, txn, rows):
         events = [self._parse_event_from_row(r) for r in rows]
@@ -333,8 +374,8 @@ class SQLBaseStore(object):
         sql = "SELECT * FROM events WHERE event_id = ?"
 
         for ev in events:
-           if hasattr(ev, "prev_state"):
-                # Load previous state_content. 
+            if hasattr(ev, "prev_state"):
+                # Load previous state_content.
                 # TODO: Should we be pulling this out above?
                 cursor = txn.execute(sql, (ev.prev_state,))
                 prevs = self.cursor_to_dict(cursor)
@@ -342,8 +383,32 @@ class SQLBaseStore(object):
                     prev = self._parse_event_from_row(prevs[0])
                     ev.prev_content = prev.content
 
+            if not hasattr(ev, "redacted"):
+                logger.debug("Doesn't have redacted key: %s", ev)
+                ev.redacted = self._has_been_redacted_txn(txn, ev)
+
+            if ev.redacted:
+                # Get the redaction event.
+                sql = "SELECT * FROM events WHERE event_id = ?"
+                txn.execute(sql, (ev.redacted,))
+
+                del_evs = self._parse_events_txn(
+                    txn, self.cursor_to_dict(txn)
+                )
+
+                if del_evs:
+                    prune_event(ev)
+                    ev.redacted_because = del_evs[0]
+
         return events
 
+    def _has_been_redacted_txn(self, txn, event):
+        sql = "SELECT event_id FROM redactions WHERE redacts = ?"
+        txn.execute(sql, (event.event_id,))
+        result = txn.fetchone()
+        return result[0] if result else None
+
+
 class Table(object):
     """ A base class used to store information about a particular table.
     """

synapse/storage/pdu.py

@@ -43,7 +43,7 @@ class PduStore(SQLBaseStore):
             PduTuple: If the pdu does not exist in the database, returns None
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_pdu_tuple, pdu_id, origin
         )
 
@@ -95,7 +95,7 @@ class PduStore(SQLBaseStore):
             list: A list of PduTuples
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_current_state_for_context,
             context
         )
@@ -143,7 +143,7 @@ class PduStore(SQLBaseStore):
             pdu_origin (str)
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._mark_as_processed, pdu_id, pdu_origin
         )
 
@@ -152,7 +152,7 @@ class PduStore(SQLBaseStore):
 
     def get_all_pdus_from_context(self, context):
         """Get a list of all PDUs for a given context."""
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_all_pdus_from_context, context,
         )
 
@@ -179,7 +179,7 @@ class PduStore(SQLBaseStore):
         Return:
             list: A list of PduTuples
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_backfill, context, pdu_list, limit
         )
 
@@ -240,7 +240,7 @@ class PduStore(SQLBaseStore):
             txn
             context (str)
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_min_depth_for_context, context
         )
 
@@ -346,7 +346,7 @@ class PduStore(SQLBaseStore):
             bool
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._is_pdu_new,
             pdu_id=pdu_id,
             origin=origin,
@@ -499,7 +499,7 @@ class StatePduStore(SQLBaseStore):
         )
 
     def get_unresolved_state_tree(self, new_state_pdu):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_unresolved_state_tree, new_state_pdu
         )
 
@@ -538,7 +538,7 @@ class StatePduStore(SQLBaseStore):
 
     def update_current_state(self, pdu_id, origin, context, pdu_type,
                              state_key):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._update_current_state,
             pdu_id, origin, context, pdu_type, state_key
         )
@@ -577,7 +577,7 @@ class StatePduStore(SQLBaseStore):
             PduEntry
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_current_state_pdu, context, pdu_type, state_key
         )
 
@@ -636,7 +636,7 @@ class StatePduStore(SQLBaseStore):
         Returns:
             bool: True if the new_pdu clobbered the current state, False if not
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._handle_new_state, new_pdu
         )
 

synapse/storage/registration.py

@@ -62,7 +62,7 @@ class RegistrationStore(SQLBaseStore):
         Raises:
             StoreError if the user_id could not be registered.
         """
-        yield self._db_pool.runInteraction(self._register, user_id, token,
+        yield self.runInteraction(self._register, user_id, token,
                                            password_hash)
 
     def _register(self, txn, user_id, token, password_hash):
@@ -99,7 +99,7 @@ class RegistrationStore(SQLBaseStore):
         Raises:
             StoreError if no user was found.
         """
-        user_id = yield self._db_pool.runInteraction(self._query_for_auth,
+        user_id = yield self.runInteraction(self._query_for_auth,
                                                      token)
         defer.returnValue(user_id)
 

synapse/storage/room.py

@@ -27,7 +27,7 @@ import logging
 logger = logging.getLogger(__name__)
 
 
-OpsLevel = collections.namedtuple("OpsLevel", ("ban_level", "kick_level"))
+OpsLevel = collections.namedtuple("OpsLevel", ("ban_level", "kick_level", "redact_level"))
 
 
 class RoomStore(SQLBaseStore):
@@ -149,7 +149,7 @@ class RoomStore(SQLBaseStore):
             defer.returnValue(None)
 
     def get_power_level(self, room_id, user_id):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_power_level,
             room_id, user_id,
         )
@@ -182,14 +182,15 @@ class RoomStore(SQLBaseStore):
             return None
 
     def get_ops_levels(self, room_id):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_ops_levels,
             room_id,
         )
 
     def _get_ops_levels(self, txn, room_id):
         sql = (
-            "SELECT ban_level, kick_level FROM room_ops_levels as r "
+            "SELECT ban_level, kick_level, redact_level "
+            "FROM room_ops_levels as r "
             "INNER JOIN current_state_events as c "
             "ON r.event_id = c.event_id "
             "WHERE c.room_id = ? "
@@ -198,7 +199,7 @@ class RoomStore(SQLBaseStore):
         rows = txn.execute(sql, (room_id,)).fetchall()
 
         if len(rows) == 1:
-            return OpsLevel(rows[0][0], rows[0][1])
+            return OpsLevel(rows[0][0], rows[0][1], rows[0][2])
         else:
             return OpsLevel(None, None)
 
@@ -326,6 +327,9 @@ class RoomStore(SQLBaseStore):
         if "ban_level" in event.content:
             content["ban_level"] = event.content["ban_level"]
 
+        if "redact_level" in event.content:
+            content["redact_level"] = event.content["redact_level"]
+
         self._simple_insert_txn(
             txn,
             "room_ops_levels",

synapse/storage/roommember.py

@@ -149,7 +149,7 @@ class RoomMemberStore(SQLBaseStore):
             membership_list (list): A list of synapse.api.constants.Membership
             values which the user must be in.
         Returns:
-            A list of dicts with "room_id" and "membership" keys.
+            A list of RoomMemberEvent objects
         """
         if not membership_list:
             return defer.succeed(None)
@@ -182,14 +182,22 @@ class RoomMemberStore(SQLBaseStore):
         )
 
     def _get_members_query_txn(self, txn, where_clause, where_values):
+        del_sql = (
+            "SELECT event_id FROM redactions WHERE redacts = e.event_id "
+            "LIMIT 1"
+        )
+
         sql = (
-            "SELECT e.* FROM events as e "
+            "SELECT e.*, (%(redacted)s) AS redacted FROM events as e "
             "INNER JOIN room_memberships as m "
             "ON e.event_id = m.event_id "
             "INNER JOIN current_state_events as c "
             "ON m.event_id = c.event_id "
-            "WHERE %s "
-        ) % (where_clause,)
+            "WHERE %(where)s "
+        ) % {
+            "redacted": del_sql,
+            "where": where_clause,
+        }
 
         txn.execute(sql, where_values)
         rows = self.cursor_to_dict(txn)
@@ -198,10 +206,11 @@ class RoomMemberStore(SQLBaseStore):
         return results
 
     @defer.inlineCallbacks
-    def user_rooms_intersect(self, user_list):
-        """ Checks whether a list of users share a room.
+    def user_rooms_intersect(self, user_id_list):
+        """ Checks whether all the users whose IDs are given in a list share a
+        room.
         """
-        user_list_clause = " OR ".join(["m.user_id = ?"] * len(user_list))
+        user_list_clause = " OR ".join(["m.user_id = ?"] * len(user_id_list))
         sql = (
             "SELECT m.room_id FROM room_memberships as m "
             "INNER JOIN current_state_events as c "
@@ -211,8 +220,8 @@ class RoomMemberStore(SQLBaseStore):
             "GROUP BY m.room_id HAVING COUNT(m.room_id) = ?"
         ) % {"clause": user_list_clause}
 
-        args = user_list
-        args.append(len(user_list))
+        args = list(user_id_list)
+        args.append(len(user_id_list))
 
         rows = yield self._execute(None, sql, *args)
 

synapse/storage/schema/delta/v4.sql

@@ -0,0 +1,12 @@
+CREATE TABLE IF NOT EXISTS redactions (
+    event_id TEXT NOT NULL,
+    redacts TEXT NOT NULL,
+    CONSTRAINT ev_uniq UNIQUE (event_id)
+);
+
+CREATE INDEX IF NOT EXISTS redactions_event_id ON redactions (event_id);
+CREATE INDEX IF NOT EXISTS redactions_redacts ON redactions (redacts);
+
+ALTER TABLE room_ops_levels ADD COLUMN redact_level INTEGER;
+
+PRAGMA user_version = 4;

synapse/storage/schema/im.sql

@@ -150,7 +150,8 @@ CREATE TABLE IF NOT EXISTS room_ops_levels(
     event_id TEXT NOT NULL,
     room_id TEXT NOT NULL,
     ban_level INTEGER,
-    kick_level INTEGER
+    kick_level INTEGER,
+    redact_level INTEGER
 );
 
 CREATE INDEX IF NOT EXISTS room_ops_levels_event_id ON room_ops_levels(event_id);

synapse/storage/schema/redactions.sql

@@ -0,0 +1,8 @@
+CREATE TABLE IF NOT EXISTS redactions (
+    event_id TEXT NOT NULL,
+    redacts TEXT NOT NULL,
+    CONSTRAINT ev_uniq UNIQUE (event_id)
+);
+
+CREATE INDEX IF NOT EXISTS redactions_event_id ON redactions (event_id);
+CREATE INDEX IF NOT EXISTS redactions_redacts ON redactions (redacts);

synapse/storage/stream.py

@@ -146,7 +146,7 @@ class StreamStore(SQLBaseStore):
         current_room_membership_sql = (
             "SELECT m.room_id FROM room_memberships as m "
             "INNER JOIN current_state_events as c ON m.event_id = c.event_id "
-            "WHERE m.user_id = ?"
+            "WHERE m.user_id = ? AND m.membership = 'join'"
         )
 
         # We also want to get any membership events about that user, e.g.
@@ -157,6 +157,11 @@ class StreamStore(SQLBaseStore):
             "WHERE m.user_id = ? "
         )
 
+        del_sql = (
+            "SELECT event_id FROM redactions WHERE redacts = e.event_id "
+            "LIMIT 1"
+        )
+
         if limit:
             limit = max(limit, MAX_STREAM_SIZE)
         else:
@@ -171,13 +176,14 @@ class StreamStore(SQLBaseStore):
             return
 
         sql = (
-            "SELECT * FROM events as e WHERE "
+            "SELECT *, (%(redacted)s) AS redacted FROM events AS e WHERE "
             "((room_id IN (%(current)s)) OR "
             "(event_id IN (%(invites)s))) "
             "AND e.stream_ordering > ? AND e.stream_ordering <= ? "
             "AND e.outlier = 0 "
             "ORDER BY stream_ordering ASC LIMIT %(limit)d "
         ) % {
+            "redacted": del_sql,
             "current": current_room_membership_sql,
             "invites": membership_sql,
             "limit": limit
@@ -224,11 +230,21 @@ class StreamStore(SQLBaseStore):
         else:
             limit_str = ""
 
+        del_sql = (
+            "SELECT event_id FROM redactions WHERE redacts = events.event_id "
+            "LIMIT 1"
+        )
+
         sql = (
-            "SELECT * FROM events "
+            "SELECT *, (%(redacted)s) AS redacted FROM events "
             "WHERE outlier = 0 AND room_id = ? AND %(bounds)s "
             "ORDER BY topological_ordering %(order)s, stream_ordering %(order)s %(limit)s "
-        ) % {"bounds": bounds, "order": order, "limit": limit_str}
+        ) % {
+            "redacted": del_sql,
+            "bounds": bounds,
+            "order": order,
+            "limit": limit_str
+        }
 
         rows = yield self._execute_and_decode(
             sql,
@@ -257,11 +273,18 @@ class StreamStore(SQLBaseStore):
                                    with_feedback=False):
         # TODO (erikj): Handle compressed feedback
 
+        del_sql = (
+            "SELECT event_id FROM redactions WHERE redacts = events.event_id "
+            "LIMIT 1"
+        )
+
         sql = (
-            "SELECT * FROM events "
+            "SELECT *, (%(redacted)s) AS redacted FROM events "
             "WHERE room_id = ? AND stream_ordering <= ? "
             "ORDER BY topological_ordering DESC, stream_ordering DESC LIMIT ? "
-        )
+        ) % {
+            "redacted": del_sql,
+        }
 
         rows = yield self._execute_and_decode(
             sql,
@@ -286,7 +309,7 @@ class StreamStore(SQLBaseStore):
         defer.returnValue(ret)
 
     def get_room_events_max_id(self):
-        return self._db_pool.runInteraction(self._get_room_events_max_id_txn)
+        return self.runInteraction(self._get_room_events_max_id_txn)
 
     def _get_room_events_max_id_txn(self, txn):
         txn.execute(

synapse/storage/transactions.py

@@ -41,7 +41,7 @@ class TransactionStore(SQLBaseStore):
             this transaction or a 2-tuple of (int, dict)
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_received_txn_response, transaction_id, origin
         )
 
@@ -72,7 +72,7 @@ class TransactionStore(SQLBaseStore):
             response_json (str)
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._set_received_txn_response,
             transaction_id, origin, code, response_dict
         )
@@ -104,7 +104,7 @@ class TransactionStore(SQLBaseStore):
             list: A list of previous transaction ids.
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._prep_send_transaction,
             transaction_id, destination, ts, pdu_list
         )
@@ -159,7 +159,7 @@ class TransactionStore(SQLBaseStore):
             code (int)
             response_json (str)
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._delivered_txn,
             transaction_id, destination, code, response_dict
         )
@@ -184,7 +184,7 @@ class TransactionStore(SQLBaseStore):
         Returns:
             list: A list of `ReceivedTransactionsTable.EntryType`
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_transactions_after, transaction_id, destination
         )
 
@@ -214,7 +214,7 @@ class TransactionStore(SQLBaseStore):
         Returns
             list: A list of PduTuple
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_pdus_after_transaction,
             transaction_id, destination
         )

tests/handlers/test_directory.py

@@ -24,6 +24,8 @@ from synapse.http.client import HttpClient
 from synapse.handlers.directory import DirectoryHandler
 from synapse.storage.directory import RoomAliasMapping
 
+from tests.utils import SQLiteMemoryDbPool
+
 
 class DirectoryHandlers(object):
     def __init__(self, hs):
@@ -33,6 +35,7 @@ class DirectoryHandlers(object):
 class DirectoryTestCase(unittest.TestCase):
     """ Tests the directory service. """
 
+    @defer.inlineCallbacks
     def setUp(self):
         self.mock_federation = Mock(spec=[
             "make_query",
@@ -43,11 +46,11 @@ class DirectoryTestCase(unittest.TestCase):
             self.query_handlers[query_type] = handler
         self.mock_federation.register_query_handler = register_query_handler
 
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
         hs = HomeServer("test",
-            datastore=Mock(spec=[
-                "get_association_from_room_alias",
-                "get_joined_hosts_for_room",
-            ]),
+            db_pool=db_pool,
             http_client=None,
             resource_for_federation=Mock(),
             replication_layer=self.mock_federation,
@@ -56,20 +59,16 @@ class DirectoryTestCase(unittest.TestCase):
 
         self.handler = hs.get_handlers().directory_handler
 
-        self.datastore = hs.get_datastore()
-
-        def hosts(room_id):
-            return defer.succeed([])
-        self.datastore.get_joined_hosts_for_room.side_effect = hosts
+        self.store = hs.get_datastore()
 
         self.my_room = hs.parse_roomalias("#my-room:test")
+        self.your_room = hs.parse_roomalias("#your-room:test")
         self.remote_room = hs.parse_roomalias("#another:remote")
 
     @defer.inlineCallbacks
     def test_get_local_association(self):
-        mocked_get = self.datastore.get_association_from_room_alias
-        mocked_get.return_value = defer.succeed(
-            RoomAliasMapping("!8765qwer:test", "#my-room:test", ["test"])
+        yield self.store.create_room_alias_association(
+            self.my_room, "!8765qwer:test", ["test"]
         )
 
         result = yield self.handler.get_association(self.my_room)
@@ -102,9 +101,8 @@ class DirectoryTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_incoming_fed_query(self):
-        mocked_get = self.datastore.get_association_from_room_alias
-        mocked_get.return_value = defer.succeed(
-            RoomAliasMapping("!8765asdf:test", "#your-room:test", ["test"])
+        yield self.store.create_room_alias_association(
+            self.your_room, "!8765asdf:test", ["test"]
         )
 
         response = yield self.query_handlers["directory"](

tests/handlers/test_federation.py

@@ -77,7 +77,7 @@ class FederationTestCase(unittest.TestCase):
         self.datastore.persist_event.assert_called_once_with(
             ANY, False, is_new_state=False
         )
-        self.notifier.on_new_room_event.assert_called_once_with(ANY)
+        self.notifier.on_new_room_event.assert_called_once_with(ANY, extra_users=[])
 
     @defer.inlineCallbacks
     def test_invite_join_target_this(self):

tests/handlers/test_presence.py

@@ -20,7 +20,9 @@ from twisted.internet import defer, reactor
 from mock import Mock, call, ANY
 import json
 
-from ..utils import MockHttpResource, MockClock, DeferredMockCallable
+from tests.utils import (
+    MockHttpResource, MockClock, DeferredMockCallable, SQLiteMemoryDbPool
+)
 
 from synapse.server import HomeServer
 from synapse.api.constants import PresenceState
@@ -60,30 +62,21 @@ class JustPresenceHandlers(object):
 class PresenceStateTestCase(unittest.TestCase):
     """ Tests presence management. """
 
+    @defer.inlineCallbacks
     def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
         hs = HomeServer("test",
-                clock=MockClock(),
-                db_pool=None,
-                datastore=Mock(spec=[
-                    "get_presence_state",
-                    "set_presence_state",
-                    "add_presence_list_pending",
-                    "set_presence_list_accepted",
-                ]),
-                handlers=None,
-                resource_for_federation=Mock(),
-                http_client=None,
-            )
+            clock=MockClock(),
+            db_pool=db_pool,
+            handlers=None,
+            resource_for_federation=Mock(),
+            http_client=None,
+        )
         hs.handlers = JustPresenceHandlers(hs)
 
-        self.datastore = hs.get_datastore()
-
-        def is_presence_visible(observed_localpart, observer_userid):
-            allow = (observed_localpart == "apple" and
-                observer_userid == "@banana:test"
-            )
-            return defer.succeed(allow)
-        self.datastore.is_presence_visible = is_presence_visible
+        self.store = hs.get_datastore()
 
         # Mock the RoomMemberHandler
         room_member_handler = Mock(spec=[])
@@ -94,6 +87,11 @@ class PresenceStateTestCase(unittest.TestCase):
         self.u_banana = hs.parse_userid("@banana:test")
         self.u_clementine = hs.parse_userid("@clementine:test")
 
+        yield self.store.create_presence(self.u_apple.localpart)
+        yield self.store.set_presence_state(
+            self.u_apple.localpart, {"state": ONLINE, "status_msg": "Online"}
+        )
+
         self.handler = hs.get_handlers().presence_handler
 
         self.room_members = []
@@ -117,7 +115,7 @@ class PresenceStateTestCase(unittest.TestCase):
 
             shared = all(map(lambda i: i in room_member_ids, userlist))
             return defer.succeed(shared)
-        self.datastore.user_rooms_intersect = user_rooms_intersect
+        self.store.user_rooms_intersect = user_rooms_intersect
 
         self.mock_start = Mock()
         self.mock_stop = Mock()
@@ -127,11 +125,6 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_get_my_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
-        )
-
         state = yield self.handler.get_state(
             target_user=self.u_apple, auth_user=self.u_apple
         )
@@ -140,13 +133,12 @@ class PresenceStateTestCase(unittest.TestCase):
             {"presence": ONLINE, "status_msg": "Online"},
             state
         )
-        mocked_get.assert_called_with("apple")
 
     @defer.inlineCallbacks
     def test_get_allowed_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
+        yield self.store.allow_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
         )
 
         state = yield self.handler.get_state(
@@ -157,15 +149,9 @@ class PresenceStateTestCase(unittest.TestCase):
             {"presence": ONLINE, "status_msg": "Online"},
             state
         )
-        mocked_get.assert_called_with("apple")
 
     @defer.inlineCallbacks
     def test_get_same_room_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
-        )
-
         self.room_members = [self.u_apple, self.u_clementine]
 
         state = yield self.handler.get_state(
@@ -179,11 +165,6 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_get_disallowed_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
-        )
-
         self.room_members = []
 
         yield self.assertFailure(
@@ -195,16 +176,17 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_set_my_state(self):
-        mocked_set = self.datastore.set_presence_state
-        mocked_set.return_value = defer.succeed({"state": OFFLINE})
-
         yield self.handler.set_state(
                 target_user=self.u_apple, auth_user=self.u_apple,
                 state={"presence": UNAVAILABLE, "status_msg": "Away"})
 
-        mocked_set.assert_called_with("apple",
-            {"state": UNAVAILABLE, "status_msg": "Away"}
+        self.assertEquals(
+            {"state": UNAVAILABLE,
+             "status_msg": "Away",
+             "mtime": 1000000},
+            (yield self.store.get_presence_state(self.u_apple.localpart))
         )
+
         self.mock_start.assert_called_with(self.u_apple,
                 state={
                     "presence": UNAVAILABLE,
@@ -222,50 +204,34 @@ class PresenceStateTestCase(unittest.TestCase):
 class PresenceInvitesTestCase(unittest.TestCase):
     """ Tests presence management. """
 
+    @defer.inlineCallbacks
     def setUp(self):
         self.mock_http_client = Mock(spec=[])
         self.mock_http_client.put_json = DeferredMockCallable()
 
         self.mock_federation_resource = MockHttpResource()
 
-        hs = HomeServer("test",
-                clock=MockClock(),
-                db_pool=None,
-                datastore=Mock(spec=[
-                    "has_presence_state",
-                    "allow_presence_visible",
-                    "add_presence_list_pending",
-                    "set_presence_list_accepted",
-                    "get_presence_list",
-                    "del_presence_list",
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
 
-                    # Bits that Federation needs
-                    "prep_send_transaction",
-                    "delivered_txn",
-                    "get_received_txn_response",
-                    "set_received_txn_response",
-                ]),
-                handlers=None,
-                resource_for_client=Mock(),
-                resource_for_federation=self.mock_federation_resource,
-                http_client=self.mock_http_client,
-            )
+        hs = HomeServer("test",
+            clock=MockClock(),
+            db_pool=db_pool,
+            handlers=None,
+            resource_for_client=Mock(),
+            resource_for_federation=self.mock_federation_resource,
+            http_client=self.mock_http_client,
+        )
         hs.handlers = JustPresenceHandlers(hs)
 
-        self.datastore = hs.get_datastore()
-
-        def has_presence_state(user_localpart):
-            return defer.succeed(
-                user_localpart in ("apple", "banana"))
-        self.datastore.has_presence_state = has_presence_state
-
-        def get_received_txn_response(*args):
-            return defer.succeed(None)
-        self.datastore.get_received_txn_response = get_received_txn_response
+        self.store = hs.get_datastore()
 
         # Some local users to test with
         self.u_apple = hs.parse_userid("@apple:test")
         self.u_banana = hs.parse_userid("@banana:test")
+        yield self.store.create_presence(self.u_apple.localpart)
+        yield self.store.create_presence(self.u_banana.localpart)
+
         # ID of a local user that does not exist
         self.u_durian = hs.parse_userid("@durian:test")
 
@@ -288,12 +254,16 @@ class PresenceInvitesTestCase(unittest.TestCase):
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_banana)
 
-        self.datastore.add_presence_list_pending.assert_called_with(
-                "apple", "@banana:test")
-        self.datastore.allow_presence_visible.assert_called_with(
-                "banana", "@apple:test")
-        self.datastore.set_presence_list_accepted.assert_called_with(
-                "apple", "@banana:test")
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 1}],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
+        self.assertTrue(
+            (yield self.store.is_presence_visible(
+                observed_localpart=self.u_banana.localpart,
+                observer_userid=self.u_apple.to_string(),
+            ))
+        )
 
         self.mock_start.assert_called_with(
                 self.u_apple, target_user=self.u_banana)
@@ -303,10 +273,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_durian)
 
-        self.datastore.add_presence_list_pending.assert_called_with(
-                "apple", "@durian:test")
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@durian:test")
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
     @defer.inlineCallbacks
     def test_invite_remote(self):
@@ -328,8 +298,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_cabbage)
 
-        self.datastore.add_presence_list_pending.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        self.assertEquals(
+            [{"observed_user_id": "@cabbage:elsewhere", "accepted": 0}],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
         yield put_json.await_calls()
 
@@ -362,8 +334,12 @@ class PresenceInvitesTestCase(unittest.TestCase):
             )
         )
 
-        self.datastore.allow_presence_visible.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        self.assertTrue(
+            (yield self.store.is_presence_visible(
+                observed_localpart=self.u_apple.localpart,
+                observer_userid=self.u_cabbage.to_string(),
+            ))
+        )
 
         yield put_json.await_calls()
 
@@ -398,6 +374,11 @@ class PresenceInvitesTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_accepted_remote(self):
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_cabbage.to_string(),
+        )
+
         yield self.mock_federation_resource.trigger("PUT",
             "/_matrix/federation/v1/send/1000000/",
             _make_edu_json("elsewhere", "m.presence_accept",
@@ -408,14 +389,21 @@ class PresenceInvitesTestCase(unittest.TestCase):
             )
         )
 
-        self.datastore.set_presence_list_accepted.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        self.assertEquals(
+            [{"observed_user_id": "@cabbage:elsewhere", "accepted": 1}],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
         self.mock_start.assert_called_with(
                 self.u_apple, target_user=self.u_cabbage)
 
     @defer.inlineCallbacks
     def test_denied_remote(self):
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid="@eggplant:elsewhere",
+        )
+
         yield self.mock_federation_resource.trigger("PUT",
             "/_matrix/federation/v1/send/1000000/",
             _make_edu_json("elsewhere", "m.presence_deny",
@@ -426,32 +414,65 @@ class PresenceInvitesTestCase(unittest.TestCase):
             )
         )
 
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@eggplant:elsewhere")
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
     @defer.inlineCallbacks
     def test_drop_local(self):
-        yield self.handler.drop(
-                observer_user=self.u_apple, observed_user=self.u_banana)
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
 
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@banana:test")
+        yield self.handler.drop(
+            observer_user=self.u_apple,
+            observed_user=self.u_banana,
+        )
+
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
         self.mock_stop.assert_called_with(
                 self.u_apple, target_user=self.u_banana)
 
     @defer.inlineCallbacks
     def test_drop_remote(self):
-        yield self.handler.drop(
-                observer_user=self.u_apple, observed_user=self.u_cabbage)
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_cabbage.to_string(),
+        )
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_cabbage.to_string(),
+        )
 
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        yield self.handler.drop(
+            observer_user=self.u_apple,
+            observed_user=self.u_cabbage,
+        )
+
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
     @defer.inlineCallbacks
     def test_get_presence_list(self):
-        self.datastore.get_presence_list.return_value = defer.succeed(
-            [{"observed_user_id": "@banana:test"}]
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
         )
 
         presence = yield self.handler.get_presence_list(
@@ -459,29 +480,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
 
         self.assertEquals([
             {"observed_user": self.u_banana,
-             "presence": OFFLINE},
+             "presence": OFFLINE,
+             "accepted": 1},
         ], presence)
 
-        self.datastore.get_presence_list.assert_called_with("apple",
-            accepted=None
-        )
-
-        self.datastore.get_presence_list.return_value = defer.succeed(
-            [{"observed_user_id": "@banana:test"}]
-        )
-
-        presence = yield self.handler.get_presence_list(
-            observer_user=self.u_apple, accepted=True
-        )
-
-        self.assertEquals([
-            {"observed_user": self.u_banana,
-             "presence": OFFLINE},
-        ], presence)
-
-        self.datastore.get_presence_list.assert_called_with("apple",
-                accepted=True)
-
 
 class PresencePushTestCase(unittest.TestCase):
     """ Tests steady-state presence status updates.

tests/handlers/test_profile.py

@@ -24,6 +24,8 @@ from synapse.server import HomeServer
 from synapse.handlers.profile import ProfileHandler
 from synapse.api.constants import Membership
 
+from tests.utils import SQLiteMemoryDbPool
+
 
 class ProfileHandlers(object):
     def __init__(self, hs):
@@ -33,6 +35,7 @@ class ProfileHandlers(object):
 class ProfileTestCase(unittest.TestCase):
     """ Tests profile management. """
 
+    @defer.inlineCallbacks
     def setUp(self):
         self.mock_federation = Mock(spec=[
             "make_query",
@@ -43,63 +46,50 @@ class ProfileTestCase(unittest.TestCase):
             self.query_handlers[query_type] = handler
         self.mock_federation.register_query_handler = register_query_handler
 
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
         hs = HomeServer("test",
-                db_pool=None,
+                db_pool=db_pool,
                 http_client=None,
-                datastore=Mock(spec=[
-                    "get_profile_displayname",
-                    "set_profile_displayname",
-                    "get_profile_avatar_url",
-                    "set_profile_avatar_url",
-                    "get_rooms_for_user_where_membership_is",
-                ]),
                 handlers=None,
                 resource_for_federation=Mock(),
                 replication_layer=self.mock_federation,
             )
         hs.handlers = ProfileHandlers(hs)
 
-        self.datastore = hs.get_datastore()
+        self.store = hs.get_datastore()
 
         self.frank = hs.parse_userid("@1234ABCD:test")
         self.bob   = hs.parse_userid("@4567:test")
         self.alice = hs.parse_userid("@alice:remote")
 
-        self.handler = hs.get_handlers().profile_handler
+        yield self.store.create_profile(self.frank.localpart)
 
-        self.mock_get_joined = (
-            self.datastore.get_rooms_for_user_where_membership_is
-        )
+        self.handler = hs.get_handlers().profile_handler
 
         # TODO(paul): Icky signal declarings.. booo
         hs.get_distributor().declare("changed_presencelike_data")
 
     @defer.inlineCallbacks
     def test_get_my_name(self):
-        mocked_get = self.datastore.get_profile_displayname
-        mocked_get.return_value = defer.succeed("Frank")
+        yield self.store.set_profile_displayname(
+            self.frank.localpart, "Frank"
+        )
 
         displayname = yield self.handler.get_displayname(self.frank)
 
         self.assertEquals("Frank", displayname)
-        mocked_get.assert_called_with("1234ABCD")
 
     @defer.inlineCallbacks
     def test_set_my_name(self):
-        mocked_set = self.datastore.set_profile_displayname
-        mocked_set.return_value = defer.succeed(())
-
-        self.mock_get_joined.return_value = defer.succeed([])
-
         yield self.handler.set_displayname(self.frank, self.frank, "Frank Jr.")
 
-        self.mock_get_joined.assert_called_once_with(
-            self.frank.to_string(),
-            [Membership.JOIN]
+        self.assertEquals(
+            (yield self.store.get_profile_displayname(self.frank.localpart)),
+            "Frank Jr."
         )
 
-        mocked_set.assert_called_with("1234ABCD", "Frank Jr.")
-
     @defer.inlineCallbacks
     def test_set_my_name_noauth(self):
         d = self.handler.set_displayname(self.frank, self.bob, "Frank Jr.")
@@ -123,40 +113,31 @@ class ProfileTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_incoming_fed_query(self):
-        mocked_get = self.datastore.get_profile_displayname
-        mocked_get.return_value = defer.succeed("Caroline")
+        yield self.store.create_profile("caroline")
+        yield self.store.set_profile_displayname("caroline", "Caroline")
 
         response = yield self.query_handlers["profile"](
             {"user_id": "@caroline:test", "field": "displayname"}
         )
 
         self.assertEquals({"displayname": "Caroline"}, response)
-        mocked_get.assert_called_with("caroline")
 
     @defer.inlineCallbacks
     def test_get_my_avatar(self):
-        mocked_get = self.datastore.get_profile_avatar_url
-        mocked_get.return_value = defer.succeed("http://my.server/me.png")
+        yield self.store.set_profile_avatar_url(
+            self.frank.localpart, "http://my.server/me.png"
+        )
 
         avatar_url = yield self.handler.get_avatar_url(self.frank)
 
         self.assertEquals("http://my.server/me.png", avatar_url)
-        mocked_get.assert_called_with("1234ABCD")
 
     @defer.inlineCallbacks
     def test_set_my_avatar(self):
-        mocked_set = self.datastore.set_profile_avatar_url
-        mocked_set.return_value = defer.succeed(())
-
-        self.mock_get_joined.return_value = defer.succeed([])
-
         yield self.handler.set_avatar_url(self.frank, self.frank,
                 "http://my.server/pic.gif")
 
-        self.mock_get_joined.assert_called_once_with(
-            self.frank.to_string(),
-            [Membership.JOIN]
+        self.assertEquals(
+            (yield self.store.get_profile_avatar_url(self.frank.localpart)),
+            "http://my.server/pic.gif"
         )
-
-
-        mocked_set.assert_called_with("1234ABCD", "http://my.server/pic.gif")

tests/storage/TESTS_NEEDED_FOR

@@ -0,0 +1,5 @@
+synapse/storage/feedback.py
+synapse/storage/keys.py
+synapse/storage/pdu.py
+synapse/storage/stream.py
+synapse/storage/transactions.py

tests/storage/test_directory.py

@@ -0,0 +1,68 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.directory import DirectoryStore
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class DirectoryStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        self.store = DirectoryStore(hs)
+
+        self.room = hs.parse_roomid("!abcde:test")
+        self.alias = hs.parse_roomalias("#my-room:test")
+
+    @defer.inlineCallbacks
+    def test_room_to_alias(self):
+        yield self.store.create_room_alias_association(
+            room_alias=self.alias,
+            room_id=self.room.to_string(),
+            servers=["test"],
+        )
+
+        self.assertEquals(
+            ["#my-room:test"],
+            (yield self.store.get_aliases_for_room(self.room.to_string()))
+        )
+
+    @defer.inlineCallbacks
+    def test_alias_to_room(self):
+        yield self.store.create_room_alias_association(
+            room_alias=self.alias,
+            room_id=self.room.to_string(),
+            servers=["test"],
+        )
+
+
+        self.assertObjectHasAttributes(
+            {"room_id": self.room.to_string(),
+             "servers": ["test"]},
+            (yield self.store.get_association_from_room_alias(self.alias))
+        )

tests/storage/test_presence.py

@@ -0,0 +1,167 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.presence import PresenceStore
+
+from tests.utils import SQLiteMemoryDbPool, MockClock
+
+
+class PresenceStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            clock=MockClock(),
+            db_pool=db_pool,
+        )
+
+        self.store = PresenceStore(hs)
+
+        self.u_apple = hs.parse_userid("@apple:test")
+        self.u_banana = hs.parse_userid("@banana:test")
+
+    @defer.inlineCallbacks
+    def test_state(self):
+        yield self.store.create_presence(
+            self.u_apple.localpart
+        )
+
+        state = yield self.store.get_presence_state(
+            self.u_apple.localpart
+        )
+
+        self.assertEquals(
+            {"state": None, "status_msg": None, "mtime": None}, state
+        )
+
+        yield self.store.set_presence_state(
+            self.u_apple.localpart, {"state": "online", "status_msg": "Here"}
+        )
+
+        state = yield self.store.get_presence_state(
+            self.u_apple.localpart
+        )
+
+        self.assertEquals(
+            {"state": "online", "status_msg": "Here", "mtime": 1000000}, state
+        )
+
+    @defer.inlineCallbacks
+    def test_visibility(self):
+        self.assertFalse((yield self.store.is_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )))
+
+        yield self.store.allow_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )
+
+        self.assertTrue((yield self.store.is_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )))
+
+        yield self.store.disallow_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )
+
+        self.assertFalse((yield self.store.is_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )))
+
+    @defer.inlineCallbacks
+    def test_presence_list(self):
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )
+
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 0}],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )
+
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 1}],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 1}],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )
+
+        yield self.store.del_presence_list(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )

tests/storage/test_profile.py

@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.profile import ProfileStore
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class ProfileStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        self.store = ProfileStore(hs)
+
+        self.u_frank = hs.parse_userid("@frank:test")
+
+    @defer.inlineCallbacks
+    def test_displayname(self):
+        yield self.store.create_profile(
+            self.u_frank.localpart
+        )
+
+        yield self.store.set_profile_displayname(
+            self.u_frank.localpart, "Frank"
+        )
+
+        self.assertEquals(
+            "Frank",
+            (yield self.store.get_profile_displayname(self.u_frank.localpart))
+        )
+
+    @defer.inlineCallbacks
+    def test_avatar_url(self):
+        yield self.store.create_profile(
+            self.u_frank.localpart
+        )
+
+        yield self.store.set_profile_avatar_url(
+                self.u_frank.localpart, "http://my.site/here"
+        )
+
+        self.assertEquals(
+            "http://my.site/here",
+            (yield self.store.get_profile_avatar_url(self.u_frank.localpart))
+        )

tests/storage/test_redaction.py

@@ -0,0 +1,262 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.constants import Membership
+from synapse.api.events.room import (
+    RoomMemberEvent, MessageEvent, RoomRedactionEvent,
+)
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RedactionTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer(
+            "test",
+            db_pool=db_pool,
+        )
+
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory()
+
+        self.u_alice = hs.parse_userid("@alice:test")
+        self.u_bob = hs.parse_userid("@bob:test")
+
+        self.room1 = hs.parse_roomid("!abc123:test")
+
+        self.depth = 1
+
+    @defer.inlineCallbacks
+    def inject_room_member(self, room, user, membership, prev_state=None,
+                           extra_content={}):
+        self.depth += 1
+
+        event = self.event_factory.create_event(
+            etype=RoomMemberEvent.TYPE,
+            user_id=user.to_string(),
+            state_key=user.to_string(),
+            room_id=room.to_string(),
+            membership=membership,
+            content={"membership": membership},
+            depth=self.depth,
+        )
+
+        event.content.update(extra_content)
+
+        if prev_state:
+            event.prev_state = prev_state
+
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            event
+        )
+
+        defer.returnValue(event)
+
+    @defer.inlineCallbacks
+    def inject_message(self, room, user, body):
+        self.depth += 1
+
+        event = self.event_factory.create_event(
+            etype=MessageEvent.TYPE,
+            user_id=user.to_string(),
+            room_id=room.to_string(),
+            content={"body": body, "msgtype": u"message"},
+            depth=self.depth,
+        )
+
+        yield self.store.persist_event(
+            event
+        )
+
+        defer.returnValue(event)
+
+    @defer.inlineCallbacks
+    def inject_redaction(self, room, event_id, user, reason):
+        event = self.event_factory.create_event(
+            etype=RoomRedactionEvent.TYPE,
+            user_id=user.to_string(),
+            room_id=room.to_string(),
+            content={"reason": reason},
+            depth=self.depth,
+            redacts=event_id,
+        )
+
+        yield self.store.persist_event(
+            event
+        )
+
+        defer.returnValue(event)
+
+    @defer.inlineCallbacks
+    def test_redact(self):
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+
+        start = yield self.store.get_room_events_max_id()
+
+        msg_event = yield self.inject_message(self.room1, self.u_alice, u"t")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_alice.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        # Check event has not been redacted:
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": MessageEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {"body": "t", "msgtype": "message"},
+            },
+            event,
+        )
+
+        self.assertFalse(hasattr(event, "redacted_because"))
+
+        # Redact event
+        reason = "Because I said so"
+        yield self.inject_redaction(
+            self.room1, msg_event.event_id, self.u_alice, reason
+        )
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_alice.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        # Check redaction
+
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": MessageEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {},
+            },
+            event,
+        )
+
+        self.assertTrue(hasattr(event, "redacted_because"))
+
+        self.assertObjectHasAttributes(
+            {
+                "type": RoomRedactionEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {"reason": reason},
+            },
+            event.redacted_because,
+        )
+
+    @defer.inlineCallbacks
+    def test_redact_join(self):
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+
+        start = yield self.store.get_room_events_max_id()
+
+        msg_event = yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN,
+            extra_content={"blue": "red"},
+        )
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_alice.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        # Check event has not been redacted:
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": RoomMemberEvent.TYPE,
+                "user_id": self.u_bob.to_string(),
+                "content": {"membership": Membership.JOIN, "blue": "red"},
+            },
+            event,
+        )
+
+        self.assertFalse(hasattr(event, "redacted_because"))
+
+        # Redact event
+        reason = "Because I said so"
+        yield self.inject_redaction(
+            self.room1, msg_event.event_id, self.u_alice, reason
+        )
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_alice.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        # Check redaction
+
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": RoomMemberEvent.TYPE,
+                "user_id": self.u_bob.to_string(),
+                "content": {"membership": Membership.JOIN},
+            },
+            event,
+        )
+
+        self.assertTrue(hasattr(event, "redacted_because"))
+
+        self.assertObjectHasAttributes(
+            {
+                "type": RoomRedactionEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {"reason": reason},
+            },
+            event.redacted_because,
+        )

tests/storage/test_registration.py

@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.registration import RegistrationStore
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RegistrationStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        self.store = RegistrationStore(hs)
+
+        self.user_id = "@my-user:test"
+        self.tokens = ["AbCdEfGhIjKlMnOpQrStUvWxYz",
+                        "BcDeFgHiJkLmNoPqRsTuVwXyZa"]
+        self.pwhash = "{xx1}123456789"
+
+    @defer.inlineCallbacks
+    def test_register(self):
+        yield self.store.register(self.user_id, self.tokens[0], self.pwhash)
+
+        self.assertEquals(
+            # TODO(paul): Surely this field should be 'user_id', not 'name'
+            #  Additionally surely it shouldn't come in a 1-element list
+            [{"name": self.user_id, "password_hash": self.pwhash}],
+            (yield self.store.get_user_by_id(self.user_id))
+        )
+
+        self.assertEquals(
+            self.user_id,
+            (yield self.store.get_user_by_token(self.tokens[0]))
+        )
+
+    @defer.inlineCallbacks
+    def test_add_tokens(self):
+        yield self.store.register(self.user_id, self.tokens[0], self.pwhash)
+        yield self.store.add_access_token_to_user(self.user_id, self.tokens[1])
+
+        self.assertEquals(
+            self.user_id,
+            (yield self.store.get_user_by_token(self.tokens[1]))
+        )
+

tests/storage/test_room.py

@@ -0,0 +1,176 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.events.room import (
+    RoomNameEvent, RoomTopicEvent
+)
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RoomStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        # We can't test RoomStore on its own without the DirectoryStore, for
+        # management of the 'room_aliases' table
+        self.store = hs.get_datastore()
+
+        self.room = hs.parse_roomid("!abcde:test")
+        self.alias = hs.parse_roomalias("#a-room-name:test")
+        self.u_creator = hs.parse_userid("@creator:test")
+
+        yield self.store.store_room(self.room.to_string(),
+            room_creator_user_id=self.u_creator.to_string(),
+            is_public=True
+        )
+
+    @defer.inlineCallbacks
+    def test_get_room(self):
+        self.assertObjectHasAttributes(
+            {"room_id": self.room.to_string(),
+             "creator": self.u_creator.to_string(),
+             "is_public": True},
+            (yield self.store.get_room(self.room.to_string()))
+        )
+
+    @defer.inlineCallbacks
+    def test_store_room_config(self):
+        yield self.store.store_room_config(self.room.to_string(),
+            visibility=False
+        )
+
+        self.assertObjectHasAttributes(
+            {"is_public": False},
+            (yield self.store.get_room(self.room.to_string()))
+        )
+
+    @defer.inlineCallbacks
+    def test_get_rooms(self):
+        # get_rooms does an INNER JOIN on the room_aliases table :(
+
+        rooms = yield self.store.get_rooms(is_public=True)
+        # Should be empty before we add the alias
+        self.assertEquals([], rooms)
+
+        yield self.store.create_room_alias_association(
+            room_alias=self.alias,
+            room_id=self.room.to_string(),
+            servers=["test"]
+        )
+
+        rooms = yield self.store.get_rooms(is_public=True)
+
+        self.assertEquals(1, len(rooms))
+        self.assertEquals({
+            "name": None,
+            "room_id": self.room.to_string(),
+            "topic": None,
+            "aliases": [self.alias.to_string()],
+        }, rooms[0])
+
+
+class RoomEventsStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        # Room events need the full datastore, for persist_event() and
+        # get_room_state()
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory();
+
+        self.room = hs.parse_roomid("!abcde:test")
+
+        yield self.store.store_room(self.room.to_string(),
+            room_creator_user_id="@creator:text",
+            is_public=True
+        )
+
+    @defer.inlineCallbacks
+    def inject_room_event(self, **kwargs):
+        yield self.store.persist_event(
+            self.event_factory.create_event(
+                room_id=self.room.to_string(),
+                **kwargs
+            )
+        )
+
+    @defer.inlineCallbacks
+    def test_room_name(self):
+        name = u"A-Room-Name"
+
+        yield self.inject_room_event(
+            etype=RoomNameEvent.TYPE,
+            name=name,
+            content={"name": name},
+            depth=1,
+        )
+
+        state = yield self.store.get_current_state(
+            room_id=self.room.to_string()
+        )
+
+        self.assertEquals(1, len(state))
+        self.assertObjectHasAttributes(
+            {"type": "m.room.name",
+             "room_id": self.room.to_string(),
+             "name": name},
+            state[0]
+        )
+
+    @defer.inlineCallbacks
+    def test_room_name(self):
+        topic = u"A place for things"
+
+        yield self.inject_room_event(
+            etype=RoomTopicEvent.TYPE,
+            topic=topic,
+            content={"topic": topic},
+            depth=1,
+        )
+
+        state = yield self.store.get_current_state(
+            room_id=self.room.to_string()
+        )
+
+        self.assertEquals(1, len(state))
+        self.assertObjectHasAttributes(
+            {"type": "m.room.topic",
+             "room_id": self.room.to_string(),
+             "topic": topic},
+            state[0]
+        )
+
+    # Not testing the various 'level' methods for now because there's lots
+    # of them and need coalescing; see JIRA SPEC-11

tests/storage/test_roommember.py

@@ -0,0 +1,157 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.constants import Membership
+from synapse.api.events.room import RoomMemberEvent
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RoomMemberStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        # We can't test the RoomMemberStore on its own without the other event
+        # storage logic
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory()
+
+        self.u_alice = hs.parse_userid("@alice:test")
+        self.u_bob = hs.parse_userid("@bob:test")
+
+        # User elsewhere on another host
+        self.u_charlie = hs.parse_userid("@charlie:elsewhere")
+
+        self.room = hs.parse_roomid("!abc123:test")
+
+    @defer.inlineCallbacks
+    def inject_room_member(self, room, user, membership):
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            self.event_factory.create_event(
+                etype=RoomMemberEvent.TYPE,
+                user_id=user.to_string(),
+                state_key=user.to_string(),
+                room_id=room.to_string(),
+                membership=membership,
+                content={"membership": membership},
+                depth=1,
+            )
+        )
+
+    @defer.inlineCallbacks
+    def test_one_member(self):
+        yield self.inject_room_member(self.room, self.u_alice, Membership.JOIN)
+
+        self.assertEquals(
+            Membership.JOIN,
+            (yield self.store.get_room_member(
+                user_id=self.u_alice.to_string(),
+                room_id=self.room.to_string(),
+            )).membership
+        )
+        self.assertEquals(
+            [self.u_alice.to_string()],
+            [m.user_id for m in (
+                yield self.store.get_room_members(self.room.to_string())
+            )]
+        )
+        self.assertEquals(
+            [self.room.to_string()],
+            [m.room_id for m in (
+                yield self.store.get_rooms_for_user_where_membership_is(
+                    self.u_alice.to_string(), [Membership.JOIN]
+                ))
+            ]
+        )
+        self.assertFalse(
+            (yield self.store.user_rooms_intersect(
+                [self.u_alice.to_string(), self.u_bob.to_string()]
+            ))
+        )
+
+    @defer.inlineCallbacks
+    def test_two_members(self):
+        yield self.inject_room_member(self.room, self.u_alice, Membership.JOIN)
+        yield self.inject_room_member(self.room, self.u_bob, Membership.JOIN)
+
+        self.assertEquals(
+            {self.u_alice.to_string(), self.u_bob.to_string()},
+            {m.user_id for m in (
+                yield self.store.get_room_members(self.room.to_string())
+            )}
+        )
+        self.assertTrue(
+            (yield self.store.user_rooms_intersect(
+                [self.u_alice.to_string(), self.u_bob.to_string()]
+            ))
+        )
+
+    @defer.inlineCallbacks
+    def test_room_hosts(self):
+        yield self.inject_room_member(self.room, self.u_alice, Membership.JOIN)
+
+        self.assertEquals(
+            ["test"],
+            (yield self.store.get_joined_hosts_for_room(self.room.to_string()))
+        )
+
+        # Should still have just one host after second join from it
+        yield self.inject_room_member(self.room, self.u_bob, Membership.JOIN)
+
+        self.assertEquals(
+            ["test"],
+            (yield self.store.get_joined_hosts_for_room(self.room.to_string()))
+        )
+
+        # Should now have two hosts after join from other host
+        yield self.inject_room_member(self.room, self.u_charlie, Membership.JOIN)
+
+        self.assertEquals(
+            {"test", "elsewhere"},
+            set((yield
+                self.store.get_joined_hosts_for_room(self.room.to_string())
+            ))
+        )
+
+        # Should still have both hosts
+        yield self.inject_room_member(self.room, self.u_alice, Membership.LEAVE)
+
+        self.assertEquals(
+            {"test", "elsewhere"},
+            set((yield
+                self.store.get_joined_hosts_for_room(self.room.to_string())
+            ))
+        )
+
+        # Should have only one host after other leaves
+        yield self.inject_room_member(self.room, self.u_charlie, Membership.LEAVE)
+
+        self.assertEquals(
+            ["test"],
+            (yield self.store.get_joined_hosts_for_room(self.room.to_string()))
+        )

tests/storage/test_stream.py

@@ -0,0 +1,226 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.constants import Membership
+from synapse.api.events.room import RoomMemberEvent, MessageEvent
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class StreamStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer(
+            "test",
+            db_pool=db_pool,
+        )
+
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory()
+
+        self.u_alice = hs.parse_userid("@alice:test")
+        self.u_bob = hs.parse_userid("@bob:test")
+
+        self.room1 = hs.parse_roomid("!abc123:test")
+        self.room2 = hs.parse_roomid("!xyx987:test")
+
+        self.depth = 1
+
+    @defer.inlineCallbacks
+    def inject_room_member(self, room, user, membership, prev_state=None):
+        self.depth += 1
+
+        event = self.event_factory.create_event(
+            etype=RoomMemberEvent.TYPE,
+            user_id=user.to_string(),
+            state_key=user.to_string(),
+            room_id=room.to_string(),
+            membership=membership,
+            content={"membership": membership},
+            depth=self.depth,
+        )
+
+        if prev_state:
+            event.prev_state = prev_state
+
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            event
+        )
+
+        defer.returnValue(event)
+
+    @defer.inlineCallbacks
+    def inject_message(self, room, user, body):
+        self.depth += 1
+
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            self.event_factory.create_event(
+                etype=MessageEvent.TYPE,
+                user_id=user.to_string(),
+                room_id=room.to_string(),
+                content={"body": body, "msgtype": u"message"},
+                depth=self.depth,
+            )
+        )
+
+    @defer.inlineCallbacks
+    def test_event_stream_get_other(self):
+        # Both bob and alice joins the room
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        # Initial stream key:
+        start = yield self.store.get_room_events_max_id()
+
+        yield self.inject_message(self.room1, self.u_alice, u"test")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_bob.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": MessageEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {"body": "test", "msgtype": "message"},
+            },
+            event,
+        )
+
+    @defer.inlineCallbacks
+    def test_event_stream_get_own(self):
+        # Both bob and alice joins the room
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        # Initial stream key:
+        start = yield self.store.get_room_events_max_id()
+
+        yield self.inject_message(self.room1, self.u_alice, u"test")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_alice.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": MessageEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {"body": "test", "msgtype": "message"},
+            },
+            event,
+        )
+
+    @defer.inlineCallbacks
+    def test_event_stream_join_leave(self):
+        # Both bob and alice joins the room
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        # Then bob leaves again.
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.LEAVE
+        )
+
+        # Initial stream key:
+        start = yield self.store.get_room_events_max_id()
+
+        yield self.inject_message(self.room1, self.u_alice, u"test")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_bob.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        # We should not get the message, as it happened *after* bob left.
+        self.assertEqual(0, len(results))
+
+    @defer.inlineCallbacks
+    def test_event_stream_prev_content(self):
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        event1 = yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+
+        start = yield self.store.get_room_events_max_id()
+
+        event2 = yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN,
+            prev_state=event1.event_id,
+        )
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_bob.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        # We should not get the message, as it happened *after* bob left.
+        self.assertEqual(1, len(results))
+
+        event = results[0]
+
+        self.assertTrue(hasattr(event, "prev_content"), msg="No prev_content key")

tests/unittest.py

@@ -71,6 +71,17 @@ class TestCase(unittest.TestCase):
             logging.getLogger().setLevel(level)
             return orig()
 
+    def assertObjectHasAttributes(self, attrs, obj):
+        """Asserts that the given object has each of the attributes given, and
+        that the value of each matches according to assertEquals."""
+        for (key, value) in attrs.items():
+            if not hasattr(obj, key):
+                raise AssertionError("Expected obj to have a '.%s'" % key)
+            try:
+                self.assertEquals(attrs[key], getattr(obj, key))
+            except AssertionError as e:
+                raise (type(e))(e.message + " for '.%s'" % key)
+
 
 def DEBUG(target):
     """A decorator to set the .loglevel attribute to logging.DEBUG.

tests/utils.py

@@ -16,12 +16,14 @@
 from synapse.http.server import HttpServer
 from synapse.api.errors import cs_error, CodeMessageException, StoreError
 from synapse.api.constants import Membership
+from synapse.storage import prepare_database
 
 from synapse.api.events.room import (
     RoomMemberEvent, MessageEvent
 )
 
 from twisted.internet import defer, reactor
+from twisted.enterprise.adbapi import ConnectionPool
 
 from collections import namedtuple
 from mock import patch, Mock
@@ -120,6 +122,18 @@ class MockClock(object):
         self.now += secs
 
 
+class SQLiteMemoryDbPool(ConnectionPool, object):
+    def __init__(self):
+        super(SQLiteMemoryDbPool, self).__init__(
+            "sqlite3", ":memory:",
+            cp_min=1,
+            cp_max=1,
+        )
+
+    def prepare(self):
+        return self.runWithConnection(prepare_database)
+
+
 class MemoryDataStore(object):
 
     Room = namedtuple(
@@ -248,7 +262,7 @@ class MemoryDataStore(object):
         return defer.succeed("invite")
 
     def get_ops_levels(self, room_id):
-        return defer.succeed((5, 5))
+        return defer.succeed((5, 5, 5))
 
 
 def _format_call(args, kwargs):

webclient/README

@@ -1,8 +1,8 @@
 Basic Usage
 -----------
 
-The web client should automatically run when running the home server. Alternatively, you can run
-it stand-alone:
+The web client should automatically run when running the home server. 
+Alternatively, you can run it stand-alone:
 
     $ python -m SimpleHTTPServer
 

webclient/app-controller.js

@@ -26,6 +26,12 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
          
     // Check current URL to avoid to display the logout button on the login page
     $scope.location = $location.path();
+
+    // disable nganimate for the local and remote video elements because ngAnimate appears
+    // to be buggy and leaves animation classes on the video elements causing them to show
+    // when they should not (their animations are pure CSS3)
+    $animate.enabled(false, angular.element('#localVideo'));
+    $animate.enabled(false, angular.element('#remoteVideo'));
     
     // Update the location state when the ng location changed
     $rootScope.$on('$routeChangeSuccess', function (event, current, previous) {
@@ -61,6 +67,16 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
         }
     };
     
+    $scope.leave = function(room_id) {
+        matrixService.leave(room_id).then(
+            function(response) {
+                console.log("Left room " + room_id);
+            },
+            function(error) {
+                console.log("Failed to leave room " + room_id + ": " + error.data.error);
+            });
+    };
+    
     // Logs the user out 
     $scope.logout = function() {
         
@@ -93,7 +109,13 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
     };
 
     $rootScope.$watch('currentCall', function(newVal, oldVal) {
-        if (!$rootScope.currentCall) return;
+        if (!$rootScope.currentCall) {
+            // This causes the still frame to be flushed out of the video elements,
+            // avoiding a flash of the last frame of the previous call when starting the next
+            angular.element('#localVideo')[0].load();
+            angular.element('#remoteVideo')[0].load();
+            return;
+        }
 
         var roomMembers = angular.copy($rootScope.events.rooms[$rootScope.currentCall.room_id].members);
         delete roomMembers[matrixService.config().user_id];
@@ -126,6 +148,7 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
             angular.element('#ringAudio')[0].pause();
             angular.element('#ringbackAudio')[0].pause();
             angular.element('#callendAudio')[0].play();
+            $scope.videoMode = undefined;
         } else if (newVal == 'ended' && oldVal == 'invite_sent' && $rootScope.currentCall.hangupParty == 'remote') {
             angular.element('#ringAudio')[0].pause();
             angular.element('#ringbackAudio')[0].pause();
@@ -138,6 +161,20 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
             angular.element('#ringbackAudio')[0].pause();
         } else if (oldVal == 'ringing') {
             angular.element('#ringAudio')[0].pause();
+        } else if (newVal == 'connected') {
+            $timeout(function() {
+                if ($scope.currentCall.type == 'video') $scope.videoMode = 'large';
+            }, 500);
+        }
+
+        if ($rootScope.currentCall && $rootScope.currentCall.type == 'video' && $rootScope.currentCall.state != 'connected') {
+            $scope.videoMode = 'mini';
+        }
+    });
+    $rootScope.$watch('currentCall.type', function(newVal, oldVal) {
+        // need to listen for this too as the type of the call won't be know when it's created
+        if ($rootScope.currentCall && $rootScope.currentCall.type == 'video' && $rootScope.currentCall.state != 'connected') {
+            $scope.videoMode = 'mini';
         }
     });
 
@@ -150,6 +187,8 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
         }
         call.onError = $scope.onCallError;
         call.onHangup = $scope.onCallHangup;
+        call.localVideoElement = angular.element('#localVideo')[0];
+        call.remoteVideoElement = angular.element('#remoteVideo')[0];
         $rootScope.currentCall = call;
     });
 
@@ -170,7 +209,7 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
     
     $rootScope.onCallError = function(errStr) {
         $scope.feedback = errStr;
-    }
+    };
 
     $rootScope.onCallHangup = function(call) {
         if (call == $rootScope.currentCall) {
@@ -178,5 +217,5 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
                 if (call == $rootScope.currentCall) $rootScope.currentCall = undefined;
             }, 4070);
         }
-    }
+    };
 }]);

webclient/app-filter.js

@@ -45,32 +45,33 @@ angular.module('matrixWebClient')
         angular.forEach(members, function(value, key) {
             value["id"] = key;
             filtered.push( value );
-            if (value["displayname"]) {
-                if (!displayNames[value["displayname"]]) {
-                    displayNames[value["displayname"]] = [];
-                }
-                displayNames[value["displayname"]].push(key);
-            }
-        });
-
-        // FIXME: we shouldn't disambiguate displayNames on every orderMembersList
-        // invocation but keep track of duplicates incrementally somewhere
-        angular.forEach(displayNames, function(value, key) {
-            if (value.length > 1) {
-                // console.log(key + ": " + value);
-                for (var i=0; i < value.length; i++) {
-                    var v = value[i];
-                    // FIXME: this permenantly rewrites the displayname for a given
-                    // room member. which means we can't reset their name if it is
-                    // no longer ambiguous!
-                    members[v].displayname += " (" + v + ")";
-                    // console.log(v + " " + members[v]);
-                };
-            }
         });
 
         filtered.sort(function (a, b) {
-            return ((a["last_active_ago"] || 10e10) > (b["last_active_ago"] || 10e10) ? 1 : -1);
+            // Sort members on their last_active absolute time
+            var aLastActiveTS = 0, bLastActiveTS = 0;
+            if (undefined !== a.last_active_ago) {
+                aLastActiveTS = a.last_updated - a.last_active_ago;
+            }
+            if (undefined !== b.last_active_ago) {
+                bLastActiveTS = b.last_updated - b.last_active_ago;
+            }
+            if (aLastActiveTS || bLastActiveTS) {
+                return bLastActiveTS - aLastActiveTS;
+            }
+            else {
+                // If they do not have last_active_ago, sort them according to their presence state
+                // Online users go first amongs members who do not have last_active_ago
+                var presenceLevels = {
+                    offline: 1,
+                    unavailable: 2,
+                    online: 4,
+                    free_for_chat: 3
+                };
+                var aPresence = (a.presence in presenceLevels) ? presenceLevels[a.presence] : 0;
+                var bPresence = (b.presence in presenceLevels) ? presenceLevels[b.presence] : 0;
+                return bPresence - aPresence;
+            }
         });
         return filtered;
     };

webclient/app.css

@@ -20,7 +20,12 @@ a:visited { color: #666; }
 a:hover   { color: #000; }
 a:active  { color: #000; }
 
-#page {
+textarea, input {
+   font-family: inherit;
+   font-size: inherit;
+}
+
+.page {
     min-height: 100%;
     margin-bottom: -32px; /* to make room for the footer */
 }
@@ -34,9 +39,15 @@ a:active  { color: #000; }
     padding-right: 20px;
 }
 
+#unsupportedBrowser {
+    padding-top: 240px;
+    text-align: center;
+}
+
 #header
 {
     position: absolute;
+    z-index: 2;
     top: 0px;
     width: 100%;
     background-color: #333;
@@ -89,6 +100,80 @@ a:active  { color: #000; }
     font-size: 80%;
 }
 
+#videoBackground {
+    position: absolute;
+    height: 100%;
+    width: 100%;
+    top: 0px;
+    left: 0px;
+    z-index: 1;
+    background-color: rgba(0,0,0,0.0);
+    pointer-events: none;
+    transition: background-color linear 500ms;
+}
+
+#videoBackground.large {
+    background-color: rgba(0,0,0,0.85);
+    pointer-events: auto;
+}
+
+#videoContainer {
+    position: relative;
+    top: 32px;
+    max-width: 1280px;
+    margin: auto;
+}
+
+#videoContainerPadding {
+    width: 1280px;
+}
+
+#localVideo {
+    position: absolute;
+    width: 128px;
+    height: 72px;
+    z-index: 1;
+    transition: left linear 500ms, top linear 500ms, width linear 500ms, height linear 500ms;
+}
+
+#localVideo.mini {
+    top: 0px;
+    left: 130px;
+}
+
+#localVideo.large {
+    top: 70px;
+    left: 20px;
+}
+
+#localVideo.ended {
+    -webkit-filter: grayscale(1);
+    filter: grayscale(1);
+}
+
+#remoteVideo {
+    position: relative;
+    height: auto;
+    transition: left linear 500ms, top linear 500ms, width linear 500ms, height linear 500ms;
+}
+
+#remoteVideo.mini {
+    left: 260px;
+    top: 0px;
+    width: 128px;
+}
+
+#remoteVideo.large {
+    left: 0px;
+    top: 50px;
+    width: 100%;
+}
+
+#remoteVideo.ended {
+    -webkit-filter: grayscale(1);
+    filter: grayscale(1);
+}
+
 #headerContent {
     color: #ccc;
     max-width: 1280px;
@@ -96,6 +181,7 @@ a:active  { color: #000; }
     text-align: right;
     height: 32px;
     line-height: 32px;
+    position: relative;
 }
 
 #headerContent a:link,

webclient/app.js

@@ -80,7 +80,24 @@ matrixWebClient.config(['$routeProvider', '$provide', '$httpProvider',
         $httpProvider.interceptors.push('AccessTokenInterceptor');
     }]);
 
-matrixWebClient.run(['$location', 'matrixService', function($location, matrixService) {
+matrixWebClient.run(['$location', '$rootScope', 'matrixService', function($location, $rootScope, matrixService) {
+
+    // Check browser support
+    // Support IE from 9.0. AngularJS needs some tricks to run on IE8 and below
+    var version = parseFloat($.browser.version);
+    if ($.browser.msie && version < 9.0) {
+        $rootScope.unsupportedBrowser = {
+            browser: navigator.userAgent,
+            reason: "Internet Explorer is supported from version 9"
+        };
+    }
+    // The app requires localStorage
+    if(typeof(Storage) === "undefined") {
+        $rootScope.unsupportedBrowser = {
+            browser: navigator.userAgent,
+            reason: "It does not support HTML local storage"
+        };
+    }
 
     // If user auth details are not in cache, go to the login page
     if (!matrixService.isUserLoggedIn() &&

webclient/components/fileInput/file-input-directive.js

@@ -31,13 +31,23 @@ angular.module('mFileInput', [])
         },
 
         link: function(scope, element, attrs, ctrl) {
-            element.bind("click", function() {
-                element.find("input")[0].click();
-                element.find("input").bind("change", function(e) {
-                    scope.selectedFile = this.files[0];
-                    scope.$apply();
+            
+            // Check if HTML5 file selection is supported
+            if (window.FileList) {
+                element.bind("click", function() {
+                    element.find("input")[0].click();
+                    element.find("input").bind("change", function(e) {
+                        scope.selectedFile = this.files[0];
+                        scope.$apply();
+                    });
                 });
-            });
+            }
+            else {
+                setTimeout(function() {
+                    element.attr("disabled", true);
+                    element.attr("title", "The app uses the HTML5 File API to send files. Your browser does not support it.");
+                }, 1);
+            }
 
             // Change the mouse icon on mouseover on this element
             element.css("cursor", "pointer");

webclient/components/matrix/event-handler-service.js

@@ -99,9 +99,9 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
     };
     reset();
 
-    var initRoom = function(room_id) {
+    var initRoom = function(room_id, room) {
         if (!(room_id in $rootScope.events.rooms)) {
-            console.log("Creating new handler entry for " + room_id);
+            console.log("Creating new rooms entry for " + room_id);
             $rootScope.events.rooms[room_id] = {
                 room_id: room_id,
                 messages: [],
@@ -112,6 +112,18 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
                 }
             };
         }
+
+        if (room) { // we got an existing room object from initialsync, seemingly.
+            // Report all other metadata of the room object (membership, inviter, visibility, ...)
+            for (var field in room) {
+                if (!room.hasOwnProperty(field)) continue;
+
+                if (-1 === ["room_id", "messages", "state"].indexOf(field)) { // why indexOf - why not ===? --Matthew
+                    $rootScope.events.rooms[room_id][field] = room[field];
+                }
+            }
+            $rootScope.events.rooms[room_id].membership = room.membership;
+        }
     };
 
     var resetRoomMessages = function(room_id) {
@@ -179,27 +191,30 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
             
             if (window.Notification && event.user_id != matrixService.config().user_id) {
                 var shouldBing = $rootScope.containsBingWord(event.content.body);
-            
-                // TODO: Binging every message when idle doesn't make much sense. Can we use this more sensibly?
-                // Unfortunately document.hidden = false on ubuntu chrome if chrome is minimised / does not have focus;
-                // true when you swap tabs though. However, for the case where the chat screen is OPEN and there is
-                // another window on top, we want to be notifying for those events. This DOES mean that there will be
-                // notifications when currently viewing the chat screen though, but that is preferable to the alternative imo.
+
+                // Ideally we would notify only when the window is hidden (i.e. document.hidden = true).
+                //
+                // However, Chrome on Linux and OSX currently returns document.hidden = false unless the window is
+                // explicitly showing a different tab.  So we need another metric to determine hiddenness - we
+                // simply use idle time.  If the user has been idle enough that their presence goes to idle, then
+                // we also display notifs when things happen.
+                //
+                // This is far far better than notifying whenever anything happens anyway, otherwise you get spammed
+                // to death with notifications when the window is in the foreground, which is horrible UX (especially
+                // if you have not defined any bingers and so get notified for everything).
                 var isIdle = (document.hidden || matrixService.presence.unavailable === mPresence.getState());
                 
-                // always bing if there are 0 bing words... apparently.
+                // We need a way to let people get notifications for everything, if they so desire.  The way to do this
+                // is to specify zero bingwords.
                 var bingWords = matrixService.config().bingWords;
                 if (bingWords === undefined || bingWords.length === 0) {
                     shouldBing = true;
                 }
                 
-                if (shouldBing) {
+                if (shouldBing && isIdle) {
                     console.log("Displaying notification for "+JSON.stringify(event));
-                    var member = $rootScope.events.rooms[event.room_id].members[event.user_id];
-                    var displayname = undefined;
-                    if (member) {
-                        displayname = member.displayname;
-                    }
+                    var member = getMember(event.room_id, event.user_id);
+                    var displayname = getUserDisplayName(event.room_id, event.user_id);
 
                     var message = event.content.body;
                     if (event.content.msgtype === "m.emote") {
@@ -207,7 +222,7 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
                     }
 
                     var notification = new window.Notification(
-                        (displayname || event.user_id) +
+                        displayname +
                         " (" + (matrixService.getRoomIdToAliasMapping(event.room_id) || event.room_id) + ")", // FIXME: don't leak room_ids here
                     {
                         "body": message,
@@ -237,12 +252,29 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
         // Exception: Do not do this if the event is a room state event because such events already come
         // as room messages events. Moreover, when they come as room messages events, they are relatively ordered
         // with other other room messages
-        if (event.content.prev !== event.content.membership && !isStateEvent) {
-            if (isLiveEvent) {
-                $rootScope.events.rooms[event.room_id].messages.push(event);
+        if (!isStateEvent) {
+            // could be a membership change, display name change, etc.
+            // Find out which one.
+            var memberChanges = undefined;
+            if (event.content.prev !== event.content.membership) {
+                memberChanges = "membership";
             }
-            else {
-                $rootScope.events.rooms[event.room_id].messages.unshift(event);
+            else if (event.prev_content && (event.prev_content.displayname !== event.content.displayname)) {
+                memberChanges = "displayname";
+            }
+
+            // mark the key which changed
+            event.changedKey = memberChanges;
+
+            // If there was a change we want to display, dump it in the message
+            // list.
+            if (memberChanges) {
+                if (isLiveEvent) {
+                    $rootScope.events.rooms[event.room_id].messages.push(event);
+                }
+                else {
+                    $rootScope.events.rooms[event.room_id].messages.unshift(event);
+                }
             }
         }
         
@@ -312,6 +344,65 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
         return index;
     };
     
+    /**
+     * Get the member object of a room member
+     * @param {String} room_id the room id
+     * @param {String} user_id the id of the user
+     * @returns {undefined | Object} the member object of this user in this room if he is part of the room
+     */
+    var getMember = function(room_id, user_id) {
+        var member;
+
+        var room = $rootScope.events.rooms[room_id];
+        if (room) {
+            member = room.members[user_id];
+        }
+        return member;
+    };
+
+    /**
+     * Return the display name of an user acccording to data already downloaded
+     * @param {String} room_id the room id
+     * @param {String} user_id the id of the user
+     * @returns {String} the user displayname or user_id if not available
+     */
+    var getUserDisplayName = function(room_id, user_id) {
+        var displayName;
+
+        // Get the user display name from the member list of the room
+        var member = getMember(room_id, user_id);
+        if (member && member.content.displayname) { // Do not consider null displayname
+            displayName = member.content.displayname;
+
+            // Disambiguate users who have the same displayname in the room
+            if (user_id !== matrixService.config().user_id) {
+                var room = $rootScope.events.rooms[room_id];
+
+                for (var member_id in room.members) {
+                    if (room.members.hasOwnProperty(member_id) && member_id !== user_id) {
+                        var member2 = room.members[member_id];
+                        if (member2.content.displayname && member2.content.displayname === displayName) {
+                            displayName = displayName + " (" + user_id + ")";
+                            break;
+                        }
+                    }
+                }
+            }
+        }
+
+        // The user may not have joined the room yet. So try to resolve display name from presence data
+        // Note: This data may not be available
+        if (undefined === displayName && user_id in $rootScope.presence) {
+            displayName = $rootScope.presence[user_id].content.displayname;
+        }
+
+        if (undefined === displayName) {
+            // By default, use the user ID
+            displayName = user_id;
+        }
+        return displayName;
+    };
+
     return {
         ROOM_CREATE_EVENT: ROOM_CREATE_EVENT,
         MSG_EVENT: MSG_EVENT,
@@ -327,6 +418,10 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
             reset();
             $rootScope.$broadcast(RESET_EVENT);
         },
+        
+        initRoom: function(room) {
+            initRoom(room.room_id, room);
+        },
     
         handleEvent: function(event, isLiveEvent, isStateEvent) {
 
@@ -479,6 +574,8 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
                 memberCount = 0;
 
                 for (var i in room.members) {
+                    if (!room.members.hasOwnProperty(i)) continue;
+
                     var member = room.members[i];
 
                     if ("join" === member.membership) {
@@ -497,15 +594,19 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
          * @returns {undefined | Object} the member object of this user in this room if he is part of the room
          */
         getMember: function(room_id, user_id) {
-            var member;
-            
-            var room = $rootScope.events.rooms[room_id];
-            if (room) {
-                member = room.members[user_id];
-            }
-            return member;
+            return getMember(room_id, user_id);
         },
         
+        /**
+         * Return the display name of an user acccording to data already downloaded
+         * @param {String} room_id the room id
+         * @param {String} user_id the id of the user
+         * @returns {String} the user displayname or user_id if not available
+         */
+        getUserDisplayName: function(room_id, user_id) {
+            return getUserDisplayName(room_id, user_id);
+        },
+
         setRoomVisibility: function(room_id, visible) {
             if (!visible) {
                 return;

webclient/components/matrix/event-stream-service.js

@@ -112,6 +112,8 @@ angular.module('eventStreamService', [])
                 var rooms = response.data.rooms;
                 for (var i = 0; i < rooms.length; ++i) {
                     var room = rooms[i];
+                    
+                    eventHandlerService.initRoom(room);
 
                     if ("messages" in room) {
                         eventHandlerService.handleRoomMessages(room.room_id, room.messages, false);
@@ -120,8 +122,6 @@ angular.module('eventStreamService', [])
                     if ("state" in room) {
                         eventHandlerService.handleEvents(room.state, false, true);
                     }
-                    
-                    eventHandlerService.setRoomVisibility(room.room_id, room.visibility);
                 }
 
                 var presence = response.data.presence;

webclient/components/matrix/matrix-call.js

@@ -40,8 +40,15 @@ window.RTCPeerConnection = window.RTCPeerConnection || window.webkitRTCPeerConne
 window.RTCSessionDescription = window.RTCSessionDescription || window.webkitRTCSessionDescription || window.mozRTCSessionDescription;
 window.RTCIceCandidate = window.RTCIceCandidate || window.webkitRTCIceCandidate || window.mozRTCIceCandidate;
 
+// Returns true if the browser supports all required features to make WebRTC call
+var isWebRTCSupported = function () {
+    return !!(navigator.getUserMedia || window.RTCPeerConnection || window.RTCSessionDescription || window.RTCIceCandidate);
+};
+
 angular.module('MatrixCall', [])
 .factory('MatrixCall', ['matrixService', 'matrixPhoneService', '$rootScope', '$timeout', function MatrixCallFactory(matrixService, matrixPhoneService, $rootScope, $timeout) {
+    $rootScope.isWebRTCSupported = isWebRTCSupported();
+
     var MatrixCall = function(room_id) {
         this.room_id = room_id;
         this.call_id = "c" + new Date().getTime();
@@ -51,17 +58,75 @@ angular.module('MatrixCall', [])
         // a queue for candidates waiting to go out. We try to amalgamate candidates into a single candidate message where possible
         this.candidateSendQueue = [];
         this.candidateSendTries = 0;
+
+        var self = this;
+        $rootScope.$watch(this.remoteVideoElement, function (oldValue, newValue) {
+            self.tryPlayRemoteStream();
+        });
+
     }
 
+    MatrixCall.getTurnServer = function() {
+        matrixService.getTurnServer().then(function(response) {
+            if (response.data.uris) {
+                console.log("Got TURN URIs: "+response.data.uris);
+                MatrixCall.turnServer = response.data;
+                $rootScope.haveTurn = true;
+                // re-fetch when we're about to reach the TTL
+                $timeout(MatrixCall.getTurnServer, MatrixCall.turnServer.ttl * 1000 * 0.9);
+            } else {
+                console.log("Got no TURN URIs from HS");
+                $rootScope.haveTurn = false;
+            }
+        }, function(error) {
+            console.log("Failed to get TURN URIs");
+            MatrixCall.turnServer = {};
+            $timeout(MatrixCall.getTurnServer, 60000);
+        });
+    }
+
+    // FIXME: we should prevent any class from being placed or accepted before this has finished
+    MatrixCall.getTurnServer();
+
     MatrixCall.CALL_TIMEOUT = 60000;
+    MatrixCall.FALLBACK_STUN_SERVER = 'stun:stun.l.google.com:19302';
 
     MatrixCall.prototype.createPeerConnection = function() {
-        var stunServer = 'stun:stun.l.google.com:19302';
         var pc;
         if (window.mozRTCPeerConnection) {
-            pc = new window.mozRTCPeerConnection({'url': stunServer});
+            var iceServers = [];
+            if (MatrixCall.turnServer) {
+                if (MatrixCall.turnServer.uris) {
+                    for (var i = 0; i < MatrixCall.turnServer.uris.length; i++) {
+                        iceServers.push({
+                            'url': MatrixCall.turnServer.uris[i],
+                            'username': MatrixCall.turnServer.username,
+                            'credential': MatrixCall.turnServer.password,
+                        });
+                    }
+                } else {
+                    console.log("No TURN server: using fallback STUN server");
+                    iceServers.push({ 'url' : MatrixCall.FALLBACK_STUN_SERVER });
+                }
+            }
+          
+            pc = new window.mozRTCPeerConnection({"iceServers":iceServers});
         } else {
-            pc = new window.RTCPeerConnection({"iceServers":[{"urls":"stun:stun.l.google.com:19302"}]});
+            var iceServers = [];
+            if (MatrixCall.turnServer) {
+                if (MatrixCall.turnServer.uris) {
+                    iceServers.push({
+                        'urls': MatrixCall.turnServer.uris,
+                        'username': MatrixCall.turnServer.username,
+                        'credential': MatrixCall.turnServer.password,
+                    });
+                } else {
+                    console.log("No TURN server: using fallback STUN server");
+                    iceServers.push({ 'urls' : MatrixCall.FALLBACK_STUN_SERVER });
+                }
+            }
+          
+            pc = new window.RTCPeerConnection({"iceServers":iceServers});
         }
         var self = this;
         pc.oniceconnectionstatechange = function() { self.onIceConnectionStateChanged(); };
@@ -71,13 +136,39 @@ angular.module('MatrixCall', [])
         return pc;
     }
 
-    MatrixCall.prototype.placeCall = function(config) {
+    MatrixCall.prototype.getUserMediaVideoContraints = function(callType) {
+        switch (callType) {
+            case 'voice':
+                return ({audio: true, video: false});
+            case 'video':
+                return ({audio: true, video: {
+                    mandatory: {
+                        minWidth: 640,
+                        maxWidth: 640,
+                        minHeight: 360,
+                        maxHeight: 360,
+                    }
+                }});
+        }
+    };
+
+    MatrixCall.prototype.placeVoiceCall = function() {
+        this.placeCallWithConstraints(this.getUserMediaVideoContraints('voice'));
+        this.type = 'voice';
+    };
+
+    MatrixCall.prototype.placeVideoCall = function(config) {
+        this.placeCallWithConstraints(this.getUserMediaVideoContraints('video'));
+        this.type = 'video';
+    };
+
+    MatrixCall.prototype.placeCallWithConstraints = function(constraints) {
         var self = this;
         matrixPhoneService.callPlaced(this);
-        navigator.getUserMedia({audio: config.audio, video: config.video}, function(s) { self.gotUserMediaForInvite(s); }, function(e) { self.getUserMediaFailed(e); });
+        navigator.getUserMedia(constraints, function(s) { self.gotUserMediaForInvite(s); }, function(e) { self.getUserMediaFailed(e); });
         this.state = 'wait_local_media';
         this.direction = 'outbound';
-        this.config = config;
+        this.config = constraints;
     };
 
     MatrixCall.prototype.initWithInvite = function(event) {
@@ -86,6 +177,17 @@ angular.module('MatrixCall', [])
         this.peerConn.setRemoteDescription(new RTCSessionDescription(this.msg.offer), this.onSetRemoteDescriptionSuccess, this.onSetRemoteDescriptionError);
         this.state = 'ringing';
         this.direction = 'inbound';
+
+        if (window.mozRTCPeerConnection) {
+            // firefox's RTCPeerConnection doesn't add streams until it starts getting media on them
+            // so we need to figure out whether a video channel has been offered by ourselves.
+            if (this.msg.offer.sdp.indexOf('m=video') > -1) {
+                this.type = 'video';
+            } else {
+                this.type = 'voice';
+            }
+        }
+
         var self = this;
         $timeout(function() {
             if (self.state == 'ringing') {
@@ -108,9 +210,24 @@ angular.module('MatrixCall', [])
 
     MatrixCall.prototype.answer = function() {
         console.log("Answering call "+this.call_id);
+
         var self = this;
+
+        var roomMembers = $rootScope.events.rooms[this.room_id].members;
+        if (roomMembers[matrixService.config().user_id].membership != 'join') {
+            console.log("We need to join the room before we can accept this call");
+            matrixService.join(this.room_id).then(function() {
+                self.answer();
+            }, function() {
+                console.log("Failed to join room: can't answer call!");
+                self.onError("Unable to join room to answer call!");
+                self.hangup();
+            });
+            return;
+        }
+
         if (!this.localAVStream && !this.waitForLocalAVStream) {
-            navigator.getUserMedia({audio: true, video: false}, function(s) { self.gotUserMediaForAnswer(s); }, function(e) { self.getUserMediaFailed(e); });
+            navigator.getUserMedia(this.getUserMediaVideoContraints(this.type), function(s) { self.gotUserMediaForAnswer(s); }, function(e) { self.getUserMediaFailed(e); });
             this.state = 'wait_local_media';
         } else if (this.localAVStream) {
             this.gotUserMediaForAnswer(this.localAVStream);
@@ -132,17 +249,24 @@ angular.module('MatrixCall', [])
         }
     };
 
-    MatrixCall.prototype.hangup = function(suppressEvent) {
+    MatrixCall.prototype.hangup = function(reason, suppressEvent) {
         console.log("Ending call "+this.call_id);
 
+        // pausing now keeps the last frame (ish) of the video call in the video element
+        // rather than it just turning black straight away
+        if (this.remoteVideoElement) this.remoteVideoElement.pause();
+        if (this.localVideoElement) this.localVideoElement.pause();
+
         this.stopAllMedia();
         if (this.peerConn) this.peerConn.close();
 
         this.hangupParty = 'local';
+        this.hangupReason = reason;
 
         var content = {
             version: 0,
             call_id: this.call_id,
+            reason: reason
         };
         this.sendEventWithRetry('m.call.hangup', content);
         this.state = 'ended';
@@ -156,6 +280,13 @@ angular.module('MatrixCall', [])
         }
         if (this.state == 'ended') return;
 
+        if (this.localVideoElement && this.type == 'video') {
+            var vidTrack = stream.getVideoTracks()[0];
+            this.localVideoElement.src = URL.createObjectURL(stream);
+            this.localVideoElement.muted = true;
+            this.localVideoElement.play();
+        }
+
         this.localAVStream = stream;
         var audioTracks = stream.getAudioTracks();
         for (var i = 0; i < audioTracks.length; i++) {
@@ -177,6 +308,13 @@ angular.module('MatrixCall', [])
     MatrixCall.prototype.gotUserMediaForAnswer = function(stream) {
         if (this.state == 'ended') return;
 
+        if (this.localVideoElement && this.type == 'video') {
+            var vidTrack = stream.getVideoTracks()[0];
+            this.localVideoElement.src = URL.createObjectURL(stream);
+            this.localVideoElement.muted = true;
+            this.localVideoElement.play();
+        }
+
         this.localAVStream = stream;
         var audioTracks = stream.getAudioTracks();
         for (var i = 0; i < audioTracks.length; i++) {
@@ -187,7 +325,7 @@ angular.module('MatrixCall', [])
         var constraints = {
             'mandatory': {
                 'OfferToReceiveAudio': true,
-                'OfferToReceiveVideo': false
+                'OfferToReceiveVideo': this.type == 'video'
             },
         };
         this.peerConn.createAnswer(function(d) { self.createdAnswer(d); }, function(e) {}, constraints);
@@ -196,14 +334,14 @@ angular.module('MatrixCall', [])
     };
 
     MatrixCall.prototype.gotLocalIceCandidate = function(event) {
-        console.log(event);
         if (event.candidate) {
+            console.log("Got local ICE "+event.candidate.sdpMid+" candidate: "+event.candidate.candidate);
             this.sendCandidate(event.candidate);
         }
     }
 
     MatrixCall.prototype.gotRemoteIceCandidate = function(cand) {
-        console.log("Got ICE candidate from remote: "+cand);
+        console.log("Got remote ICE "+cand.sdpMid+" candidate: "+cand.candidate);
         if (this.state == 'ended') {
             console.log("Ignoring remote ICE candidate because call has ended");
             return;
@@ -218,6 +356,7 @@ angular.module('MatrixCall', [])
         this.state = 'connecting';
     };
 
+
     MatrixCall.prototype.gotLocalOffer = function(description) {
         console.log("Created offer: "+description);
 
@@ -239,8 +378,7 @@ angular.module('MatrixCall', [])
         var self = this;
         $timeout(function() {
             if (self.state == 'invite_sent') {
-                self.hangupReason = 'invite_timeout';
-                self.hangup();
+                self.hangup('invite_timeout');
             }
         }, MatrixCall.CALL_TIMEOUT);
 
@@ -269,7 +407,7 @@ angular.module('MatrixCall', [])
     };
 
     MatrixCall.prototype.getUserMediaFailed = function() {
-        this.onError("Couldn't start capturing audio! Is your microphone set up?");
+        this.onError("Couldn't start capturing! Is your microphone set up?");
         this.hangup();
     };
 
@@ -283,6 +421,8 @@ angular.module('MatrixCall', [])
                 self.state = 'connected';
                 self.didConnect = true;
             });
+        } else if (this.peerConn.iceConnectionState == 'failed') {
+            this.hangup('ice_failed');
         }
     };
 
@@ -305,6 +445,14 @@ angular.module('MatrixCall', [])
 
         this.remoteAVStream = s;
 
+        if (this.direction == 'inbound') {
+            if (s.getVideoTracks().length > 0) {
+                this.type = 'video';
+            } else {
+                this.type = 'voice';
+            }
+        }
+
         var self = this;
         forAllTracksOnStream(s, function(t) {
             // not currently implemented in chrome
@@ -314,9 +462,16 @@ angular.module('MatrixCall', [])
         event.stream.onended = function(e) { self.onRemoteStreamEnded(e); }; 
         // not currently implemented in chrome
         event.stream.onstarted = function(e) { self.onRemoteStreamStarted(e); };
-        var player = new Audio();
-        player.src = URL.createObjectURL(s);
-        player.play();
+
+        this.tryPlayRemoteStream();
+    };
+
+    MatrixCall.prototype.tryPlayRemoteStream = function(event) {
+        if (this.remoteVideoElement && this.remoteAVStream) {
+            var player = this.remoteVideoElement;
+            player.src = URL.createObjectURL(this.remoteAVStream);
+            player.play();
+        }
     };
 
     MatrixCall.prototype.onRemoteStreamStarted = function(event) {
@@ -345,12 +500,15 @@ angular.module('MatrixCall', [])
         });
     };
 
-    MatrixCall.prototype.onHangupReceived = function() {
+    MatrixCall.prototype.onHangupReceived = function(msg) {
         console.log("Hangup received");
+        if (this.remoteVideoElement) this.remoteVideoElement.pause();
+        if (this.localVideoElement) this.localVideoElement.pause();
         this.state = 'ended';
         this.hangupParty = 'remote';
+        this.hangupReason = msg.reason;
         this.stopAllMedia();
-        if (this.peerConn.signalingState != 'closed') this.peerConn.close();
+        if (this.peerConn && this.peerConn.signalingState != 'closed') this.peerConn.close();
         if (this.onHangup) this.onHangup(this);
     };
 
@@ -361,13 +519,15 @@ angular.module('MatrixCall', [])
             newCall.waitForLocalAVStream = true;
         } else if (this.state == 'create_offer') {
             console.log("Handing local stream to new call");
-            newCall.localAVStream = this.localAVStream;
+            newCall.gotUserMediaForAnswer(this.localAVStream);
             delete(this.localAVStream);
         } else if (this.state == 'invite_sent') {
             console.log("Handing local stream to new call");
-            newCall.localAVStream = this.localAVStream;
+            newCall.gotUserMediaForAnswer(this.localAVStream);
             delete(this.localAVStream);
         }
+        newCall.localVideoElement = this.localVideoElement;
+        newCall.remoteVideoElement = this.remoteVideoElement;
         this.successor = newCall;
         this.hangup(true);
     };
@@ -429,7 +589,7 @@ angular.module('MatrixCall', [])
         }
 
         if (this.candidateSendTries > 5) {
-            console.log("Failed to send candidates on attempt "+ev.tries+". Giving up for now.");
+            console.log("Failed to send candidates on attempt "+this.candidateSendTries+". Giving up for now.");
             this.candidateSendTries = 0;
             return;
         }

webclient/components/matrix/matrix-filter.js

@@ -19,7 +19,7 @@
 angular.module('matrixFilter', [])
 
 // Compute the room name according to information we have
-.filter('mRoomName', ['$rootScope', 'matrixService', function($rootScope, matrixService) {
+.filter('mRoomName', ['$rootScope', 'matrixService', 'eventHandlerService', function($rootScope, matrixService, eventHandlerService) {
     return function(room_id) {
         var roomName;
 
@@ -31,60 +31,80 @@ angular.module('matrixFilter', [])
         if (room) {
             // Get name from room state date
             var room_name_event = room["m.room.name"];
+
+            // Determine if it is a public room
+            var isPublicRoom = false;
+            if (room["m.room.join_rules"] && room["m.room.join_rules"].content) {
+                isPublicRoom = ("public" === room["m.room.join_rules"].content.join_rule);
+            }
+
             if (room_name_event) {
                 roomName = room_name_event.content.name;
             }
             else if (alias) {
                 roomName = alias;
             }
-            else if (room.members) {
+            else if (room.members && !isPublicRoom) {    // Do not rename public room
+            
+                var user_id = matrixService.config().user_id;
+
                 // Else, build the name from its users
-                // FIXME: Is it still required?
                 // Limit the room renaming to 1:1 room
                 if (2 === Object.keys(room.members).length) {
                     for (var i in room.members) {
-                        var member = room.members[i];
-                        if (member.state_key !== matrixService.config().user_id) {
+                        if (!room.members.hasOwnProperty(i)) continue;
 
-                            if (member.state_key in $rootScope.presence) {
-                                // If the user is available in presence, use the displayname there
-                                // as it is the most uptodate
-                                roomName = $rootScope.presence[member.state_key].content.displayname;
-                            }
-                            else if (member.content.displayname) {
-                                roomName = member.content.displayname;
-                            }
-                            else {
-                                roomName = member.state_key;
-                            }
+                        var member = room.members[i];
+                        if (member.state_key !== user_id) {
+                            roomName = eventHandlerService.getUserDisplayName(room_id, member.state_key);
+                            break;
                         }
                     }
                 }
-                else if (1 === Object.keys(room.members).length) {
-                    // The other member may be in the invite list, get all invited users
-                    var invitedUserIDs = [];
-                    for (var i in room.messages) {
-                        var message = room.messages[i];
-                        if ("m.room.member" === message.type && "invite" === message.membership) {
-                            // Make sure there is no duplicate user
-                            if (-1 === invitedUserIDs.indexOf(message.state_key)) {
-                                invitedUserIDs.push(message.state_key);
-                            }
-                        } 
-                    }
+                else if (Object.keys(room.members).length <= 1) {
+                    
+                    var otherUserId;
 
-                    // For now, only 1:1 room needs to be renamed. It means only 1 invited user
-                    if (1 === invitedUserIDs.length) {
-                        var userID = invitedUserIDs[0];
-
-                        // Try to resolve his displayname in presence global data
-                        if (userID in $rootScope.presence) {
-                            roomName = $rootScope.presence[userID].content.displayname;
-                        }
-                        else {
-                            roomName = userID;
-                        }
+                    if (Object.keys(room.members)[0] && Object.keys(room.members)[0] !== user_id) {
+                        otherUserId = Object.keys(room.members)[0];
                     }
+                    else {
+                        // it's got to be an invite, or failing that a self-chat;
+                        otherUserId = room.inviter || user_id;
+/*                        
+                        // XXX: This should all be unnecessary now thanks to using the /rooms/<room>/roomid API
+
+                        // The other member may be in the invite list, get all invited users
+                        var invitedUserIDs = [];
+                        
+                        // XXX: *SURELY* we shouldn't have to trawl through the whole messages list to
+                        // find invite - surely the other user should be in room.members with state invited? :/ --Matthew
+                        for (var i in room.messages) {
+                            var message = room.messages[i];
+                            if ("m.room.member" === message.type && "invite" === message.content.membership) {
+                                // Filter out the current user
+                                var member_id = message.state_key;
+                                if (member_id === user_id) {
+                                    member_id = message.user_id;
+                                }
+                                if (member_id !== user_id) {
+                                    // Make sure there is no duplicate user
+                                    if (-1 === invitedUserIDs.indexOf(member_id)) {
+                                        invitedUserIDs.push(member_id);
+                                    }
+                                }
+                            } 
+                        }
+
+                        // For now, only 1:1 room needs to be renamed. It means only 1 invited user
+                        if (1 === invitedUserIDs.length) {
+                            otherUserId = invitedUserIDs[0];
+                        }
+*/                        
+                    }
+                    
+                    // Get the user display name
+                    roomName = eventHandlerService.getUserDisplayName(room_id, otherUserId);
                 }
             }
         }
@@ -97,43 +117,23 @@ angular.module('matrixFilter', [])
         if (undefined === roomName) {
             // By default, use the room ID
             roomName = room_id;
+
+            // XXX: this is *INCREDIBLY* heavy logging for a function that calls every single
+            // time any kind of digest runs which refreshes a room name...
+            // commenting it out for now.
+
+            // Log some information that lead to this leak
+            // console.log("Room ID leak for " + room_id);
+            // console.log("room object: " + JSON.stringify(room, undefined, 4));   
         }
 
         return roomName;
     };
 }])
 
-// Compute the user display name in a room according to the data already downloaded
-.filter('mUserDisplayName', ['$rootScope', function($rootScope) {
+// Return the user display name
+.filter('mUserDisplayName', ['eventHandlerService', function(eventHandlerService) {
     return function(user_id, room_id) {
-        var displayName;
-    
-        // Try to find the user name among presence data
-        // Warning: that means we have received before a presence event for this
-        // user which cannot be guaranted.
-        // However, if we get the info by this way, we are sure this is the latest user display name
-        // See FIXME comment below
-        if (user_id in $rootScope.presence) {
-            displayName = $rootScope.presence[user_id].content.displayname;
-        }
-            
-        // FIXME: Would like to use the display name as defined in room members of the room.
-        // But this information is the display name of the user when he has joined the room.
-        // It does not take into account user display name update
-        if (room_id) {
-            var room = $rootScope.events.rooms[room_id];
-            if (room && (user_id in room.members)) {
-                var member = room.members[user_id];
-                if (member.content.displayname) {
-                    displayName = member.content.displayname;
-                }
-            }
-        }
-        
-        if (undefined === displayName) {
-            // By default, use the user ID
-            displayName = user_id;
-        }
-        return displayName;
+        return eventHandlerService.getUserDisplayName(room_id, user_id);
     };
 }]);

webclient/components/matrix/matrix-phone-service.js

@@ -59,6 +59,16 @@ angular.module('matrixPhoneService', [])
 
             var MatrixCall = $injector.get('MatrixCall');
             var call = new MatrixCall(event.room_id);
+
+            if (!isWebRTCSupported()) {
+                console.log("Incoming call ID "+msg.call_id+" but this browser doesn't support WebRTC");
+                // don't hang up the call: there could be other clients connected that do support WebRTC and declining the
+                // the call on their behalf would be really annoying.
+                // instead, we broadcast a fake call event with a non-functional call object
+                $rootScope.$broadcast(matrixPhoneService.INCOMING_CALL_EVENT, call);
+                return;
+            }
+
             call.call_id = msg.call_id;
             call.initWithInvite(event);
             matrixPhoneService.allCalls[call.call_id] = call;
@@ -135,7 +145,7 @@ angular.module('matrixPhoneService', [])
                 call.initWithHangup(event);
                 matrixPhoneService.allCalls[msg.call_id] = call;
             } else {
-                call.onHangupReceived();
+                call.onHangupReceived(msg);
                 delete(matrixPhoneService.allCalls[msg.call_id]);
             }
         }

webclient/components/matrix/matrix-service.js

@@ -264,7 +264,13 @@ angular.module('matrixService', [])
 
             return doRequest("GET", path, params);
         },
-
+        
+        // get room state for a specific room
+        roomState: function(room_id) {
+            var path = "/rooms/" + room_id + "/state";
+            return doRequest("GET", path);
+        },
+        
         // Joins a room
         join: function(room_id) {
             return this.membershipChange(room_id, undefined, "join");
@@ -697,11 +703,10 @@ angular.module('matrixService', [])
         createRoomIdToAliasMapping: function(roomId, alias) {
             roomIdToAlias[roomId] = alias;
             aliasToRoomId[alias] = roomId;
-            // localStorage.setItem(MAPPING_PREFIX+roomId, alias);
         },
         
         getRoomIdToAliasMapping: function(roomId) {
-            var alias = roomIdToAlias[roomId]; // was localStorage.getItem(MAPPING_PREFIX+roomId)
+            var alias = roomIdToAlias[roomId];
             //console.log("looking for alias for " + roomId + "; found: " + alias);
             return alias;
         },
@@ -762,6 +767,10 @@ angular.module('matrixService', [])
             var deferred = $q.defer();
             deferred.reject({data:{error: "Invalid room: " + room_id}});
             return deferred.promise;
+        },
+
+        getTurnServer: function() {
+            return doRequest("GET", "/voip/turnServer");
         }
 
     };

webclient/home/home-controller.js

@@ -42,6 +42,10 @@ angular.module('HomeController', ['matrixService', 'eventHandlerService', 'Recen
         displayName: "",
         avatarUrl: ""
     };
+    
+    $scope.newChat = {
+        user: ""
+    };
 
     var refresh = function() {
         
@@ -82,18 +86,24 @@ angular.module('HomeController', ['matrixService', 'eventHandlerService', 'Recen
     
     // Go to a room
     $scope.goToRoom = function(room_id) {
-        // Simply open the room page on this room id
-        //$location.url("room/" + room_id);
         matrixService.join(room_id).then(
             function(response) {
+                var final_room_id = room_id;
                 if (response.data.hasOwnProperty("room_id")) {
-                    if (response.data.room_id != room_id) {
-                        $location.url("room/" + response.data.room_id);
-                        return;
-                     }
+                    final_room_id = response.data.room_id;
                 }
 
-                $location.url("room/" + room_id);
+                // TODO: factor out the common housekeeping whenever we try to join a room or alias
+                matrixService.roomState(final_room_id).then(
+                    function(response) {
+                        eventHandlerService.handleEvents(response.data, false, true);
+                    },
+                    function(error) {
+                        $scope.feedback = "Failed to get room state for: " + final_room_id;
+                    }
+                );                                        
+
+                $location.url("room/" + final_room_id);
             },
             function(error) {
                 $scope.feedback = "Can't join room: " + JSON.stringify(error.data);
@@ -104,6 +114,15 @@ angular.module('HomeController', ['matrixService', 'eventHandlerService', 'Recen
     $scope.joinAlias = function(room_alias) {
         matrixService.joinAlias(room_alias).then(
             function(response) {
+                // TODO: factor out the common housekeeping whenever we try to join a room or alias
+                matrixService.roomState(response.room_id).then(
+                    function(response) {
+                        eventHandlerService.handleEvents(response.data, false, true);
+                    },
+                    function(error) {
+                        $scope.feedback = "Failed to get room state for: " + response.room_id;
+                    }
+                );                                        
                 // Go to this room
                 $location.url("room/" + room_alias);
             },
@@ -112,6 +131,32 @@ angular.module('HomeController', ['matrixService', 'eventHandlerService', 'Recen
             }
         );
     };
+    
+    // FIXME: factor this out between user-controller and home-controller etc.
+    $scope.messageUser = function() {    
+        
+        // FIXME: create a new room every time, for now
+        
+        matrixService.create(null, 'private').then(
+            function(response) { 
+                // This room has been created. Refresh the rooms list
+                var room_id = response.data.room_id;
+                console.log("Created room with id: "+ room_id);
+                
+                matrixService.invite(room_id, $scope.newChat.user).then(
+                    function() {
+                        $scope.feedback = "Invite sent successfully";
+                        $scope.$parent.goToPage("/room/" + room_id);
+                    },
+                    function(reason) {
+                        $scope.feedback = "Failure: " + JSON.stringify(reason);
+                    });
+            },
+            function(error) {
+                $scope.feedback = "Failure: " + JSON.stringify(error.data);
+            });                
+    };
+    
  
     $scope.onInit = function() {
         // Load profile data

webclient/home/home.html

@@ -17,7 +17,7 @@
             <div>{{ config.user_id }}</div>
         </div>
     </div>
-    
+        
     <h3>Recent conversations</h3>
         <div ng-include="'recents/recents.html'"></div>
     <br/>
@@ -52,17 +52,24 @@
     
     <div>
         <form>
-            <input size="40" ng-model="newRoom.room_alias" ng-enter="createNewRoom(newRoom.room_alias, newRoom.private)" placeholder="(e.g. foo_channel)"/>
+            <input size="40" ng-model="newRoom.room_alias" ng-enter="createNewRoom(newRoom.room_alias, newRoom.private)" placeholder="(e.g. foo)"/>
             <input type="checkbox" ng-model="newRoom.private">private
             <button ng-disabled="!newRoom.room_alias" ng-click="createNewRoom(newRoom.room_alias, newRoom.private)">Create room</button>    
         </form>
     </div>
     <div>
         <form>
-            <input size="40" ng-model="joinAlias.room_alias" ng-enter="joinAlias(joinAlias.room_alias)" placeholder="(e.g. #foo_channel:example.org)"/>
+            <input size="40" ng-model="joinAlias.room_alias" ng-enter="joinAlias(joinAlias.room_alias)" placeholder="(e.g. #foo:example.org)"/>
             <button ng-disabled="!joinAlias.room_alias" ng-click="joinAlias(joinAlias.room_alias)">Join room</button>    
         </form>
     </div>
+    <div>
+        <form>
+            <input size="40" ng-model="newChat.user" ng-enter="messageUser()" placeholder="e.g. @user:domain.com"/>
+            <button ng-disabled="!newChat.user" ng-click="messageUser()">Message user</button>
+        </form>
+    </div>
+    
     <br/>
     
     {{ feedback }}

webclient/index.html

@@ -45,6 +45,13 @@
 </head>
 
 <body>
+    <div id="videoBackground" ng-class="videoMode">
+        <div id="videoContainer" ng-class="videoMode">
+            <div id="videoContainerPadding"></div>
+            <video id="localVideo" ng-class="[videoMode, currentCall.state]" ng-show="currentCall && currentCall.type == 'video' && (currentCall.state == 'connected' || currentCall.state == 'connecting' || currentCall.state == 'invite_sent' || currentCall.state == 'ended')"></video>
+            <video id="remoteVideo" ng-class="[videoMode, currentCall.state]" ng-show="currentCall && currentCall.type == 'video' && (currentCall.state == 'connected' || (currentCall.state == 'ended' && currentCall.didConnect))"></video>
+        </div>
+    </div>
 
     <div id="header">
         <!-- Do not show buttons on the login page -->
@@ -58,20 +65,22 @@
                     <br />
                     <span id="callState">
                         <span ng-show="currentCall.state == 'invite_sent'">Calling...</span>
-                        <span ng-show="currentCall.state == 'ringing'">Incoming Call</span>
+                        <span ng-show="currentCall.state == 'ringing' && currentCall && currentCall.type == 'video'">Incoming Video Call</span>
+                        <span ng-show="currentCall.state == 'ringing' && currentCall && currentCall.type == 'voice'">Incoming Voice Call</span>
                         <span ng-show="currentCall.state == 'connecting'">Call Connecting...</span>
                         <span ng-show="currentCall.state == 'connected'">Call Connected</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'remote'">Call Rejected</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local' && currentCall.hangupReason == undefined">Call Canceled</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local' && currentCall.hangupReason == 'invite_timeout'">User Not Responding</span>
-                        <span ng-show="currentCall.state == 'ended' && currentCall.didConnect && currentCall.direction == 'outbound'">Call Ended</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'inbound'">Call Canceled</span>
-                        <span ng-show="currentCall.state == 'ended' && currentCall.didConnect && currentCall.direction == 'inbound'">Call Ended</span>
+                        <span ng-show="currentCall.state == 'ended' && currentCall.hangupReason == 'ice_failed'">Media Connection Failed{{ haveTurn ? "" : " (VoIP relaying unsupported by Home Server)" }}</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'remote'">Call Rejected</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local'">Call Canceled</span>
+                        <span ng-show="currentCall.state == 'ended' && currentCall.hangupReason == 'invite_timeout' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local'">User Not Responding</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && currentCall.didConnect && currentCall.direction == 'outbound'">Call Ended</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && !currentCall.didConnect && currentCall.direction == 'inbound'">Call Canceled</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && currentCall.didConnect && currentCall.direction == 'inbound'">Call Ended</span>
                         <span ng-show="currentCall.state == 'wait_local_media'">Waiting for media permission...</span>
                     </span>
                 </div>
                 <span ng-show="currentCall.state == 'ringing'">
-                    <button ng-click="answerCall()">Answer</button>
+                    <button ng-click="answerCall()" ng-disabled="!isWebRTCSupported" title="{{isWebRTCSupported ? '' : 'Your browser does not support VoIP' }}">Answer {{ currentCall.type }} call</button>
                     <button ng-click="hangupCall()">Reject</button>
                 </span>
                 <button ng-click="hangupCall()" ng-show="currentCall && currentCall.state != 'ringing' && currentCall.state != 'ended' && currentCall.state != 'fledgling'">Hang up</button>
@@ -92,6 +101,7 @@
                     <source src="media/busy.mp3" type="audio/mpeg" />
                 </audio>
             </div>
+
             <a href id="headerUserId" ng-click='goToUserPage(user_id)'>{{ user_id }}</a>
             &nbsp;
             <button ng-click='goToPage("/")'>Home</button>
@@ -100,9 +110,20 @@
         </div>
     </div>
 
-    <div id="page" ng-view></div>
+    <div class="page" ng-hide="unsupportedBrowser" ng-view></div>
 
-    <div id="footer" ng-hide="location.indexOf('/room') == 0">
+    <div class="page" ng-show="unsupportedBrowser">
+        <div id="unsupportedBrowser" ng-show="unsupportedBrowser">
+            Sorry, your browser is not supported. <br/>
+                Reason: {{ unsupportedBrowser.reason }}
+
+            <br/><br/>
+            Your browser: <br/>
+            {{ unsupportedBrowser.browser }}
+        </div>
+    </div>
+
+    <div id="footer" ng-hide="location.indexOf('/room') === 0">
         <div id="footerContent">
             &copy; 2014 Matrix.org
         </div>

webclient/login/login.html

@@ -23,7 +23,7 @@
                     <br/>
                     <input id="password" size="32" type="password" ng-model="account.password" placeholder="Password"/>
                     <br/><br/>
-                    <button ng-click="login()" ng-disabled="!account.user_id || !account.password || !account.homeserver">Login</button>
+                    <button id="login" ng-click="login()" ng-disabled="!account.user_id || !account.password || !account.homeserver">Login</button>
                     <br/><br/>
                 </div>
 

webclient/recents/recents-filter.js

@@ -37,6 +37,10 @@ angular.module('RecentsController')
 
                 filtered.push(room);
             }
+            else if ("invite" === room.membership) {
+                // The only information we have about the room is that the user has been invited
+                filtered.push(room);
+            }
         });
 
         // And time sort them

webclient/recents/recents.html

@@ -2,7 +2,7 @@
     <table class="recentsTable">
         <tbody ng-repeat="(index, room) in events.rooms | orderRecents" 
                ng-click="goToPage('room/' + (room.room_alias ? room.room_alias : room.room_id) )" 
-               class ="recentsRoom" 
+               class="recentsRoom" 
                ng-class="{'recentsRoomSelected': (room.room_id === recentsSelectedRoomID)}">                                           
             <tr>
                 <td ng-class="room['m.room.join_rules'].content.join_rule == 'public' ? 'recentsRoomName recentsPublicRoom' : 'recentsRoomName'">
@@ -19,6 +19,8 @@
                     {{ lastMsg = eventHandlerService.getLastMessage(room.room_id, true);"" }}
 
                     {{ (lastMsg.ts) | date:'MMM d HH:mm' }}
+                    
+                    <img ng-click="leave(room.room_id); $event.stopPropagation();" src="img/close.png" width="10" height="10" style="margin-bottom: -1px; margin-left: 2px;" alt="close"/>
                 </td>
             </tr>
 
@@ -31,28 +33,35 @@
                     
                     <div ng-hide="room.membership === 'invite'" ng-switch="lastMsg.type">
                         <div ng-switch-when="m.room.member">
-                            <span ng-if="'join' === lastMsg.content.membership">
-                                {{ lastMsg.state_key | mUserDisplayName: room.room_id}} joined
-                            </span>
-                            <span ng-if="'leave' === lastMsg.content.membership">
-                                <span ng-if="lastMsg.user_id === lastMsg.state_key">
-                                    {{lastMsg.state_key | mUserDisplayName: room.room_id }} left
+                            <span ng-switch="lastMsg.changedKey">
+                                <span ng-switch-when="membership">
+                                    <span ng-if="'join' === lastMsg.content.membership">
+                                        {{ lastMsg.state_key | mUserDisplayName: room.room_id }} joined
+                                    </span>
+                                    <span ng-if="'leave' === lastMsg.content.membership">
+                                        <span ng-if="lastMsg.user_id === lastMsg.state_key">
+                                            {{lastMsg.state_key | mUserDisplayName: room.room_id }} left
+                                        </span>
+                                        <span ng-if="lastMsg.user_id !== lastMsg.state_key">
+                                            {{ lastMsg.user_id | mUserDisplayName: room.room_id }}
+                                            {{ {"join": "kicked", "ban": "unbanned"}[lastMsg.content.prev] }}
+                                            {{ lastMsg.state_key | mUserDisplayName: room.room_id }}
+                                        </span>
+                                        <span ng-if="'join' === lastMsg.content.prev && lastMsg.content.reason">
+                                            : {{ lastMsg.content.reason }}
+                                        </span>
+                                    </span>
+                                    <span ng-if="'invite' === lastMsg.content.membership || 'ban' === lastMsg.content.membership">
+                                        {{ lastMsg.user_id | mUserDisplayName: room.room_id }}
+                                        {{ {"invite": "invited", "ban": "banned"}[lastMsg.content.membership] }}
+                                        {{ lastMsg.state_key | mUserDisplayName: room.room_id }}
+                                        <span ng-if="'ban' === lastMsg.content.prev && lastMsg.content.reason">
+                                            : {{ lastMsg.content.reason }}
+                                        </span>
+                                    </span>         
                                 </span>
-                                <span ng-if="lastMsg.user_id !== lastMsg.state_key">
-                                    {{ lastMsg.user_id | mUserDisplayName: room.room_id }}
-                                    {{ {"join": "kicked", "ban": "unbanned"}[lastMsg.content.prev] }}
-                                    {{ lastMsg.state_key | mUserDisplayName: room.room_id }}
-                                </span>
-                                <span ng-if="'join' === lastMsg.content.prev && lastMsg.content.reason">
-                                    : {{ lastMsg.content.reason }}
-                                </span>
-                            </span>
-                            <span ng-if="'invite' === lastMsg.content.membership || 'ban' === lastMsg.content.membership">
-                                {{ lastMsg.user_id | mUserDisplayName: room.room_id }}
-                                {{ {"invite": "invited", "ban": "banned"}[lastMsg.content.membership] }}
-                                {{ lastMsg.state_key | mUserDisplayName: room.room_id }}
-                                <span ng-if="'ban' === lastMsg.content.prev && lastMsg.content.reason">
-                                    : {{ lastMsg.content.reason }}
+                                <span ng-switch-when="displayname">
+                                    {{ lastMsg.user_id }} changed their display name from {{ lastMsg.prev_content.displayname }} to {{ lastMsg.content.displayname }}
                                 </span>
                             </span>
                         </div>

webclient/room/room-controller.js

@@ -15,8 +15,8 @@ limitations under the License.
 */
 
 angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
-.controller('RoomController', ['$filter', '$scope', '$timeout', '$routeParams', '$location', '$rootScope', 'matrixService', 'eventHandlerService', 'mFileUpload', 'matrixPhoneService', 'MatrixCall',
-                               function($filter, $scope, $timeout, $routeParams, $location, $rootScope, matrixService, eventHandlerService, mFileUpload, matrixPhoneService, MatrixCall) {
+.controller('RoomController', ['$filter', '$scope', '$timeout', '$routeParams', '$location', '$rootScope', 'matrixService', 'mPresence', 'eventHandlerService', 'mFileUpload', 'matrixPhoneService', 'MatrixCall',
+                               function($filter, $scope, $timeout, $routeParams, $location, $rootScope, matrixService, mPresence, eventHandlerService, mFileUpload, matrixPhoneService, MatrixCall) {
    'use strict';
     var MESSAGES_PER_PAGINATION = 30;
     var THUMBNAIL_SIZE = 320;
@@ -33,7 +33,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         paginating: false, // used to avoid concurrent pagination requests pulling in dup contents
         stream_failure: undefined, // the response when the stream fails
         waiting_for_joined_event: false,  // true when the join request is pending. Back to false once the corresponding m.room.member event is received
-        messages_visibility: "hidden" // In order to avoid flickering when scrolling down the message table at the page opening, delay the message table display
+        messages_visibility: "hidden", // In order to avoid flickering when scrolling down the message table at the page opening, delay the message table display
     };
     $scope.members = {};
     $scope.autoCompleting = false;
@@ -189,7 +189,6 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 // Notify when a user joins
                 if ((document.hidden  || matrixService.presence.unavailable === mPresence.getState())
                         && event.state_key !== $scope.state.user_id  && "join" === event.membership) {
-                    debugger;
                     var notification = new window.Notification(
                         event.content.displayname +
                         " (" + (matrixService.getRoomIdToAliasMapping(event.room_id) || event.room_id) + ")", // FIXME: don't leak room_ids here
@@ -401,6 +400,8 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         // Find the max power level
         var maxPowerLevel = 0;
         for (var i in $scope.members) {
+            if (!$scope.members.hasOwnProperty(i)) continue;
+
             var member = $scope.members[i];
             if (member.powerLevel) {
                 maxPowerLevel = Math.max(maxPowerLevel, member.powerLevel);
@@ -410,6 +411,8 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         // Normalized them on a 0..100% scale to be use in css width
         if (maxPowerLevel) {
             for (var i in $scope.members) {
+                if (!$scope.members.hasOwnProperty(i)) continue;
+
                 var member = $scope.members[i];
                 member.powerLevelNorm = (member.powerLevel * 100) / maxPowerLevel;
             }
@@ -417,14 +420,16 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
     };
 
     $scope.send = function() {
-        if (undefined === $scope.textInput || $scope.textInput === "") {
+        var input = $('#mainInput').val();
+        
+        if (undefined === input || input === "") {
             return;
         }
         
         scrollToBottom(true);
 
         // Store the command in the history
-        history.push($scope.textInput);
+        history.push(input);
 
         var promise;
         var cmd;
@@ -432,13 +437,11 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         var echo = false;
         
         // Check for IRC style commands first
-        var line = $scope.textInput;
-        
         // trim any trailing whitespace, as it can confuse the parser for IRC-style commands
-        line = line.replace(/\s+$/, "");
+        input = input.replace(/\s+$/, "");
         
-        if (line[0] === "/" && line[1] !== "/") {
-            var bits = line.match(/^(\S+?)( +(.*))?$/);
+        if (input[0] === "/" && input[1] !== "/") {
+            var bits = input.match(/^(\S+?)( +(.*))?$/);
             cmd = bits[1];
             args = bits[3];
             
@@ -480,6 +483,15 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                             else {
                                 promise = matrixService.joinAlias(room_alias).then(
                                     function(response) {
+                                        // TODO: factor out the common housekeeping whenever we try to join a room or alias
+                                        matrixService.roomState(response.room_id).then(
+                                            function(response) {
+                                                eventHandlerService.handleEvents(response.data, false, true);
+                                            },
+                                            function(error) {
+                                                $scope.feedback = "Failed to get room state for: " + response.room_id;
+                                            }
+                                        );                                        
                                         $location.url("room/" + room_alias);
                                     },
                                     function(error) {
@@ -581,7 +593,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         // By default send this as a message unless it's an IRC-style command
         if (!promise && !cmd) {
             // Make the request
-            promise = matrixService.sendTextMessage($scope.room_id, line);
+            promise = matrixService.sendTextMessage($scope.room_id, input);
             echo = true;
         }
         
@@ -590,7 +602,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
             // To do so, create a minimalist fake text message event and add it to the in-memory list of room messages
             var echoMessage = {
                 content: {
-                    body: (cmd === "/me" ? args : line),
+                    body: (cmd === "/me" ? args : input),
                     hsob_ts: new Date().getTime(), // fake a timestamp
                     msgtype: (cmd === "/me" ? "m.emote" : "m.text"),
                 },
@@ -600,7 +612,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 echo_msg_state: "messagePending"     // Add custom field to indicate the state of this fake message to HTML
             };
 
-            $scope.textInput = "";
+            $('#mainInput').val('');
             $rootScope.events.rooms[$scope.room_id].messages.push(echoMessage);
             scrollToBottom();
         }
@@ -620,7 +632,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                         echoMessage.event_id = response.data.event_id;
                     }
                     else {
-                        $scope.textInput = "";
+                        $('#mainInput').val('');
                     }         
                 },
                 function(error) {
@@ -703,13 +715,24 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 // The room members is available in the data fetched by initialSync
                 if ($rootScope.events.rooms[$scope.room_id]) {
 
-                    // There is no need to do a 1st pagination (initialSync provided enough to fill a page)
-                    $scope.state.first_pagination = false;
+                    var messages = $rootScope.events.rooms[$scope.room_id].messages;
+
+                    if (0 === messages.length
+                    || (1 === messages.length && "m.room.member" === messages[0].type && "invite" === messages[0].content.membership && $scope.state.user_id === messages[0].state_key)) {
+                        // If we just joined a room, we won't have this history from initial sync, so we should try to paginate it anyway    
+                        $scope.state.first_pagination = true;
+                    }
+                    else {
+                        // There is no need to do a 1st pagination (initialSync provided enough to fill a page)
+                        $scope.state.first_pagination = false;
+                    }
 
                     var members = $rootScope.events.rooms[$scope.room_id].members;
 
                     // Update the member list
                     for (var i in members) {
+                        if (!members.hasOwnProperty(i)) continue;
+
                         var member = members[i];
                         updateMemberList(member);
                     }
@@ -727,6 +750,16 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                     $scope.state.waiting_for_joined_event = true;
                     matrixService.join($scope.room_id).then(
                         function() {
+                            // TODO: factor out the common housekeeping whenever we try to join a room or alias
+                            matrixService.roomState($scope.room_id).then(
+                                function(response) {
+                                    eventHandlerService.handleEvents(response.data, false, true);
+                                },
+                                function(error) {
+                                    console.error("Failed to get room state for: " + $scope.room_id);
+                                }
+                            );                                        
+                            
                             // onInit3 will be called once the joined m.room.member event is received from the events stream
                             // This avoids to get the joined information twice in parallel:
                             //    - one from the events stream
@@ -735,6 +768,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                         },
                         function(reason) {
                             console.log("Can't join room: " + JSON.stringify(reason));
+                            // FIXME: what if it wasn't a perms problem?
                             $scope.state.permission_denied = "You do not have permission to join this room";
                         });
                 }
@@ -804,7 +838,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         
         matrixService.leave($scope.room_id).then(
             function(response) {
-                console.log("Left room ");
+                console.log("Left room " + $scope.room_id);
                 $location.url("home");
             },
             function(error) {
@@ -854,7 +888,9 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         var call = new MatrixCall($scope.room_id);
         call.onError = $rootScope.onCallError;
         call.onHangup = $rootScope.onCallHangup;
-        call.placeCall({audio: true, video: false});
+        // remote video element is used for playing audio in voice calls
+        call.remoteVideoElement = angular.element('#remoteVideo')[0];
+        call.placeVoiceCall();
         $rootScope.currentCall = call;
     };
 
@@ -862,7 +898,9 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         var call = new MatrixCall($scope.room_id);
         call.onError = $rootScope.onCallError;
         call.onHangup = $rootScope.onCallHangup;
-        call.placeCall({audio: true, video: true});
+        call.localVideoElement = angular.element('#localVideo')[0];
+        call.remoteVideoElement = angular.element('#remoteVideo')[0];
+        call.placeVideoCall();
         $rootScope.currentCall = call;
     };
 
@@ -904,11 +942,11 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
 
             if (-1 === this.position) {
                 // User starts to go to into the history, save the current line
-                this.typingMessage = $scope.textInput;
+                this.typingMessage = $('#mainInput').val();
             }
             else {
                 // If the user modified this line in history, keep the change
-                this.data[this.position] = $scope.textInput;
+                this.data[this.position] = $('#mainInput').val();
             }
 
             // Bounds the new position to valid data
@@ -919,11 +957,11 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
 
             if (-1 !== this.position) {
                 // Show the message from the history
-                $scope.textInput = this.data[this.position];
+                $('#mainInput').val(this.data[this.position]);
             }
             else if (undefined !== this.typingMessage) {
                 // Go back to the message the user started to type
-                $scope.textInput = this.typingMessage;
+                $('#mainInput').val(this.typingMessage);
             }
         }
     };

webclient/room/room-directive.js

@@ -21,39 +21,62 @@ angular.module('RoomController')
     return function (scope, element, attrs) {
         element.bind("keydown keypress", function (event) {
             // console.log("event: " + event.which);
-            if (event.which === 9) {
+            var TAB = 9;
+            var SHIFT = 16;
+            var keypressCode = event.which;
+            if (keypressCode === TAB) {
                 if (!scope.tabCompleting) { // cache our starting text
-                    // console.log("caching " + element[0].value);
                     scope.tabCompleteOriginal = element[0].value;
                     scope.tabCompleting = true;
+                    scope.tabCompleteIndex = 0;
                 }
                 
+                // loop in the right direction
                 if (event.shiftKey) {
                     scope.tabCompleteIndex--;
                     if (scope.tabCompleteIndex < 0) {
-                        scope.tabCompleteIndex = 0;
+                        // wrap to the last search match, and fix up to a real
+                        // index value after we've matched
+                        scope.tabCompleteIndex = Number.MAX_VALUE;
                     }
                 }
                 else {
                     scope.tabCompleteIndex++;
                 }
                 
+
                 var searchIndex = 0;
                 var targetIndex = scope.tabCompleteIndex;
                 var text = scope.tabCompleteOriginal;
                 
-                // console.log("targetIndex: " + targetIndex + ", text=" + text);
+                // console.log("targetIndex: " + targetIndex + ", 
+                // text=" + text);
                                     
-                // FIXME: use the correct regexp to recognise userIDs
+                // FIXME: use the correct regexp to recognise userIDs --M
+                //
+                // XXX: I don't really know what the point of this is. You
+                // WANT to match freeform text given you want to match display
+                // names AND user IDs. Surely you just want to get the last
+                // word out of the input text and that's that?
+                // Am I missing something here? -- Kegan
+                //
+                // You're not missing anything - my point was that we should
+                // explicitly define the syntax for user IDs /somewhere/.
+                // Meanwhile as long as the delimeters are well defined, we
+                // could just pick "the last word".  But to know what the
+                // correct delimeters are, we probably do need a formal
+                // syntax for user IDs to refer to... --Matthew
+                
                 var search = /@?([a-zA-Z0-9_\-:\.]+)$/.exec(text);
-                if (targetIndex === 0) {
-                    element[0].value = text;
-                    
-                    // Force angular to wake up and update the input ng-model by firing up input event
+
+                if (targetIndex === 0) { // 0 is always the original text
+                    element[0].value = text;                    
+                    // Force angular to wake up and update the input ng-model 
+                    // by firing up input event
                     angular.element(element[0]).triggerHandler('input');
                 }
                 else if (search && search[1]) {
-                    // console.log("search found: " + search);
+                    // console.log("search found: " + search+" from "+text);
                     var expansion;
                     
                     // FIXME: could do better than linear search here
@@ -68,6 +91,7 @@ angular.module('RoomController')
                     if (searchIndex < targetIndex) { // then search raw mxids
                         angular.forEach(scope.members, function(item, name) {
                             if (searchIndex < targetIndex) {
+                                // === 1 because mxids are @username
                                 if (name.toLowerCase().indexOf(search[1].toLowerCase()) === 1) {
                                     expansion = name;
                                     searchIndex++;
@@ -76,18 +100,22 @@ angular.module('RoomController')
                         });
                     }
                     
-                    if (searchIndex === targetIndex) {
-                        // xchat-style tab complete
+                    if (searchIndex === targetIndex || 
+                            targetIndex === Number.MAX_VALUE) {
+                        // xchat-style tab complete, add a colon if tab 
+                        // completing at the start of the text
                         if (search[0].length === text.length)
-                            expansion += " : ";
+                            expansion += ": ";
                         else
                             expansion += " ";
                         element[0].value = text.replace(/@?([a-zA-Z0-9_\-:\.]+)$/, expansion);
                         // cancel blink
                         element[0].className = "";     
-                        
-                        // Force angular to wake up and update the input ng-model by firing up input event
-                        angular.element(element[0]).triggerHandler('input');
+                        if (targetIndex === Number.MAX_VALUE) {
+                            // wrap the index around to the last index found
+                            scope.tabCompleteIndex = searchIndex;
+                            targetIndex = searchIndex;
+                        }
                     }
                     else {
                         // console.log("wrapped!");
@@ -97,23 +125,40 @@ angular.module('RoomController')
                         }, 150);
                         element[0].value = text;
                         scope.tabCompleteIndex = 0;
-                        
-                        // Force angular to wake up and update the input ng-model by firing up input event
-                        angular.element(element[0]).triggerHandler('input');
                     }
+
+                    // Force angular to wak up and update the input ng-model by
+                    // firing up input event
+                    angular.element(element[0]).triggerHandler('input');
                 }
                 else {
                     scope.tabCompleteIndex = 0;
                 }
+                // prevent the default TAB operation (typically focus shifting)
                 event.preventDefault();
             }
-            else if (event.which !== 16 && scope.tabCompleting) {
+            else if (keypressCode !== SHIFT && scope.tabCompleting) {
                 scope.tabCompleting = false;
                 scope.tabCompleteIndex = 0;
             }
         });
     };
 }])
+.directive('commandHistory', [ function() {
+    return function (scope, element, attrs) {
+        element.bind("keydown", function (event) {
+            var keycodePressed = event.which;
+            var UP_ARROW = 38;
+            var DOWN_ARROW = 40;
+            if (keycodePressed === UP_ARROW) {
+                scope.history.goUp(event);
+            }
+            else if (keycodePressed === DOWN_ARROW) {
+                scope.history.goDown(event);
+            } 
+        });
+    }
+}])
 
 // A directive to anchor the scroller position at the bottom when the browser is resizing.
 // When the screen resizes, the bottom of the element remains the same, not the top.

webclient/room/room.html

@@ -48,7 +48,15 @@
                          width="80" height="80"/>
                     <img class="userAvatarGradient" src="img/gradient.png" title="{{ member.id }}" width="80" height="24"/>
                     <div class="userPowerLevel" ng-style="{'width': member.powerLevelNorm +'%'}"></div>
-                    <div class="userName">{{ member.displayname || member.id.substr(0, member.id.indexOf(':')) }}<br/>{{ member.displayname ? "" : member.id.substr(member.id.indexOf(':')) }}</div>
+                    <div class="userName">
+                        <div ng-show="member.displayname">
+                            {{ member.id | mUserDisplayName: room_id }}
+                        </div>
+                        <div ng-hide="member.displayname">
+                            {{ member.id.substr(0, member.id.indexOf(':')) }}<br/>
+                            {{ member.id.substr(member.id.indexOf(':')) }}
+                        </div>
+                    </div>
                 </td>
                 <td class="userPresence" ng-class="(member.presence === 'online' ? 'online' : (member.presence === 'unavailable' ? 'unavailable' : '')) + ' ' + (member.membership == 'invite' ? 'invited' : '')">
                     <span ng-show="member.last_active_ago">{{ member.last_active_ago + (now - member.last_updated) | duration }}<br/>ago</span>
@@ -65,7 +73,7 @@
             <tr ng-repeat="msg in events.rooms[room_id].messages"
                 ng-class="(events.rooms[room_id].messages[$index + 1].user_id !== msg.user_id ? 'differentUser' : '') + (msg.user_id === state.user_id ? ' mine' : '')" scroll-item>
                 <td class="leftBlock">
-                    <div class="sender" ng-hide="events.rooms[room_id].messages[$index - 1].user_id === msg.user_id">{{ members[msg.user_id].displayname || msg.user_id }}</div>
+                    <div class="sender" ng-hide="events.rooms[room_id].messages[$index - 1].user_id === msg.user_id"> {{ msg.user_id | mUserDisplayName: room_id }}</div>
                     <div class="timestamp"
                          ng-class="msg.echo_msg_state">
                         {{ (msg.content.hsob_ts || msg.ts) | date:'MMM d HH:mm' }}
@@ -77,10 +85,10 @@
                 </td>
                 <td ng-class="(!msg.content.membership && ('m.room.topic' !== msg.type && 'm.room.name' !== msg.type))? (msg.content.msgtype === 'm.emote' ? 'emote text' : 'text') : 'membership text'">
                     <div class="bubble">
-                        <span ng-if="'join' === msg.content.membership">
+                        <span ng-if="'join' === msg.content.membership && msg.changedKey === 'membership'">
                             {{ members[msg.state_key].displayname || msg.state_key }} joined
                         </span>
-                        <span ng-if="'leave' === msg.content.membership">
+                        <span ng-if="'leave' === msg.content.membership && msg.changedKey === 'membership'">
                             <span ng-if="msg.user_id === msg.state_key">
                                 {{ members[msg.state_key].displayname || msg.state_key }} left
                             </span>
@@ -93,7 +101,8 @@
                                 </span>
                             </span>
                         </span>
-                        <span ng-if="'invite' === msg.content.membership || 'ban' === msg.content.membership">
+                        <span ng-if="'invite' === msg.content.membership && msg.changedKey === 'membership' || 
+                                     'ban' === msg.content.membership && msg.changedKey === 'membership'">
                             {{ members[msg.user_id].displayname || msg.user_id }}
                             {{ {"invite": "invited", "ban": "banned"}[msg.content.membership] }}
                             {{ members[msg.state_key].displayname || msg.state_key }}
@@ -101,6 +110,9 @@
                                 : {{ msg.content.reason }}
                             </span>
                         </span>                        
+                        <span ng-if="msg.changedKey === 'displayname'">
+                            {{ msg.user_id }} changed their display name from {{ msg.prev_content.displayname }} to {{ msg.content.displayname }}
+                        </span>
                         
                         <span ng-show='msg.content.msgtype === "m.emote"'
                               ng-class="msg.echo_msg_state"
@@ -111,8 +123,8 @@
                               ng-class="containsBingWord(msg.content.body) && msg.user_id != state.user_id ? msg.echo_msg_state + ' messageBing' : msg.echo_msg_state"
                               ng-bind-html="((msg.content.msgtype === 'm.text') ? msg.content.body : '') | linky:'_blank'"/>
 
-                        <span ng-show='msg.type === "m.call.invite" && msg.user_id == state.user_id'>Outgoing Call</span>
-                        <span ng-show='msg.type === "m.call.invite" && msg.user_id != state.user_id'>Incoming Call</span>
+                        <span ng-show='msg.type === "m.call.invite" && msg.user_id == state.user_id'>Outgoing Call{{ isWebRTCSupported ? '' : ' (But your browser does not support VoIP)' }}</span>
+                        <span ng-show='msg.type === "m.call.invite" && msg.user_id != state.user_id'>Incoming Call{{ isWebRTCSupported ? '' : ' (But your browser does not support VoIP)' }}</span>
 
                         <div ng-show='msg.content.msgtype === "m.image"'>
                             <div ng-hide='msg.content.thumbnail_url' ng-style="msg.content.body.h && { 'height' : (msg.content.body.h < 320) ? msg.content.body.h : 320}">
@@ -157,10 +169,9 @@
                         {{ state.user_id }} 
                     </td>
                     <td width="*">
-                        <textarea id="mainInput" rows="1" ng-model="textInput" ng-enter="send()"
+                        <textarea id="mainInput" rows="1" ng-enter="send()"
                                   ng-disabled="state.permission_denied"
-                                  ng-keydown="(38 === $event.which) ? history.goUp($event) : ((40 === $event.which) ? history.goDown($event) : 0)"
-                                  ng-focus="true" autocomplete="off" tab-complete/>
+                                  ng-focus="true" autocomplete="off" tab-complete command-history/>
                     </td>
                     <td id="buttonsCell">
                         <button ng-click="send()" ng-disabled="state.permission_denied">Send</button>
@@ -176,7 +187,20 @@
                         <button ng-click="inviteUser()" ng-disabled="state.permission_denied">Invite</button>
                 </span>
                 <button ng-click="leaveRoom()" ng-disabled="state.permission_denied">Leave</button>
-                <button ng-click="startVoiceCall()" ng-show="(currentCall == undefined || currentCall.state == 'ended') && memberCount() == 2" ng-disabled="state.permission_denied">Voice Call</button>
+                <button ng-click="startVoiceCall()"
+                        ng-show="(currentCall == undefined || currentCall.state == 'ended')"
+                        ng-disabled="state.permission_denied || !isWebRTCSupported || memberCount() != 2"
+                        title ="{{ !isWebRTCSupported ? 'VoIP requires webRTC but your browser does not support it' : (memberCount() == 2 ? '' : 'VoIP calls can only be made in rooms with two participants') }}"
+                        >
+                    Voice Call
+                </button>
+                <button ng-click="startVideoCall()"
+                        ng-show="(currentCall == undefined || currentCall.state == 'ended')"
+                        ng-disabled="state.permission_denied || !isWebRTCSupported || memberCount() != 2"
+                        title ="{{ !isWebRTCSupported ? 'VoIP requires webRTC but your browser does not support it' : (memberCount() == 2 ? '' : 'VoIP calls can only be made in rooms with two participants') }}"
+                        >
+                    Video Call
+                </button>
             </div>
         
             {{ feedback }}

webclient/settings/settings.html

@@ -51,10 +51,10 @@
         <h3>Desktop notifications</h3>
         <div class="section" ng-switch="settings.notifications">
             <div ng-switch-when="granted">
-                Notifications are enabled. You will be alerted when a message contains your user ID or display name.
+                Notifications are enabled.
                 <div class="section">
-                    <h4>Additional words to alert on:</h4>
-                    <p>Leave blank to alert on all messages.</p>
+                    <h4>Specific words to alert on:</h4>
+                    <p>Leave blank to alert on all messages. Your username & display name always alerts.</p>
                     <input size=40 name="bingWords" ng-model="settings.bingWords" ng-list placeholder="Enter words separated with , (supports regex)"
                     ng-blur="saveBingWords()"/>
                     <ul>

webclient/test/README

@@ -0,0 +1,33 @@
+Requires:
+ - nodejs/npm
+ - npm install karma
+ - npm install jasmine
+ - npm install protractor (e2e testing)
+
+Setting up continuous integration / run the unit tests (make sure you're in
+this directory so it can find the config file):
+  karma start
+
+
+Setting up e2e tests (only if you don't have a selenium server to run the tests
+on. If you do, edit the config to point to that url):
+
+  webdriver-manager update
+  webdriver-manager start
+
+  Create a file "environment-protractor.js" in this directory and type:
+    module.exports = {
+        seleniumAddress: 'http://localhost:4444/wd/hub',
+        baseUrl: "http://localhost:8008",
+        username: "YOUR_TEST_USERNAME",
+        password: "YOUR_TEST_PASSWORD"
+    }
+
+Running e2e tests:
+  protractor protractor.conf.js
+
+NOTE: This will create a public room on the target home server.
+
+
+
+

webclient/test/e2e/home.spec.js

@@ -0,0 +1,16 @@
+var env = require("../environment-protractor.js");
+
+describe("home page", function() {
+
+    beforeEach(function() {
+        ptor = protractor.getInstance();
+        // FIXME we use longpoll on the event stream, and I can't get $interval
+        // playing nicely with it. Patches welcome to fix this.
+        ptor.ignoreSynchronization = true;
+    }); 
+
+    it("should have a title", function() {
+        browser.get(env.baseUrl);
+        expect(browser.getTitle()).toEqual("[matrix]");
+    });
+});

webclient/test/karma.conf.js

@@ -0,0 +1,82 @@
+// Karma configuration
+// Generated on Thu Sep 18 2014 14:25:57 GMT+0100 (BST)
+
+module.exports = function(config) {
+  config.set({
+
+    // base path that will be used to resolve all patterns (eg. files, exclude)
+    basePath: '',
+
+
+    // frameworks to use
+    // available frameworks: https://npmjs.org/browse/keyword/karma-adapter
+    frameworks: ['jasmine'],
+
+
+    // list of files / patterns to load in the browser
+    // XXX: Order is important, and doing /js/angular* makes the tests not run :/
+    files: [
+      '../js/jquery*',
+      '../js/angular.js',
+      '../js/angular-mocks.js',
+      '../js/angular-route.js',
+      '../js/angular-animate.js',
+      '../js/angular-sanitize.js',
+      '../js/ng-infinite-scroll-matrix.js',
+      '../login/**/*.*',
+      '../room/**/*.*',
+      '../components/**/*.*',
+      '../user/**/*.*',
+      '../home/**/*.*',
+      '../recents/**/*.*',
+      '../settings/**/*.*',
+      '../app.js',
+      '../app*',
+      './unit/**/*.js'
+    ],
+
+
+    // list of files to exclude
+    exclude: [
+    ],
+
+
+    // preprocess matching files before serving them to the browser
+    // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
+    preprocessors: {
+    },
+
+
+    // test results reporter to use
+    // possible values: 'dots', 'progress'
+    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
+    reporters: ['progress'],
+
+
+    // web server port
+    port: 9876,
+
+
+    // enable / disable colors in the output (reporters and logs)
+    colors: true,
+
+
+    // level of logging
+    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
+    logLevel: config.LOG_DEBUG,
+
+
+    // enable / disable watching file and executing tests whenever any file changes
+    autoWatch: true,
+
+
+    // start these browsers
+    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
+    browsers: ['Chrome'],
+
+
+    // Continuous Integration mode
+    // if true, Karma captures browsers, runs the tests and exits
+    singleRun: false
+  });
+};

webclient/test/protractor.conf.js

@@ -0,0 +1,18 @@
+var env = require("./environment-protractor.js");
+exports.config = {
+    seleniumAddress: env.seleniumAddress,
+    specs: ['e2e/*.spec.js'],
+    onPrepare: function() {
+        browser.driver.get(env.baseUrl);
+        browser.driver.findElement(by.id("user_id")).sendKeys(env.username);
+        browser.driver.findElement(by.id("password")).sendKeys(env.password);
+        browser.driver.findElement(by.id("login")).click();
+
+        // wait till the login is done, detect via url change
+        browser.driver.wait(function() {
+            return browser.driver.getCurrentUrl().then(function(url) {
+                return !(/login/.test(url))
+            });
+        });
+    }
+}

webclient/test/unit/user-controller.spec.js

@@ -0,0 +1,57 @@
+describe("UserCtrl", function() {
+    var scope, ctrl, matrixService, routeParams, $q, $timeout;
+    var userId = "@foo:bar";
+    var displayName = "Foo";
+    var avatarUrl = "avatar.url";
+    
+    beforeEach(module('matrixWebClient'));
+
+    beforeEach(function() {
+
+        inject(function($rootScope, $injector, $controller, _$q_, _$timeout_) {
+            $q = _$q_;
+            $timeout = _$timeout_;
+
+            matrixService = {
+                config: function() {
+                    return {
+                        user_id: userId
+                    };
+                },
+
+                getDisplayName: function(uid) {
+                    var d = $q.defer();
+                    d.resolve({
+                        data: {
+                            displayname: displayName
+                        }
+                    });
+                    return d.promise;
+                },
+
+                getProfilePictureUrl: function(uid) {
+                    var d = $q.defer();
+                    d.resolve({
+                        data: {
+                            avatar_url: avatarUrl
+                        }
+                    });
+                    return d.promise;
+                }
+            };
+            scope = $rootScope.$new();
+            routeParams = {
+                user_matrix_id: userId
+            };
+            ctrl = $controller('UserController', {
+                '$scope': scope, 
+                '$routeParams': routeParams, 
+                'matrixService': matrixService
+            });
+        });
+    });
+
+    it('should display your user id', function() {
+        expect(scope.user_id).toEqual(userId);
+    });
+});

webclient/user/user-controller.js

@@ -38,7 +38,8 @@ angular.module('UserController', ['matrixService'])
             $scope.user.avatar_url = response.data.avatar_url;
         }
     );
-    
+
+    // FIXME: factor this out between user-controller and home-controller etc.
     $scope.messageUser = function() {    
         
         // FIXME: create a new room every time, for now