Bump versions and changelog

This uses the ignoreSynchronization flag because of the longpoll on the event
stream. It would be better to use $interval, but couldn't get that to
*reliably* work when testing. I suspect that $interval won't help us here,
since there is genuinely an open $http connection, as we're doing a long
poll. https://github.com/angular/protractor/issues/49 for more info.

.gitignore

@@ -25,5 +25,6 @@ graph/*.png
 graph/*.dot
 
 webclient/config.js
+webclient/test/environment-protractor.js
 
 uploads

CHANGES.rst

@@ -1,3 +1,39 @@
+Changes in synapse 0.3.3 (2014-09-22)
+=====================================
+
+Homeserver:
+ * Fix bug where you continued to get events for rooms you had left.
+
+Webclient:
+ * Add support for video calls with basic UI.
+ * Fix bug where one to one chats were named after your display name rather
+   than the other person's.
+ * Fix bug which caused lag when typing in the textarea.
+ * Refuse to run on browsers we know won't work.
+ * Trigger pagination when joining new rooms.
+ * Fix bug where we sometimes didn't display invitations in recents.
+ * Automatically join room when accepting a VoIP call.
+ * Disable outgoing and reject incoming calls on browsers we don't support
+   VoIP in.
+ * Don't display desktop notifications for messages in the room you are
+   non-idle and speaking in.
+
+Changes in synapse 0.3.2 (2014-09-18)
+=====================================
+
+Webclient:
+ * Fix bug where an empty "bing words" list in old accounts didn't send
+   notifications when it should have done.
+
+Changes in synapse 0.3.1 (2014-09-18)
+=====================================
+This is a release to hotfix v0.3.0 to fix two regressions.
+
+Webclient:
+ * Fix a regression where we sometimes displayed duplicate events.
+ * Fix a regression where we didn't immediately remove rooms you were
+   banned in from the recents list.
+
 Changes in synapse 0.3.0 (2014-09-18)
 =====================================
 See UPGRADE for information about changes to the client server API, including
@@ -25,7 +61,6 @@ Webclient:
  * The recents list now differentiates between public & private rooms.
  * Fix bug where when switching between rooms the pagination flickered before
    the view jumped to the bottom of the screen.
- * Add support for password resets.
  * Add bing word support.
 
 Registration API:

README.rst

@@ -102,7 +102,7 @@ service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc.
 
 Synapse ships with two basic demo Matrix clients: webclient (a basic group chat
 web client demo implemented in AngularJS) and cmdclient (a basic Python
-commandline utility which lets you easily see what the JSON APIs are up to).
+command line utility which lets you easily see what the JSON APIs are up to).
 
 We'd like to invite you to take a look at the Matrix spec, try to run a
 homeserver, and join the existing Matrix chatrooms already out there, experiment
@@ -122,7 +122,7 @@ Homeserver Installation
 First, the dependencies need to be installed.  Start by installing 
 'python2.7-dev' and the various tools of the compiler toolchain.
 
-  Installing prerequisites on ubuntu::
+  Installing prerequisites on Ubuntu::
 
     $ sudo apt-get install build-essential python2.7-dev libffi-dev
 
@@ -151,8 +151,8 @@ you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and
 installing it. Installing PyNaCl using pip may also work (remember to remove any
 other versions installed by setuputils in, for example, ~/.local/lib).
 
-On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'`` you will
-need to ``export CFLAGS=-Qunused-arguments``.
+On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'`` 
+you will need to ``export CFLAGS=-Qunused-arguments``.
 
 This will run a process of downloading and installing into your
 user's .local/lib directory all of the required dependencies that are
@@ -203,9 +203,10 @@ For the first form, simply pass the required hostname (of the machine) as the
         --generate-config
     $ python synapse/app/homeserver.py --config-path homeserver.config
     
-Alternatively, you can run synapse via synctl - running ``synctl start`` to generate a
-homeserver.yaml config file, where you can then edit server-name to specify
-machine.my.domain.name, and then set the actual server running again with synctl start.
+Alternatively, you can run synapse via synctl - running ``synctl start`` to 
+generate a homeserver.yaml config file, where you can then edit server-name to 
+specify machine.my.domain.name, and then set the actual server running again 
+with synctl start.
 
 For the second form, first create your SRV record and publish it in DNS. This
 needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname
@@ -293,7 +294,8 @@ track 3PID logins and publish end-user public keys.
 
 It's currently early days for identity servers as Matrix is not yet using 3PIDs
 as the primary means of identity and E2E encryption is not complete. As such,
-we are running a single identity server (http://matrix.org:8090) at the current time.
+we are running a single identity server (http://matrix.org:8090) at the current 
+time.
 
 
 Where's the spec?!

VERSION

@@ -1 +1 @@
-0.3.0
+0.3.3

synapse/__init__.py

@@ -16,4 +16,4 @@
 """ This is a reference implementation of a synapse home server.
 """
 
-__version__ = "0.3.0"
+__version__ = "0.3.3"

synapse/api/events/__init__.py

@@ -156,7 +156,8 @@ class SynapseEvent(JsonEncodedObject):
                     return "Missing %s key" % key
 
                 if type(content[key]) != type(template[key]):
-                    return "Key %s is of the wrong type." % key
+                    return "Key %s is of the wrong type (got %s, want %s)" % (
+                        key, type(content[key]), type(template[key]))
 
                 if type(content[key]) == dict:
                     # we must go deeper

synapse/app/homeserver.py

@@ -14,7 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.storage import read_schema
+from synapse.storage import prepare_database
 
 from synapse.server import HomeServer
 
@@ -36,30 +36,14 @@ from daemonize import Daemonize
 import twisted.manhole.telnet
 
 import logging
-import sqlite3
 import os
 import re
 import sys
+import sqlite3
 
 logger = logging.getLogger(__name__)
 
 
-SCHEMAS = [
-    "transactions",
-    "pdu",
-    "users",
-    "profiles",
-    "presence",
-    "im",
-    "room_aliases",
-]
-
-
-# Remember to update this number every time an incompatible change is made to
-# database schema files, so the users will be informed on server restarts.
-SCHEMA_VERSION = 3
-
-
 class SynapseHomeServer(HomeServer):
 
     def build_http_client(self):
@@ -80,52 +64,12 @@ class SynapseHomeServer(HomeServer):
         )
 
     def build_db_pool(self):
-        """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we
-        don't have to worry about overwriting existing content.
-        """
-        logging.info("Preparing database: %s...", self.db_name)
-
-        with sqlite3.connect(self.db_name) as db_conn:
-            c = db_conn.cursor()
-            c.execute("PRAGMA user_version")
-            row = c.fetchone()
-
-            if row and row[0]:
-                user_version = row[0]
-
-                if user_version > SCHEMA_VERSION:
-                    raise ValueError("Cannot use this database as it is too " +
-                        "new for the server to understand"
-                    )
-                elif user_version < SCHEMA_VERSION:
-                    logging.info("Upgrading database from version %d",
-                        user_version
-                    )
-
-                    # Run every version since after the current version.
-                    for v in range(user_version + 1, SCHEMA_VERSION + 1):
-                        sql_script = read_schema("delta/v%d" % (v))
-                        c.executescript(sql_script)
-
-                    db_conn.commit()
-
-            else:
-                for sql_loc in SCHEMAS:
-                    sql_script = read_schema(sql_loc)
-
-                    c.executescript(sql_script)
-                db_conn.commit()
-                c.execute("PRAGMA user_version = %d" % SCHEMA_VERSION)
-
-            c.close()
-
-        logging.info("Database prepared in %s.", self.db_name)
-
-        pool = adbapi.ConnectionPool(
-            'sqlite3', self.db_name, check_same_thread=False,
-            cp_min=1, cp_max=1)
-
-        return pool
+        return adbapi.ConnectionPool(
+            "sqlite3", self.get_db_name(),
+            check_same_thread=False,
+            cp_min=1,
+            cp_max=1
+        )
 
     def create_resource_tree(self, web_client, redirect_root_to_web_client):
         """Create the resource tree for this Home Server.
@@ -230,10 +174,6 @@ class SynapseHomeServer(HomeServer):
             logger.info("Synapse now listening on port %d", unsecure_port)
 
 
-def run():
-    reactor.run()
-
-
 def setup():
     config = HomeServerConfig.load_config(
         "Synapse Homeserver",
@@ -268,7 +208,15 @@ def setup():
         web_client=config.webclient,
         redirect_root_to_web_client=True,
     )
-    hs.start_listening(config.bind_port, config.unsecure_port)
+
+    db_name = hs.get_db_name()
+
+    logging.info("Preparing database: %s...", db_name)
+
+    with sqlite3.connect(db_name) as db_conn:
+        prepare_database(db_conn)
+
+    logging.info("Database prepared in %s.", db_name)
 
     hs.get_db_pool()
 
@@ -279,12 +227,14 @@ def setup():
         f.namespace['hs'] = hs
         reactor.listenTCP(config.manhole, f, interface='127.0.0.1')
 
+    hs.start_listening(config.bind_port, config.unsecure_port)
+
     if config.daemonize:
         print config.pid_file
         daemon = Daemonize(
             app="synapse-homeserver",
             pid=config.pid_file,
-            action=run,
+            action=reactor.run,
             auto_close_fds=False,
             verbose=True,
             logger=logger,
@@ -292,7 +242,7 @@ def setup():
 
         daemon.start()
     else:
-        run()
+        reactor.run()
 
 
 if __name__ == '__main__':

synapse/config/captcha.py

@@ -14,13 +14,16 @@
 
 from ._base import Config
 
+
 class CaptchaConfig(Config):
 
     def __init__(self, args):
         super(CaptchaConfig, self).__init__(args)
         self.recaptcha_private_key = args.recaptcha_private_key
         self.enable_registration_captcha = args.enable_registration_captcha
-        self.captcha_ip_origin_is_x_forwarded = args.captcha_ip_origin_is_x_forwarded
+        self.captcha_ip_origin_is_x_forwarded = (
+            args.captcha_ip_origin_is_x_forwarded
+        )
 
     @classmethod
     def add_arguments(cls, parser):
@@ -32,11 +35,12 @@ class CaptchaConfig(Config):
         )
         group.add_argument(
             "--enable-registration-captcha", type=bool, default=False,
-            help="Enables ReCaptcha checks when registering, preventing signup "+
-            "unless a captcha is answered. Requires a valid ReCaptcha public/private key."
+            help="Enables ReCaptcha checks when registering, preventing signup"
+            + " unless a captcha is answered. Requires a valid ReCaptcha "
+            + "public/private key."
         )
         group.add_argument(
             "--captcha_ip_origin_is_x_forwarded", type=bool, default=False,
-            help="When checking captchas, use the X-Forwarded-For (XFF) header as the client IP "+
-            "and not the actual client IP."
+            help="When checking captchas, use the X-Forwarded-For (XFF) header"
+            + " as the client IP and not the actual client IP."
         )

synapse/server.py

@@ -58,6 +58,7 @@ class BaseHomeServer(object):
     DEPENDENCIES = [
         'clock',
         'http_client',
+        'db_name',
         'db_pool',
         'persistence_service',
         'replication_layer',

synapse/storage/__init__.py

@@ -47,6 +47,23 @@ import os
 
 logger = logging.getLogger(__name__)
 
+
+SCHEMAS = [
+    "transactions",
+    "pdu",
+    "users",
+    "profiles",
+    "presence",
+    "im",
+    "room_aliases",
+]
+
+
+# Remember to update this number every time an incompatible change is made to
+# database schema files, so the users will be informed on server restarts.
+SCHEMA_VERSION = 3
+
+
 class _RollbackButIsFineException(Exception):
     """ This exception is used to rollback a transaction without implying
     something went wrong.
@@ -78,7 +95,7 @@ class DataStore(RoomMemberStore, RoomStore,
             stream_ordering = self.min_token
 
         try:
-            yield self._db_pool.runInteraction(
+            yield self.runInteraction(
                 self._persist_pdu_event_txn,
                 pdu=pdu,
                 event=event,
@@ -291,7 +308,7 @@ class DataStore(RoomMemberStore, RoomStore,
                 prev_state_pdu=prev_state_pdu,
             )
 
-        return self._db_pool.runInteraction(_snapshot)
+        return self.runInteraction(_snapshot)
 
 
 class Snapshot(object):
@@ -361,3 +378,42 @@ def read_schema(schema):
     """
     with open(schema_path(schema)) as schema_file:
         return schema_file.read()
+
+
+def prepare_database(db_conn):
+    """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we
+    don't have to worry about overwriting existing content.
+    """
+    c = db_conn.cursor()
+    c.execute("PRAGMA user_version")
+    row = c.fetchone()
+
+    if row and row[0]:
+        user_version = row[0]
+
+        if user_version > SCHEMA_VERSION:
+            raise ValueError("Cannot use this database as it is too " +
+                "new for the server to understand"
+            )
+        elif user_version < SCHEMA_VERSION:
+            logging.info("Upgrading database from version %d",
+                user_version
+            )
+
+            # Run every version since after the current version.
+            for v in range(user_version + 1, SCHEMA_VERSION + 1):
+                sql_script = read_schema("delta/v%d" % (v))
+                c.executescript(sql_script)
+
+            db_conn.commit()
+
+    else:
+        for sql_loc in SCHEMAS:
+            sql_script = read_schema(sql_loc)
+
+            c.executescript(sql_script)
+        db_conn.commit()
+        c.execute("PRAGMA user_version = %d" % SCHEMA_VERSION)
+
+    c.close()
+

synapse/storage/_base.py

@@ -26,6 +26,44 @@ import json
 
 logger = logging.getLogger(__name__)
 
+sql_logger = logging.getLogger("synapse.storage.SQL")
+
+
+class LoggingTransaction(object):
+    """An object that almost-transparently proxies for the 'txn' object
+    passed to the constructor. Adds logging to the .execute() method."""
+    __slots__ = ["txn"]
+
+    def __init__(self, txn):
+        object.__setattr__(self, "txn", txn)
+
+    def __getattribute__(self, name):
+        if name == "execute":
+            return object.__getattribute__(self, "execute")
+
+        return getattr(object.__getattribute__(self, "txn"), name)
+
+    def __setattr__(self, name, value):
+        setattr(object.__getattribute__(self, "txn"), name, value)
+
+    def execute(self, sql, *args, **kwargs):
+        # TODO(paul): Maybe use 'info' and 'debug' for values?
+        sql_logger.debug("[SQL] %s", sql)
+        try:
+            if args and args[0]:
+                values = args[0]
+                sql_logger.debug("[SQL values] " +
+                    ", ".join(("<%s>",) * len(values)), *values)
+        except:
+            # Don't let logging failures stop SQL from working
+            pass
+
+        # TODO(paul): Here would be an excellent place to put some timing
+        #   measurements, and log (warning?) slow queries.
+        return object.__getattribute__(self, "txn").execute(
+            sql, *args, **kwargs
+        )
+
 
 class SQLBaseStore(object):
 
@@ -35,6 +73,13 @@ class SQLBaseStore(object):
         self.event_factory = hs.get_event_factory()
         self._clock = hs.get_clock()
 
+    def runInteraction(self, func, *args, **kwargs):
+        """Wraps the .runInteraction() method on the underlying db_pool."""
+        def inner_func(txn, *args, **kwargs):
+            return func(LoggingTransaction(txn), *args, **kwargs)
+
+        return self._db_pool.runInteraction(inner_func, *args, **kwargs)
+
     def cursor_to_dict(self, cursor):
         """Converts a SQL cursor into an list of dicts.
 
@@ -60,11 +105,6 @@ class SQLBaseStore(object):
         Returns:
             The result of decoder(results)
         """
-        logger.debug(
-            "[SQL] %s  Args=%s Func=%s",
-            query, args, decoder.__name__ if decoder else None
-        )
-
         def interaction(txn):
             cursor = txn.execute(query, args)
             if decoder:
@@ -72,7 +112,7 @@ class SQLBaseStore(object):
             else:
                 return cursor.fetchall()
 
-        return self._db_pool.runInteraction(interaction)
+        return self.runInteraction(interaction)
 
     def _execute_and_decode(self, query, *args):
         return self._execute(self.cursor_to_dict, query, *args)
@@ -88,7 +128,7 @@ class SQLBaseStore(object):
             values : dict of new column names and values for them
             or_replace : bool; if True performs an INSERT OR REPLACE
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._simple_insert_txn, table, values, or_replace=or_replace
         )
 
@@ -172,7 +212,7 @@ class SQLBaseStore(object):
             txn.execute(sql, keyvalues.values())
             return txn.fetchall()
 
-        res = yield self._db_pool.runInteraction(func)
+        res = yield self.runInteraction(func)
 
         defer.returnValue([r[0] for r in res])
 
@@ -195,7 +235,7 @@ class SQLBaseStore(object):
             txn.execute(sql, keyvalues.values())
             return self.cursor_to_dict(txn)
 
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _simple_update_one(self, table, keyvalues, updatevalues,
                            retcols=None):
@@ -263,7 +303,7 @@ class SQLBaseStore(object):
                     raise StoreError(500, "More than one row matched")
 
             return ret
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _simple_delete_one(self, table, keyvalues):
         """Executes a DELETE query on the named table, expecting to delete a
@@ -284,7 +324,7 @@ class SQLBaseStore(object):
                 raise StoreError(404, "No row found")
             if txn.rowcount > 1:
                 raise StoreError(500, "more than one row matched")
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _simple_max_id(self, table):
         """Executes a SELECT query on the named table, expecting to return the
@@ -302,7 +342,7 @@ class SQLBaseStore(object):
                 return 0
             return max_id
 
-        return self._db_pool.runInteraction(func)
+        return self.runInteraction(func)
 
     def _parse_event_from_row(self, row_dict):
         d = copy.deepcopy({k: v for k, v in row_dict.items() if v})
@@ -325,7 +365,7 @@ class SQLBaseStore(object):
         )
 
     def _parse_events(self, rows):
-        return self._db_pool.runInteraction(self._parse_events_txn, rows)
+        return self.runInteraction(self._parse_events_txn, rows)
 
     def _parse_events_txn(self, txn, rows):
         events = [self._parse_event_from_row(r) for r in rows]

synapse/storage/pdu.py

@@ -43,7 +43,7 @@ class PduStore(SQLBaseStore):
             PduTuple: If the pdu does not exist in the database, returns None
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_pdu_tuple, pdu_id, origin
         )
 
@@ -95,7 +95,7 @@ class PduStore(SQLBaseStore):
             list: A list of PduTuples
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_current_state_for_context,
             context
         )
@@ -143,7 +143,7 @@ class PduStore(SQLBaseStore):
             pdu_origin (str)
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._mark_as_processed, pdu_id, pdu_origin
         )
 
@@ -152,7 +152,7 @@ class PduStore(SQLBaseStore):
 
     def get_all_pdus_from_context(self, context):
         """Get a list of all PDUs for a given context."""
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_all_pdus_from_context, context,
         )
 
@@ -179,7 +179,7 @@ class PduStore(SQLBaseStore):
         Return:
             list: A list of PduTuples
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_backfill, context, pdu_list, limit
         )
 
@@ -240,7 +240,7 @@ class PduStore(SQLBaseStore):
             txn
             context (str)
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_min_depth_for_context, context
         )
 
@@ -346,7 +346,7 @@ class PduStore(SQLBaseStore):
             bool
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._is_pdu_new,
             pdu_id=pdu_id,
             origin=origin,
@@ -499,7 +499,7 @@ class StatePduStore(SQLBaseStore):
         )
 
     def get_unresolved_state_tree(self, new_state_pdu):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_unresolved_state_tree, new_state_pdu
         )
 
@@ -538,7 +538,7 @@ class StatePduStore(SQLBaseStore):
 
     def update_current_state(self, pdu_id, origin, context, pdu_type,
                              state_key):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._update_current_state,
             pdu_id, origin, context, pdu_type, state_key
         )
@@ -577,7 +577,7 @@ class StatePduStore(SQLBaseStore):
             PduEntry
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_current_state_pdu, context, pdu_type, state_key
         )
 
@@ -636,7 +636,7 @@ class StatePduStore(SQLBaseStore):
         Returns:
             bool: True if the new_pdu clobbered the current state, False if not
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._handle_new_state, new_pdu
         )
 

synapse/storage/registration.py

@@ -62,7 +62,7 @@ class RegistrationStore(SQLBaseStore):
         Raises:
             StoreError if the user_id could not be registered.
         """
-        yield self._db_pool.runInteraction(self._register, user_id, token,
+        yield self.runInteraction(self._register, user_id, token,
                                            password_hash)
 
     def _register(self, txn, user_id, token, password_hash):
@@ -99,7 +99,7 @@ class RegistrationStore(SQLBaseStore):
         Raises:
             StoreError if no user was found.
         """
-        user_id = yield self._db_pool.runInteraction(self._query_for_auth,
+        user_id = yield self.runInteraction(self._query_for_auth,
                                                      token)
         defer.returnValue(user_id)
 

synapse/storage/room.py

@@ -149,7 +149,7 @@ class RoomStore(SQLBaseStore):
             defer.returnValue(None)
 
     def get_power_level(self, room_id, user_id):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_power_level,
             room_id, user_id,
         )
@@ -182,7 +182,7 @@ class RoomStore(SQLBaseStore):
             return None
 
     def get_ops_levels(self, room_id):
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_ops_levels,
             room_id,
         )

synapse/storage/roommember.py

@@ -149,7 +149,7 @@ class RoomMemberStore(SQLBaseStore):
             membership_list (list): A list of synapse.api.constants.Membership
             values which the user must be in.
         Returns:
-            A list of dicts with "room_id" and "membership" keys.
+            A list of RoomMemberEvent objects
         """
         if not membership_list:
             return defer.succeed(None)
@@ -198,10 +198,11 @@ class RoomMemberStore(SQLBaseStore):
         return results
 
     @defer.inlineCallbacks
-    def user_rooms_intersect(self, user_list):
-        """ Checks whether a list of users share a room.
+    def user_rooms_intersect(self, user_id_list):
+        """ Checks whether all the users whose IDs are given in a list share a
+        room.
         """
-        user_list_clause = " OR ".join(["m.user_id = ?"] * len(user_list))
+        user_list_clause = " OR ".join(["m.user_id = ?"] * len(user_id_list))
         sql = (
             "SELECT m.room_id FROM room_memberships as m "
             "INNER JOIN current_state_events as c "
@@ -211,8 +212,8 @@ class RoomMemberStore(SQLBaseStore):
             "GROUP BY m.room_id HAVING COUNT(m.room_id) = ?"
         ) % {"clause": user_list_clause}
 
-        args = user_list
-        args.append(len(user_list))
+        args = list(user_id_list)
+        args.append(len(user_id_list))
 
         rows = yield self._execute(None, sql, *args)
 

synapse/storage/stream.py

@@ -146,7 +146,7 @@ class StreamStore(SQLBaseStore):
         current_room_membership_sql = (
             "SELECT m.room_id FROM room_memberships as m "
             "INNER JOIN current_state_events as c ON m.event_id = c.event_id "
-            "WHERE m.user_id = ?"
+            "WHERE m.user_id = ? AND m.membership = 'join'"
         )
 
         # We also want to get any membership events about that user, e.g.
@@ -286,7 +286,7 @@ class StreamStore(SQLBaseStore):
         defer.returnValue(ret)
 
     def get_room_events_max_id(self):
-        return self._db_pool.runInteraction(self._get_room_events_max_id_txn)
+        return self.runInteraction(self._get_room_events_max_id_txn)
 
     def _get_room_events_max_id_txn(self, txn):
         txn.execute(

synapse/storage/transactions.py

@@ -41,7 +41,7 @@ class TransactionStore(SQLBaseStore):
             this transaction or a 2-tuple of (int, dict)
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_received_txn_response, transaction_id, origin
         )
 
@@ -72,7 +72,7 @@ class TransactionStore(SQLBaseStore):
             response_json (str)
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._set_received_txn_response,
             transaction_id, origin, code, response_dict
         )
@@ -104,7 +104,7 @@ class TransactionStore(SQLBaseStore):
             list: A list of previous transaction ids.
         """
 
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._prep_send_transaction,
             transaction_id, destination, ts, pdu_list
         )
@@ -159,7 +159,7 @@ class TransactionStore(SQLBaseStore):
             code (int)
             response_json (str)
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._delivered_txn,
             transaction_id, destination, code, response_dict
         )
@@ -184,7 +184,7 @@ class TransactionStore(SQLBaseStore):
         Returns:
             list: A list of `ReceivedTransactionsTable.EntryType`
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_transactions_after, transaction_id, destination
         )
 
@@ -214,7 +214,7 @@ class TransactionStore(SQLBaseStore):
         Returns
             list: A list of PduTuple
         """
-        return self._db_pool.runInteraction(
+        return self.runInteraction(
             self._get_pdus_after_transaction,
             transaction_id, destination
         )

tests/handlers/test_directory.py

@@ -24,6 +24,8 @@ from synapse.http.client import HttpClient
 from synapse.handlers.directory import DirectoryHandler
 from synapse.storage.directory import RoomAliasMapping
 
+from tests.utils import SQLiteMemoryDbPool
+
 
 class DirectoryHandlers(object):
     def __init__(self, hs):
@@ -33,6 +35,7 @@ class DirectoryHandlers(object):
 class DirectoryTestCase(unittest.TestCase):
     """ Tests the directory service. """
 
+    @defer.inlineCallbacks
     def setUp(self):
         self.mock_federation = Mock(spec=[
             "make_query",
@@ -43,11 +46,11 @@ class DirectoryTestCase(unittest.TestCase):
             self.query_handlers[query_type] = handler
         self.mock_federation.register_query_handler = register_query_handler
 
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
         hs = HomeServer("test",
-            datastore=Mock(spec=[
-                "get_association_from_room_alias",
-                "get_joined_hosts_for_room",
-            ]),
+            db_pool=db_pool,
             http_client=None,
             resource_for_federation=Mock(),
             replication_layer=self.mock_federation,
@@ -56,20 +59,16 @@ class DirectoryTestCase(unittest.TestCase):
 
         self.handler = hs.get_handlers().directory_handler
 
-        self.datastore = hs.get_datastore()
-
-        def hosts(room_id):
-            return defer.succeed([])
-        self.datastore.get_joined_hosts_for_room.side_effect = hosts
+        self.store = hs.get_datastore()
 
         self.my_room = hs.parse_roomalias("#my-room:test")
+        self.your_room = hs.parse_roomalias("#your-room:test")
         self.remote_room = hs.parse_roomalias("#another:remote")
 
     @defer.inlineCallbacks
     def test_get_local_association(self):
-        mocked_get = self.datastore.get_association_from_room_alias
-        mocked_get.return_value = defer.succeed(
-            RoomAliasMapping("!8765qwer:test", "#my-room:test", ["test"])
+        yield self.store.create_room_alias_association(
+            self.my_room, "!8765qwer:test", ["test"]
         )
 
         result = yield self.handler.get_association(self.my_room)
@@ -102,9 +101,8 @@ class DirectoryTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_incoming_fed_query(self):
-        mocked_get = self.datastore.get_association_from_room_alias
-        mocked_get.return_value = defer.succeed(
-            RoomAliasMapping("!8765asdf:test", "#your-room:test", ["test"])
+        yield self.store.create_room_alias_association(
+            self.your_room, "!8765asdf:test", ["test"]
         )
 
         response = yield self.query_handlers["directory"](

tests/handlers/test_presence.py

@@ -20,7 +20,9 @@ from twisted.internet import defer, reactor
 from mock import Mock, call, ANY
 import json
 
-from ..utils import MockHttpResource, MockClock, DeferredMockCallable
+from tests.utils import (
+    MockHttpResource, MockClock, DeferredMockCallable, SQLiteMemoryDbPool
+)
 
 from synapse.server import HomeServer
 from synapse.api.constants import PresenceState
@@ -60,30 +62,21 @@ class JustPresenceHandlers(object):
 class PresenceStateTestCase(unittest.TestCase):
     """ Tests presence management. """
 
+    @defer.inlineCallbacks
     def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
         hs = HomeServer("test",
-                clock=MockClock(),
-                db_pool=None,
-                datastore=Mock(spec=[
-                    "get_presence_state",
-                    "set_presence_state",
-                    "add_presence_list_pending",
-                    "set_presence_list_accepted",
-                ]),
-                handlers=None,
-                resource_for_federation=Mock(),
-                http_client=None,
-            )
+            clock=MockClock(),
+            db_pool=db_pool,
+            handlers=None,
+            resource_for_federation=Mock(),
+            http_client=None,
+        )
         hs.handlers = JustPresenceHandlers(hs)
 
-        self.datastore = hs.get_datastore()
-
-        def is_presence_visible(observed_localpart, observer_userid):
-            allow = (observed_localpart == "apple" and
-                observer_userid == "@banana:test"
-            )
-            return defer.succeed(allow)
-        self.datastore.is_presence_visible = is_presence_visible
+        self.store = hs.get_datastore()
 
         # Mock the RoomMemberHandler
         room_member_handler = Mock(spec=[])
@@ -94,6 +87,11 @@ class PresenceStateTestCase(unittest.TestCase):
         self.u_banana = hs.parse_userid("@banana:test")
         self.u_clementine = hs.parse_userid("@clementine:test")
 
+        yield self.store.create_presence(self.u_apple.localpart)
+        yield self.store.set_presence_state(
+            self.u_apple.localpart, {"state": ONLINE, "status_msg": "Online"}
+        )
+
         self.handler = hs.get_handlers().presence_handler
 
         self.room_members = []
@@ -117,7 +115,7 @@ class PresenceStateTestCase(unittest.TestCase):
 
             shared = all(map(lambda i: i in room_member_ids, userlist))
             return defer.succeed(shared)
-        self.datastore.user_rooms_intersect = user_rooms_intersect
+        self.store.user_rooms_intersect = user_rooms_intersect
 
         self.mock_start = Mock()
         self.mock_stop = Mock()
@@ -127,11 +125,6 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_get_my_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
-        )
-
         state = yield self.handler.get_state(
             target_user=self.u_apple, auth_user=self.u_apple
         )
@@ -140,13 +133,12 @@ class PresenceStateTestCase(unittest.TestCase):
             {"presence": ONLINE, "status_msg": "Online"},
             state
         )
-        mocked_get.assert_called_with("apple")
 
     @defer.inlineCallbacks
     def test_get_allowed_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
+        yield self.store.allow_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
         )
 
         state = yield self.handler.get_state(
@@ -157,15 +149,9 @@ class PresenceStateTestCase(unittest.TestCase):
             {"presence": ONLINE, "status_msg": "Online"},
             state
         )
-        mocked_get.assert_called_with("apple")
 
     @defer.inlineCallbacks
     def test_get_same_room_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
-        )
-
         self.room_members = [self.u_apple, self.u_clementine]
 
         state = yield self.handler.get_state(
@@ -179,11 +165,6 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_get_disallowed_state(self):
-        mocked_get = self.datastore.get_presence_state
-        mocked_get.return_value = defer.succeed(
-            {"state": ONLINE, "status_msg": "Online"}
-        )
-
         self.room_members = []
 
         yield self.assertFailure(
@@ -195,16 +176,17 @@ class PresenceStateTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_set_my_state(self):
-        mocked_set = self.datastore.set_presence_state
-        mocked_set.return_value = defer.succeed({"state": OFFLINE})
-
         yield self.handler.set_state(
                 target_user=self.u_apple, auth_user=self.u_apple,
                 state={"presence": UNAVAILABLE, "status_msg": "Away"})
 
-        mocked_set.assert_called_with("apple",
-            {"state": UNAVAILABLE, "status_msg": "Away"}
+        self.assertEquals(
+            {"state": UNAVAILABLE,
+             "status_msg": "Away",
+             "mtime": 1000000},
+            (yield self.store.get_presence_state(self.u_apple.localpart))
         )
+
         self.mock_start.assert_called_with(self.u_apple,
                 state={
                     "presence": UNAVAILABLE,
@@ -222,50 +204,34 @@ class PresenceStateTestCase(unittest.TestCase):
 class PresenceInvitesTestCase(unittest.TestCase):
     """ Tests presence management. """
 
+    @defer.inlineCallbacks
     def setUp(self):
         self.mock_http_client = Mock(spec=[])
         self.mock_http_client.put_json = DeferredMockCallable()
 
         self.mock_federation_resource = MockHttpResource()
 
-        hs = HomeServer("test",
-                clock=MockClock(),
-                db_pool=None,
-                datastore=Mock(spec=[
-                    "has_presence_state",
-                    "allow_presence_visible",
-                    "add_presence_list_pending",
-                    "set_presence_list_accepted",
-                    "get_presence_list",
-                    "del_presence_list",
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
 
-                    # Bits that Federation needs
-                    "prep_send_transaction",
-                    "delivered_txn",
-                    "get_received_txn_response",
-                    "set_received_txn_response",
-                ]),
-                handlers=None,
-                resource_for_client=Mock(),
-                resource_for_federation=self.mock_federation_resource,
-                http_client=self.mock_http_client,
-            )
+        hs = HomeServer("test",
+            clock=MockClock(),
+            db_pool=db_pool,
+            handlers=None,
+            resource_for_client=Mock(),
+            resource_for_federation=self.mock_federation_resource,
+            http_client=self.mock_http_client,
+        )
         hs.handlers = JustPresenceHandlers(hs)
 
-        self.datastore = hs.get_datastore()
-
-        def has_presence_state(user_localpart):
-            return defer.succeed(
-                user_localpart in ("apple", "banana"))
-        self.datastore.has_presence_state = has_presence_state
-
-        def get_received_txn_response(*args):
-            return defer.succeed(None)
-        self.datastore.get_received_txn_response = get_received_txn_response
+        self.store = hs.get_datastore()
 
         # Some local users to test with
         self.u_apple = hs.parse_userid("@apple:test")
         self.u_banana = hs.parse_userid("@banana:test")
+        yield self.store.create_presence(self.u_apple.localpart)
+        yield self.store.create_presence(self.u_banana.localpart)
+
         # ID of a local user that does not exist
         self.u_durian = hs.parse_userid("@durian:test")
 
@@ -288,12 +254,16 @@ class PresenceInvitesTestCase(unittest.TestCase):
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_banana)
 
-        self.datastore.add_presence_list_pending.assert_called_with(
-                "apple", "@banana:test")
-        self.datastore.allow_presence_visible.assert_called_with(
-                "banana", "@apple:test")
-        self.datastore.set_presence_list_accepted.assert_called_with(
-                "apple", "@banana:test")
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 1}],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
+        self.assertTrue(
+            (yield self.store.is_presence_visible(
+                observed_localpart=self.u_banana.localpart,
+                observer_userid=self.u_apple.to_string(),
+            ))
+        )
 
         self.mock_start.assert_called_with(
                 self.u_apple, target_user=self.u_banana)
@@ -303,10 +273,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_durian)
 
-        self.datastore.add_presence_list_pending.assert_called_with(
-                "apple", "@durian:test")
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@durian:test")
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
     @defer.inlineCallbacks
     def test_invite_remote(self):
@@ -328,8 +298,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
         yield self.handler.send_invite(
                 observer_user=self.u_apple, observed_user=self.u_cabbage)
 
-        self.datastore.add_presence_list_pending.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        self.assertEquals(
+            [{"observed_user_id": "@cabbage:elsewhere", "accepted": 0}],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
         yield put_json.await_calls()
 
@@ -362,8 +334,12 @@ class PresenceInvitesTestCase(unittest.TestCase):
             )
         )
 
-        self.datastore.allow_presence_visible.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        self.assertTrue(
+            (yield self.store.is_presence_visible(
+                observed_localpart=self.u_apple.localpart,
+                observer_userid=self.u_cabbage.to_string(),
+            ))
+        )
 
         yield put_json.await_calls()
 
@@ -398,6 +374,11 @@ class PresenceInvitesTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_accepted_remote(self):
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_cabbage.to_string(),
+        )
+
         yield self.mock_federation_resource.trigger("PUT",
             "/_matrix/federation/v1/send/1000000/",
             _make_edu_json("elsewhere", "m.presence_accept",
@@ -408,14 +389,21 @@ class PresenceInvitesTestCase(unittest.TestCase):
             )
         )
 
-        self.datastore.set_presence_list_accepted.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        self.assertEquals(
+            [{"observed_user_id": "@cabbage:elsewhere", "accepted": 1}],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
         self.mock_start.assert_called_with(
                 self.u_apple, target_user=self.u_cabbage)
 
     @defer.inlineCallbacks
     def test_denied_remote(self):
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid="@eggplant:elsewhere",
+        )
+
         yield self.mock_federation_resource.trigger("PUT",
             "/_matrix/federation/v1/send/1000000/",
             _make_edu_json("elsewhere", "m.presence_deny",
@@ -426,32 +414,65 @@ class PresenceInvitesTestCase(unittest.TestCase):
             )
         )
 
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@eggplant:elsewhere")
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
     @defer.inlineCallbacks
     def test_drop_local(self):
-        yield self.handler.drop(
-                observer_user=self.u_apple, observed_user=self.u_banana)
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
 
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@banana:test")
+        yield self.handler.drop(
+            observer_user=self.u_apple,
+            observed_user=self.u_banana,
+        )
+
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
         self.mock_stop.assert_called_with(
                 self.u_apple, target_user=self.u_banana)
 
     @defer.inlineCallbacks
     def test_drop_remote(self):
-        yield self.handler.drop(
-                observer_user=self.u_apple, observed_user=self.u_cabbage)
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_cabbage.to_string(),
+        )
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_cabbage.to_string(),
+        )
 
-        self.datastore.del_presence_list.assert_called_with(
-                "apple", "@cabbage:elsewhere")
+        yield self.handler.drop(
+            observer_user=self.u_apple,
+            observed_user=self.u_cabbage,
+        )
+
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(self.u_apple.localpart))
+        )
 
     @defer.inlineCallbacks
     def test_get_presence_list(self):
-        self.datastore.get_presence_list.return_value = defer.succeed(
-            [{"observed_user_id": "@banana:test"}]
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
         )
 
         presence = yield self.handler.get_presence_list(
@@ -459,29 +480,10 @@ class PresenceInvitesTestCase(unittest.TestCase):
 
         self.assertEquals([
             {"observed_user": self.u_banana,
-             "presence": OFFLINE},
+             "presence": OFFLINE,
+             "accepted": 1},
         ], presence)
 
-        self.datastore.get_presence_list.assert_called_with("apple",
-            accepted=None
-        )
-
-        self.datastore.get_presence_list.return_value = defer.succeed(
-            [{"observed_user_id": "@banana:test"}]
-        )
-
-        presence = yield self.handler.get_presence_list(
-            observer_user=self.u_apple, accepted=True
-        )
-
-        self.assertEquals([
-            {"observed_user": self.u_banana,
-             "presence": OFFLINE},
-        ], presence)
-
-        self.datastore.get_presence_list.assert_called_with("apple",
-                accepted=True)
-
 
 class PresencePushTestCase(unittest.TestCase):
     """ Tests steady-state presence status updates.

tests/handlers/test_profile.py

@@ -24,6 +24,8 @@ from synapse.server import HomeServer
 from synapse.handlers.profile import ProfileHandler
 from synapse.api.constants import Membership
 
+from tests.utils import SQLiteMemoryDbPool
+
 
 class ProfileHandlers(object):
     def __init__(self, hs):
@@ -33,6 +35,7 @@ class ProfileHandlers(object):
 class ProfileTestCase(unittest.TestCase):
     """ Tests profile management. """
 
+    @defer.inlineCallbacks
     def setUp(self):
         self.mock_federation = Mock(spec=[
             "make_query",
@@ -43,63 +46,50 @@ class ProfileTestCase(unittest.TestCase):
             self.query_handlers[query_type] = handler
         self.mock_federation.register_query_handler = register_query_handler
 
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
         hs = HomeServer("test",
-                db_pool=None,
+                db_pool=db_pool,
                 http_client=None,
-                datastore=Mock(spec=[
-                    "get_profile_displayname",
-                    "set_profile_displayname",
-                    "get_profile_avatar_url",
-                    "set_profile_avatar_url",
-                    "get_rooms_for_user_where_membership_is",
-                ]),
                 handlers=None,
                 resource_for_federation=Mock(),
                 replication_layer=self.mock_federation,
             )
         hs.handlers = ProfileHandlers(hs)
 
-        self.datastore = hs.get_datastore()
+        self.store = hs.get_datastore()
 
         self.frank = hs.parse_userid("@1234ABCD:test")
         self.bob   = hs.parse_userid("@4567:test")
         self.alice = hs.parse_userid("@alice:remote")
 
-        self.handler = hs.get_handlers().profile_handler
+        yield self.store.create_profile(self.frank.localpart)
 
-        self.mock_get_joined = (
-            self.datastore.get_rooms_for_user_where_membership_is
-        )
+        self.handler = hs.get_handlers().profile_handler
 
         # TODO(paul): Icky signal declarings.. booo
         hs.get_distributor().declare("changed_presencelike_data")
 
     @defer.inlineCallbacks
     def test_get_my_name(self):
-        mocked_get = self.datastore.get_profile_displayname
-        mocked_get.return_value = defer.succeed("Frank")
+        yield self.store.set_profile_displayname(
+            self.frank.localpart, "Frank"
+        )
 
         displayname = yield self.handler.get_displayname(self.frank)
 
         self.assertEquals("Frank", displayname)
-        mocked_get.assert_called_with("1234ABCD")
 
     @defer.inlineCallbacks
     def test_set_my_name(self):
-        mocked_set = self.datastore.set_profile_displayname
-        mocked_set.return_value = defer.succeed(())
-
-        self.mock_get_joined.return_value = defer.succeed([])
-
         yield self.handler.set_displayname(self.frank, self.frank, "Frank Jr.")
 
-        self.mock_get_joined.assert_called_once_with(
-            self.frank.to_string(),
-            [Membership.JOIN]
+        self.assertEquals(
+            (yield self.store.get_profile_displayname(self.frank.localpart)),
+            "Frank Jr."
         )
 
-        mocked_set.assert_called_with("1234ABCD", "Frank Jr.")
-
     @defer.inlineCallbacks
     def test_set_my_name_noauth(self):
         d = self.handler.set_displayname(self.frank, self.bob, "Frank Jr.")
@@ -123,40 +113,31 @@ class ProfileTestCase(unittest.TestCase):
 
     @defer.inlineCallbacks
     def test_incoming_fed_query(self):
-        mocked_get = self.datastore.get_profile_displayname
-        mocked_get.return_value = defer.succeed("Caroline")
+        yield self.store.create_profile("caroline")
+        yield self.store.set_profile_displayname("caroline", "Caroline")
 
         response = yield self.query_handlers["profile"](
             {"user_id": "@caroline:test", "field": "displayname"}
         )
 
         self.assertEquals({"displayname": "Caroline"}, response)
-        mocked_get.assert_called_with("caroline")
 
     @defer.inlineCallbacks
     def test_get_my_avatar(self):
-        mocked_get = self.datastore.get_profile_avatar_url
-        mocked_get.return_value = defer.succeed("http://my.server/me.png")
+        yield self.store.set_profile_avatar_url(
+            self.frank.localpart, "http://my.server/me.png"
+        )
 
         avatar_url = yield self.handler.get_avatar_url(self.frank)
 
         self.assertEquals("http://my.server/me.png", avatar_url)
-        mocked_get.assert_called_with("1234ABCD")
 
     @defer.inlineCallbacks
     def test_set_my_avatar(self):
-        mocked_set = self.datastore.set_profile_avatar_url
-        mocked_set.return_value = defer.succeed(())
-
-        self.mock_get_joined.return_value = defer.succeed([])
-
         yield self.handler.set_avatar_url(self.frank, self.frank,
                 "http://my.server/pic.gif")
 
-        self.mock_get_joined.assert_called_once_with(
-            self.frank.to_string(),
-            [Membership.JOIN]
+        self.assertEquals(
+            (yield self.store.get_profile_avatar_url(self.frank.localpart)),
+            "http://my.server/pic.gif"
         )
-
-
-        mocked_set.assert_called_with("1234ABCD", "http://my.server/pic.gif")

tests/storage/TESTS_NEEDED_FOR

@@ -0,0 +1,5 @@
+synapse/storage/feedback.py
+synapse/storage/keys.py
+synapse/storage/pdu.py
+synapse/storage/stream.py
+synapse/storage/transactions.py

tests/storage/test_directory.py

@@ -0,0 +1,68 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.directory import DirectoryStore
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class DirectoryStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        self.store = DirectoryStore(hs)
+
+        self.room = hs.parse_roomid("!abcde:test")
+        self.alias = hs.parse_roomalias("#my-room:test")
+
+    @defer.inlineCallbacks
+    def test_room_to_alias(self):
+        yield self.store.create_room_alias_association(
+            room_alias=self.alias,
+            room_id=self.room.to_string(),
+            servers=["test"],
+        )
+
+        self.assertEquals(
+            ["#my-room:test"],
+            (yield self.store.get_aliases_for_room(self.room.to_string()))
+        )
+
+    @defer.inlineCallbacks
+    def test_alias_to_room(self):
+        yield self.store.create_room_alias_association(
+            room_alias=self.alias,
+            room_id=self.room.to_string(),
+            servers=["test"],
+        )
+
+
+        self.assertObjectHasAttributes(
+            {"room_id": self.room.to_string(),
+             "servers": ["test"]},
+            (yield self.store.get_association_from_room_alias(self.alias))
+        )

tests/storage/test_presence.py

@@ -0,0 +1,167 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.presence import PresenceStore
+
+from tests.utils import SQLiteMemoryDbPool, MockClock
+
+
+class PresenceStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            clock=MockClock(),
+            db_pool=db_pool,
+        )
+
+        self.store = PresenceStore(hs)
+
+        self.u_apple = hs.parse_userid("@apple:test")
+        self.u_banana = hs.parse_userid("@banana:test")
+
+    @defer.inlineCallbacks
+    def test_state(self):
+        yield self.store.create_presence(
+            self.u_apple.localpart
+        )
+
+        state = yield self.store.get_presence_state(
+            self.u_apple.localpart
+        )
+
+        self.assertEquals(
+            {"state": None, "status_msg": None, "mtime": None}, state
+        )
+
+        yield self.store.set_presence_state(
+            self.u_apple.localpart, {"state": "online", "status_msg": "Here"}
+        )
+
+        state = yield self.store.get_presence_state(
+            self.u_apple.localpart
+        )
+
+        self.assertEquals(
+            {"state": "online", "status_msg": "Here", "mtime": 1000000}, state
+        )
+
+    @defer.inlineCallbacks
+    def test_visibility(self):
+        self.assertFalse((yield self.store.is_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )))
+
+        yield self.store.allow_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )
+
+        self.assertTrue((yield self.store.is_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )))
+
+        yield self.store.disallow_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )
+
+        self.assertFalse((yield self.store.is_presence_visible(
+            observed_localpart=self.u_apple.localpart,
+            observer_userid=self.u_banana.to_string(),
+        )))
+
+    @defer.inlineCallbacks
+    def test_presence_list(self):
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )
+
+        yield self.store.add_presence_list_pending(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 0}],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )
+
+        yield self.store.set_presence_list_accepted(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 1}],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [{"observed_user_id": "@banana:test", "accepted": 1}],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )
+
+        yield self.store.del_presence_list(
+            observer_localpart=self.u_apple.localpart,
+            observed_userid=self.u_banana.to_string(),
+        )
+
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+            ))
+        )
+        self.assertEquals(
+            [],
+            (yield self.store.get_presence_list(
+                observer_localpart=self.u_apple.localpart,
+                accepted=True,
+            ))
+        )

tests/storage/test_profile.py

@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.profile import ProfileStore
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class ProfileStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        self.store = ProfileStore(hs)
+
+        self.u_frank = hs.parse_userid("@frank:test")
+
+    @defer.inlineCallbacks
+    def test_displayname(self):
+        yield self.store.create_profile(
+            self.u_frank.localpart
+        )
+
+        yield self.store.set_profile_displayname(
+            self.u_frank.localpart, "Frank"
+        )
+
+        self.assertEquals(
+            "Frank",
+            (yield self.store.get_profile_displayname(self.u_frank.localpart))
+        )
+
+    @defer.inlineCallbacks
+    def test_avatar_url(self):
+        yield self.store.create_profile(
+            self.u_frank.localpart
+        )
+
+        yield self.store.set_profile_avatar_url(
+                self.u_frank.localpart, "http://my.site/here"
+        )
+
+        self.assertEquals(
+            "http://my.site/here",
+            (yield self.store.get_profile_avatar_url(self.u_frank.localpart))
+        )

tests/storage/test_registration.py

@@ -0,0 +1,69 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.storage.registration import RegistrationStore
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RegistrationStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        self.store = RegistrationStore(hs)
+
+        self.user_id = "@my-user:test"
+        self.tokens = ["AbCdEfGhIjKlMnOpQrStUvWxYz",
+                        "BcDeFgHiJkLmNoPqRsTuVwXyZa"]
+        self.pwhash = "{xx1}123456789"
+
+    @defer.inlineCallbacks
+    def test_register(self):
+        yield self.store.register(self.user_id, self.tokens[0], self.pwhash)
+
+        self.assertEquals(
+            # TODO(paul): Surely this field should be 'user_id', not 'name'
+            #  Additionally surely it shouldn't come in a 1-element list
+            [{"name": self.user_id, "password_hash": self.pwhash}],
+            (yield self.store.get_user_by_id(self.user_id))
+        )
+
+        self.assertEquals(
+            self.user_id,
+            (yield self.store.get_user_by_token(self.tokens[0]))
+        )
+
+    @defer.inlineCallbacks
+    def test_add_tokens(self):
+        yield self.store.register(self.user_id, self.tokens[0], self.pwhash)
+        yield self.store.add_access_token_to_user(self.user_id, self.tokens[1])
+
+        self.assertEquals(
+            self.user_id,
+            (yield self.store.get_user_by_token(self.tokens[1]))
+        )
+

tests/storage/test_room.py

@@ -0,0 +1,176 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.events.room import (
+    RoomNameEvent, RoomTopicEvent
+)
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RoomStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        # We can't test RoomStore on its own without the DirectoryStore, for
+        # management of the 'room_aliases' table
+        self.store = hs.get_datastore()
+
+        self.room = hs.parse_roomid("!abcde:test")
+        self.alias = hs.parse_roomalias("#a-room-name:test")
+        self.u_creator = hs.parse_userid("@creator:test")
+
+        yield self.store.store_room(self.room.to_string(),
+            room_creator_user_id=self.u_creator.to_string(),
+            is_public=True
+        )
+
+    @defer.inlineCallbacks
+    def test_get_room(self):
+        self.assertObjectHasAttributes(
+            {"room_id": self.room.to_string(),
+             "creator": self.u_creator.to_string(),
+             "is_public": True},
+            (yield self.store.get_room(self.room.to_string()))
+        )
+
+    @defer.inlineCallbacks
+    def test_store_room_config(self):
+        yield self.store.store_room_config(self.room.to_string(),
+            visibility=False
+        )
+
+        self.assertObjectHasAttributes(
+            {"is_public": False},
+            (yield self.store.get_room(self.room.to_string()))
+        )
+
+    @defer.inlineCallbacks
+    def test_get_rooms(self):
+        # get_rooms does an INNER JOIN on the room_aliases table :(
+
+        rooms = yield self.store.get_rooms(is_public=True)
+        # Should be empty before we add the alias
+        self.assertEquals([], rooms)
+
+        yield self.store.create_room_alias_association(
+            room_alias=self.alias,
+            room_id=self.room.to_string(),
+            servers=["test"]
+        )
+
+        rooms = yield self.store.get_rooms(is_public=True)
+
+        self.assertEquals(1, len(rooms))
+        self.assertEquals({
+            "name": None,
+            "room_id": self.room.to_string(),
+            "topic": None,
+            "aliases": [self.alias.to_string()],
+        }, rooms[0])
+
+
+class RoomEventsStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        # Room events need the full datastore, for persist_event() and
+        # get_room_state()
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory();
+
+        self.room = hs.parse_roomid("!abcde:test")
+
+        yield self.store.store_room(self.room.to_string(),
+            room_creator_user_id="@creator:text",
+            is_public=True
+        )
+
+    @defer.inlineCallbacks
+    def inject_room_event(self, **kwargs):
+        yield self.store.persist_event(
+            self.event_factory.create_event(
+                room_id=self.room.to_string(),
+                **kwargs
+            )
+        )
+
+    @defer.inlineCallbacks
+    def test_room_name(self):
+        name = u"A-Room-Name"
+
+        yield self.inject_room_event(
+            etype=RoomNameEvent.TYPE,
+            name=name,
+            content={"name": name},
+            depth=1,
+        )
+
+        state = yield self.store.get_current_state(
+            room_id=self.room.to_string()
+        )
+
+        self.assertEquals(1, len(state))
+        self.assertObjectHasAttributes(
+            {"type": "m.room.name",
+             "room_id": self.room.to_string(),
+             "name": name},
+            state[0]
+        )
+
+    @defer.inlineCallbacks
+    def test_room_name(self):
+        topic = u"A place for things"
+
+        yield self.inject_room_event(
+            etype=RoomTopicEvent.TYPE,
+            topic=topic,
+            content={"topic": topic},
+            depth=1,
+        )
+
+        state = yield self.store.get_current_state(
+            room_id=self.room.to_string()
+        )
+
+        self.assertEquals(1, len(state))
+        self.assertObjectHasAttributes(
+            {"type": "m.room.topic",
+             "room_id": self.room.to_string(),
+             "topic": topic},
+            state[0]
+        )
+
+    # Not testing the various 'level' methods for now because there's lots
+    # of them and need coalescing; see JIRA SPEC-11

tests/storage/test_roommember.py

@@ -0,0 +1,157 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.constants import Membership
+from synapse.api.events.room import RoomMemberEvent
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class RoomMemberStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer("test",
+            db_pool=db_pool,
+        )
+
+        # We can't test the RoomMemberStore on its own without the other event
+        # storage logic
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory()
+
+        self.u_alice = hs.parse_userid("@alice:test")
+        self.u_bob = hs.parse_userid("@bob:test")
+
+        # User elsewhere on another host
+        self.u_charlie = hs.parse_userid("@charlie:elsewhere")
+
+        self.room = hs.parse_roomid("!abc123:test")
+
+    @defer.inlineCallbacks
+    def inject_room_member(self, room, user, membership):
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            self.event_factory.create_event(
+                etype=RoomMemberEvent.TYPE,
+                user_id=user.to_string(),
+                state_key=user.to_string(),
+                room_id=room.to_string(),
+                membership=membership,
+                content={"membership": membership},
+                depth=1,
+            )
+        )
+
+    @defer.inlineCallbacks
+    def test_one_member(self):
+        yield self.inject_room_member(self.room, self.u_alice, Membership.JOIN)
+
+        self.assertEquals(
+            Membership.JOIN,
+            (yield self.store.get_room_member(
+                user_id=self.u_alice.to_string(),
+                room_id=self.room.to_string(),
+            )).membership
+        )
+        self.assertEquals(
+            [self.u_alice.to_string()],
+            [m.user_id for m in (
+                yield self.store.get_room_members(self.room.to_string())
+            )]
+        )
+        self.assertEquals(
+            [self.room.to_string()],
+            [m.room_id for m in (
+                yield self.store.get_rooms_for_user_where_membership_is(
+                    self.u_alice.to_string(), [Membership.JOIN]
+                ))
+            ]
+        )
+        self.assertFalse(
+            (yield self.store.user_rooms_intersect(
+                [self.u_alice.to_string(), self.u_bob.to_string()]
+            ))
+        )
+
+    @defer.inlineCallbacks
+    def test_two_members(self):
+        yield self.inject_room_member(self.room, self.u_alice, Membership.JOIN)
+        yield self.inject_room_member(self.room, self.u_bob, Membership.JOIN)
+
+        self.assertEquals(
+            {self.u_alice.to_string(), self.u_bob.to_string()},
+            {m.user_id for m in (
+                yield self.store.get_room_members(self.room.to_string())
+            )}
+        )
+        self.assertTrue(
+            (yield self.store.user_rooms_intersect(
+                [self.u_alice.to_string(), self.u_bob.to_string()]
+            ))
+        )
+
+    @defer.inlineCallbacks
+    def test_room_hosts(self):
+        yield self.inject_room_member(self.room, self.u_alice, Membership.JOIN)
+
+        self.assertEquals(
+            ["test"],
+            (yield self.store.get_joined_hosts_for_room(self.room.to_string()))
+        )
+
+        # Should still have just one host after second join from it
+        yield self.inject_room_member(self.room, self.u_bob, Membership.JOIN)
+
+        self.assertEquals(
+            ["test"],
+            (yield self.store.get_joined_hosts_for_room(self.room.to_string()))
+        )
+
+        # Should now have two hosts after join from other host
+        yield self.inject_room_member(self.room, self.u_charlie, Membership.JOIN)
+
+        self.assertEquals(
+            {"test", "elsewhere"},
+            set((yield
+                self.store.get_joined_hosts_for_room(self.room.to_string())
+            ))
+        )
+
+        # Should still have both hosts
+        yield self.inject_room_member(self.room, self.u_alice, Membership.LEAVE)
+
+        self.assertEquals(
+            {"test", "elsewhere"},
+            set((yield
+                self.store.get_joined_hosts_for_room(self.room.to_string())
+            ))
+        )
+
+        # Should have only one host after other leaves
+        yield self.inject_room_member(self.room, self.u_charlie, Membership.LEAVE)
+
+        self.assertEquals(
+            ["test"],
+            (yield self.store.get_joined_hosts_for_room(self.room.to_string()))
+        )

tests/storage/test_stream.py

@@ -0,0 +1,226 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from tests import unittest
+from twisted.internet import defer
+
+from synapse.server import HomeServer
+from synapse.api.constants import Membership
+from synapse.api.events.room import RoomMemberEvent, MessageEvent
+
+from tests.utils import SQLiteMemoryDbPool
+
+
+class StreamStoreTestCase(unittest.TestCase):
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        db_pool = SQLiteMemoryDbPool()
+        yield db_pool.prepare()
+
+        hs = HomeServer(
+            "test",
+            db_pool=db_pool,
+        )
+
+        self.store = hs.get_datastore()
+        self.event_factory = hs.get_event_factory()
+
+        self.u_alice = hs.parse_userid("@alice:test")
+        self.u_bob = hs.parse_userid("@bob:test")
+
+        self.room1 = hs.parse_roomid("!abc123:test")
+        self.room2 = hs.parse_roomid("!xyx987:test")
+
+        self.depth = 1
+
+    @defer.inlineCallbacks
+    def inject_room_member(self, room, user, membership, prev_state=None):
+        self.depth += 1
+
+        event = self.event_factory.create_event(
+            etype=RoomMemberEvent.TYPE,
+            user_id=user.to_string(),
+            state_key=user.to_string(),
+            room_id=room.to_string(),
+            membership=membership,
+            content={"membership": membership},
+            depth=self.depth,
+        )
+
+        if prev_state:
+            event.prev_state = prev_state
+
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            event
+        )
+
+        defer.returnValue(event)
+
+    @defer.inlineCallbacks
+    def inject_message(self, room, user, body):
+        self.depth += 1
+
+        # Have to create a join event using the eventfactory
+        yield self.store.persist_event(
+            self.event_factory.create_event(
+                etype=MessageEvent.TYPE,
+                user_id=user.to_string(),
+                room_id=room.to_string(),
+                content={"body": body, "msgtype": u"message"},
+                depth=self.depth,
+            )
+        )
+
+    @defer.inlineCallbacks
+    def test_event_stream_get_other(self):
+        # Both bob and alice joins the room
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        # Initial stream key:
+        start = yield self.store.get_room_events_max_id()
+
+        yield self.inject_message(self.room1, self.u_alice, u"test")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_bob.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": MessageEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {"body": "test", "msgtype": "message"},
+            },
+            event,
+        )
+
+    @defer.inlineCallbacks
+    def test_event_stream_get_own(self):
+        # Both bob and alice joins the room
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        # Initial stream key:
+        start = yield self.store.get_room_events_max_id()
+
+        yield self.inject_message(self.room1, self.u_alice, u"test")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_alice.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        self.assertEqual(1, len(results))
+
+        event = results[0]
+
+        self.assertObjectHasAttributes(
+            {
+                "type": MessageEvent.TYPE,
+                "user_id": self.u_alice.to_string(),
+                "content": {"body": "test", "msgtype": "message"},
+            },
+            event,
+        )
+
+    @defer.inlineCallbacks
+    def test_event_stream_join_leave(self):
+        # Both bob and alice joins the room
+        yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        # Then bob leaves again.
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.LEAVE
+        )
+
+        # Initial stream key:
+        start = yield self.store.get_room_events_max_id()
+
+        yield self.inject_message(self.room1, self.u_alice, u"test")
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_bob.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        # We should not get the message, as it happened *after* bob left.
+        self.assertEqual(0, len(results))
+
+    @defer.inlineCallbacks
+    def test_event_stream_prev_content(self):
+        yield self.inject_room_member(
+            self.room1, self.u_bob, Membership.JOIN
+        )
+
+        event1 = yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN
+        )
+
+        start = yield self.store.get_room_events_max_id()
+
+        event2 = yield self.inject_room_member(
+            self.room1, self.u_alice, Membership.JOIN,
+            prev_state=event1.event_id,
+        )
+
+        end = yield self.store.get_room_events_max_id()
+
+        results, _ = yield self.store.get_room_events_stream(
+            self.u_bob.to_string(),
+            start,
+            end,
+            None,  # Is currently ignored
+        )
+
+        # We should not get the message, as it happened *after* bob left.
+        self.assertEqual(1, len(results))
+
+        event = results[0]
+
+        self.assertTrue(hasattr(event, "prev_content"), msg="No prev_content key")

tests/unittest.py

@@ -71,6 +71,17 @@ class TestCase(unittest.TestCase):
             logging.getLogger().setLevel(level)
             return orig()
 
+    def assertObjectHasAttributes(self, attrs, obj):
+        """Asserts that the given object has each of the attributes given, and
+        that the value of each matches according to assertEquals."""
+        for (key, value) in attrs.items():
+            if not hasattr(obj, key):
+                raise AssertionError("Expected obj to have a '.%s'" % key)
+            try:
+                self.assertEquals(attrs[key], getattr(obj, key))
+            except AssertionError as e:
+                raise (type(e))(e.message + " for '.%s'" % key)
+
 
 def DEBUG(target):
     """A decorator to set the .loglevel attribute to logging.DEBUG.

tests/utils.py

@@ -16,12 +16,14 @@
 from synapse.http.server import HttpServer
 from synapse.api.errors import cs_error, CodeMessageException, StoreError
 from synapse.api.constants import Membership
+from synapse.storage import prepare_database
 
 from synapse.api.events.room import (
     RoomMemberEvent, MessageEvent
 )
 
 from twisted.internet import defer, reactor
+from twisted.enterprise.adbapi import ConnectionPool
 
 from collections import namedtuple
 from mock import patch, Mock
@@ -120,6 +122,18 @@ class MockClock(object):
         self.now += secs
 
 
+class SQLiteMemoryDbPool(ConnectionPool, object):
+    def __init__(self):
+        super(SQLiteMemoryDbPool, self).__init__(
+            "sqlite3", ":memory:",
+            cp_min=1,
+            cp_max=1,
+        )
+
+    def prepare(self):
+        return self.runWithConnection(prepare_database)
+
+
 class MemoryDataStore(object):
 
     Room = namedtuple(

webclient/README

@@ -1,8 +1,8 @@
 Basic Usage
 -----------
 
-The web client should automatically run when running the home server. Alternatively, you can run
-it stand-alone:
+The web client should automatically run when running the home server. 
+Alternatively, you can run it stand-alone:
 
     $ python -m SimpleHTTPServer
 

webclient/app-controller.js

@@ -26,6 +26,12 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
          
     // Check current URL to avoid to display the logout button on the login page
     $scope.location = $location.path();
+
+    // disable nganimate for the local and remote video elements because ngAnimate appears
+    // to be buggy and leaves animation classes on the video elements causing them to show
+    // when they should not (their animations are pure CSS3)
+    $animate.enabled(false, angular.element('#localVideo'));
+    $animate.enabled(false, angular.element('#remoteVideo'));
     
     // Update the location state when the ng location changed
     $rootScope.$on('$routeChangeSuccess', function (event, current, previous) {
@@ -93,7 +99,13 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
     };
 
     $rootScope.$watch('currentCall', function(newVal, oldVal) {
-        if (!$rootScope.currentCall) return;
+        if (!$rootScope.currentCall) {
+            // This causes the still frame to be flushed out of the video elements,
+            // avoiding a flash of the last frame of the previous call when starting the next
+            angular.element('#localVideo')[0].load();
+            angular.element('#remoteVideo')[0].load();
+            return;
+        }
 
         var roomMembers = angular.copy($rootScope.events.rooms[$rootScope.currentCall.room_id].members);
         delete roomMembers[matrixService.config().user_id];
@@ -126,6 +138,7 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
             angular.element('#ringAudio')[0].pause();
             angular.element('#ringbackAudio')[0].pause();
             angular.element('#callendAudio')[0].play();
+            $scope.videoMode = undefined;
         } else if (newVal == 'ended' && oldVal == 'invite_sent' && $rootScope.currentCall.hangupParty == 'remote') {
             angular.element('#ringAudio')[0].pause();
             angular.element('#ringbackAudio')[0].pause();
@@ -138,6 +151,20 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
             angular.element('#ringbackAudio')[0].pause();
         } else if (oldVal == 'ringing') {
             angular.element('#ringAudio')[0].pause();
+        } else if (newVal == 'connected') {
+            $timeout(function() {
+                if ($scope.currentCall.type == 'video') $scope.videoMode = 'large';
+            }, 500);
+        }
+
+        if ($rootScope.currentCall && $rootScope.currentCall.type == 'video' && $rootScope.currentCall.state != 'connected') {
+            $scope.videoMode = 'mini';
+        }
+    });
+    $rootScope.$watch('currentCall.type', function(newVal, oldVal) {
+        // need to listen for this too as the type of the call won't be know when it's created
+        if ($rootScope.currentCall && $rootScope.currentCall.type == 'video' && $rootScope.currentCall.state != 'connected') {
+            $scope.videoMode = 'mini';
         }
     });
 
@@ -150,6 +177,8 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
         }
         call.onError = $scope.onCallError;
         call.onHangup = $scope.onCallHangup;
+        call.localVideoElement = angular.element('#localVideo')[0];
+        call.remoteVideoElement = angular.element('#remoteVideo')[0];
         $rootScope.currentCall = call;
     });
 
@@ -170,7 +199,7 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
     
     $rootScope.onCallError = function(errStr) {
         $scope.feedback = errStr;
-    }
+    };
 
     $rootScope.onCallHangup = function(call) {
         if (call == $rootScope.currentCall) {
@@ -178,5 +207,5 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even
                 if (call == $rootScope.currentCall) $rootScope.currentCall = undefined;
             }, 4070);
         }
-    }
+    };
 }]);

webclient/app.css

@@ -20,7 +20,12 @@ a:visited { color: #666; }
 a:hover   { color: #000; }
 a:active  { color: #000; }
 
-#page {
+textarea, input {
+   font-family: inherit;
+   font-size: inherit;
+}
+
+.page {
     min-height: 100%;
     margin-bottom: -32px; /* to make room for the footer */
 }
@@ -34,9 +39,15 @@ a:active  { color: #000; }
     padding-right: 20px;
 }
 
+#unsupportedBrowser {
+    padding-top: 240px;
+    text-align: center;
+}
+
 #header
 {
     position: absolute;
+    z-index: 2;
     top: 0px;
     width: 100%;
     background-color: #333;
@@ -89,6 +100,80 @@ a:active  { color: #000; }
     font-size: 80%;
 }
 
+#videoBackground {
+    position: absolute;
+    height: 100%;
+    width: 100%;
+    top: 0px;
+    left: 0px;
+    z-index: 1;
+    background-color: rgba(0,0,0,0.0);
+    pointer-events: none;
+    transition: background-color linear 500ms;
+}
+
+#videoBackground.large {
+    background-color: rgba(0,0,0,0.85);
+    pointer-events: auto;
+}
+
+#videoContainer {
+    position: relative;
+    top: 32px;
+    max-width: 1280px;
+    margin: auto;
+}
+
+#videoContainerPadding {
+    width: 1280px;
+}
+
+#localVideo {
+    position: absolute;
+    width: 128px;
+    height: 72px;
+    z-index: 1;
+    transition: left linear 500ms, top linear 500ms, width linear 500ms, height linear 500ms;
+}
+
+#localVideo.mini {
+    top: 0px;
+    left: 130px;
+}
+
+#localVideo.large {
+    top: 70px;
+    left: 20px;
+}
+
+#localVideo.ended {
+    -webkit-filter: grayscale(1);
+    filter: grayscale(1);
+}
+
+#remoteVideo {
+    position: relative;
+    height: auto;
+    transition: left linear 500ms, top linear 500ms, width linear 500ms, height linear 500ms;
+}
+
+#remoteVideo.mini {
+    left: 260px;
+    top: 0px;
+    width: 128px;
+}
+
+#remoteVideo.large {
+    left: 0px;
+    top: 50px;
+    width: 100%;
+}
+
+#remoteVideo.ended {
+    -webkit-filter: grayscale(1);
+    filter: grayscale(1);
+}
+
 #headerContent {
     color: #ccc;
     max-width: 1280px;
@@ -96,6 +181,7 @@ a:active  { color: #000; }
     text-align: right;
     height: 32px;
     line-height: 32px;
+    position: relative;
 }
 
 #headerContent a:link,

webclient/app.js

@@ -80,7 +80,24 @@ matrixWebClient.config(['$routeProvider', '$provide', '$httpProvider',
         $httpProvider.interceptors.push('AccessTokenInterceptor');
     }]);
 
-matrixWebClient.run(['$location', 'matrixService', function($location, matrixService) {
+matrixWebClient.run(['$location', '$rootScope', 'matrixService', function($location, $rootScope, matrixService) {
+
+    // Check browser support
+    // Support IE from 9.0. AngularJS needs some tricks to run on IE8 and below
+    var version = parseFloat($.browser.version);
+    if ($.browser.msie && version < 9.0) {
+        $rootScope.unsupportedBrowser = {
+            browser: navigator.userAgent,
+            reason: "Internet Explorer is supported from version 9"
+        };
+    }
+    // The app requires localStorage
+    if(typeof(Storage) === "undefined") {
+        $rootScope.unsupportedBrowser = {
+            browser: navigator.userAgent,
+            reason: "It does not support HTML local storage"
+        };
+    }
 
     // If user auth details are not in cache, go to the login page
     if (!matrixService.isUserLoggedIn() &&

webclient/components/fileInput/file-input-directive.js

@@ -31,13 +31,23 @@ angular.module('mFileInput', [])
         },
 
         link: function(scope, element, attrs, ctrl) {
-            element.bind("click", function() {
-                element.find("input")[0].click();
-                element.find("input").bind("change", function(e) {
-                    scope.selectedFile = this.files[0];
-                    scope.$apply();
+            
+            // Check if HTML5 file selection is supported
+            if (window.FileList) {
+                element.bind("click", function() {
+                    element.find("input")[0].click();
+                    element.find("input").bind("change", function(e) {
+                        scope.selectedFile = this.files[0];
+                        scope.$apply();
+                    });
                 });
-            });
+            }
+            else {
+                setTimeout(function() {
+                    element.attr("disabled", true);
+                    element.attr("title", "The app uses the HTML5 File API to send files. Your browser does not support it.");
+                }, 1);
+            }
 
             // Change the mouse icon on mouseover on this element
             element.css("cursor", "pointer");

webclient/components/matrix/event-handler-service.js

@@ -99,7 +99,7 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
     };
     reset();
 
-    var initRoom = function(room_id) {
+    var initRoom = function(room_id, room) {
         if (!(room_id in $rootScope.events.rooms)) {
             console.log("Creating new handler entry for " + room_id);
             $rootScope.events.rooms[room_id] = {
@@ -112,6 +112,16 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
                 }
             };
         }
+
+        if (room) {
+            // Report all other metadata of the room object (membership, inviter, visibility, ...)
+            for (var field in room) {
+                if (-1 === ["room_id", "messages", "state"].indexOf(field)) {
+                    $rootScope.events.rooms[room_id][field] = room[field];
+                }
+            }
+            $rootScope.events.rooms[room_id].membership = room.membership;
+        }
     };
 
     var resetRoomMessages = function(room_id) {
@@ -179,21 +189,27 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
             
             if (window.Notification && event.user_id != matrixService.config().user_id) {
                 var shouldBing = $rootScope.containsBingWord(event.content.body);
-            
-                // TODO: Binging every message when idle doesn't make much sense. Can we use this more sensibly?
-                // Unfortunately document.hidden = false on ubuntu chrome if chrome is minimised / does not have focus;
-                // true when you swap tabs though. However, for the case where the chat screen is OPEN and there is
-                // another window on top, we want to be notifying for those events. This DOES mean that there will be
-                // notifications when currently viewing the chat screen though, but that is preferable to the alternative imo.
+
+                // Ideally we would notify only when the window is hidden (i.e. document.hidden = true).
+                //
+                // However, Chrome on Linux and OSX currently returns document.hidden = false unless the window is
+                // explicitly showing a different tab.  So we need another metric to determine hiddenness - we
+                // simply use idle time.  If the user has been idle enough that their presence goes to idle, then
+                // we also display notifs when things happen.
+                //
+                // This is far far better than notifying whenever anything happens anyway, otherwise you get spammed
+                // to death with notifications when the window is in the foreground, which is horrible UX (especially
+                // if you have not defined any bingers and so get notified for everything).
                 var isIdle = (document.hidden || matrixService.presence.unavailable === mPresence.getState());
                 
-                // always bing if there are 0 bing words... apparently.
+                // We need a way to let people get notifications for everything, if they so desire.  The way to do this
+                // is to specify zero bingwords.
                 var bingWords = matrixService.config().bingWords;
-                if (bingWords && bingWords.length === 0) {
+                if (bingWords === undefined || bingWords.length === 0) {
                     shouldBing = true;
                 }
                 
-                if (shouldBing) {
+                if (shouldBing && isIdle) {
                     console.log("Displaying notification for "+JSON.stringify(event));
                     var member = $rootScope.events.rooms[event.room_id].members[event.user_id];
                     var displayname = undefined;
@@ -232,20 +248,12 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
     };
     
     var handleRoomMember = function(event, isLiveEvent, isStateEvent) {
-        // if the server is stupidly re-relaying a no-op join, discard it.
-        if (event.prev_content && 
-            event.content.membership === "join" &&
-            event.content.membership === event.prev_content.membership)
-        {
-            return;
-        }
         
         // add membership changes as if they were a room message if something interesting changed
         // Exception: Do not do this if the event is a room state event because such events already come
         // as room messages events. Moreover, when they come as room messages events, they are relatively ordered
-        // with other other room messages XXX This is no longer true, you only get a single event, not a room message event.
-        // FIXME: This possibly reintroduces multiple join messages.
-        if (event.content.prev !== event.content.membership) { // && !isStateEvent
+        // with other other room messages
+        if (event.content.prev !== event.content.membership && !isStateEvent) {
             if (isLiveEvent) {
                 $rootScope.events.rooms[event.room_id].messages.push(event);
             }
@@ -335,6 +343,10 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
             reset();
             $rootScope.$broadcast(RESET_EVENT);
         },
+        
+        initRoom: function(room) {
+            initRoom(room.room_id, room);
+        },
     
         handleEvent: function(event, isLiveEvent, isStateEvent) {
 
@@ -376,7 +388,6 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
                         handleMessage(event, isLiveEvent);
                         break;
                     case "m.room.member":
-                        isStateEvent = true;
                         handleRoomMember(event, isLiveEvent, isStateEvent);
                         break;
                     case "m.presence":
@@ -406,8 +417,6 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
         // isLiveEvents determines whether notifications should be shown, whether
         // messages get appended to the start/end of lists, etc.
         handleEvents: function(events, isLiveEvents, isStateEvents) {
-            // XXX FIXME TODO: isStateEvents is being left as undefined sometimes. It makes no sense
-            // to have isStateEvents as an arg, since things like m.room.member are ALWAYS state events.
             for (var i=0; i<events.length; i++) {
                 this.handleEvent(events[i], isLiveEvents, isStateEvents);
             }
@@ -423,7 +432,6 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
             if (dir && 'b' === dir) {
                 // paginateBackMessages requests messages to be in reverse chronological order
                 for (var i=0; i<events.length; i++) {
-                    // FIXME: Being live != being state
                     this.handleEvent(events[i], isLiveEvents, isLiveEvents);
                 }
                 
@@ -433,7 +441,6 @@ function(matrixService, $rootScope, $q, $timeout, mPresence) {
             else {
                 // InitialSync returns messages in chronological order
                 for (var i=events.length - 1; i>=0; i--) {
-                    // FIXME: Being live != being state
                     this.handleEvent(events[i], isLiveEvents, isLiveEvents);
                 }
                 // Store where to start pagination

webclient/components/matrix/event-stream-service.js

@@ -112,6 +112,8 @@ angular.module('eventStreamService', [])
                 var rooms = response.data.rooms;
                 for (var i = 0; i < rooms.length; ++i) {
                     var room = rooms[i];
+                    
+                    eventHandlerService.initRoom(room);
 
                     if ("messages" in room) {
                         eventHandlerService.handleRoomMessages(room.room_id, room.messages, false);
@@ -120,8 +122,6 @@ angular.module('eventStreamService', [])
                     if ("state" in room) {
                         eventHandlerService.handleEvents(room.state, false, true);
                     }
-                    
-                    eventHandlerService.setRoomVisibility(room.room_id, room.visibility);
                 }
 
                 var presence = response.data.presence;

webclient/components/matrix/matrix-call.js

@@ -40,8 +40,15 @@ window.RTCPeerConnection = window.RTCPeerConnection || window.webkitRTCPeerConne
 window.RTCSessionDescription = window.RTCSessionDescription || window.webkitRTCSessionDescription || window.mozRTCSessionDescription;
 window.RTCIceCandidate = window.RTCIceCandidate || window.webkitRTCIceCandidate || window.mozRTCIceCandidate;
 
+// Returns true if the browser supports all required features to make WebRTC call
+var isWebRTCSupported = function () {
+    return !!(navigator.getUserMedia || window.RTCPeerConnection || window.RTCSessionDescription || window.RTCIceCandidate);
+};
+
 angular.module('MatrixCall', [])
 .factory('MatrixCall', ['matrixService', 'matrixPhoneService', '$rootScope', '$timeout', function MatrixCallFactory(matrixService, matrixPhoneService, $rootScope, $timeout) {
+    $rootScope.isWebRTCSupported = isWebRTCSupported();
+
     var MatrixCall = function(room_id) {
         this.room_id = room_id;
         this.call_id = "c" + new Date().getTime();
@@ -51,6 +58,12 @@ angular.module('MatrixCall', [])
         // a queue for candidates waiting to go out. We try to amalgamate candidates into a single candidate message where possible
         this.candidateSendQueue = [];
         this.candidateSendTries = 0;
+
+        var self = this;
+        $rootScope.$watch(this.remoteVideoElement, function (oldValue, newValue) {
+            self.tryPlayRemoteStream();
+        });
+
     }
 
     MatrixCall.CALL_TIMEOUT = 60000;
@@ -71,13 +84,39 @@ angular.module('MatrixCall', [])
         return pc;
     }
 
-    MatrixCall.prototype.placeCall = function(config) {
+    MatrixCall.prototype.getUserMediaVideoContraints = function(callType) {
+        switch (callType) {
+            case 'voice':
+                return ({audio: true, video: false});
+            case 'video':
+                return ({audio: true, video: {
+                    mandatory: {
+                        minWidth: 640,
+                        maxWidth: 640,
+                        minHeight: 360,
+                        maxHeight: 360,
+                    }
+                }});
+        }
+    };
+
+    MatrixCall.prototype.placeVoiceCall = function() {
+        this.placeCallWithConstraints(this.getUserMediaVideoContraints('voice'));
+        this.type = 'voice';
+    };
+
+    MatrixCall.prototype.placeVideoCall = function(config) {
+        this.placeCallWithConstraints(this.getUserMediaVideoContraints('video'));
+        this.type = 'video';
+    };
+
+    MatrixCall.prototype.placeCallWithConstraints = function(constraints) {
         var self = this;
         matrixPhoneService.callPlaced(this);
-        navigator.getUserMedia({audio: config.audio, video: config.video}, function(s) { self.gotUserMediaForInvite(s); }, function(e) { self.getUserMediaFailed(e); });
+        navigator.getUserMedia(constraints, function(s) { self.gotUserMediaForInvite(s); }, function(e) { self.getUserMediaFailed(e); });
         this.state = 'wait_local_media';
         this.direction = 'outbound';
-        this.config = config;
+        this.config = constraints;
     };
 
     MatrixCall.prototype.initWithInvite = function(event) {
@@ -86,6 +125,17 @@ angular.module('MatrixCall', [])
         this.peerConn.setRemoteDescription(new RTCSessionDescription(this.msg.offer), this.onSetRemoteDescriptionSuccess, this.onSetRemoteDescriptionError);
         this.state = 'ringing';
         this.direction = 'inbound';
+
+        if (window.mozRTCPeerConnection) {
+            // firefox's RTCPeerConnection doesn't add streams until it starts getting media on them
+            // so we need to figure out whether a video channel has been offered by ourselves.
+            if (this.msg.offer.sdp.indexOf('m=video') > -1) {
+                this.type = 'video';
+            } else {
+                this.type = 'voice';
+            }
+        }
+
         var self = this;
         $timeout(function() {
             if (self.state == 'ringing') {
@@ -108,9 +158,24 @@ angular.module('MatrixCall', [])
 
     MatrixCall.prototype.answer = function() {
         console.log("Answering call "+this.call_id);
+
         var self = this;
+
+        var roomMembers = $rootScope.events.rooms[this.room_id].members;
+        if (roomMembers[matrixService.config().user_id].membership != 'join') {
+            console.log("We need to join the room before we can accept this call");
+            matrixService.join(this.room_id).then(function() {
+                self.answer();
+            }, function() {
+                console.log("Failed to join room: can't answer call!");
+                self.onError("Unable to join room to answer call!");
+                self.hangup();
+            });
+            return;
+        }
+
         if (!this.localAVStream && !this.waitForLocalAVStream) {
-            navigator.getUserMedia({audio: true, video: false}, function(s) { self.gotUserMediaForAnswer(s); }, function(e) { self.getUserMediaFailed(e); });
+            navigator.getUserMedia(this.getUserMediaVideoContraints(this.type), function(s) { self.gotUserMediaForAnswer(s); }, function(e) { self.getUserMediaFailed(e); });
             this.state = 'wait_local_media';
         } else if (this.localAVStream) {
             this.gotUserMediaForAnswer(this.localAVStream);
@@ -132,17 +197,24 @@ angular.module('MatrixCall', [])
         }
     };
 
-    MatrixCall.prototype.hangup = function(suppressEvent) {
+    MatrixCall.prototype.hangup = function(reason, suppressEvent) {
         console.log("Ending call "+this.call_id);
 
+        // pausing now keeps the last frame (ish) of the video call in the video element
+        // rather than it just turning black straight away
+        if (this.remoteVideoElement) this.remoteVideoElement.pause();
+        if (this.localVideoElement) this.localVideoElement.pause();
+
         this.stopAllMedia();
         if (this.peerConn) this.peerConn.close();
 
         this.hangupParty = 'local';
+        this.hangupReason = reason;
 
         var content = {
             version: 0,
             call_id: this.call_id,
+            reason: reason
         };
         this.sendEventWithRetry('m.call.hangup', content);
         this.state = 'ended';
@@ -156,6 +228,13 @@ angular.module('MatrixCall', [])
         }
         if (this.state == 'ended') return;
 
+        if (this.localVideoElement && this.type == 'video') {
+            var vidTrack = stream.getVideoTracks()[0];
+            this.localVideoElement.src = URL.createObjectURL(stream);
+            this.localVideoElement.muted = true;
+            this.localVideoElement.play();
+        }
+
         this.localAVStream = stream;
         var audioTracks = stream.getAudioTracks();
         for (var i = 0; i < audioTracks.length; i++) {
@@ -177,6 +256,13 @@ angular.module('MatrixCall', [])
     MatrixCall.prototype.gotUserMediaForAnswer = function(stream) {
         if (this.state == 'ended') return;
 
+        if (this.localVideoElement && this.type == 'video') {
+            var vidTrack = stream.getVideoTracks()[0];
+            this.localVideoElement.src = URL.createObjectURL(stream);
+            this.localVideoElement.muted = true;
+            this.localVideoElement.play();
+        }
+
         this.localAVStream = stream;
         var audioTracks = stream.getAudioTracks();
         for (var i = 0; i < audioTracks.length; i++) {
@@ -187,7 +273,7 @@ angular.module('MatrixCall', [])
         var constraints = {
             'mandatory': {
                 'OfferToReceiveAudio': true,
-                'OfferToReceiveVideo': false
+                'OfferToReceiveVideo': this.type == 'video'
             },
         };
         this.peerConn.createAnswer(function(d) { self.createdAnswer(d); }, function(e) {}, constraints);
@@ -196,14 +282,14 @@ angular.module('MatrixCall', [])
     };
 
     MatrixCall.prototype.gotLocalIceCandidate = function(event) {
-        console.log(event);
         if (event.candidate) {
+            console.log("Got local ICE "+event.candidate.sdpMid+" candidate: "+event.candidate.candidate);
             this.sendCandidate(event.candidate);
         }
     }
 
     MatrixCall.prototype.gotRemoteIceCandidate = function(cand) {
-        console.log("Got ICE candidate from remote: "+cand);
+        console.log("Got remote ICE "+cand.sdpMid+" candidate: "+cand.candidate);
         if (this.state == 'ended') {
             console.log("Ignoring remote ICE candidate because call has ended");
             return;
@@ -218,6 +304,7 @@ angular.module('MatrixCall', [])
         this.state = 'connecting';
     };
 
+
     MatrixCall.prototype.gotLocalOffer = function(description) {
         console.log("Created offer: "+description);
 
@@ -239,8 +326,7 @@ angular.module('MatrixCall', [])
         var self = this;
         $timeout(function() {
             if (self.state == 'invite_sent') {
-                self.hangupReason = 'invite_timeout';
-                self.hangup();
+                self.hangup('invite_timeout');
             }
         }, MatrixCall.CALL_TIMEOUT);
 
@@ -269,7 +355,7 @@ angular.module('MatrixCall', [])
     };
 
     MatrixCall.prototype.getUserMediaFailed = function() {
-        this.onError("Couldn't start capturing audio! Is your microphone set up?");
+        this.onError("Couldn't start capturing! Is your microphone set up?");
         this.hangup();
     };
 
@@ -283,6 +369,8 @@ angular.module('MatrixCall', [])
                 self.state = 'connected';
                 self.didConnect = true;
             });
+        } else if (this.peerConn.iceConnectionState == 'failed') {
+            this.hangup('ice_failed');
         }
     };
 
@@ -305,6 +393,14 @@ angular.module('MatrixCall', [])
 
         this.remoteAVStream = s;
 
+        if (this.direction == 'inbound') {
+            if (s.getVideoTracks().length > 0) {
+                this.type = 'video';
+            } else {
+                this.type = 'voice';
+            }
+        }
+
         var self = this;
         forAllTracksOnStream(s, function(t) {
             // not currently implemented in chrome
@@ -314,9 +410,16 @@ angular.module('MatrixCall', [])
         event.stream.onended = function(e) { self.onRemoteStreamEnded(e); }; 
         // not currently implemented in chrome
         event.stream.onstarted = function(e) { self.onRemoteStreamStarted(e); };
-        var player = new Audio();
-        player.src = URL.createObjectURL(s);
-        player.play();
+
+        this.tryPlayRemoteStream();
+    };
+
+    MatrixCall.prototype.tryPlayRemoteStream = function(event) {
+        if (this.remoteVideoElement && this.remoteAVStream) {
+            var player = this.remoteVideoElement;
+            player.src = URL.createObjectURL(this.remoteAVStream);
+            player.play();
+        }
     };
 
     MatrixCall.prototype.onRemoteStreamStarted = function(event) {
@@ -345,12 +448,15 @@ angular.module('MatrixCall', [])
         });
     };
 
-    MatrixCall.prototype.onHangupReceived = function() {
+    MatrixCall.prototype.onHangupReceived = function(msg) {
         console.log("Hangup received");
+        if (this.remoteVideoElement) this.remoteVideoElement.pause();
+        if (this.localVideoElement) this.localVideoElement.pause();
         this.state = 'ended';
         this.hangupParty = 'remote';
+        this.hangupReason = msg.reason;
         this.stopAllMedia();
-        if (this.peerConn.signalingState != 'closed') this.peerConn.close();
+        if (this.peerConn && this.peerConn.signalingState != 'closed') this.peerConn.close();
         if (this.onHangup) this.onHangup(this);
     };
 
@@ -361,13 +467,15 @@ angular.module('MatrixCall', [])
             newCall.waitForLocalAVStream = true;
         } else if (this.state == 'create_offer') {
             console.log("Handing local stream to new call");
-            newCall.localAVStream = this.localAVStream;
+            newCall.gotUserMediaForAnswer(this.localAVStream);
             delete(this.localAVStream);
         } else if (this.state == 'invite_sent') {
             console.log("Handing local stream to new call");
-            newCall.localAVStream = this.localAVStream;
+            newCall.gotUserMediaForAnswer(this.localAVStream);
             delete(this.localAVStream);
         }
+        newCall.localVideoElement = this.localVideoElement;
+        newCall.remoteVideoElement = this.remoteVideoElement;
         this.successor = newCall;
         this.hangup(true);
     };
@@ -429,7 +537,7 @@ angular.module('MatrixCall', [])
         }
 
         if (this.candidateSendTries > 5) {
-            console.log("Failed to send candidates on attempt "+ev.tries+". Giving up for now.");
+            console.log("Failed to send candidates on attempt "+this.candidateSendTries+". Giving up for now.");
             this.candidateSendTries = 0;
             return;
         }

webclient/components/matrix/matrix-filter.js

@@ -38,13 +38,15 @@ angular.module('matrixFilter', [])
                 roomName = alias;
             }
             else if (room.members) {
+
+                var user_id = matrixService.config().user_id;
+
                 // Else, build the name from its users
-                // FIXME: Is it still required?
                 // Limit the room renaming to 1:1 room
                 if (2 === Object.keys(room.members).length) {
                     for (var i in room.members) {
                         var member = room.members[i];
-                        if (member.state_key !== matrixService.config().user_id) {
+                        if (member.state_key !== user_id) {
 
                             if (member.state_key in $rootScope.presence) {
                                 // If the user is available in presence, use the displayname there
@@ -61,29 +63,43 @@ angular.module('matrixFilter', [])
                     }
                 }
                 else if (1 === Object.keys(room.members).length) {
-                    // The other member may be in the invite list, get all invited users
-                    var invitedUserIDs = [];
-                    for (var i in room.messages) {
-                        var message = room.messages[i];
-                        if ("m.room.member" === message.type && "invite" === message.membership) {
-                            // Make sure there is no duplicate user
-                            if (-1 === invitedUserIDs.indexOf(message.state_key)) {
-                                invitedUserIDs.push(message.state_key);
-                            }
-                        } 
+                    var otherUserId;
+
+                    if (Object.keys(room.members)[0] !== user_id) {
+                        otherUserId = Object.keys(room.members)[0];
+                    }
+                    else {
+                        // The other member may be in the invite list, get all invited users
+                        var invitedUserIDs = [];
+                        for (var i in room.messages) {
+                            var message = room.messages[i];
+                            if ("m.room.member" === message.type && "invite" === message.membership) {
+                                // Filter out the current user
+                                var member_id = message.state_key;
+                                if (member_id === user_id) {
+                                    member_id = message.user_id;
+                                }
+                                if (member_id !== user_id) {
+                                    // Make sure there is no duplicate user
+                                    if (-1 === invitedUserIDs.indexOf(member_id)) {
+                                        invitedUserIDs.push(member_id);
+                                    }
+                                }
+                            } 
+                        }
+
+                        // For now, only 1:1 room needs to be renamed. It means only 1 invited user
+                        if (1 === invitedUserIDs.length) {
+                            otherUserId = invitedUserIDs[0];
+                        }
                     }
 
-                    // For now, only 1:1 room needs to be renamed. It means only 1 invited user
-                    if (1 === invitedUserIDs.length) {
-                        var userID = invitedUserIDs[0];
-
-                        // Try to resolve his displayname in presence global data
-                        if (userID in $rootScope.presence) {
-                            roomName = $rootScope.presence[userID].content.displayname;
-                        }
-                        else {
-                            roomName = userID;
-                        }
+                    // Try to resolve his displayname in presence global data
+                    if (otherUserId in $rootScope.presence) {
+                        roomName = $rootScope.presence[otherUserId].content.displayname;
+                    }
+                    else {
+                        roomName = otherUserId;
                     }
                 }
             }
@@ -97,6 +113,14 @@ angular.module('matrixFilter', [])
         if (undefined === roomName) {
             // By default, use the room ID
             roomName = room_id;
+
+            // XXX: this is *INCREDIBLY* heavy logging for a function that calls every single
+            // time any kind of digest runs which refreshes a room name...
+            // commenting it out for now.
+
+            // Log some information that lead to this leak
+            // console.log("Room ID leak for " + room_id);
+            // console.log("room object: " + JSON.stringify(room, undefined, 4));   
         }
 
         return roomName;

webclient/components/matrix/matrix-phone-service.js

@@ -59,6 +59,16 @@ angular.module('matrixPhoneService', [])
 
             var MatrixCall = $injector.get('MatrixCall');
             var call = new MatrixCall(event.room_id);
+
+            if (!isWebRTCSupported()) {
+                console.log("Incoming call ID "+msg.call_id+" but this browser doesn't support WebRTC");
+                // don't hang up the call: there could be other clients connected that do support WebRTC and declining the
+                // the call on their behalf would be really annoying.
+                // instead, we broadcast a fake call event with a non-functional call object
+                $rootScope.$broadcast(matrixPhoneService.INCOMING_CALL_EVENT, call);
+                return;
+            }
+
             call.call_id = msg.call_id;
             call.initWithInvite(event);
             matrixPhoneService.allCalls[call.call_id] = call;
@@ -135,7 +145,7 @@ angular.module('matrixPhoneService', [])
                 call.initWithHangup(event);
                 matrixPhoneService.allCalls[msg.call_id] = call;
             } else {
-                call.onHangupReceived();
+                call.onHangupReceived(msg);
                 delete(matrixPhoneService.allCalls[msg.call_id]);
             }
         }

webclient/index.html

@@ -45,6 +45,13 @@
 </head>
 
 <body>
+    <div id="videoBackground" ng-class="videoMode">
+        <div id="videoContainer" ng-class="videoMode">
+            <div id="videoContainerPadding"></div>
+            <video id="localVideo" ng-class="[videoMode, currentCall.state]" ng-show="currentCall && currentCall.type == 'video' && (currentCall.state == 'connected' || currentCall.state == 'connecting' || currentCall.state == 'invite_sent' || currentCall.state == 'ended')"></video>
+            <video id="remoteVideo" ng-class="[videoMode, currentCall.state]" ng-show="currentCall && currentCall.type == 'video' && (currentCall.state == 'connected' || (currentCall.state == 'ended' && currentCall.didConnect))"></video>
+        </div>
+    </div>
 
     <div id="header">
         <!-- Do not show buttons on the login page -->
@@ -58,20 +65,22 @@
                     <br />
                     <span id="callState">
                         <span ng-show="currentCall.state == 'invite_sent'">Calling...</span>
-                        <span ng-show="currentCall.state == 'ringing'">Incoming Call</span>
+                        <span ng-show="currentCall.state == 'ringing' && currentCall && currentCall.type == 'video'">Incoming Video Call</span>
+                        <span ng-show="currentCall.state == 'ringing' && currentCall && currentCall.type == 'voice'">Incoming Voice Call</span>
                         <span ng-show="currentCall.state == 'connecting'">Call Connecting...</span>
                         <span ng-show="currentCall.state == 'connected'">Call Connected</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'remote'">Call Rejected</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local' && currentCall.hangupReason == undefined">Call Canceled</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local' && currentCall.hangupReason == 'invite_timeout'">User Not Responding</span>
-                        <span ng-show="currentCall.state == 'ended' && currentCall.didConnect && currentCall.direction == 'outbound'">Call Ended</span>
-                        <span ng-show="currentCall.state == 'ended' && !currentCall.didConnect && currentCall.direction == 'inbound'">Call Canceled</span>
-                        <span ng-show="currentCall.state == 'ended' && currentCall.didConnect && currentCall.direction == 'inbound'">Call Ended</span>
+                        <span ng-show="currentCall.state == 'ended' && currentCall.hangupReason == 'ice_failed'">Media Connection Failed</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'remote'">Call Rejected</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local'">Call Canceled</span>
+                        <span ng-show="currentCall.state == 'ended' && currentCall.hangupReason == 'invite_timeout' && !currentCall.didConnect && currentCall.direction == 'outbound' && currentCall.hangupParty == 'local'">User Not Responding</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && currentCall.didConnect && currentCall.direction == 'outbound'">Call Ended</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && !currentCall.didConnect && currentCall.direction == 'inbound'">Call Canceled</span>
+                        <span ng-show="currentCall.state == 'ended' && !currentCall.hangupReason && currentCall.didConnect && currentCall.direction == 'inbound'">Call Ended</span>
                         <span ng-show="currentCall.state == 'wait_local_media'">Waiting for media permission...</span>
                     </span>
                 </div>
                 <span ng-show="currentCall.state == 'ringing'">
-                    <button ng-click="answerCall()">Answer</button>
+                    <button ng-click="answerCall()" ng-disabled="!isWebRTCSupported" title="{{isWebRTCSupported ? '' : 'Your browser does not support VoIP' }}">Answer {{ currentCall.type }} call</button>
                     <button ng-click="hangupCall()">Reject</button>
                 </span>
                 <button ng-click="hangupCall()" ng-show="currentCall && currentCall.state != 'ringing' && currentCall.state != 'ended' && currentCall.state != 'fledgling'">Hang up</button>
@@ -92,6 +101,7 @@
                     <source src="media/busy.mp3" type="audio/mpeg" />
                 </audio>
             </div>
+
             <a href id="headerUserId" ng-click='goToUserPage(user_id)'>{{ user_id }}</a>
             &nbsp;
             <button ng-click='goToPage("/")'>Home</button>
@@ -100,9 +110,20 @@
         </div>
     </div>
 
-    <div id="page" ng-view></div>
+    <div class="page" ng-hide="unsupportedBrowser" ng-view></div>
 
-    <div id="footer" ng-hide="location.indexOf('/room') == 0">
+    <div class="page" ng-show="unsupportedBrowser">
+        <div id="unsupportedBrowser" ng-show="unsupportedBrowser">
+            Sorry, your browser is not supported. <br/>
+                Reason: {{ unsupportedBrowser.reason }}
+
+            <br/><br/>
+            Your browser: <br/>
+            {{ unsupportedBrowser.browser }}
+        </div>
+    </div>
+
+    <div id="footer" ng-hide="location.indexOf('/room') === 0">
         <div id="footerContent">
             &copy; 2014 Matrix.org
         </div>

webclient/login/login.html

@@ -23,7 +23,7 @@
                     <br/>
                     <input id="password" size="32" type="password" ng-model="account.password" placeholder="Password"/>
                     <br/><br/>
-                    <button ng-click="login()" ng-disabled="!account.user_id || !account.password || !account.homeserver">Login</button>
+                    <button id="login" ng-click="login()" ng-disabled="!account.user_id || !account.password || !account.homeserver">Login</button>
                     <br/><br/>
                 </div>
 

webclient/recents/recents-filter.js

@@ -19,14 +19,13 @@
 angular.module('RecentsController')
 .filter('orderRecents', ["matrixService", "eventHandlerService", function(matrixService, eventHandlerService) {
     return function(rooms) {
-
         var user_id = matrixService.config().user_id;
 
         // Transform the dict into an array
         // The key, room_id, is already in value objects
         var filtered = [];
         angular.forEach(rooms, function(room, room_id) {
-
+            
             // Show the room only if the user has joined it or has been invited
             // (ie, do not show it if he has been banned)
             var member = eventHandlerService.getMember(room_id, user_id);
@@ -35,8 +34,13 @@ angular.module('RecentsController')
                 // Count users here
                 // TODO: Compute it directly in eventHandlerService
                 room.numUsersInRoom = eventHandlerService.getUsersCountInRoom(room_id);
+
+                filtered.push(room);
+            }
+            else if ("invite" === room.membership) {
+                // The only information we have about the room is that the user has been invited
+                filtered.push(room);
             }
-            filtered.push(room);
         });
 
         // And time sort them
@@ -60,4 +64,4 @@ angular.module('RecentsController')
         });
         return filtered;
     };
-}]);
+}]);

webclient/room/room-controller.js

@@ -15,8 +15,8 @@ limitations under the License.
 */
 
 angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
-.controller('RoomController', ['$filter', '$scope', '$timeout', '$routeParams', '$location', '$rootScope', 'matrixService', 'eventHandlerService', 'mFileUpload', 'matrixPhoneService', 'MatrixCall',
-                               function($filter, $scope, $timeout, $routeParams, $location, $rootScope, matrixService, eventHandlerService, mFileUpload, matrixPhoneService, MatrixCall) {
+.controller('RoomController', ['$filter', '$scope', '$timeout', '$routeParams', '$location', '$rootScope', 'matrixService', 'mPresence', 'eventHandlerService', 'mFileUpload', 'matrixPhoneService', 'MatrixCall',
+                               function($filter, $scope, $timeout, $routeParams, $location, $rootScope, matrixService, mPresence, eventHandlerService, mFileUpload, matrixPhoneService, MatrixCall) {
    'use strict';
     var MESSAGES_PER_PAGINATION = 30;
     var THUMBNAIL_SIZE = 320;
@@ -33,7 +33,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         paginating: false, // used to avoid concurrent pagination requests pulling in dup contents
         stream_failure: undefined, // the response when the stream fails
         waiting_for_joined_event: false,  // true when the join request is pending. Back to false once the corresponding m.room.member event is received
-        messages_visibility: "hidden" // In order to avoid flickering when scrolling down the message table at the page opening, delay the message table display
+        messages_visibility: "hidden", // In order to avoid flickering when scrolling down the message table at the page opening, delay the message table display
     };
     $scope.members = {};
     $scope.autoCompleting = false;
@@ -189,7 +189,6 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 // Notify when a user joins
                 if ((document.hidden  || matrixService.presence.unavailable === mPresence.getState())
                         && event.state_key !== $scope.state.user_id  && "join" === event.membership) {
-                    debugger;
                     var notification = new window.Notification(
                         event.content.displayname +
                         " (" + (matrixService.getRoomIdToAliasMapping(event.room_id) || event.room_id) + ")", // FIXME: don't leak room_ids here
@@ -417,14 +416,16 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
     };
 
     $scope.send = function() {
-        if (undefined === $scope.textInput || $scope.textInput === "") {
+        var input = $('#mainInput').val();
+        
+        if (undefined === input || input === "") {
             return;
         }
         
         scrollToBottom(true);
 
         // Store the command in the history
-        history.push($scope.textInput);
+        history.push(input);
 
         var promise;
         var cmd;
@@ -432,13 +433,11 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         var echo = false;
         
         // Check for IRC style commands first
-        var line = $scope.textInput;
-        
         // trim any trailing whitespace, as it can confuse the parser for IRC-style commands
-        line = line.replace(/\s+$/, "");
+        input = input.replace(/\s+$/, "");
         
-        if (line[0] === "/" && line[1] !== "/") {
-            var bits = line.match(/^(\S+?)( +(.*))?$/);
+        if (input[0] === "/" && input[1] !== "/") {
+            var bits = input.match(/^(\S+?)( +(.*))?$/);
             cmd = bits[1];
             args = bits[3];
             
@@ -581,7 +580,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         // By default send this as a message unless it's an IRC-style command
         if (!promise && !cmd) {
             // Make the request
-            promise = matrixService.sendTextMessage($scope.room_id, line);
+            promise = matrixService.sendTextMessage($scope.room_id, input);
             echo = true;
         }
         
@@ -590,7 +589,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
             // To do so, create a minimalist fake text message event and add it to the in-memory list of room messages
             var echoMessage = {
                 content: {
-                    body: (cmd === "/me" ? args : line),
+                    body: (cmd === "/me" ? args : input),
                     hsob_ts: new Date().getTime(), // fake a timestamp
                     msgtype: (cmd === "/me" ? "m.emote" : "m.text"),
                 },
@@ -600,7 +599,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 echo_msg_state: "messagePending"     // Add custom field to indicate the state of this fake message to HTML
             };
 
-            $scope.textInput = "";
+            $('#mainInput').val('');
             $rootScope.events.rooms[$scope.room_id].messages.push(echoMessage);
             scrollToBottom();
         }
@@ -620,7 +619,7 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                         echoMessage.event_id = response.data.event_id;
                     }
                     else {
-                        $scope.textInput = "";
+                        $('#mainInput').val('');
                     }         
                 },
                 function(error) {
@@ -704,7 +703,13 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
                 if ($rootScope.events.rooms[$scope.room_id]) {
 
                     // There is no need to do a 1st pagination (initialSync provided enough to fill a page)
-                    $scope.state.first_pagination = false;
+                    if ($rootScope.events.rooms[$scope.room_id].messages.length) {
+                        $scope.state.first_pagination = false;
+                    }
+                    else {
+                        // except if we just joined a room, we won't have this history from initial sync, so we should try to paginate it anyway                        
+                        $scope.state.first_pagination = true;
+                    }
 
                     var members = $rootScope.events.rooms[$scope.room_id].members;
 
@@ -854,7 +859,9 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         var call = new MatrixCall($scope.room_id);
         call.onError = $rootScope.onCallError;
         call.onHangup = $rootScope.onCallHangup;
-        call.placeCall({audio: true, video: false});
+        // remote video element is used for playing audio in voice calls
+        call.remoteVideoElement = angular.element('#remoteVideo')[0];
+        call.placeVoiceCall();
         $rootScope.currentCall = call;
     };
 
@@ -862,7 +869,9 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
         var call = new MatrixCall($scope.room_id);
         call.onError = $rootScope.onCallError;
         call.onHangup = $rootScope.onCallHangup;
-        call.placeCall({audio: true, video: true});
+        call.localVideoElement = angular.element('#localVideo')[0];
+        call.remoteVideoElement = angular.element('#remoteVideo')[0];
+        call.placeVideoCall();
         $rootScope.currentCall = call;
     };
 
@@ -904,11 +913,11 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
 
             if (-1 === this.position) {
                 // User starts to go to into the history, save the current line
-                this.typingMessage = $scope.textInput;
+                this.typingMessage = $('#mainInput').val();
             }
             else {
                 // If the user modified this line in history, keep the change
-                this.data[this.position] = $scope.textInput;
+                this.data[this.position] = $('#mainInput').val();
             }
 
             // Bounds the new position to valid data
@@ -919,11 +928,11 @@ angular.module('RoomController', ['ngSanitize', 'matrixFilter', 'mFileInput'])
 
             if (-1 !== this.position) {
                 // Show the message from the history
-                $scope.textInput = this.data[this.position];
+                $('#mainInput').val(this.data[this.position]);
             }
             else if (undefined !== this.typingMessage) {
                 // Go back to the message the user started to type
-                $scope.textInput = this.typingMessage;
+                $('#mainInput').val(this.typingMessage);
             }
         }
     };

webclient/room/room.html

@@ -111,8 +111,8 @@
                               ng-class="containsBingWord(msg.content.body) && msg.user_id != state.user_id ? msg.echo_msg_state + ' messageBing' : msg.echo_msg_state"
                               ng-bind-html="((msg.content.msgtype === 'm.text') ? msg.content.body : '') | linky:'_blank'"/>
 
-                        <span ng-show='msg.type === "m.call.invite" && msg.user_id == state.user_id'>Outgoing Call</span>
-                        <span ng-show='msg.type === "m.call.invite" && msg.user_id != state.user_id'>Incoming Call</span>
+                        <span ng-show='msg.type === "m.call.invite" && msg.user_id == state.user_id'>Outgoing Call{{ isWebRTCSupported ? '' : ' (But your browser does not support VoIP)' }}</span>
+                        <span ng-show='msg.type === "m.call.invite" && msg.user_id != state.user_id'>Incoming Call{{ isWebRTCSupported ? '' : ' (But your browser does not support VoIP)' }}</span>
 
                         <div ng-show='msg.content.msgtype === "m.image"'>
                             <div ng-hide='msg.content.thumbnail_url' ng-style="msg.content.body.h && { 'height' : (msg.content.body.h < 320) ? msg.content.body.h : 320}">
@@ -157,7 +157,7 @@
                         {{ state.user_id }} 
                     </td>
                     <td width="*">
-                        <textarea id="mainInput" rows="1" ng-model="textInput" ng-enter="send()"
+                        <textarea id="mainInput" rows="1" ng-enter="send()"
                                   ng-disabled="state.permission_denied"
                                   ng-keydown="(38 === $event.which) ? history.goUp($event) : ((40 === $event.which) ? history.goDown($event) : 0)"
                                   ng-focus="true" autocomplete="off" tab-complete/>
@@ -176,7 +176,20 @@
                         <button ng-click="inviteUser()" ng-disabled="state.permission_denied">Invite</button>
                 </span>
                 <button ng-click="leaveRoom()" ng-disabled="state.permission_denied">Leave</button>
-                <button ng-click="startVoiceCall()" ng-show="(currentCall == undefined || currentCall.state == 'ended') && memberCount() == 2" ng-disabled="state.permission_denied">Voice Call</button>
+                <button ng-click="startVoiceCall()"
+                        ng-show="(currentCall == undefined || currentCall.state == 'ended')"
+                        ng-disabled="state.permission_denied || !isWebRTCSupported || memberCount() != 2"
+                        title ="{{ !isWebRTCSupported ? 'VoIP requires webRTC but your browser does not support it' : (memberCount() == 2 ? '' : 'VoIP calls can only be made in rooms with two participants') }}"
+                        >
+                    Voice Call
+                </button>
+                <button ng-click="startVideoCall()"
+                        ng-show="(currentCall == undefined || currentCall.state == 'ended')"
+                        ng-disabled="state.permission_denied || !isWebRTCSupported || memberCount() != 2"
+                        title ="{{ !isWebRTCSupported ? 'VoIP requires webRTC but your browser does not support it' : (memberCount() == 2 ? '' : 'VoIP calls can only be made in rooms with two participants') }}"
+                        >
+                    Video Call
+                </button>
             </div>
         
             {{ feedback }}

webclient/settings/settings.html

@@ -51,10 +51,10 @@
         <h3>Desktop notifications</h3>
         <div class="section" ng-switch="settings.notifications">
             <div ng-switch-when="granted">
-                Notifications are enabled. You will be alerted when a message contains your user ID or display name.
+                Notifications are enabled.
                 <div class="section">
-                    <h4>Additional words to alert on:</h4>
-                    <p>Leave blank to alert on all messages.</p>
+                    <h4>Specific words to alert on:</h4>
+                    <p>Leave blank to alert on all messages. Your username & display name always alerts.</p>
                     <input size=40 name="bingWords" ng-model="settings.bingWords" ng-list placeholder="Enter words separated with , (supports regex)"
                     ng-blur="saveBingWords()"/>
                     <ul>

webclient/test/README

@@ -0,0 +1,33 @@
+Requires:
+ - nodejs/npm
+ - npm install karma
+ - npm install jasmine
+ - npm install protractor (e2e testing)
+
+Setting up continuous integration / run the unit tests (make sure you're in
+this directory so it can find the config file):
+  karma start
+
+
+Setting up e2e tests (only if you don't have a selenium server to run the tests
+on. If you do, edit the config to point to that url):
+
+  webdriver-manager update
+  webdriver-manager start
+
+  Create a file "environment-protractor.js" in this directory and type:
+    module.exports = {
+        seleniumAddress: 'http://localhost:4444/wd/hub',
+        baseUrl: "http://localhost:8008",
+        username: "YOUR_TEST_USERNAME",
+        password: "YOUR_TEST_PASSWORD"
+    }
+
+Running e2e tests:
+  protractor protractor.conf.js
+
+NOTE: This will create a public room on the target home server.
+
+
+
+

webclient/test/e2e/home.spec.js

@@ -0,0 +1,16 @@
+var env = require("../environment-protractor.js");
+
+describe("home page", function() {
+
+    beforeEach(function() {
+        ptor = protractor.getInstance();
+        // FIXME we use longpoll on the event stream, and I can't get $interval
+        // playing nicely with it. Patches welcome to fix this.
+        ptor.ignoreSynchronization = true;
+    }); 
+
+    it("should have a title", function() {
+        browser.get(env.baseUrl);
+        expect(browser.getTitle()).toEqual("[matrix]");
+    });
+});

webclient/test/karma.conf.js

@@ -0,0 +1,82 @@
+// Karma configuration
+// Generated on Thu Sep 18 2014 14:25:57 GMT+0100 (BST)
+
+module.exports = function(config) {
+  config.set({
+
+    // base path that will be used to resolve all patterns (eg. files, exclude)
+    basePath: '',
+
+
+    // frameworks to use
+    // available frameworks: https://npmjs.org/browse/keyword/karma-adapter
+    frameworks: ['jasmine'],
+
+
+    // list of files / patterns to load in the browser
+    // XXX: Order is important, and doing /js/angular* makes the tests not run :/
+    files: [
+      '../js/jquery*',
+      '../js/angular.js',
+      '../js/angular-mocks.js',
+      '../js/angular-route.js',
+      '../js/angular-animate.js',
+      '../js/angular-sanitize.js',
+      '../js/ng-infinite-scroll-matrix.js',
+      '../login/**/*.*',
+      '../room/**/*.*',
+      '../components/**/*.*',
+      '../user/**/*.*',
+      '../home/**/*.*',
+      '../recents/**/*.*',
+      '../settings/**/*.*',
+      '../app.js',
+      '../app*',
+      './unit/**/*.js'
+    ],
+
+
+    // list of files to exclude
+    exclude: [
+    ],
+
+
+    // preprocess matching files before serving them to the browser
+    // available preprocessors: https://npmjs.org/browse/keyword/karma-preprocessor
+    preprocessors: {
+    },
+
+
+    // test results reporter to use
+    // possible values: 'dots', 'progress'
+    // available reporters: https://npmjs.org/browse/keyword/karma-reporter
+    reporters: ['progress'],
+
+
+    // web server port
+    port: 9876,
+
+
+    // enable / disable colors in the output (reporters and logs)
+    colors: true,
+
+
+    // level of logging
+    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
+    logLevel: config.LOG_DEBUG,
+
+
+    // enable / disable watching file and executing tests whenever any file changes
+    autoWatch: true,
+
+
+    // start these browsers
+    // available browser launchers: https://npmjs.org/browse/keyword/karma-launcher
+    browsers: ['Chrome'],
+
+
+    // Continuous Integration mode
+    // if true, Karma captures browsers, runs the tests and exits
+    singleRun: false
+  });
+};

webclient/test/protractor.conf.js

@@ -0,0 +1,18 @@
+var env = require("./environment-protractor.js");
+exports.config = {
+    seleniumAddress: env.seleniumAddress,
+    specs: ['e2e/*.spec.js'],
+    onPrepare: function() {
+        browser.driver.get(env.baseUrl);
+        browser.driver.findElement(by.id("user_id")).sendKeys(env.username);
+        browser.driver.findElement(by.id("password")).sendKeys(env.password);
+        browser.driver.findElement(by.id("login")).click();
+
+        // wait till the login is done, detect via url change
+        browser.driver.wait(function() {
+            return browser.driver.getCurrentUrl().then(function(url) {
+                return !(/login/.test(url))
+            });
+        });
+    }
+}

webclient/test/unit/user-controller.spec.js

@@ -0,0 +1,57 @@
+describe("UserCtrl", function() {
+    var scope, ctrl, matrixService, routeParams, $q, $timeout;
+    var userId = "@foo:bar";
+    var displayName = "Foo";
+    var avatarUrl = "avatar.url";
+    
+    beforeEach(module('matrixWebClient'));
+
+    beforeEach(function() {
+
+        inject(function($rootScope, $injector, $controller, _$q_, _$timeout_) {
+            $q = _$q_;
+            $timeout = _$timeout_;
+
+            matrixService = {
+                config: function() {
+                    return {
+                        user_id: userId
+                    };
+                },
+
+                getDisplayName: function(uid) {
+                    var d = $q.defer();
+                    d.resolve({
+                        data: {
+                            displayname: displayName
+                        }
+                    });
+                    return d.promise;
+                },
+
+                getProfilePictureUrl: function(uid) {
+                    var d = $q.defer();
+                    d.resolve({
+                        data: {
+                            avatar_url: avatarUrl
+                        }
+                    });
+                    return d.promise;
+                }
+            };
+            scope = $rootScope.$new();
+            routeParams = {
+                user_matrix_id: userId
+            };
+            ctrl = $controller('UserController', {
+                '$scope': scope, 
+                '$routeParams': routeParams, 
+                'matrixService': matrixService
+            });
+        });
+    });
+
+    it('should display your user id', function() {
+        expect(scope.user_id).toEqual(userId);
+    });
+});