In get_recent_events_for_room, get the full event json rather than just the event_ids so we don't have to do so many db queries

Don't perform the check in auth.is_host_in_room but instead do it in _do_join
and also assert that there are no m.room.members in the room before doing so.

.gitignore

@@ -38,3 +38,5 @@ graph/*.dot
 **/webclient/test/environment-protractor.js
 
 uploads
+
+.idea/

CHANGES.rst

@@ -1,3 +1,32 @@
+Changes in synapse 0.6.1 (2015-01-07)
+=====================================
+
+ * Major optimizations to improve performance of initial sync and event sending
+   in large rooms (by up to 10x)
+ * Media repository now includes a Content-Length header on media downloads.
+ * Improve quality of thumbnails by changing resizing algorithm.
+
+Changes in synapse 0.6.0 (2014-12-16)
+=====================================
+
+ * Add new API for media upload and download that supports thumbnailing.
+ * Replicate media uploads over multiple homeservers so media is always served
+   to clients from their local homeserver.  This obsoletes the
+   --content-addr parameter and confusion over accessing content directly
+   from remote homeservers.
+ * Implement exponential backoff when retrying federation requests when
+   sending to remote homeservers which are offline.
+ * Implement typing notifications.
+ * Fix bugs where we sent events with invalid signatures due to bugs where
+   we incorrectly persisted events.
+ * Improve performance of database queries involving retrieving events.
+
+Changes in synapse 0.5.4a (2014-12-13)
+======================================
+
+ * Fix bug while generating the error message when a file path specified in
+   the config doesn't exist.
+
 Changes in synapse 0.5.4 (2014-12-03)
 =====================================
 

README.rst

@@ -108,6 +108,18 @@ To install the synapse homeserver run::
 This installs synapse, along with the libraries it uses, into
 ``$HOME/.local/lib/`` on Linux or ``$HOME/Library/Python/2.7/lib/`` on OSX.
 
+Your python may not give priority to locally installed libraries over system
+libraries, in which case you must add your local packages to your python path::
+
+    $ # on Linux:
+    $ export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages:$PYTHONPATH
+
+    $ # on OSX:
+    $ export PYTHONPATH=$HOME/Library/Python/2.7/lib/python/site-packages:$PYTHONPATH
+
+For reliable VoIP calls to be routed via this homeserver, you MUST configure
+a TURN server.  See docs/turn-howto.rst for details.
+
 Troubleshooting Installation
 ----------------------------
 
@@ -133,6 +145,37 @@ failing, e.g.::
 On OSX, if you encounter clang: error: unknown argument: '-mno-fused-madd' you
 will need to export CFLAGS=-Qunused-arguments.
 
+Windows Install
+---------------
+Synapse can be installed on Cygwin. It requires the following Cygwin packages:
+
+ - gcc
+ - git
+ - libffi-devel
+ - openssl (and openssl-devel, python-openssl)
+ - python
+ - python-setuptools
+ 
+The content repository requires additional packages and will be unable to process
+uploads without them:
+ - libjpeg8
+ - libjpeg8-devel
+ - zlib
+If you choose to install Synapse without these packages, you will need to reinstall
+``pillow`` for changes to be applied, e.g. ``pip uninstall pillow`` ``pip install
+pillow --user``
+
+Troubleshooting:
+
+- You may need to upgrade ``setuptools`` to get this to work correctly:
+  ``pip install setuptools --upgrade``.
+- You may encounter errors indicating that ``ffi.h`` is missing, even with
+  ``libffi-devel`` installed. If you do, copy the ``.h`` files:
+  ``cp /usr/lib/libffi-3.0.13/include/*.h /usr/include``
+- You may need to install libsodium from source in order to install PyNacl. If
+  you do, you may need to create a symlink to ``libsodium.a`` so ``ld`` can find
+  it: ``ln -s /usr/local/lib/libsodium.a /usr/lib/libsodium.a``
+
 Running Your Homeserver
 =======================
 
@@ -208,6 +251,11 @@ Upgrading an existing homeserver
 IMPORTANT: Before upgrading an existing homeserver to a new version, please
 refer to UPGRADE.rst for any additional instructions.
 
+Otherwise, simply re-install the new codebase over the current one - e.g.
+by ``pip install --user --process-dependency-links
+https://github.com/matrix-org/synapse/tarball/master``
+if using pip, or by ``git pull`` if running off a git working copy.
+
 
 Setting up Federation
 =====================

UPGRADE.rst

@@ -1,3 +1,23 @@
+Upgrading to v0.6.0
+===================
+
+To pull in new dependencies, run::
+
+    python setup.py develop --user
+
+This update includes a change to the database schema. To upgrade you first need
+to upgrade the database by running::
+
+    python scripts/upgrade_db_to_v0.6.0.py <db> <server_name> <signing_key>
+
+Where `<db>` is the location of the database, `<server_name>` is the
+server name as specified in the synapse configuration, and `<signing_key>` is
+the location of the signing key as specified in the synapse configuration.
+
+This may take some time to complete. Failures of signatures and content hashes
+can safely be ignored.
+
+
 Upgrading to v0.5.1
 ===================
 
@@ -32,7 +52,7 @@ resulting conflicts during the upgrade process.
 Before running the command the homeserver should be first completely 
 shutdown. To run it, simply specify the location of the database, e.g.:
 
-  ./database-prepare-for-0.5.0.sh "homeserver.db"
+  ./scripts/database-prepare-for-0.5.0.sh "homeserver.db"
 
 Once this has successfully completed it will be safe to restart the 
 homeserver. You may notice that the homeserver takes a few seconds longer to 
@@ -127,7 +147,7 @@ rooms the home server was a member of and room alias mappings.
 Before running the command the homeserver should be first completely 
 shutdown. To run it, simply specify the location of the database, e.g.:
 
-  ./database-prepare-for-0.0.1.sh "homeserver.db"
+  ./scripts/database-prepare-for-0.0.1.sh "homeserver.db"
 
 Once this has successfully completed it will be safe to restart the 
 homeserver. You may notice that the homeserver takes a few seconds longer to 

VERSION

@@ -1 +1 @@
-0.5.4
+0.6.1b

contrib/graph/graph2.py

@@ -0,0 +1,156 @@
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import sqlite3
+import pydot
+import cgi
+import json
+import datetime
+import argparse
+
+from synapse.events import FrozenEvent
+
+
+def make_graph(db_name, room_id, file_prefix, limit):
+    conn = sqlite3.connect(db_name)
+
+    sql = (
+        "SELECT json FROM event_json as j "
+        "INNER JOIN events as e ON e.event_id = j.event_id "
+        "WHERE j.room_id = ?"
+    )
+
+    args = [room_id]
+
+    if limit:
+        sql += (
+            " ORDER BY topological_ordering DESC, stream_ordering DESC "
+            "LIMIT ?"
+        )
+
+        args.append(limit)
+
+    c = conn.execute(sql, args)
+
+    events = [FrozenEvent(json.loads(e[0])) for e in c.fetchall()]
+
+    events.sort(key=lambda e: e.depth)
+
+    node_map = {}
+    state_groups = {}
+
+    graph = pydot.Dot(graph_name="Test")
+
+    for event in events:
+        c = conn.execute(
+            "SELECT state_group FROM event_to_state_groups "
+            "WHERE event_id = ?",
+            (event.event_id,)
+        )
+
+        res = c.fetchone()
+        state_group = res[0] if res else None
+
+        if state_group is not None:
+            state_groups.setdefault(state_group, []).append(event.event_id)
+
+        t = datetime.datetime.fromtimestamp(
+            float(event.origin_server_ts) / 1000
+        ).strftime('%Y-%m-%d %H:%M:%S,%f')
+
+        content = json.dumps(event.get_dict()["content"])
+
+        label = (
+            "<"
+            "<b>%(name)s </b><br/>"
+            "Type: <b>%(type)s </b><br/>"
+            "State key: <b>%(state_key)s </b><br/>"
+            "Content: <b>%(content)s </b><br/>"
+            "Time: <b>%(time)s </b><br/>"
+            "Depth: <b>%(depth)s </b><br/>"
+            "State group: %(state_group)s<br/>"
+            ">"
+        ) % {
+            "name": event.event_id,
+            "type": event.type,
+            "state_key": event.get("state_key", None),
+            "content": cgi.escape(content, quote=True),
+            "time": t,
+            "depth": event.depth,
+            "state_group": state_group,
+        }
+
+        node = pydot.Node(
+            name=event.event_id,
+            label=label,
+        )
+
+        node_map[event.event_id] = node
+        graph.add_node(node)
+
+    for event in events:
+        for prev_id, _ in event.prev_events:
+            try:
+                end_node = node_map[prev_id]
+            except:
+                end_node = pydot.Node(
+                    name=prev_id,
+                    label="<<b>%s</b>>" % (prev_id,),
+                )
+
+                node_map[prev_id] = end_node
+                graph.add_node(end_node)
+
+            edge = pydot.Edge(node_map[event.event_id], end_node)
+            graph.add_edge(edge)
+
+    for group, event_ids in state_groups.items():
+        if len(event_ids) <= 1:
+            continue
+
+        cluster = pydot.Cluster(
+            str(group),
+            label="<State Group: %s>" % (str(group),)
+        )
+
+        for event_id in event_ids:
+            cluster.add_node(node_map[event_id])
+
+        graph.add_subgraph(cluster)
+
+    graph.write('%s.dot' % file_prefix, format='raw', prog='dot')
+    graph.write_svg("%s.svg" % file_prefix, prog='dot')
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Generate a PDU graph for a given room by talking "
+                    "to the given homeserver to get the list of PDUs. \n"
+                    "Requires pydot."
+    )
+    parser.add_argument(
+        "-p", "--prefix", dest="prefix",
+        help="String to prefix output files with",
+        default="graph_output"
+    )
+    parser.add_argument(
+        "-l", "--limit",
+        help="Only retrieve the last N events.",
+    )
+    parser.add_argument('db')
+    parser.add_argument('room')
+
+    args = parser.parse_args()
+
+    make_graph(args.db, args.room, args.prefix, args.limit)

docs/code_style.rst

@@ -1,10 +1,14 @@
 Basically, PEP8
 
-- Max line width: 80 chars.
+- NEVER tabs. 4 spaces to indent.
+- Max line width: 79 chars (with flexibility to overflow by a "few chars" if
+  the overflowing content is not semantically significant and avoids an
+  explosion of vertical whitespace).
 - Use camel case for class and type names
 - Use underscores for functions and variables.
 - Use double quotes.
-- Use parentheses instead of '\' for line continuation where ever possible (which is pretty much everywhere)
+- Use parentheses instead of '\\' for line continuation where ever possible
+  (which is pretty much everywhere)
 - There should be max a single new line between:
     - statements
     - functions in a class
@@ -14,5 +18,32 @@ Basically, PEP8
     - a single space after a comma
     - a single space before and after for '=' when used as assignment
     - no spaces before and after for '=' for default values and keyword arguments.
+- Indenting must follow PEP8; either hanging indent or multiline-visual indent
+  depending on the size and shape of the arguments and what makes more sense to
+  the author. In other words, both this::
 
-Comments should follow the google code style. This is so that we can generate documentation with sphinx (http://sphinxcontrib-napoleon.readthedocs.org/en/latest/) 
+    print("I am a fish %s" % "moo")
+
+  and this::
+
+    print("I am a fish %s" %
+          "moo")
+
+  and this::
+
+    print(
+        "I am a fish %s" %
+        "moo"
+    )
+
+  ...are valid, although given each one takes up 2x more vertical space than
+  the previous, it's up to the author's discretion as to which layout makes most
+  sense for their function invocation.  (e.g. if they want to add comments
+  per-argument, or put expressions in the arguments, or group related arguments
+  together, or want to deliberately extend or preserve vertical/horizontal
+  space)
+
+Comments should follow the google code style. This is so that we can generate
+documentation with sphinx (http://sphinxcontrib-napoleon.readthedocs.org/en/latest/) 
+
+Code should pass pep8 --max-line-length=100 without any warnings.

docs/media_repository.rst

@@ -0,0 +1,27 @@
+Media Repository 
+================
+
+*Synapse implementation-specific details for the media repository*
+
+The media repository is where attachments and avatar photos are stored.
+It stores attachment content and thumbnails for media uploaded by local users.
+It caches attachment content and thumbnails for media uploaded by remote users.
+
+Storage
+-------
+
+Each item of media is assigned a ``media_id`` when it is uploaded.
+The ``media_id`` is a randomly chosen, URL safe 24 character string.
+Metadata such as the MIME type, upload time and length are stored in the
+sqlite3 database indexed by ``media_id``.
+Content is stored on the filesystem under a ``"local_content"`` directory.
+Thumbnails are stored under a ``"local_thumbnails"`` directory.
+The item with ``media_id`` ``"aabbccccccccdddddddddddd"`` is stored under
+``"local_content/aa/bb/ccccccccdddddddddddd"``. Its thumbnail with width
+``128`` and height ``96`` and type ``"image/jpeg"`` is stored under
+``"local_thumbnails/aa/bb/ccccccccdddddddddddd/128-96-image-jpeg"``
+Remote content is cached under ``"remote_content"`` directory. Each item of
+remote content is assigned a local "``filesystem_id``" to ensure that the
+directory structure ``"remote_content/server_name/aa/bb/ccccccccdddddddddddd"``
+is appropriate. Thumbnails for remote content are stored under
+``"remote_thumbnails/server_name/..."``

jsfiddles/create_room_send_msg/demo.css

@@ -1,17 +0,0 @@
-.loggedin {
-    visibility: hidden;
-}
-
-p {
-    font-family: monospace;
-}
-
-table
-{
-    border-spacing:5px;
-}
-
-th,td
-{
-    padding:5px;
-}

jsfiddles/create_room_send_msg/demo.html

@@ -1,30 +0,0 @@
-<div>
-    <p>This room creation / message sending demo requires a home server to be running on http://localhost:8008</p>
-</div>
-<form class="loginForm">
-    <input type="text" id="userLogin" placeholder="Username"></input>
-    <input type="password" id="passwordLogin" placeholder="Password"></input>
-    <input type="button" class="login" value="Login"></input>
-</form>
-<div class="loggedin">
-    <form class="createRoomForm">
-        <input type="text" id="roomAlias" placeholder="Room alias (optional)"></input>
-        <input type="button" class="createRoom" value="Create Room"></input>
-    </form>
-    <form class="sendMessageForm">
-        <input type="text" id="roomId" placeholder="Room ID"></input>
-        <input type="text" id="messageBody" placeholder="Message body"></input>
-        <input type="button" class="sendMessage" value="Send Message"></input>
-    </form>
-    <table id="rooms">
-        <tbody>
-            <tr>
-                <th>Room ID</th>
-                <th>My state</th>
-                <th>Room Alias</th>
-                <th>Latest message</th>
-            </tr>
-        </tbody>
-    </table>
-</div>
-    

jsfiddles/create_room_send_msg/demo.js

@@ -1,113 +0,0 @@
-var accountInfo = {};
-
-var showLoggedIn = function(data) {
-    accountInfo = data;
-    getCurrentRoomList();
-    $(".loggedin").css({visibility: "visible"});
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            var errMsg = "To try this, you need a home server running!";
-            var errJson = $.parseJSON(err.responseText);
-            if (errJson) {
-                errMsg = JSON.stringify(errJson);   
-            }
-            alert(errMsg);  
-        }
-    }); 
-});
-
-var getCurrentRoomList = function() {
-    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
-    $.getJSON(url, function(data) {
-        var rooms = data.rooms;
-        for (var i=0; i<rooms.length; ++i) {
-            rooms[i].latest_message = rooms[i].messages.chunk[0].content.body;
-            addRoom(rooms[i]);   
-        }
-    }).fail(function(err) {
-        alert(JSON.stringify($.parseJSON(err.responseText)));
-    });
-};
-
-$('.createRoom').live('click', function() {
-    var roomAlias = $("#roomAlias").val();
-    var data = {};
-    if (roomAlias.length > 0) {
-        data.room_alias_name = roomAlias;   
-    }
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            data.membership = "join"; // you are automatically joined into every room you make.
-            data.latest_message = "";
-            addRoom(data);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-var addRoom = function(data) {
-    row = "<tr>" +
-        "<td>"+data.room_id+"</td>" +
-        "<td>"+data.membership+"</td>" +
-        "<td>"+data.room_alias+"</td>" +
-        "<td>"+data.latest_message+"</td>" +
-        "</tr>";
-    $("#rooms").append(row);
-};
-
-$('.sendMessage').live('click', function() {
-    var roomId = $("#roomId").val();
-    var body = $("#messageBody").val();
-    var msgId = $.now();
-    
-    if (roomId.length === 0 || body.length === 0) {
-        return;
-    }
-    
-    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomid", encodeURIComponent(roomId));
-    
-    var data = {
-        msgtype: "m.text",
-        body: body
-    };
-    
-    $.ajax({
-        url: url,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            $("#messageBody").val("");
-            // wipe the table and reload it. Using the event stream would be the best
-            // solution but that is out of scope of this fiddle.
-            $("#rooms").find("tr:gt(0)").remove();
-            getCurrentRoomList();
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});

jsfiddles/event_stream/demo.css

@@ -1,17 +0,0 @@
-.loggedin {
-    visibility: hidden;
-}
-
-p {
-    font-family: monospace;
-}
-
-table
-{
-    border-spacing:5px;
-}
-
-th,td
-{
-    padding:5px;
-}

jsfiddles/event_stream/demo.html

@@ -1,23 +0,0 @@
-<div>
-    <p>This event stream demo requires a home server to be running on http://localhost:8008</p>
-</div>
-<form class="loginForm">
-    <input type="text" id="userLogin" placeholder="Username"></input>
-    <input type="password" id="passwordLogin" placeholder="Password"></input>
-    <input type="button" class="login" value="Login"></input>
-</form>
-<div class="loggedin">
-    <form class="sendMessageForm">
-        <input type="button" class="sendMessage" value="Send random message"></input>
-    </form>
-    <p id="streamErrorText"></p>
-    <table id="rooms">
-        <tbody>
-            <tr>
-                <th>Room ID</th>
-                <th>Latest message</th>
-            </tr>
-        </tbody>
-    </table>
-</div>
-    

jsfiddles/event_stream/demo.js

@@ -1,145 +0,0 @@
-var accountInfo = {};
-
-var eventStreamInfo = {
-    from: "END"
-};
-
-var roomInfo = [];
-
-var longpollEventStream = function() {
-    var url = "http://localhost:8008/_matrix/client/api/v1/events?access_token=$token&from=$from";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$from", eventStreamInfo.from);
-    
-    $.getJSON(url, function(data) {
-        eventStreamInfo.from = data.end;
-        
-        var hasNewLatestMessage = false;
-        for (var i=0; i<data.chunk.length; ++i) {
-            if (data.chunk[i].type === "m.room.message") {
-                for (var j=0; j<roomInfo.length; ++j) {
-                    if (roomInfo[j].room_id === data.chunk[i].room_id) {
-                        roomInfo[j].latest_message = data.chunk[i].content.body;
-                        hasNewLatestMessage = true;
-                    }
-                }
-            }
-        }
-        
-        if (hasNewLatestMessage) {
-           setRooms(roomInfo);
-        }
-        $("#streamErrorText").text("");
-        longpollEventStream();
-    }).fail(function(err) {
-        $("#streamErrorText").text("Event stream error: "+JSON.stringify($.parseJSON(err.responseText)));
-        setTimeout(longpollEventStream, 5000);
-    });
-};
-
-var showLoggedIn = function(data) {
-    accountInfo = data;
-    longpollEventStream();
-    getCurrentRoomList();
-    $(".loggedin").css({visibility: "visible"});
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            $("#rooms").find("tr:gt(0)").remove();
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            var errMsg = "To try this, you need a home server running!";
-            var errJson = $.parseJSON(err.responseText);
-            if (errJson) {
-                errMsg = JSON.stringify(errJson);   
-            }
-            alert(errMsg);  
-        }
-    }); 
-});
-
-var getCurrentRoomList = function() {
-    $("#roomId").val("");
-    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
-    $.getJSON(url, function(data) {
-        var rooms = data.rooms;
-        for (var i=0; i<rooms.length; ++i) {
-            if ("messages" in rooms[i]) {
-                rooms[i].latest_message = rooms[i].messages.chunk[0].content.body;   
-            }
-        }
-        roomInfo = rooms;
-        setRooms(roomInfo);  
-    }).fail(function(err) {
-        alert(JSON.stringify($.parseJSON(err.responseText)));
-    });
-};
-
-$('.sendMessage').live('click', function() {
-    if (roomInfo.length === 0) {
-        alert("There is no room to send a message to!");
-        return;
-    }
-    
-    var index = Math.floor(Math.random() * roomInfo.length);
-    
-    sendMessage(roomInfo[index].room_id);
-});
-
-var sendMessage = function(roomId) {
-    var body = "jsfiddle message @" + $.now();
-    
-    if (roomId.length === 0) {
-        return;
-    }
-    
-    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomid", encodeURIComponent(roomId));
-    
-    var data = {
-        msgtype: "m.text",
-        body: body
-    };
-    
-    $.ajax({
-        url: url,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            $("#messageBody").val("");
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-};
-
-var setRooms = function(roomList) {
-    // wipe existing entries
-    $("#rooms").find("tr:gt(0)").remove();
-    
-    var rows = "";
-    for (var i=0; i<roomList.length; ++i) {
-        row = "<tr>" +
-            "<td>"+roomList[i].room_id+"</td>" +
-            "<td>"+roomList[i].latest_message+"</td>" +
-            "</tr>";  
-        rows += row;
-    }
-    
-    $("#rooms").append(rows);
-};
-

jsfiddles/example_app/demo.css

@@ -1,43 +0,0 @@
-.roomListDashboard, .roomContents, .sendMessageForm {
-    visibility: hidden;
-}
-
-.roomList {
-    background-color: #909090;
-}
-
-.messageWrapper {
-    background-color: #EEEEEE;
-    height: 400px;
-    overflow: scroll;
-}
-
-.membersWrapper {
-    background-color: #EEEEEE;
-    height: 200px;
-    width: 50%;
-    overflow: scroll;
-}
-
-.textEntry {
-    width: 100%
-}
-
-p {
-    font-family: monospace;
-}
-
-table
-{
-    border-spacing:5px;
-}
-
-th,td
-{
-    padding:5px;
-}
-
-.roomList tr:not(:first-child):hover {
-    background-color: orange;
-    cursor: pointer;
-}

jsfiddles/example_app/demo.details

@@ -1,7 +0,0 @@
- name: Example Matrix Client
- description: Includes login, live event streaming, creating rooms, sending messages and viewing member lists.
- authors:
-   - matrix.org
- resources:
-   - http://matrix.org
- normalize_css: no

jsfiddles/example_app/demo.html

@@ -1,56 +0,0 @@
-<div class="signUp">
-    <p>Matrix example application: Requires a local home server running at http://localhost:8008</p>
-    <form class="registrationForm">
-        <p>No account? Register:</p>
-        <input type="text" id="userReg" placeholder="Username"></input>
-        <input type="password" id="passwordReg" placeholder="Password"></input>
-        <input type="button" class="register" value="Register"></input>
-    </form>
-    <form class="loginForm">
-        <p>Got an account? Login:</p>
-        <input type="text" id="userLogin" placeholder="Username"></input>
-        <input type="password" id="passwordLogin" placeholder="Password"></input>
-        <input type="button" class="login" value="Login"></input>
-    </form>
-</div>
-
-<div class="roomListDashboard">
-    <form class="createRoomForm">
-        <input type="text" id="roomAlias" placeholder="Room alias"></input>
-        <input type="button" class="createRoom" value="Create Room"></input>
-    </form>
-    <table id="rooms" class="roomList">
-        <tbody>
-            <tr>
-                <th>Room</th>
-                <th>My state</th>
-                <th>Latest message</th>
-            </tr>
-        </tbody>
-    </table>
-</div>
-
-<div class="roomContents">
-    <p id="roomName">Select a room</p>
-    <div class="messageWrapper">
-        <table id="messages">
-            <tbody>
-            </tbody>
-        </table>
-    </div>
-    <form class="sendMessageForm">
-        <input type="text" class="textEntry" id="body" placeholder="Enter text here..." onkeydown="javascript:if (event.keyCode == 13) document.getElementById('sendMsg').focus()"></input>
-        <input type="button" class="sendMessage" id="sendMsg" value="Send"></input>
-    </form>
-</div>
-
-<div>
-    <p>Member list:</p>
-    <div class="membersWrapper">
-        <table id="members">
-            <tbody>
-            </tbody>
-        </table>
-    </div>
-</div>
-    

jsfiddles/example_app/demo.js

@@ -1,327 +0,0 @@
-var accountInfo = {};
-
-var eventStreamInfo = {
-    from: "END"
-};
-
-var roomInfo = [];
-var memberInfo = [];
-var viewingRoomId;
-
-// ************** Event Streaming **************
-var longpollEventStream = function() {
-    var url = "http://localhost:8008/_matrix/client/api/v1/events?access_token=$token&from=$from";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$from", eventStreamInfo.from);
-
-    $.getJSON(url, function(data) {
-        eventStreamInfo.from = data.end;
-        
-        var hasNewLatestMessage = false;
-        var updatedMemberList = false;
-        var i=0;
-        var j=0;
-        for (i=0; i<data.chunk.length; ++i) {
-            if (data.chunk[i].type === "m.room.message") {
-                console.log("Got new message: " + JSON.stringify(data.chunk[i]));
-                if (viewingRoomId === data.chunk[i].room_id) {
-                    addMessage(data.chunk[i]);
-                }
-                
-                for (j=0; j<roomInfo.length; ++j) {
-                    if (roomInfo[j].room_id === data.chunk[i].room_id) {
-                        roomInfo[j].latest_message = data.chunk[i].content.body;
-                        hasNewLatestMessage = true;
-                    }
-                }
-            }
-            else if (data.chunk[i].type === "m.room.member") {
-                if (viewingRoomId === data.chunk[i].room_id) {
-                    console.log("Got new member: " + JSON.stringify(data.chunk[i]));
-                    addMessage(data.chunk[i]);
-                    for (j=0; j<memberInfo.length; ++j) {
-                        if (memberInfo[j].state_key === data.chunk[i].state_key) {
-                            memberInfo[j] = data.chunk[i];
-                            updatedMemberList = true;
-                            break;
-                        }
-                    }
-                    if (!updatedMemberList) {
-                        memberInfo.push(data.chunk[i]);  
-                        updatedMemberList = true;
-                    }
-                }
-                if (data.chunk[i].state_key === accountInfo.user_id) {
-                    getCurrentRoomList(); // update our join/invite list
-                }
-            }
-            else {
-                console.log("Discarding: " + JSON.stringify(data.chunk[i]));
-            }
-        }
-        
-        if (hasNewLatestMessage) {
-           setRooms(roomInfo);
-        }
-        if (updatedMemberList) {
-            $("#members").empty();
-            for (i=0; i<memberInfo.length; ++i) { 
-                addMember(memberInfo[i]);
-            }
-        }
-        longpollEventStream();
-    }).fail(function(err) {
-        setTimeout(longpollEventStream, 5000);
-    });
-};
-
-// ************** Registration and Login **************
-var onLoggedIn = function(data) {
-    accountInfo = data;
-    longpollEventStream();
-    getCurrentRoomList();
-    $(".roomListDashboard").css({visibility: "visible"});
-    $(".roomContents").css({visibility: "visible"});
-    $(".signUp").css({display: "none"});
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            onLoggedIn(data);
-        },
-        error: function(err) {
-            alert("Unable to login: is the home server running?");  
-        }
-    }); 
-});
-
-$('.register').live('click', function() {
-    var user = $("#userReg").val();
-    var password = $("#passwordReg").val();
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/register",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            onLoggedIn(data);
-        },
-        error: function(err) {
-            var msg = "Is the home server running?";
-            var errJson = $.parseJSON(err.responseText);
-            if (errJson !== null) {
-                msg = errJson.error;   
-            }
-            alert("Unable to register: "+msg);  
-        }
-    });
-});
-
-// ************** Creating a room ******************
-$('.createRoom').live('click', function() {
-    var roomAlias = $("#roomAlias").val();
-    var data = {};
-    if (roomAlias.length > 0) {
-        data.room_alias_name = roomAlias;   
-    }
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(response) {
-            $("#roomAlias").val("");
-            response.membership = "join"; // you are automatically joined into every room you make.
-            response.latest_message = "";
-            
-            roomInfo.push(response);
-            setRooms(roomInfo);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-// ************** Getting current state **************
-var getCurrentRoomList = function() {
-    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
-    $.getJSON(url, function(data) {
-        var rooms = data.rooms;
-        for (var i=0; i<rooms.length; ++i) {
-            if ("messages" in rooms[i]) {
-                rooms[i].latest_message = rooms[i].messages.chunk[0].content.body;   
-            }
-        }
-        roomInfo = rooms;
-        setRooms(roomInfo);  
-    }).fail(function(err) {
-        alert(JSON.stringify($.parseJSON(err.responseText)));
-    });
-};
-
-var loadRoomContent = function(roomId) {
-    console.log("loadRoomContent " + roomId);
-    viewingRoomId = roomId;
-    $("#roomName").text("Room: "+roomId);
-    $(".sendMessageForm").css({visibility: "visible"});
-    getMessages(roomId);
-    getMemberList(roomId);
-};
-
-var getMessages = function(roomId) {
-    $("#messages").empty();
-    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/" + 
-              encodeURIComponent(roomId) + "/messages?access_token=" + accountInfo.access_token + "&from=END&dir=b&limit=10";
-    $.getJSON(url, function(data) {
-        for (var i=data.chunk.length-1; i>=0; --i) {
-            addMessage(data.chunk[i]);   
-        }
-    });
-};
-
-var getMemberList = function(roomId) {
-    $("#members").empty();
-    memberInfo = [];
-    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/" + 
-              encodeURIComponent(roomId) + "/members?access_token=" + accountInfo.access_token;
-    $.getJSON(url, function(data) {
-        for (var i=0; i<data.chunk.length; ++i) {
-            memberInfo.push(data.chunk[i]);
-            addMember(data.chunk[i]);   
-        }
-    });
-};
-
-// ************** Sending messages **************
-$('.sendMessage').live('click', function() {
-    if (viewingRoomId === undefined) {
-        alert("There is no room to send a message to!");
-        return;
-    }
-    var body = $("#body").val();
-    sendMessage(viewingRoomId, body);
-});
-
-var sendMessage = function(roomId, body) {
-    var msgId = $.now();
-    
-    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomid", encodeURIComponent(roomId));
-    
-    var data = {
-        msgtype: "m.text",
-        body: body
-    };
-    
-    $.ajax({
-        url: url,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            $("#body").val("");
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    });
-};
-
-// ************** Navigation and DOM manipulation **************
-var setRooms = function(roomList) {
-    // wipe existing entries
-    $("#rooms").find("tr:gt(0)").remove();
-    
-    var rows = "";
-    for (var i=0; i<roomList.length; ++i) {
-        row = "<tr>" +
-              "<td>"+roomList[i].room_id+"</td>" +
-              "<td>"+roomList[i].membership+"</td>" +
-              "<td>"+roomList[i].latest_message+"</td>" +
-              "</tr>";  
-        rows += row;
-    }
-    
-    $("#rooms").append(rows);
-    
-    $('#rooms').find("tr").click(function(){
-        var roomId = $(this).find('td:eq(0)').text();
-        var membership = $(this).find('td:eq(1)').text();
-        if (membership !== "join") {
-            console.log("Joining room " + roomId); 
-            var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/join?access_token=$token";
-            url = url.replace("$token", accountInfo.access_token);
-            url = url.replace("$roomid", encodeURIComponent(roomId));
-            $.ajax({
-                url: url,
-                type: "POST",
-                contentType: "application/json; charset=utf-8",
-                data: JSON.stringify({membership: "join"}),
-                dataType: "json",
-                success: function(data) {
-                    loadRoomContent(roomId);
-                    getCurrentRoomList();
-                },
-                error: function(err) {
-                    alert(JSON.stringify($.parseJSON(err.responseText)));  
-                }
-            });
-        }
-        else {
-            loadRoomContent(roomId);
-        }
-    });
-};
-
-var addMessage = function(data) {
-
-    var msg = data.content.body;
-    if (data.type === "m.room.member") {
-        if (data.content.membership === undefined) {
-            return;
-        }
-        if (data.content.membership === "invite") {
-            msg = "<em>invited " + data.state_key + " to the room</em>";
-        }
-        else if (data.content.membership === "join") {
-            msg = "<em>joined the room</em>";
-        }
-        else if (data.content.membership === "leave") {
-            msg = "<em>left the room</em>";
-        }
-        else if (data.content.membership === "ban") {
-            msg = "<em>was banned from the room</em>";
-        }
-    }
-    if (msg === undefined) {
-        return;
-    }
-
-    var row = "<tr>" +
-              "<td>"+data.user_id+"</td>" +
-              "<td>"+msg+"</td>" +
-              "</tr>"; 
-    $("#messages").append(row);
-};
-
-var addMember = function(data) {
-    var row = "<tr>" +
-              "<td>"+data.state_key+"</td>" +
-              "<td>"+data.content.membership+"</td>" +
-              "</tr>"; 
-    $("#members").append(row);
-};
-

jsfiddles/register_login/demo.css

@@ -1,7 +0,0 @@
-.loggedin {
-    visibility: hidden;
-}
-
-p {
-    font-family: monospace;
-}

jsfiddles/register_login/demo.html

@@ -1,20 +0,0 @@
-<div>
-    <p>This registration/login demo requires a home server to be running on http://localhost:8008</p>
-</div>
-<form class="registrationForm">
-    <input type="text" id="user" placeholder="Username"></input>
-    <input type="password" id="password" placeholder="Password"></input>
-    <input type="button" class="register" value="Register"></input>
-</form>
-<form class="loginForm">
-    <input type="text" id="userLogin" placeholder="Username"></input>
-    <input type="password" id="passwordLogin" placeholder="Password"></input>
-    <input type="button" class="login" value="Login"></input>
-</form>
-<div class="loggedin">
-    <p id="welcomeText"></p>
-    <input type="button" class="testToken" value="Test token"></input>
-    <input type="button" class="logout" value="Logout"></input>
-    <p id="imSyncText"></p>
-</div>
-    

jsfiddles/register_login/demo.js

@@ -1,79 +0,0 @@
-var accountInfo = {};
-
-var showLoggedIn = function(data) {
-    accountInfo = data;
-    $(".loggedin").css({visibility: "visible"});
-    $("#welcomeText").text("Welcome " + accountInfo.user_id+". Your access token is:  " +
-                           accountInfo.access_token);    
-};
-
-$('.register').live('click', function() {
-    var user = $("#user").val();
-    var password = $("#password").val();
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/register",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            var errMsg = "To try this, you need a home server running!";
-            var errJson = $.parseJSON(err.responseText);
-            if (errJson) {
-                errMsg = JSON.stringify(errJson);   
-            }
-            alert(errMsg);   
-        }
-    });
-});
-
-var login = function(user, password) {
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            var errMsg = "To try this, you need a home server running!";
-            var errJson = $.parseJSON(err.responseText);
-            if (errJson) {
-                errMsg = JSON.stringify(errJson);   
-            }
-            alert(errMsg);  
-        }
-    });  
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.getJSON("http://localhost:8008/_matrix/client/api/v1/login", function(data) {
-        if (data.flows[0].type !== "m.login.password") {
-            alert("I don't know how to login with this type: " + data.type);
-            return;
-        }
-        login(user, password);
-    });
-});
-
-$('.logout').live('click', function() {
-    accountInfo = {};
-    $("#imSyncText").text("");
-    $(".loggedin").css({visibility: "hidden"});
-});
-
-$('.testToken').live('click', function() {
-    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
-    $.getJSON(url, function(data) {
-         $("#imSyncText").text(JSON.stringify(data, undefined, 2));
-    }).fail(function(err) {
-        $("#imSyncText").text(JSON.stringify($.parseJSON(err.responseText)));
-    });
-});

jsfiddles/room_memberships/demo.css

@@ -1,17 +0,0 @@
-.loggedin {
-    visibility: hidden;
-}
-
-p {
-    font-family: monospace;
-}
-
-table
-{
-    border-spacing:5px;
-}
-
-th,td
-{
-    padding:5px;
-}

jsfiddles/room_memberships/demo.html

@@ -1,37 +0,0 @@
-<div>
-    <p>This room membership demo requires a home server to be running on http://localhost:8008</p>
-</div>
-<form class="loginForm">
-    <input type="text" id="userLogin" placeholder="Username"></input>
-    <input type="password" id="passwordLogin" placeholder="Password"></input>
-    <input type="button" class="login" value="Login"></input>
-</form>
-<div class="loggedin">
-    <form class="createRoomForm">
-        <input type="button" class="createRoom" value="Create Room"></input>
-    </form>
-    <form class="changeMembershipForm">
-        <input type="text" id="roomId" placeholder="Room ID"></input>
-        <input type="text" id="targetUser" placeholder="Target User ID"></input>
-        <select id="membership">
-            <option value="invite">invite</option>
-            <option value="join">join</option>
-            <option value="leave">leave</option>
-        </select>
-        <input type="button" class="changeMembership" value="Change Membership"></input>
-    </form>
-    <form class="joinAliasForm">
-        <input type="text" id="roomAlias" placeholder="Room Alias (#name:domain)"></input>
-        <input type="button" class="joinAlias" value="Join via Alias"></input>
-    </form>
-    <table id="rooms">
-        <tbody>
-            <tr>
-                <th>Room ID</th>
-                <th>My state</th>
-                <th>Room Alias</th>
-            </tr>
-        </tbody>
-    </table>
-</div>
-    

jsfiddles/room_memberships/demo.js

@@ -1,141 +0,0 @@
-var accountInfo = {};
-
-var showLoggedIn = function(data) {
-    accountInfo = data;
-    getCurrentRoomList();
-    $(".loggedin").css({visibility: "visible"});
-    $("#membership").change(function() {
-    if ($("#membership").val() === "invite") {
-        $("#targetUser").css({visibility: "visible"});
-    }
-    else {
-        $("#targetUser").css({visibility: "hidden"});
-    }
-});
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            $("#rooms").find("tr:gt(0)").remove();
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            var errMsg = "To try this, you need a home server running!";
-            var errJson = $.parseJSON(err.responseText);
-            if (errJson) {
-                errMsg = JSON.stringify(errJson);   
-            }
-            alert(errMsg); 
-        }
-    }); 
-});
-
-var getCurrentRoomList = function() {
-    $("#roomId").val("");
-    // wipe the table and reload it. Using the event stream would be the best
-    // solution but that is out of scope of this fiddle.
-    $("#rooms").find("tr:gt(0)").remove();
-    
-    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
-    $.getJSON(url, function(data) {
-        var rooms = data.rooms;
-        for (var i=0; i<rooms.length; ++i) {
-            addRoom(rooms[i]);   
-        }
-    }).fail(function(err) {
-        alert(JSON.stringify($.parseJSON(err.responseText)));
-    });
-};
-
-$('.createRoom').live('click', function() {
-    var data = {};
-    $.ajax({
-        url: "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            data.membership = "join"; // you are automatically joined into every room you make.
-            data.latest_message = "";
-            addRoom(data);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-var addRoom = function(data) {
-    row = "<tr>" +
-        "<td>"+data.room_id+"</td>" +
-        "<td>"+data.membership+"</td>" +
-        "<td>"+data.room_alias+"</td>" +
-        "</tr>";
-    $("#rooms").append(row);
-};
-
-$('.changeMembership').live('click', function() {
-    var roomId = $("#roomId").val();
-    var member = $("#targetUser").val();
-    var membership = $("#membership").val();
-    
-    if (roomId.length === 0) {
-        return;
-    }
-    
-    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/$membership?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomid", encodeURIComponent(roomId));
-    url = url.replace("$membership", membership);
-    
-    var data = {};
-    
-    if (membership === "invite") {
-        data = {
-            user_id: member
-        };
-    }
-
-    $.ajax({
-        url: url,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            getCurrentRoomList();
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-$('.joinAlias').live('click', function() {
-    var roomAlias = $("#roomAlias").val();
-    var url = "http://localhost:8008/_matrix/client/api/v1/join/$roomalias?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomalias", encodeURIComponent(roomAlias));
-    $.ajax({
-        url: url,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({}),
-        dataType: "json",
-        success: function(data) {
-            getCurrentRoomList();
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});

scripts/check_event_hash.py

@@ -18,6 +18,9 @@ class dictobj(dict):
     def get_full_dict(self):
         return dict(self)
 
+    def get_pdu_json(self):
+        return dict(self)
+
 
 def main():
     parser = argparse.ArgumentParser()

scripts/copyrighter-sql.pl

@@ -0,0 +1,33 @@
+#!/usr/bin/perl -pi
+# Copyright 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+$copyright = <<EOT;
+/* Copyright 2015 OpenMarket Ltd
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+EOT
+
+s/^(# -\*- coding: utf-8 -\*-\n)?/$1$copyright/ if ($. == 1);

scripts/copyrighter.pl

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 $copyright = <<EOT;
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

scripts/federation_client.py

@@ -0,0 +1,143 @@
+import nacl.signing
+import json
+import base64
+import requests
+import sys
+import srvlookup
+
+
+def encode_base64(input_bytes):
+    """Encode bytes as a base64 string without any padding."""
+
+    input_len = len(input_bytes)
+    output_len = 4 * ((input_len + 2) // 3) + (input_len + 2) % 3 - 2
+    output_bytes = base64.b64encode(input_bytes)
+    output_string = output_bytes[:output_len].decode("ascii")
+    return output_string
+
+
+def decode_base64(input_string):
+    """Decode a base64 string to bytes inferring padding from the length of the
+    string."""
+
+    input_bytes = input_string.encode("ascii")
+    input_len = len(input_bytes)
+    padding = b"=" * (3 - ((input_len + 3) % 4))
+    output_len = 3 * ((input_len + 2) // 4) + (input_len + 2) % 4 - 2
+    output_bytes = base64.b64decode(input_bytes + padding)
+    return output_bytes[:output_len]
+
+
+def encode_canonical_json(value):
+    return json.dumps(
+         value,
+         # Encode code-points outside of ASCII as UTF-8 rather than \u escapes
+         ensure_ascii=False,
+         # Remove unecessary white space.
+         separators=(',',':'),
+         # Sort the keys of dictionaries.
+         sort_keys=True,
+         # Encode the resulting unicode as UTF-8 bytes.
+     ).encode("UTF-8")
+
+
+def sign_json(json_object, signing_key, signing_name):
+    signatures = json_object.pop("signatures", {})
+    unsigned = json_object.pop("unsigned", None)
+
+    signed = signing_key.sign(encode_canonical_json(json_object))
+    signature_base64 = encode_base64(signed.signature)
+
+    key_id = "%s:%s" % (signing_key.alg, signing_key.version)
+    signatures.setdefault(signing_name, {})[key_id] = signature_base64
+
+    json_object["signatures"] = signatures
+    if unsigned is not None:
+        json_object["unsigned"] = unsigned
+
+    return json_object
+
+
+NACL_ED25519 = "ed25519"
+
+def decode_signing_key_base64(algorithm, version, key_base64):
+    """Decode a base64 encoded signing key
+    Args:
+        algorithm (str): The algorithm the key is for (currently "ed25519").
+        version (str): Identifies this key out of the keys for this entity.
+        key_base64 (str): Base64 encoded bytes of the key.
+    Returns:
+        A SigningKey object.
+    """
+    if algorithm == NACL_ED25519:
+        key_bytes = decode_base64(key_base64)
+        key = nacl.signing.SigningKey(key_bytes)
+        key.version = version
+        key.alg = NACL_ED25519
+        return key
+    else:
+        raise ValueError("Unsupported algorithm %s" % (algorithm,))
+
+
+def read_signing_keys(stream):
+    """Reads a list of keys from a stream
+    Args:
+        stream : A stream to iterate for keys.
+    Returns:
+        list of SigningKey objects.
+    """
+    keys = []
+    for line in stream:
+        algorithm, version, key_base64 = line.split()
+        keys.append(decode_signing_key_base64(algorithm, version, key_base64))
+    return keys
+
+
+def lookup(destination, path):
+    if ":" in destination:
+        return "https://%s%s" % (destination, path)
+    else:
+        srv = srvlookup.lookup("matrix", "tcp", destination)[0]
+        return "https://%s:%d%s" % (srv.host, srv.port, path)
+
+def get_json(origin_name, origin_key, destination, path):
+    request_json = {
+        "method": "GET",
+        "uri": path,
+        "origin": origin_name,
+        "destination": destination,
+    }
+
+    signed_json = sign_json(request_json, origin_key, origin_name)
+
+    authorization_headers = []
+
+    for key, sig in signed_json["signatures"][origin_name].items():
+        authorization_headers.append(bytes(
+            "X-Matrix origin=%s,key=\"%s\",sig=\"%s\"" % (
+                origin_name, key, sig,
+            )
+        ))
+
+    result = requests.get(
+        lookup(destination, path),
+        headers={"Authorization": authorization_headers[0]},
+        verify=False,
+    )
+    return result.json()
+
+
+def main():
+    origin_name, keyfile, destination, path = sys.argv[1:]
+
+    with open(keyfile) as f:
+        key = read_signing_keys(f)[0]
+
+    result = get_json(
+        origin_name, key, destination, "/_matrix/federation/v1/" + path
+    )
+
+    json.dump(result, sys.stdout)
+
+if __name__ == "__main__":
+    main()

scripts/upgrade_db_to_v0.6.0.py

@@ -0,0 +1,331 @@
+
+from synapse.storage import SCHEMA_VERSION, read_schema
+from synapse.storage._base import SQLBaseStore
+from synapse.storage.signatures import SignatureStore
+from synapse.storage.event_federation import EventFederationStore
+
+from syutil.base64util import encode_base64, decode_base64
+
+from synapse.crypto.event_signing import compute_event_signature
+
+from synapse.events.builder import EventBuilder
+from synapse.events.utils import prune_event
+
+from synapse.crypto.event_signing import check_event_content_hash
+
+from syutil.crypto.jsonsign import (
+    verify_signed_json, SignatureVerifyException,
+)
+from syutil.crypto.signing_key import decode_verify_key_bytes
+
+from syutil.jsonutil import encode_canonical_json
+
+import argparse
+# import dns.resolver
+import hashlib
+import httplib
+import json
+import sqlite3
+import syutil
+import urllib2
+
+
+delta_sql = """
+CREATE TABLE IF NOT EXISTS event_json(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    internal_metadata NOT NULL,
+    json BLOB NOT NULL,
+    CONSTRAINT ev_j_uniq UNIQUE (event_id)
+);
+
+CREATE INDEX IF NOT EXISTS event_json_id ON event_json(event_id);
+CREATE INDEX IF NOT EXISTS event_json_room_id ON event_json(room_id);
+
+PRAGMA user_version = 10;
+"""
+
+
+class Store(object):
+    _get_event_signatures_txn = SignatureStore.__dict__["_get_event_signatures_txn"]
+    _get_event_content_hashes_txn = SignatureStore.__dict__["_get_event_content_hashes_txn"]
+    _get_event_reference_hashes_txn = SignatureStore.__dict__["_get_event_reference_hashes_txn"]
+    _get_prev_event_hashes_txn = SignatureStore.__dict__["_get_prev_event_hashes_txn"]
+    _get_prev_events_and_state = EventFederationStore.__dict__["_get_prev_events_and_state"]
+    _get_auth_events = EventFederationStore.__dict__["_get_auth_events"]
+    cursor_to_dict = SQLBaseStore.__dict__["cursor_to_dict"]
+    _simple_select_onecol_txn = SQLBaseStore.__dict__["_simple_select_onecol_txn"]
+    _simple_select_list_txn = SQLBaseStore.__dict__["_simple_select_list_txn"]
+    _simple_insert_txn = SQLBaseStore.__dict__["_simple_insert_txn"]
+
+    def _generate_event_json(self, txn, rows):
+        events = []
+        for row in rows:
+            d = dict(row)
+
+            d.pop("stream_ordering", None)
+            d.pop("topological_ordering", None)
+            d.pop("processed", None)
+
+            if "origin_server_ts" not in d:
+                d["origin_server_ts"] = d.pop("ts", 0)
+            else:
+                d.pop("ts", 0)
+
+            d.pop("prev_state", None)
+            d.update(json.loads(d.pop("unrecognized_keys")))
+
+            d["sender"] = d.pop("user_id")
+
+            d["content"] = json.loads(d["content"])
+
+            if "age_ts" not in d:
+                # For compatibility
+                d["age_ts"] = d.get("origin_server_ts", 0)
+
+            d.setdefault("unsigned", {})["age_ts"] = d.pop("age_ts")
+
+            outlier = d.pop("outlier", False)
+
+            # d.pop("membership", None)
+
+            d.pop("state_hash", None)
+
+            d.pop("replaces_state", None)
+
+            b = EventBuilder(d)
+            b.internal_metadata.outlier = outlier
+
+            events.append(b)
+
+        for i, ev in enumerate(events):
+            signatures = self._get_event_signatures_txn(
+                txn, ev.event_id,
+            )
+
+            ev.signatures = {
+                n: {
+                    k: encode_base64(v) for k, v in s.items()
+                }
+                for n, s in signatures.items()
+            }
+
+            hashes = self._get_event_content_hashes_txn(
+                txn, ev.event_id,
+            )
+
+            ev.hashes = {
+                k: encode_base64(v) for k, v in hashes.items()
+            }
+
+            prevs = self._get_prev_events_and_state(txn, ev.event_id)
+
+            ev.prev_events = [
+                (e_id, h)
+                for e_id, h, is_state in prevs
+                if is_state == 0
+            ]
+
+            # ev.auth_events = self._get_auth_events(txn, ev.event_id)
+
+            hashes = dict(ev.auth_events)
+
+            for e_id, hash in ev.prev_events:
+                if e_id in hashes and not hash:
+                    hash.update(hashes[e_id])
+            #
+            # if hasattr(ev, "state_key"):
+            #     ev.prev_state = [
+            #         (e_id, h)
+            #         for e_id, h, is_state in prevs
+            #         if is_state == 1
+            #     ]
+
+        return [e.build() for e in events]
+
+
+store = Store()
+
+
+# def get_key(server_name):
+#     print "Getting keys for: %s" % (server_name,)
+#     targets = []
+#     if ":" in server_name:
+#         target, port = server_name.split(":")
+#         targets.append((target, int(port)))
+#     try:
+#         answers = dns.resolver.query("_matrix._tcp." + server_name, "SRV")
+#         for srv in answers:
+#             targets.append((srv.target, srv.port))
+#     except dns.resolver.NXDOMAIN:
+#         targets.append((server_name, 8448))
+#     except:
+#         print "Failed to lookup keys for %s" % (server_name,)
+#         return {}
+#
+#     for target, port in targets:
+#         url = "https://%s:%i/_matrix/key/v1" % (target, port)
+#         try:
+#             keys = json.load(urllib2.urlopen(url, timeout=2))
+#             verify_keys = {}
+#             for key_id, key_base64 in keys["verify_keys"].items():
+#                 verify_key = decode_verify_key_bytes(
+#                     key_id, decode_base64(key_base64)
+#                 )
+#                 verify_signed_json(keys, server_name, verify_key)
+#                 verify_keys[key_id] = verify_key
+#             print "Got keys for: %s" % (server_name,)
+#             return verify_keys
+#         except urllib2.URLError:
+#             pass
+#         except urllib2.HTTPError:
+#             pass
+#         except httplib.HTTPException:
+#             pass
+#
+#     print "Failed to get keys for %s" % (server_name,)
+#     return {}
+
+
+def reinsert_events(cursor, server_name, signing_key):
+    print "Running delta: v10"
+
+    cursor.executescript(delta_sql)
+
+    cursor.execute(
+        "SELECT * FROM events ORDER BY rowid ASC"
+    )
+
+    print "Getting events..."
+
+    rows = store.cursor_to_dict(cursor)
+
+    events = store._generate_event_json(cursor, rows)
+
+    print "Got events from DB."
+
+    algorithms = {
+        "sha256": hashlib.sha256,
+    }
+
+    key_id = "%s:%s" % (signing_key.alg, signing_key.version)
+    verify_key = signing_key.verify_key
+    verify_key.alg = signing_key.alg
+    verify_key.version = signing_key.version
+
+    server_keys = {
+        server_name: {
+            key_id: verify_key
+        }
+    }
+
+    i = 0
+    N = len(events)
+
+    for event in events:
+        if i % 100 == 0:
+            print "Processed: %d/%d events" % (i,N,)
+        i += 1
+
+        # for alg_name in event.hashes:
+        #     if check_event_content_hash(event, algorithms[alg_name]):
+        #         pass
+        #     else:
+        #         pass
+        #         print "FAIL content hash %s %s" % (alg_name, event.event_id, )
+
+        have_own_correctly_signed = False
+        for host, sigs in event.signatures.items():
+            pruned = prune_event(event)
+
+            for key_id in sigs:
+                if host not in server_keys:
+                    server_keys[host] = {}  # get_key(host)
+                if key_id in server_keys[host]:
+                    try:
+                        verify_signed_json(
+                            pruned.get_pdu_json(),
+                            host,
+                            server_keys[host][key_id]
+                        )
+
+                        if host == server_name:
+                            have_own_correctly_signed = True
+                    except SignatureVerifyException:
+                        print "FAIL signature check %s %s" % (
+                            key_id, event.event_id
+                        )
+
+        # TODO: Re sign with our own server key
+        if not have_own_correctly_signed:
+            sigs = compute_event_signature(event, server_name, signing_key)
+            event.signatures.update(sigs)
+
+            pruned = prune_event(event)
+
+            for key_id in event.signatures[server_name]:
+                verify_signed_json(
+                    pruned.get_pdu_json(),
+                    server_name,
+                    server_keys[server_name][key_id]
+                )
+
+        event_json = encode_canonical_json(
+            event.get_dict()
+        ).decode("UTF-8")
+
+        metadata_json = encode_canonical_json(
+            event.internal_metadata.get_dict()
+        ).decode("UTF-8")
+
+        store._simple_insert_txn(
+            cursor,
+            table="event_json",
+            values={
+                "event_id": event.event_id,
+                "room_id": event.room_id,
+                "internal_metadata": metadata_json,
+                "json": event_json,
+            },
+            or_replace=True,
+        )
+
+
+def main(database, server_name, signing_key):
+    conn = sqlite3.connect(database)
+    cursor = conn.cursor()
+
+    # Do other deltas:
+    cursor.execute("PRAGMA user_version")
+    row = cursor.fetchone()
+
+    if row and row[0]:
+        user_version = row[0]
+        # Run every version since after the current version.
+        for v in range(user_version + 1, 10):
+            print "Running delta: %d" % (v,)
+            sql_script = read_schema("delta/v%d" % (v,))
+            cursor.executescript(sql_script)
+
+    reinsert_events(cursor, server_name, signing_key)
+
+    conn.commit()
+
+    print "Success!"
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("database")
+    parser.add_argument("server_name")
+    parser.add_argument(
+        "signing_key", type=argparse.FileType('r'),
+    )
+    args = parser.parse_args()
+
+    signing_key = syutil.crypto.signing_key.read_signing_keys(
+        args.signing_key
+    )
+
+    main(args.database, args.server_name, signing_key[0])

setup.py

@@ -32,7 +32,7 @@ setup(
     description="Reference Synapse Home Server",
     install_requires=[
         "syutil==0.0.2",
-        "matrix_angular_sdk==0.5.1",
+        "matrix_angular_sdk==0.6.0",
         "Twisted>=14.0.0",
         "service_identity>=1.0.0",
         "pyopenssl>=0.14",
@@ -41,11 +41,13 @@ setup(
         "pynacl",
         "daemonize",
         "py-bcrypt",
+        "frozendict>=0.4",
+        "pillow",
     ],
     dependency_links=[
         "https://github.com/matrix-org/syutil/tarball/v0.0.2#egg=syutil-0.0.2",
         "https://github.com/pyca/pynacl/tarball/d4d3175589b892f6ea7c22f466e0e223853516fa#egg=pynacl-0.3.0",
-        "https://github.com/matrix-org/matrix-angular-sdk/tarball/v0.5.1/#egg=matrix_angular_sdk-0.5.1",
+        "https://github.com/matrix-org/matrix-angular-sdk/tarball/v0.6.0/#egg=matrix_angular_sdk-0.6.0",
     ],
     setup_requires=[
         "setuptools_trial",
@@ -59,6 +61,6 @@ setup(
     entry_points="""
     [console_scripts]
     synctl=synapse.app.synctl:main
-    synapse-homeserver=synapse.app.homeserver:run
+    synapse-homeserver=synapse.app.homeserver:main
     """
 )

synapse/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-""" This is a reference implementation of a synapse home server.
+""" This is a reference implementation of a Matrix home server.
 """
 
-__version__ = "0.5.4"
+__version__ = "0.6.1b"

synapse/api/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/api/auth.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,14 +17,10 @@
 
 from twisted.internet import defer
 
-from synapse.api.constants import Membership, JoinRules
+from synapse.api.constants import EventTypes, Membership, JoinRules
 from synapse.api.errors import AuthError, StoreError, Codes, SynapseError
-from synapse.api.events.room import (
-    RoomMemberEvent, RoomPowerLevelsEvent, RoomRedactionEvent,
-    RoomJoinRulesEvent, RoomCreateEvent, RoomAliasesEvent,
-)
 from synapse.util.logutils import log_function
-from syutil.base64util import encode_base64
+from synapse.util.async import run_on_reactor
 
 import logging
 
@@ -53,15 +49,17 @@ class Auth(object):
                 logger.warn("Trusting event: %s", event.event_id)
                 return True
 
-            if event.type == RoomCreateEvent.TYPE:
+            if event.type == EventTypes.Create:
                 # FIXME
                 return True
 
             # FIXME: Temp hack
-            if event.type == RoomAliasesEvent.TYPE:
+            if event.type == EventTypes.Aliases:
                 return True
 
-            if event.type == RoomMemberEvent.TYPE:
+            logger.debug("Auth events: %s", auth_events)
+
+            if event.type == EventTypes.Member:
                 allowed = self.is_membership_change_allowed(
                     event, auth_events
                 )
@@ -74,10 +72,10 @@ class Auth(object):
             self.check_event_sender_in_room(event, auth_events)
             self._can_send_event(event, auth_events)
 
-            if event.type == RoomPowerLevelsEvent.TYPE:
+            if event.type == EventTypes.PowerLevels:
                 self._check_power_levels(event, auth_events)
 
-            if event.type == RoomRedactionEvent.TYPE:
+            if event.type == EventTypes.Redaction:
                 self._check_redaction(event, auth_events)
 
             logger.debug("Allowing! %s", event)
@@ -93,7 +91,7 @@ class Auth(object):
     def check_joined_room(self, room_id, user_id):
         member = yield self.state.get_current_state(
             room_id=room_id,
-            event_type=RoomMemberEvent.TYPE,
+            event_type=EventTypes.Member,
             state_key=user_id
         )
         self._check_joined_room(member, user_id, room_id)
@@ -104,7 +102,7 @@ class Auth(object):
         curr_state = yield self.state.get_current_state(room_id)
 
         for event in curr_state:
-            if event.type == RoomMemberEvent.TYPE:
+            if event.type == EventTypes.Member:
                 try:
                     if self.hs.parse_userid(event.state_key).domain != host:
                         continue
@@ -118,7 +116,7 @@ class Auth(object):
         defer.returnValue(False)
 
     def check_event_sender_in_room(self, event, auth_events):
-        key = (RoomMemberEvent.TYPE, event.user_id, )
+        key = (EventTypes.Member, event.user_id, )
         member_event = auth_events.get(key)
 
         return self._check_joined_room(
@@ -140,7 +138,7 @@ class Auth(object):
         # Check if this is the room creator joining:
         if len(event.prev_events) == 1 and Membership.JOIN == membership:
             # Get room creation event:
-            key = (RoomCreateEvent.TYPE, "", )
+            key = (EventTypes.Create, "", )
             create = auth_events.get(key)
             if create and event.prev_events[0][0] == create.event_id:
                 if create.content["creator"] == event.state_key:
@@ -149,19 +147,19 @@ class Auth(object):
         target_user_id = event.state_key
 
         # get info about the caller
-        key = (RoomMemberEvent.TYPE, event.user_id, )
+        key = (EventTypes.Member, event.user_id, )
         caller = auth_events.get(key)
 
         caller_in_room = caller and caller.membership == Membership.JOIN
         caller_invited = caller and caller.membership == Membership.INVITE
 
         # get info about the target
-        key = (RoomMemberEvent.TYPE, target_user_id, )
+        key = (EventTypes.Member, target_user_id, )
         target = auth_events.get(key)
 
         target_in_room = target and target.membership == Membership.JOIN
 
-        key = (RoomJoinRulesEvent.TYPE, "", )
+        key = (EventTypes.JoinRules, "", )
         join_rule_event = auth_events.get(key)
         if join_rule_event:
             join_rule = join_rule_event.content.get(
@@ -256,7 +254,7 @@ class Auth(object):
         return True
 
     def _get_power_level_from_event_state(self, event, user_id, auth_events):
-        key = (RoomPowerLevelsEvent.TYPE, "", )
+        key = (EventTypes.PowerLevels, "", )
         power_level_event = auth_events.get(key)
         level = None
         if power_level_event:
@@ -264,7 +262,7 @@ class Auth(object):
             if not level:
                 level = power_level_event.content.get("users_default", 0)
         else:
-            key = (RoomCreateEvent.TYPE, "", )
+            key = (EventTypes.Create, "", )
             create_event = auth_events.get(key)
             if (create_event is not None and
                     create_event.content["creator"] == user_id):
@@ -273,7 +271,7 @@ class Auth(object):
         return level
 
     def _get_ops_level_from_event_state(self, event, auth_events):
-        key = (RoomPowerLevelsEvent.TYPE, "", )
+        key = (EventTypes.PowerLevels, "", )
         power_level_event = auth_events.get(key)
 
         if power_level_event:
@@ -351,29 +349,31 @@ class Auth(object):
         return self.store.is_server_admin(user)
 
     @defer.inlineCallbacks
-    def add_auth_events(self, event):
-        if event.type == RoomCreateEvent.TYPE:
-            event.auth_events = []
+    def add_auth_events(self, builder, context):
+        yield run_on_reactor()
+
+        if builder.type == EventTypes.Create:
+            builder.auth_events = []
             return
 
-        auth_events = []
+        auth_ids = []
 
-        key = (RoomPowerLevelsEvent.TYPE, "", )
-        power_level_event = event.old_state_events.get(key)
+        key = (EventTypes.PowerLevels, "", )
+        power_level_event = context.current_state.get(key)
 
         if power_level_event:
-            auth_events.append(power_level_event.event_id)
+            auth_ids.append(power_level_event.event_id)
 
-        key = (RoomJoinRulesEvent.TYPE, "", )
-        join_rule_event = event.old_state_events.get(key)
+        key = (EventTypes.JoinRules, "", )
+        join_rule_event = context.current_state.get(key)
 
-        key = (RoomMemberEvent.TYPE, event.user_id, )
-        member_event = event.old_state_events.get(key)
+        key = (EventTypes.Member, builder.user_id, )
+        member_event = context.current_state.get(key)
 
-        key = (RoomCreateEvent.TYPE, "", )
-        create_event = event.old_state_events.get(key)
+        key = (EventTypes.Create, "", )
+        create_event = context.current_state.get(key)
         if create_event:
-            auth_events.append(create_event.event_id)
+            auth_ids.append(create_event.event_id)
 
         if join_rule_event:
             join_rule = join_rule_event.content.get("join_rule")
@@ -381,33 +381,37 @@ class Auth(object):
         else:
             is_public = False
 
-        if event.type == RoomMemberEvent.TYPE:
-            e_type = event.content["membership"]
+        if builder.type == EventTypes.Member:
+            e_type = builder.content["membership"]
             if e_type in [Membership.JOIN, Membership.INVITE]:
                 if join_rule_event:
-                    auth_events.append(join_rule_event.event_id)
+                    auth_ids.append(join_rule_event.event_id)
 
+            if e_type == Membership.JOIN:
                 if member_event and not is_public:
-                    auth_events.append(member_event.event_id)
+                    auth_ids.append(member_event.event_id)
+            else:
+                if member_event:
+                    auth_ids.append(member_event.event_id)
         elif member_event:
             if member_event.content["membership"] == Membership.JOIN:
-                auth_events.append(member_event.event_id)
+                auth_ids.append(member_event.event_id)
 
-        hashes = yield self.store.get_event_reference_hashes(
-            auth_events
+        auth_events_entries = yield self.store.add_event_hashes(
+            auth_ids
         )
-        hashes = [
-            {
-                k: encode_base64(v) for k, v in h.items()
-                if k == "sha256"
-            }
-            for h in hashes
-        ]
-        event.auth_events = zip(auth_events, hashes)
+
+        builder.auth_events = auth_events_entries
+
+        context.auth_events = {
+            k: v
+            for k, v in context.current_state.items()
+            if v.event_id in auth_ids
+        }
 
     @log_function
     def _can_send_event(self, event, auth_events):
-        key = (RoomPowerLevelsEvent.TYPE, "", )
+        key = (EventTypes.PowerLevels, "", )
         send_level_event = auth_events.get(key)
         send_level = None
         if send_level_event:

synapse/api/constants.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -59,3 +59,18 @@ class LoginType(object):
     EMAIL_URL = u"m.login.email.url"
     EMAIL_IDENTITY = u"m.login.email.identity"
     RECAPTCHA = u"m.login.recaptcha"
+
+
+class EventTypes(object):
+    Member = "m.room.member"
+    Create = "m.room.create"
+    JoinRules = "m.room.join_rules"
+    PowerLevels = "m.room.power_levels"
+    Aliases = "m.room.aliases"
+    Redaction = "m.room.redaction"
+    Feedback = "m.room.message.feedback"
+
+    # These are used for validation
+    Message = "m.room.message"
+    Topic = "m.room.topic"
+    Name = "m.room.name"

synapse/api/errors.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -34,6 +34,7 @@ class Codes(object):
     LIMIT_EXCEEDED = "M_LIMIT_EXCEEDED"
     CAPTCHA_NEEDED = "M_CAPTCHA_NEEDED"
     CAPTCHA_INVALID = "M_CAPTCHA_INVALID"
+    TOO_LARGE = "M_TOO_LARGE"
 
 
 class CodeMessageException(Exception):

synapse/api/events/__init__.py

@@ -1,148 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.util.jsonobject import JsonEncodedObject
-
-
-def serialize_event(hs, e):
-    # FIXME(erikj): To handle the case of presence events and the like
-    if not isinstance(e, SynapseEvent):
-        return e
-
-    # Should this strip out None's?
-    d = {k: v for k, v in e.get_dict().items()}
-    if "age_ts" in d:
-        d["age"] = int(hs.get_clock().time_msec()) - d["age_ts"]
-        del d["age_ts"]
-
-    return d
-
-
-class SynapseEvent(JsonEncodedObject):
-
-    """Base class for Synapse events. These are JSON objects which must abide
-    by a certain well-defined structure.
-    """
-
-    # Attributes that are currently assumed by the federation side:
-    # Mandatory:
-    # - event_id
-    # - room_id
-    # - type
-    # - is_state
-    #
-    # Optional:
-    # - state_key (mandatory when is_state is True)
-    # - prev_events (these can be filled out by the federation layer itself.)
-    # - prev_state
-
-    valid_keys = [
-        "event_id",
-        "type",
-        "room_id",
-        "user_id",  # sender/initiator
-        "content",  # HTTP body, JSON
-        "state_key",
-        "age_ts",
-        "prev_content",
-        "replaces_state",
-        "redacted_because",
-        "origin_server_ts",
-    ]
-
-    internal_keys = [
-        "is_state",
-        "depth",
-        "destinations",
-        "origin",
-        "outlier",
-        "redacted",
-        "prev_events",
-        "hashes",
-        "signatures",
-        "prev_state",
-        "auth_events",
-        "state_hash",
-    ]
-
-    required_keys = [
-        "event_id",
-        "room_id",
-        "content",
-    ]
-
-    outlier = False
-
-    def __init__(self, raises=True, **kwargs):
-        super(SynapseEvent, self).__init__(**kwargs)
-        # if "content" in kwargs:
-        #     self.check_json(self.content, raises=raises)
-
-    def get_content_template(self):
-        """ Retrieve the JSON template for this event as a dict.
-
-        The template must be a dict representing the JSON to match. Only
-        required keys should be present. The values of the keys in the template
-        are checked via type() to the values of the same keys in the actual
-        event JSON.
-
-        NB: If loading content via json.loads, you MUST define strings as
-        unicode.
-
-        For example:
-            Content:
-                {
-                    "name": u"bob",
-                    "age": 18,
-                    "friends": [u"mike", u"jill"]
-                }
-            Template:
-                {
-                    "name": u"string",
-                    "age": 0,
-                    "friends": [u"string"]
-                }
-            The values "string" and 0 could be anything, so long as the types
-            are the same as the content.
-        """
-        raise NotImplementedError("get_content_template not implemented.")
-
-    def get_pdu_json(self, time_now=None):
-        pdu_json = self.get_full_dict()
-        pdu_json.pop("destinations", None)
-        pdu_json.pop("outlier", None)
-        pdu_json.pop("replaces_state", None)
-        pdu_json.pop("redacted", None)
-        pdu_json.pop("prev_content", None)
-        state_hash = pdu_json.pop("state_hash", None)
-        if state_hash is not None:
-            pdu_json.setdefault("unsigned", {})["state_hash"] = state_hash
-        content = pdu_json.get("content", {})
-        content.pop("prev", None)
-        if time_now is not None and "age_ts" in pdu_json:
-            age = time_now - pdu_json["age_ts"]
-            pdu_json.setdefault("unsigned", {})["age"] = int(age)
-            del pdu_json["age_ts"]
-        user_id = pdu_json.pop("user_id")
-        pdu_json["sender"] = user_id
-        return pdu_json
-
-
-class SynapseStateEvent(SynapseEvent):
-
-    def __init__(self, **kwargs):
-        if "state_key" not in kwargs:
-            kwargs["state_key"] = ""
-        super(SynapseStateEvent, self).__init__(**kwargs)

synapse/api/events/factory.py

@@ -1,90 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.api.events.room import (
-    RoomTopicEvent, MessageEvent, RoomMemberEvent, FeedbackEvent,
-    InviteJoinEvent, RoomConfigEvent, RoomNameEvent, GenericEvent,
-    RoomPowerLevelsEvent, RoomJoinRulesEvent,
-    RoomCreateEvent,
-    RoomRedactionEvent,
-)
-
-from synapse.types import EventID
-
-from synapse.util.stringutils import random_string
-
-
-class EventFactory(object):
-
-    _event_classes = [
-        RoomTopicEvent,
-        RoomNameEvent,
-        MessageEvent,
-        RoomMemberEvent,
-        FeedbackEvent,
-        InviteJoinEvent,
-        RoomConfigEvent,
-        RoomPowerLevelsEvent,
-        RoomJoinRulesEvent,
-        RoomCreateEvent,
-        RoomRedactionEvent,
-    ]
-
-    def __init__(self, hs):
-        self._event_list = {}  # dict of TYPE to event class
-        for event_class in EventFactory._event_classes:
-            self._event_list[event_class.TYPE] = event_class
-
-        self.clock = hs.get_clock()
-        self.hs = hs
-
-        self.event_id_count = 0
-
-    def create_event_id(self):
-        i = str(self.event_id_count)
-        self.event_id_count += 1
-
-        local_part = str(int(self.clock.time())) + i + random_string(5)
-
-        e_id = EventID.create_local(local_part, self.hs)
-
-        return e_id.to_string()
-
-    def create_event(self, etype=None, **kwargs):
-        kwargs["type"] = etype
-        if "event_id" not in kwargs:
-            kwargs["event_id"] = self.create_event_id()
-            kwargs["origin"] = self.hs.hostname
-        else:
-            ev_id = self.hs.parse_eventid(kwargs["event_id"])
-            kwargs["origin"] = ev_id.domain
-
-        if "origin_server_ts" not in kwargs:
-            kwargs["origin_server_ts"] = int(self.clock.time_msec())
-
-        # The "age" key is a delta timestamp that should be converted into an
-        # absolute timestamp the minute we see it.
-        if "age" in kwargs:
-            kwargs["age_ts"] = int(self.clock.time_msec()) - int(kwargs["age"])
-            del kwargs["age"]
-        elif "age_ts" not in kwargs:
-            kwargs["age_ts"] = int(self.clock.time_msec())
-
-        if etype in self._event_list:
-            handler = self._event_list[etype]
-        else:
-            handler = GenericEvent
-
-        return handler(**kwargs)

synapse/api/events/room.py

@@ -1,170 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.api.constants import Feedback, Membership
-from synapse.api.errors import SynapseError
-from . import SynapseEvent, SynapseStateEvent
-
-
-class GenericEvent(SynapseEvent):
-    def get_content_template(self):
-        return {}
-
-
-class RoomTopicEvent(SynapseEvent):
-    TYPE = "m.room.topic"
-
-    internal_keys = SynapseEvent.internal_keys + [
-        "topic",
-    ]
-
-    def __init__(self, **kwargs):
-        kwargs["state_key"] = ""
-        if "topic" in kwargs["content"]:
-            kwargs["topic"] = kwargs["content"]["topic"]
-        super(RoomTopicEvent, self).__init__(**kwargs)
-
-    def get_content_template(self):
-        return {"topic": u"string"}
-
-
-class RoomNameEvent(SynapseEvent):
-    TYPE = "m.room.name"
-
-    internal_keys = SynapseEvent.internal_keys + [
-        "name",
-    ]
-
-    def __init__(self, **kwargs):
-        kwargs["state_key"] = ""
-        if "name" in kwargs["content"]:
-            kwargs["name"] = kwargs["content"]["name"]
-        super(RoomNameEvent, self).__init__(**kwargs)
-
-    def get_content_template(self):
-        return {"name": u"string"}
-
-
-class RoomMemberEvent(SynapseEvent):
-    TYPE = "m.room.member"
-
-    valid_keys = SynapseEvent.valid_keys + [
-        # target is the state_key
-        "membership",  # action
-    ]
-
-    def __init__(self, **kwargs):
-        if "membership" not in kwargs:
-            kwargs["membership"] = kwargs.get("content", {}).get("membership")
-        if not kwargs["membership"] in Membership.LIST:
-            raise SynapseError(400, "Bad membership value.")
-        super(RoomMemberEvent, self).__init__(**kwargs)
-
-    def get_content_template(self):
-        return {"membership": u"string"}
-
-
-class MessageEvent(SynapseEvent):
-    TYPE = "m.room.message"
-
-    valid_keys = SynapseEvent.valid_keys + [
-        "msg_id",  # unique per room + user combo
-    ]
-
-    def __init__(self, **kwargs):
-        super(MessageEvent, self).__init__(**kwargs)
-
-    def get_content_template(self):
-        return {"msgtype": u"string"}
-
-
-class FeedbackEvent(SynapseEvent):
-    TYPE = "m.room.message.feedback"
-
-    valid_keys = SynapseEvent.valid_keys
-
-    def __init__(self, **kwargs):
-        super(FeedbackEvent, self).__init__(**kwargs)
-        if not kwargs["content"]["type"] in Feedback.LIST:
-            raise SynapseError(400, "Bad feedback value.")
-
-    def get_content_template(self):
-        return {
-            "type": u"string",
-            "target_event_id": u"string"
-        }
-
-
-class InviteJoinEvent(SynapseEvent):
-    TYPE = "m.room.invite_join"
-
-    valid_keys = SynapseEvent.valid_keys + [
-        # target_user_id is the state_key
-        "target_host",
-    ]
-
-    def __init__(self, **kwargs):
-        super(InviteJoinEvent, self).__init__(**kwargs)
-
-    def get_content_template(self):
-        return {}
-
-
-class RoomConfigEvent(SynapseEvent):
-    TYPE = "m.room.config"
-
-    def __init__(self, **kwargs):
-        kwargs["state_key"] = ""
-        super(RoomConfigEvent, self).__init__(**kwargs)
-
-    def get_content_template(self):
-        return {}
-
-
-class RoomCreateEvent(SynapseStateEvent):
-    TYPE = "m.room.create"
-
-    def get_content_template(self):
-        return {}
-
-
-class RoomJoinRulesEvent(SynapseStateEvent):
-    TYPE = "m.room.join_rules"
-
-    def get_content_template(self):
-        return {}
-
-
-class RoomPowerLevelsEvent(SynapseStateEvent):
-    TYPE = "m.room.power_levels"
-
-    def get_content_template(self):
-        return {}
-
-
-class RoomAliasesEvent(SynapseStateEvent):
-    TYPE = "m.room.aliases"
-
-    def get_content_template(self):
-        return {}
-
-
-class RoomRedactionEvent(SynapseEvent):
-    TYPE = "m.room.redaction"
-
-    valid_keys = SynapseEvent.valid_keys + ["redacts"]
-
-    def get_content_template(self):
-        return {}

synapse/api/events/utils.py

@@ -1,85 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from .room import (
-    RoomMemberEvent, RoomJoinRulesEvent, RoomPowerLevelsEvent,
-    RoomAliasesEvent, RoomCreateEvent,
-)
-
-
-def prune_event(event):
-    """ Returns a pruned version of the given event, which removes all keys we
-    don't know about or think could potentially be dodgy.
-
-    This is used when we "redact" an event. We want to remove all fields that
-    the user has specified, but we do want to keep necessary information like
-    type, state_key etc.
-    """
-    event_type = event.type
-
-    allowed_keys = [
-        "event_id",
-        "user_id",
-        "room_id",
-        "hashes",
-        "signatures",
-        "content",
-        "type",
-        "state_key",
-        "depth",
-        "prev_events",
-        "prev_state",
-        "auth_events",
-        "origin",
-        "origin_server_ts",
-    ]
-
-    new_content = {}
-
-    def add_fields(*fields):
-        for field in fields:
-            if field in event.content:
-                new_content[field] = event.content[field]
-
-    if event_type == RoomMemberEvent.TYPE:
-        add_fields("membership")
-    elif event_type == RoomCreateEvent.TYPE:
-        add_fields("creator")
-    elif event_type == RoomJoinRulesEvent.TYPE:
-        add_fields("join_rule")
-    elif event_type == RoomPowerLevelsEvent.TYPE:
-        add_fields(
-            "users",
-            "users_default",
-            "events",
-            "events_default",
-            "events_default",
-            "state_default",
-            "ban",
-            "kick",
-            "redact",
-        )
-    elif event_type == RoomAliasesEvent.TYPE:
-        add_fields("aliases")
-
-    allowed_fields = {
-        k: v
-        for k, v in event.get_full_dict().items()
-        if k in allowed_keys
-    }
-
-    allowed_fields["content"] = new_content
-
-    return type(event)(**allowed_fields)

synapse/api/events/validator.py

@@ -1,87 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.api.errors import SynapseError, Codes
-
-
-class EventValidator(object):
-    def __init__(self, hs):
-        pass
-
-    def validate(self, event):
-        """Checks the given JSON content abides by the rules of the template.
-
-        Args:
-            content : A JSON object to check.
-            raises: True to raise a SynapseError if the check fails.
-        Returns:
-            True if the content passes the template. Returns False if the check
-            fails and raises=False.
-        Raises:
-            SynapseError if the check fails and raises=True.
-        """
-        # recursively call to inspect each layer
-        err_msg = self._check_json_template(
-            event.content,
-            event.get_content_template()
-        )
-        if err_msg:
-            raise SynapseError(400, err_msg, Codes.BAD_JSON)
-        else:
-            return True
-
-    def _check_json_template(self, content, template):
-        """Check content and template matches.
-
-        If the template is a dict, each key in the dict will be validated with
-        the content, else it will just compare the types of content and
-        template. This basic type check is required because this function will
-        be recursively called and could be called with just strs or ints.
-
-        Args:
-            content: The content to validate.
-            template: The validation template.
-        Returns:
-            str: An error message if the validation fails, else None.
-        """
-        if type(content) != type(template):
-            return "Mismatched types: %s" % template
-
-        if type(template) == dict:
-            for key in template:
-                if key not in content:
-                    return "Missing %s key" % key
-
-                if type(content[key]) != type(template[key]):
-                    return "Key %s is of the wrong type (got %s, want %s)" % (
-                        key, type(content[key]), type(template[key]))
-
-                if type(content[key]) == dict:
-                    # we must go deeper
-                    msg = self._check_json_template(
-                        content[key],
-                        template[key]
-                    )
-                    if msg:
-                        return msg
-                elif type(content[key]) == list:
-                    # make sure each item type in content matches the template
-                    for entry in content[key]:
-                        msg = self._check_json_template(
-                            entry,
-                            template[key][0]
-                        )
-                        if msg:
-                            return msg

synapse/api/ratelimiting.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/api/urls.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,3 +20,4 @@ FEDERATION_PREFIX = "/_matrix/federation/v1"
 WEB_CLIENT_PREFIX = "/_matrix/client"
 CONTENT_REPO_PREFIX = "/_matrix/content"
 SERVER_KEY_PREFIX = "/_matrix/key/v1"
+MEDIA_PREFIX = "/_matrix/media/v1"

synapse/app/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/app/homeserver.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,22 +14,25 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.storage import prepare_database
+from synapse.storage import prepare_database, UpgradeDatabaseException
 
 from synapse.server import HomeServer
 
+from synapse.python_dependencies import check_requirements
+
 from twisted.internet import reactor
 from twisted.enterprise import adbapi
 from twisted.web.resource import Resource
 from twisted.web.static import File
 from twisted.web.server import Site
 from synapse.http.server import JsonResource, RootRedirect
-from synapse.http.content_repository import ContentRepoResource
+from synapse.media.v0.content_repository import ContentRepoResource
+from synapse.media.v1.media_repository import MediaRepositoryResource
 from synapse.http.server_key_resource import LocalKey
 from synapse.http.matrixfederationclient import MatrixFederationHttpClient
 from synapse.api.urls import (
     CLIENT_PREFIX, FEDERATION_PREFIX, WEB_CLIENT_PREFIX, CONTENT_REPO_PREFIX,
-    SERVER_KEY_PREFIX,
+    SERVER_KEY_PREFIX, MEDIA_PREFIX
 )
 from synapse.config.homeserver import HomeServerConfig
 from synapse.crypto import context_factory
@@ -38,6 +41,8 @@ from synapse.util.logcontext import LoggingContext
 from daemonize import Daemonize
 import twisted.manhole.telnet
 
+import synapse
+
 import logging
 import os
 import re
@@ -69,6 +74,9 @@ class SynapseHomeServer(HomeServer):
             self, self.upload_dir, self.auth, self.content_addr
         )
 
+    def build_resource_for_media_repository(self):
+        return MediaRepositoryResource(self)
+
     def build_resource_for_server_key(self):
         return LocalKey(self)
 
@@ -99,6 +107,7 @@ class SynapseHomeServer(HomeServer):
             (FEDERATION_PREFIX, self.get_resource_for_federation()),
             (CONTENT_REPO_PREFIX, self.get_resource_for_content_repo()),
             (SERVER_KEY_PREFIX, self.get_resource_for_server_key()),
+            (MEDIA_PREFIX, self.get_resource_for_media_repository()),
         ]
         if web_client:
             logger.info("Adding the web client.")
@@ -193,7 +202,10 @@ def setup():
 
     config.setup_logging()
 
+    check_requirements()
+
     logger.info("Server hostname: %s", config.server_name)
+    logger.info("Server version: %s", synapse.__version__)
 
     if re.search(":[0-9]+$", config.server_name):
         domain_with_port = config.server_name
@@ -223,12 +235,26 @@ def setup():
 
     logger.info("Preparing database: %s...", db_name)
 
-    with sqlite3.connect(db_name) as db_conn:
-        prepare_database(db_conn)
+    try:
+        with sqlite3.connect(db_name) as db_conn:
+            prepare_database(db_conn)
+    except UpgradeDatabaseException:
+        sys.stderr.write(
+            "\nFailed to upgrade database.\n"
+            "Have you checked for version specific instructions in"
+            " UPGRADES.rst?\n"
+        )
+        sys.exit(1)
 
     logger.info("Database prepared in %s.", db_name)
 
-    hs.get_db_pool()
+    db_pool = hs.get_db_pool()
+
+    if db_name == ":memory:":
+        # Memory databases will need to be setup each time they are opened.
+        reactor.callWhenRunning(
+            db_pool.runWithConnection, prepare_database
+        )
 
     if config.manhole:
         f = twisted.manhole.telnet.ShellFactory()
@@ -265,6 +291,7 @@ def run():
 
 def main():
     with LoggingContext("main"):
+        check_requirements()
         setup()
 
 

synapse/app/synctl.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/_base.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -44,18 +44,32 @@ class Config(object):
             )
         if not os.path.exists(file_path):
             raise ConfigError(
-                "File % config for %s doesn't exist."
+                "File %s config for %s doesn't exist."
                 " Try running again with --generate-config"
-                % (config_name,)
+                % (file_path, config_name,)
             )
         return cls.abspath(file_path)
 
+    @staticmethod
+    def ensure_directory(dir_path):
+        if not os.path.exists(dir_path):
+            os.makedirs(dir_path)
+        if not os.path.isdir(dir_path):
+            raise ConfigError(
+                "%s is not a directory" % (dir_path,)
+            )
+        return dir_path
+
     @classmethod
     def read_file(cls, file_path, config_name):
         cls.check_file(file_path, config_name)
         with open(file_path) as file_stream:
             return file_stream.read()
 
+    @staticmethod
+    def default_path(name):
+        return os.path.abspath(os.path.join(os.path.curdir, name))
+
     @staticmethod
     def read_config_file(file_path):
         with open(file_path) as file_stream:

synapse/config/captcha.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/database.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,7 +20,10 @@ import os
 class DatabaseConfig(Config):
     def __init__(self, args):
         super(DatabaseConfig, self).__init__(args)
-        self.database_path = self.abspath(args.database_path)
+        if args.database_path == ":memory:":
+            self.database_path = ":memory:"
+        else:
+            self.database_path = self.abspath(args.database_path)
 
     @classmethod
     def add_arguments(cls, parser):

synapse/config/email.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/homeserver.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/logger.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -36,7 +36,7 @@ class LoggingConfig(Config):
             help="The verbosity level."
         )
         logging_group.add_argument(
-            '-f', '--log-file', dest="log_file", default=None,
+            '-f', '--log-file', dest="log_file", default="homeserver.log",
             help="File to log to."
         )
         logging_group.add_argument(
@@ -66,7 +66,10 @@ class LoggingConfig(Config):
 
             formatter = logging.Formatter(log_format)
             if self.log_file:
-                handler = logging.FileHandler(self.log_file)
+                # TODO: Customisable file size / backup count
+                handler = logging.handlers.RotatingFileHandler(
+                    self.log_file, maxBytes=(1000 * 1000 * 100), backupCount=3
+                )
             else:
                 handler = logging.StreamHandler()
             handler.setFormatter(formatter)

synapse/config/ratelimiting.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/repository.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014, 2015 matrix.org
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,6 +20,8 @@ class ContentRepositoryConfig(Config):
     def __init__(self, args):
         super(ContentRepositoryConfig, self).__init__(args)
         self.max_upload_size = self.parse_size(args.max_upload_size)
+        self.max_image_pixels = self.parse_size(args.max_image_pixels)
+        self.media_store_path = self.ensure_directory(args.media_store_path)
 
     def parse_size(self, string):
         sizes = {"K": 1024, "M": 1024 * 1024}
@@ -35,5 +37,12 @@ class ContentRepositoryConfig(Config):
         super(ContentRepositoryConfig, cls).add_arguments(parser)
         db_group = parser.add_argument_group("content_repository")
         db_group.add_argument(
-            "--max-upload-size", default="1M"
+            "--max-upload-size", default="10M"
+        )
+        db_group.add_argument(
+            "--media-store-path", default=cls.default_path("media_store")
+        )
+        db_group.add_argument(
+            "--max-image-pixels", default="32M",
+            help="Maximum number of pixels that will be thumbnailed"
         )

synapse/config/server.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -47,8 +47,12 @@ class ServerConfig(Config):
     def add_arguments(cls, parser):
         super(ServerConfig, cls).add_arguments(parser)
         server_group = parser.add_argument_group("server")
-        server_group.add_argument("-H", "--server-name", default="localhost",
-                                  help="The name of the server")
+        server_group.add_argument(
+            "-H", "--server-name", default="localhost",
+            help="The domain name of the server, with optional explicit port. "
+                 "This is used by remote servers to connect to this server, "
+                 "e.g. matrix.org, localhost:8080, etc."
+        )
         server_group.add_argument("--signing-key-path",
                                   help="The signing key to sign messages with")
         server_group.add_argument("-p", "--bind-port", metavar="PORT",

synapse/config/tls.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/voip.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/context_factory.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/event_signing.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,7 @@
 # limitations under the License.
 
 
-from synapse.api.events.utils import prune_event
+from synapse.events.utils import prune_event
 from syutil.jsonutil import encode_canonical_json
 from syutil.base64util import encode_base64, decode_base64
 from syutil.crypto.jsonsign import sign_json
@@ -29,17 +29,17 @@ logger = logging.getLogger(__name__)
 
 def check_event_content_hash(event, hash_algorithm=hashlib.sha256):
     """Check whether the hash for this PDU matches the contents"""
-    computed_hash = _compute_content_hash(event, hash_algorithm)
-    logger.debug("Expecting hash: %s", encode_base64(computed_hash.digest()))
-    if computed_hash.name not in event.hashes:
+    name, expected_hash = compute_content_hash(event, hash_algorithm)
+    logger.debug("Expecting hash: %s", encode_base64(expected_hash))
+    if name not in event.hashes:
         raise SynapseError(
             400,
             "Algorithm %s not in hashes %s" % (
-                computed_hash.name, list(event.hashes),
+                name, list(event.hashes),
             ),
             Codes.UNAUTHORIZED,
         )
-    message_hash_base64 = event.hashes[computed_hash.name]
+    message_hash_base64 = event.hashes[name]
     try:
         message_hash_bytes = decode_base64(message_hash_base64)
     except:
@@ -48,10 +48,10 @@ def check_event_content_hash(event, hash_algorithm=hashlib.sha256):
             "Invalid base64: %s" % (message_hash_base64,),
             Codes.UNAUTHORIZED,
         )
-    return message_hash_bytes == computed_hash.digest()
+    return message_hash_bytes == expected_hash
 
 
-def _compute_content_hash(event, hash_algorithm):
+def compute_content_hash(event, hash_algorithm):
     event_json = event.get_pdu_json()
     event_json.pop("age_ts", None)
     event_json.pop("unsigned", None)
@@ -59,8 +59,11 @@ def _compute_content_hash(event, hash_algorithm):
     event_json.pop("hashes", None)
     event_json.pop("outlier", None)
     event_json.pop("destinations", None)
+
     event_json_bytes = encode_canonical_json(event_json)
-    return hash_algorithm(event_json_bytes)
+
+    hashed = hash_algorithm(event_json_bytes)
+    return (hashed.name, hashed.digest())
 
 
 def compute_event_reference_hash(event, hash_algorithm=hashlib.sha256):
@@ -79,27 +82,28 @@ def compute_event_signature(event, signature_name, signing_key):
     redact_json = tmp_event.get_pdu_json()
     redact_json.pop("age_ts", None)
     redact_json.pop("unsigned", None)
-    logger.debug("Signing event: %s", redact_json)
+    logger.debug("Signing event: %s", encode_canonical_json(redact_json))
     redact_json = sign_json(redact_json, signature_name, signing_key)
+    logger.debug("Signed event: %s", encode_canonical_json(redact_json))
     return redact_json["signatures"]
 
 
 def add_hashes_and_signatures(event, signature_name, signing_key,
                               hash_algorithm=hashlib.sha256):
-    if hasattr(event, "old_state_events"):
-        state_json_bytes = encode_canonical_json(
-            [e.event_id for e in event.old_state_events.values()]
-        )
-        hashed = hash_algorithm(state_json_bytes)
-        event.state_hash = {
-            hashed.name: encode_base64(hashed.digest())
-        }
+    # if hasattr(event, "old_state_events"):
+    #     state_json_bytes = encode_canonical_json(
+    #         [e.event_id for e in event.old_state_events.values()]
+    #     )
+    #     hashed = hash_algorithm(state_json_bytes)
+    #     event.state_hash = {
+    #         hashed.name: encode_base64(hashed.digest())
+    #     }
 
-    hashed = _compute_content_hash(event, hash_algorithm=hash_algorithm)
+    name, digest = compute_content_hash(event, hash_algorithm=hash_algorithm)
 
     if not hasattr(event, "hashes"):
         event.hashes = {}
-    event.hashes[hashed.name] = encode_base64(hashed.digest())
+    event.hashes[name] = encode_base64(digest)
 
     event.signatures = compute_event_signature(
         event,

synapse/crypto/keyclient.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/keyring.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/events/__init__.py

@@ -0,0 +1,153 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014, 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.util.frozenutils import freeze, unfreeze
+
+
+class _EventInternalMetadata(object):
+    def __init__(self, internal_metadata_dict):
+        self.__dict__ = internal_metadata_dict
+
+    def get_dict(self):
+        return dict(self.__dict__)
+
+    def is_outlier(self):
+        return hasattr(self, "outlier") and self.outlier
+
+
+def _event_dict_property(key):
+    def getter(self):
+        return self._event_dict[key]
+
+    def setter(self, v):
+        self._event_dict[key] = v
+
+    def delete(self):
+        del self._event_dict[key]
+
+    return property(
+        getter,
+        setter,
+        delete,
+    )
+
+
+class EventBase(object):
+    def __init__(self, event_dict, signatures={}, unsigned={},
+                 internal_metadata_dict={}):
+        self.signatures = signatures
+        self.unsigned = unsigned
+
+        self._event_dict = event_dict
+
+        self.internal_metadata = _EventInternalMetadata(
+            internal_metadata_dict
+        )
+
+    auth_events = _event_dict_property("auth_events")
+    depth = _event_dict_property("depth")
+    content = _event_dict_property("content")
+    event_id = _event_dict_property("event_id")
+    hashes = _event_dict_property("hashes")
+    origin = _event_dict_property("origin")
+    origin_server_ts = _event_dict_property("origin_server_ts")
+    prev_events = _event_dict_property("prev_events")
+    prev_state = _event_dict_property("prev_state")
+    redacts = _event_dict_property("redacts")
+    room_id = _event_dict_property("room_id")
+    sender = _event_dict_property("sender")
+    state_key = _event_dict_property("state_key")
+    type = _event_dict_property("type")
+    user_id = _event_dict_property("sender")
+
+    @property
+    def membership(self):
+        return self.content["membership"]
+
+    def is_state(self):
+        return hasattr(self, "state_key")
+
+    def get_dict(self):
+        d = dict(self._event_dict)
+        d.update({
+            "signatures": self.signatures,
+            "unsigned": self.unsigned,
+        })
+
+        return d
+
+    def get(self, key, default):
+        return self._event_dict.get(key, default)
+
+    def get_internal_metadata_dict(self):
+        return self.internal_metadata.get_dict()
+
+    def get_pdu_json(self, time_now=None):
+        pdu_json = self.get_dict()
+
+        if time_now is not None and "age_ts" in pdu_json["unsigned"]:
+            age = time_now - pdu_json["unsigned"]["age_ts"]
+            pdu_json.setdefault("unsigned", {})["age"] = int(age)
+            del pdu_json["unsigned"]["age_ts"]
+
+        return pdu_json
+
+    def __set__(self, instance, value):
+        raise AttributeError("Unrecognized attribute %s" % (instance,))
+
+
+class FrozenEvent(EventBase):
+    def __init__(self, event_dict, internal_metadata_dict={}):
+        event_dict = dict(event_dict)
+
+        # Signatures is a dict of dicts, and this is faster than doing a
+        # copy.deepcopy
+        signatures = {
+            name: {sig_id: sig for sig_id, sig in sigs.items()}
+            for name, sigs in event_dict.pop("signatures", {}).items()
+        }
+
+        unsigned = dict(event_dict.pop("unsigned", {}))
+
+        frozen_dict = freeze(event_dict)
+
+        super(FrozenEvent, self).__init__(
+            frozen_dict,
+            signatures=signatures,
+            unsigned=unsigned,
+            internal_metadata_dict=internal_metadata_dict,
+        )
+
+    @staticmethod
+    def from_event(event):
+        e = FrozenEvent(
+            event.get_pdu_json()
+        )
+
+        e.internal_metadata = event.internal_metadata
+
+        return e
+
+    def get_dict(self):
+        # We need to unfreeze what we return
+        return unfreeze(super(FrozenEvent, self).get_dict())
+
+    def __str__(self):
+        return self.__repr__()
+
+    def __repr__(self):
+        return "<FrozenEvent event_id='%s', type='%s', state_key='%s'>" % (
+            self.event_id, self.type, self.get("state_key", None),
+        )

synapse/events/builder.py

@@ -0,0 +1,71 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014, 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from . import EventBase, FrozenEvent
+
+from synapse.types import EventID
+
+from synapse.util.stringutils import random_string
+
+import copy
+
+
+class EventBuilder(EventBase):
+    def __init__(self, key_values={}):
+        signatures = copy.deepcopy(key_values.pop("signatures", {}))
+        unsigned = copy.deepcopy(key_values.pop("unsigned", {}))
+
+        super(EventBuilder, self).__init__(
+            key_values,
+            signatures=signatures,
+            unsigned=unsigned
+        )
+
+    def build(self):
+        return FrozenEvent.from_event(self)
+
+
+class EventBuilderFactory(object):
+    def __init__(self, clock, hostname):
+        self.clock = clock
+        self.hostname = hostname
+
+        self.event_id_count = 0
+
+    def create_event_id(self):
+        i = str(self.event_id_count)
+        self.event_id_count += 1
+
+        local_part = str(int(self.clock.time())) + i + random_string(5)
+
+        e_id = EventID.create(local_part, self.hostname)
+
+        return e_id.to_string()
+
+    def new(self, key_values={}):
+        key_values["event_id"] = self.create_event_id()
+
+        time_now = int(self.clock.time_msec())
+
+        key_values.setdefault("origin", self.hostname)
+        key_values.setdefault("origin_server_ts", time_now)
+
+        key_values.setdefault("unsigned", {})
+        age = key_values["unsigned"].pop("age", 0)
+        key_values["unsigned"].setdefault("age_ts", time_now - age)
+
+        key_values["signatures"] = {}
+
+        return EventBuilder(key_values=key_values,)

synapse/events/snapshot.py

@@ -0,0 +1,22 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014, 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+class EventContext(object):
+
+    def __init__(self, current_state=None, auth_events=None):
+        self.current_state = current_state
+        self.auth_events = auth_events
+        self.state_group = None

synapse/events/utils.py

@@ -0,0 +1,141 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014, 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.api.constants import EventTypes
+from . import EventBase
+
+
+def prune_event(event):
+    """ Returns a pruned version of the given event, which removes all keys we
+    don't know about or think could potentially be dodgy.
+
+    This is used when we "redact" an event. We want to remove all fields that
+    the user has specified, but we do want to keep necessary information like
+    type, state_key etc.
+    """
+    event_type = event.type
+
+    allowed_keys = [
+        "event_id",
+        "sender",
+        "room_id",
+        "hashes",
+        "signatures",
+        "content",
+        "type",
+        "state_key",
+        "depth",
+        "prev_events",
+        "prev_state",
+        "auth_events",
+        "origin",
+        "origin_server_ts",
+        "membership",
+    ]
+
+    new_content = {}
+
+    def add_fields(*fields):
+        for field in fields:
+            if field in event.content:
+                new_content[field] = event.content[field]
+
+    if event_type == EventTypes.Member:
+        add_fields("membership")
+    elif event_type == EventTypes.Create:
+        add_fields("creator")
+    elif event_type == EventTypes.JoinRules:
+        add_fields("join_rule")
+    elif event_type == EventTypes.PowerLevels:
+        add_fields(
+            "users",
+            "users_default",
+            "events",
+            "events_default",
+            "events_default",
+            "state_default",
+            "ban",
+            "kick",
+            "redact",
+        )
+    elif event_type == EventTypes.Aliases:
+        add_fields("aliases")
+
+    allowed_fields = {
+        k: v
+        for k, v in event.get_dict().items()
+        if k in allowed_keys
+    }
+
+    allowed_fields["content"] = new_content
+
+    allowed_fields["unsigned"] = {}
+
+    if "age_ts" in event.unsigned:
+        allowed_fields["unsigned"]["age_ts"] = event.unsigned["age_ts"]
+
+    return type(event)(allowed_fields)
+
+
+def serialize_event(hs, e, client_event=True):
+    # FIXME(erikj): To handle the case of presence events and the like
+    if not isinstance(e, EventBase):
+        return e
+
+    # Should this strip out None's?
+    d = {k: v for k, v in e.get_dict().items()}
+
+    if not client_event:
+        # set the age and keep all other keys
+        if "age_ts" in d["unsigned"]:
+            now = int(hs.get_clock().time_msec())
+            d["unsigned"]["age"] = now - d["unsigned"]["age_ts"]
+        return d
+
+    if "age_ts" in d["unsigned"]:
+        now = int(hs.get_clock().time_msec())
+        d["age"] = now - d["unsigned"]["age_ts"]
+        del d["unsigned"]["age_ts"]
+
+    d["user_id"] = d.pop("sender", None)
+
+    if "redacted_because" in e.unsigned:
+        d["redacted_because"] = serialize_event(
+            hs, e.unsigned["redacted_because"]
+        )
+
+        del d["unsigned"]["redacted_because"]
+
+    if "redacted_by" in e.unsigned:
+        d["redacted_by"] = e.unsigned["redacted_by"]
+        del d["unsigned"]["redacted_by"]
+
+    if "replaces_state" in e.unsigned:
+        d["replaces_state"] = e.unsigned["replaces_state"]
+        del d["unsigned"]["replaces_state"]
+
+    if "prev_content" in e.unsigned:
+        d["prev_content"] = e.unsigned["prev_content"]
+        del d["unsigned"]["prev_content"]
+
+    del d["auth_events"]
+    del d["prev_events"]
+    del d["hashes"]
+    del d["signatures"]
+    d.pop("depth", None)
+    d.pop("unsigned", None)
+    d.pop("origin", None)
+
+    return d

synapse/events/validator.py

@@ -0,0 +1,92 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014, 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.types import EventID, RoomID, UserID
+from synapse.api.errors import SynapseError
+from synapse.api.constants import EventTypes, Membership
+
+
+class EventValidator(object):
+
+    def validate(self, event):
+        EventID.from_string(event.event_id)
+        RoomID.from_string(event.room_id)
+
+        required = [
+            # "auth_events",
+            "content",
+            # "hashes",
+            "origin",
+            # "prev_events",
+            "sender",
+            "type",
+        ]
+
+        for k in required:
+            if not hasattr(event, k):
+                raise SynapseError(400, "Event does not have key %s" % (k,))
+
+        # Check that the following keys have string values
+        strings = [
+            "origin",
+            "sender",
+            "type",
+        ]
+
+        if hasattr(event, "state_key"):
+            strings.append("state_key")
+
+        for s in strings:
+            if not isinstance(getattr(event, s), basestring):
+                raise SynapseError(400, "Not '%s' a string type" % (s,))
+
+        if event.type == EventTypes.Member:
+            if "membership" not in event.content:
+                raise SynapseError(400, "Content has not membership key")
+
+            if event.content["membership"] not in Membership.LIST:
+                raise SynapseError(400, "Invalid membership key")
+
+        # Check that the following keys have dictionary values
+        # TODO
+
+        # Check that the following keys have the correct format for DAGs
+        # TODO
+
+    def validate_new(self, event):
+        self.validate(event)
+
+        UserID.from_string(event.sender)
+
+        if event.type == EventTypes.Message:
+            strings = [
+                "body",
+                "msgtype",
+            ]
+
+            self._ensure_strings(event.content, strings)
+
+        elif event.type == EventTypes.Topic:
+            self._ensure_strings(event.content, ["topic"])
+
+        elif event.type == EventTypes.Name:
+            self._ensure_strings(event.content, ["name"])
+
+    def _ensure_strings(self, d, keys):
+        for s in keys:
+            if s not in d:
+                raise SynapseError(400, "'%s' not in content" % (s,))
+            if not isinstance(d[s], basestring):
+                raise SynapseError(400, "Not '%s' a string type" % (s,))

synapse/federation/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/persistence.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/replication.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -25,6 +25,7 @@ from .persistence import TransactionActions
 
 from synapse.util.logutils import log_function
 from synapse.util.logcontext import PreserveLoggingContext
+from synapse.events import FrozenEvent
 
 import logging
 
@@ -73,7 +74,7 @@ class ReplicationLayer(object):
 
         self._clock = hs.get_clock()
 
-        self.event_factory = hs.get_event_factory()
+        self.event_builder_factory = hs.get_event_builder_factory()
 
     def set_handler(self, handler):
         """Sets the handler that the replication layer will use to communicate
@@ -112,7 +113,7 @@ class ReplicationLayer(object):
         self.query_handlers[query_type] = handler
 
     @log_function
-    def send_pdu(self, pdu):
+    def send_pdu(self, pdu, destinations):
         """Informs the replication layer about a new PDU generated within the
         home server that should be transmitted to others.
 
@@ -131,7 +132,7 @@ class ReplicationLayer(object):
         logger.debug("[%s] transaction_layer.enqueue_pdu... ", pdu.event_id)
 
         # TODO, add errback, etc.
-        self._transaction_queue.enqueue_pdu(pdu, order)
+        self._transaction_queue.enqueue_pdu(pdu, destinations, order)
 
         logger.debug(
             "[%s] transaction_layer.enqueue_pdu... done",
@@ -255,37 +256,39 @@ class ReplicationLayer(object):
 
     @defer.inlineCallbacks
     @log_function
-    def get_state_for_context(self, destination, context, event_id=None):
-        """Requests all of the `current` state PDUs for a given context from
+    def get_state_for_room(self, destination, room_id, event_id):
+        """Requests all of the `current` state PDUs for a given room from
         a remote home server.
 
         Args:
             destination (str): The remote homeserver to query for the state.
-            context (str): The context we're interested in.
+            room_id (str): The id of the room we're interested in.
+            event_id (str): The id of the event we want the state at.
 
         Returns:
             Deferred: Results in a list of PDUs.
         """
 
-        transaction_data = yield self.transport_layer.get_context_state(
-            destination,
-            context,
-            event_id=event_id,
+        result = yield self.transport_layer.get_room_state(
+            destination, room_id, event_id=event_id,
         )
 
-        transaction = Transaction(**transaction_data)
         pdus = [
-            self.event_from_pdu_json(p, outlier=True)
-            for p in transaction.pdus
+            self.event_from_pdu_json(p, outlier=True) for p in result["pdus"]
         ]
 
-        defer.returnValue(pdus)
+        auth_chain = [
+            self.event_from_pdu_json(p, outlier=True)
+            for p in result.get("auth_chain", [])
+        ]
+
+        defer.returnValue((pdus, auth_chain))
 
     @defer.inlineCallbacks
     @log_function
-    def get_event_auth(self, destination, context, event_id):
+    def get_event_auth(self, destination, room_id, event_id):
         res = yield self.transport_layer.get_event_auth(
-            destination, context, event_id,
+            destination, room_id, event_id,
         )
 
         auth_chain = [
@@ -299,9 +302,9 @@ class ReplicationLayer(object):
 
     @defer.inlineCallbacks
     @log_function
-    def on_backfill_request(self, origin, context, versions, limit):
+    def on_backfill_request(self, origin, room_id, versions, limit):
         pdus = yield self.handler.on_backfill_request(
-            origin, context, versions, limit
+            origin, room_id, versions, limit
         )
 
         defer.returnValue((200, self._transaction_from_pdus(pdus).get_dict()))
@@ -334,7 +337,7 @@ class ReplicationLayer(object):
             defer.returnValue(response)
             return
 
-        logger.debug("[%s] Transacition is new", transaction.transaction_id)
+        logger.debug("[%s] Transaction is new", transaction.transaction_id)
 
         with PreserveLoggingContext():
             dl = []
@@ -375,17 +378,21 @@ class ReplicationLayer(object):
 
     @defer.inlineCallbacks
     @log_function
-    def on_context_state_request(self, origin, context, event_id):
+    def on_context_state_request(self, origin, room_id, event_id):
         if event_id:
             pdus = yield self.handler.get_state_for_pdu(
-                origin,
-                context,
-                event_id,
+                origin, room_id, event_id,
+            )
+            auth_chain = yield self.store.get_auth_chain(
+                [pdu.event_id for pdu in pdus]
             )
         else:
             raise NotImplementedError("Specify an event")
 
-        defer.returnValue((200, self._transaction_from_pdus(pdus).get_dict()))
+        defer.returnValue((200, {
+            "pdus": [pdu.get_pdu_json() for pdu in pdus],
+            "auth_chain": [pdu.get_pdu_json() for pdu in auth_chain],
+        }))
 
     @defer.inlineCallbacks
     @log_function
@@ -402,7 +409,7 @@ class ReplicationLayer(object):
     @defer.inlineCallbacks
     @log_function
     def on_pull_request(self, origin, versions):
-        raise NotImplementedError("Pull transacions not implemented")
+        raise NotImplementedError("Pull transactions not implemented")
 
     @defer.inlineCallbacks
     def on_query_request(self, query_type, args):
@@ -411,34 +418,27 @@ class ReplicationLayer(object):
             defer.returnValue((200, response))
         else:
             defer.returnValue(
-                (404, "No handler for Query type '%s'" % (query_type, ))
+                (404, "No handler for Query type '%s'" % (query_type,))
             )
 
     @defer.inlineCallbacks
-    def on_make_join_request(self, context, user_id):
-        pdu = yield self.handler.on_make_join_request(context, user_id)
+    def on_make_join_request(self, room_id, user_id):
+        pdu = yield self.handler.on_make_join_request(room_id, user_id)
         time_now = self._clock.time_msec()
-        defer.returnValue({
-            "event": pdu.get_pdu_json(time_now),
-        })
+        defer.returnValue({"event": pdu.get_pdu_json(time_now)})
 
     @defer.inlineCallbacks
     def on_invite_request(self, origin, content):
         pdu = self.event_from_pdu_json(content)
         ret_pdu = yield self.handler.on_invite_request(origin, pdu)
         time_now = self._clock.time_msec()
-        defer.returnValue(
-            (
-                200,
-                {
-                    "event": ret_pdu.get_pdu_json(time_now),
-                }
-            )
-        )
+        defer.returnValue((200, {"event": ret_pdu.get_pdu_json(time_now)}))
 
     @defer.inlineCallbacks
     def on_send_join_request(self, origin, content):
+        logger.debug("on_send_join_request: content: %s", content)
         pdu = self.event_from_pdu_json(content)
+        logger.debug("on_send_join_request: pdu sigs: %s", pdu.signatures)
         res_pdus = yield self.handler.on_send_join_request(origin, pdu)
         time_now = self._clock.time_msec()
         defer.returnValue((200, {
@@ -449,26 +449,17 @@ class ReplicationLayer(object):
         }))
 
     @defer.inlineCallbacks
-    def on_event_auth(self, origin, context, event_id):
+    def on_event_auth(self, origin, room_id, event_id):
         time_now = self._clock.time_msec()
         auth_pdus = yield self.handler.on_event_auth(event_id)
-        defer.returnValue(
-            (
-                200,
-                {
-                    "auth_chain": [
-                        a.get_pdu_json(time_now) for a in auth_pdus
-                    ],
-                }
-            )
-        )
+        defer.returnValue((200, {
+            "auth_chain": [a.get_pdu_json(time_now) for a in auth_pdus],
+        }))
 
     @defer.inlineCallbacks
-    def make_join(self, destination, context, user_id):
+    def make_join(self, destination, room_id, user_id):
         ret = yield self.transport_layer.make_join(
-            destination=destination,
-            context=context,
-            user_id=user_id,
+            destination, room_id, user_id
         )
 
         pdu_dict = ret["event"]
@@ -481,10 +472,10 @@ class ReplicationLayer(object):
     def send_join(self, destination, pdu):
         time_now = self._clock.time_msec()
         _, content = yield self.transport_layer.send_join(
-            destination,
-            pdu.room_id,
-            pdu.event_id,
-            pdu.get_pdu_json(time_now),
+            destination=destination,
+            room_id=pdu.room_id,
+            event_id=pdu.event_id,
+            content=pdu.get_pdu_json(time_now),
         )
 
         logger.debug("Got content: %s", content)
@@ -494,9 +485,6 @@ class ReplicationLayer(object):
             for p in content.get("state", [])
         ]
 
-        # FIXME: We probably want to do something with the auth_chain given
-        # to us
-
         auth_chain = [
             self.event_from_pdu_json(p, outlier=True)
             for p in content.get("auth_chain", [])
@@ -510,11 +498,11 @@ class ReplicationLayer(object):
         })
 
     @defer.inlineCallbacks
-    def send_invite(self, destination, context, event_id, pdu):
+    def send_invite(self, destination, room_id, event_id, pdu):
         time_now = self._clock.time_msec()
         code, content = yield self.transport_layer.send_invite(
             destination=destination,
-            context=context,
+            room_id=room_id,
             event_id=event_id,
             content=pdu.get_pdu_json(time_now),
         )
@@ -557,13 +545,21 @@ class ReplicationLayer(object):
             origin, pdu.event_id, do_auth=False
         )
 
-        if existing and (not existing.outlier or pdu.outlier):
+        already_seen = (
+            existing and (
+                not existing.internal_metadata.is_outlier()
+                or pdu.internal_metadata.is_outlier()
+            )
+        )
+        if already_seen:
             logger.debug("Already seen pdu %s", pdu.event_id)
             defer.returnValue({})
             return
 
         state = None
 
+        auth_chain = []
+
         # We need to make sure we have all the auth events.
         # for e_id, _ in pdu.auth_events:
         #     exists = yield self._get_persisted_pdu(
@@ -595,7 +591,7 @@ class ReplicationLayer(object):
         #             )
 
         # Get missing pdus if necessary.
-        if not pdu.outlier:
+        if not pdu.internal_metadata.is_outlier():
             # We only backfill backwards to the min depth.
             min_depth = yield self.handler.get_min_depth_for_context(
                 pdu.room_id
@@ -636,7 +632,7 @@ class ReplicationLayer(object):
                     "_handle_new_pdu getting state for %s",
                     pdu.room_id
                 )
-                state = yield self.get_state_for_context(
+                state, auth_chain = yield self.get_state_for_room(
                     origin, pdu.room_id, pdu.event_id,
                 )
 
@@ -646,6 +642,7 @@ class ReplicationLayer(object):
                 pdu,
                 backfilled=backfilled,
                 state=state,
+                auth_chain=auth_chain,
             )
         else:
             ret = None
@@ -658,19 +655,14 @@ class ReplicationLayer(object):
         return "<ReplicationLayer(%s)>" % self.server_name
 
     def event_from_pdu_json(self, pdu_json, outlier=False):
-        #TODO: Check we have all the PDU keys here
-        pdu_json.setdefault("hashes", {})
-        pdu_json.setdefault("signatures", {})
-        sender = pdu_json.pop("sender", None)
-        if sender is not None:
-            pdu_json["user_id"] = sender
-        state_hash = pdu_json.get("unsigned", {}).pop("state_hash", None)
-        if state_hash is not None:
-            pdu_json["state_hash"] = state_hash
-        return self.event_factory.create_event(
-            pdu_json["type"], outlier=outlier, **pdu_json
+        event = FrozenEvent(
+            pdu_json
         )
 
+        event.internal_metadata.outlier = outlier
+
+        return event
+
 
 class _TransactionQueue(object):
     """This class makes sure we only have one transaction in flight at
@@ -685,6 +677,7 @@ class _TransactionQueue(object):
         self.transport_layer = transport_layer
 
         self._clock = hs.get_clock()
+        self.store = hs.get_datastore()
 
         # Is a mapping from destinations -> deferreds. Used to keep track
         # of which destinations have transactions in flight and when they are
@@ -705,15 +698,14 @@ class _TransactionQueue(object):
 
     @defer.inlineCallbacks
     @log_function
-    def enqueue_pdu(self, pdu, order):
+    def enqueue_pdu(self, pdu, destinations, order):
         # We loop through all destinations to see whether we already have
         # a transaction in progress. If we do, stick it in the pending_pdus
         # table and we'll get back to it later.
 
-        destinations = set([
-            d for d in pdu.destinations
-            if d != self.server_name
-        ])
+        destinations = set(destinations)
+        destinations.discard(self.server_name)
+        destinations.discard("localhost")
 
         logger.debug("Sending to: %s", str(destinations))
 
@@ -728,8 +720,14 @@ class _TransactionQueue(object):
                 (pdu, deferred, order)
             )
 
+            def eb(failure):
+                if not deferred.called:
+                    deferred.errback(failure)
+                else:
+                    logger.warn("Failed to send pdu", failure)
+
             with PreserveLoggingContext():
-                self._attempt_new_transaction(destination)
+                self._attempt_new_transaction(destination).addErrback(eb)
 
             deferreds.append(deferred)
 
@@ -739,6 +737,9 @@ class _TransactionQueue(object):
     def enqueue_edu(self, edu):
         destination = edu.destination
 
+        if destination == self.server_name:
+            return
+
         deferred = defer.Deferred()
         self.pending_edus_by_dest.setdefault(destination, []).append(
             (edu, deferred)
@@ -748,7 +749,7 @@ class _TransactionQueue(object):
             if not deferred.called:
                 deferred.errback(failure)
             else:
-                logger.exception("Failed to send edu", failure)
+                logger.warn("Failed to send edu", failure)
 
         with PreserveLoggingContext():
             self._attempt_new_transaction(destination).addErrback(eb)
@@ -770,18 +771,54 @@ class _TransactionQueue(object):
     @defer.inlineCallbacks
     @log_function
     def _attempt_new_transaction(self, destination):
+
+        (retry_last_ts, retry_interval) = (0, 0)
+        retry_timings = yield self.store.get_destination_retry_timings(
+            destination
+        )
+        if retry_timings:
+            (retry_last_ts, retry_interval) = (
+                retry_timings.retry_last_ts, retry_timings.retry_interval
+            )
+            if retry_last_ts + retry_interval > int(self._clock.time_msec()):
+                logger.info(
+                    "TX [%s] not ready for retry yet - "
+                    "dropping transaction for now",
+                    destination,
+                )
+                return
+            else:
+                logger.info("TX [%s] is ready for retry", destination)
+
+        logger.info("TX [%s] _attempt_new_transaction", destination)
+
         if destination in self.pending_transactions:
+            # XXX: pending_transactions can get stuck on by a never-ending
+            # request at which point pending_pdus_by_dest just keeps growing.
+            # we need application-layer timeouts of some flavour of these
+            # requests
             return
 
-        #  list of (pending_pdu, deferred, order)
+        # list of (pending_pdu, deferred, order)
         pending_pdus = self.pending_pdus_by_dest.pop(destination, [])
         pending_edus = self.pending_edus_by_dest.pop(destination, [])
         pending_failures = self.pending_failures_by_dest.pop(destination, [])
 
+        if pending_pdus:
+            logger.info("TX [%s] len(pending_pdus_by_dest[dest]) = %d",
+                        destination, len(pending_pdus))
+
         if not pending_pdus and not pending_edus and not pending_failures:
             return
 
-        logger.debug("TX [%s] Attempting new transaction", destination)
+        logger.debug(
+            "TX [%s] Attempting new transaction"
+            " (pdus: %d, edus: %d, failures: %d)",
+            destination,
+            len(pending_pdus),
+            len(pending_edus),
+            len(pending_failures)
+        )
 
         # Sort based on the order field
         pending_pdus.sort(key=lambda t: t[2])
@@ -814,7 +851,11 @@ class _TransactionQueue(object):
             yield self.transaction_actions.prepare_to_send(transaction)
 
             logger.debug("TX [%s] Persisted transaction", destination)
-            logger.debug("TX [%s] Sending transaction...", destination)
+            logger.info(
+                "TX [%s] Sending transaction [%s]",
+                destination,
+                transaction.transaction_id,
+            )
 
             # Actually send the transaction
 
@@ -835,6 +876,8 @@ class _TransactionQueue(object):
                 transaction, json_data_cb
             )
 
+            logger.info("TX [%s] got %d response", destination, code)
+
             logger.debug("TX [%s] Sent transaction", destination)
             logger.debug("TX [%s] Marking as delivered...", destination)
 
@@ -847,8 +890,14 @@ class _TransactionQueue(object):
 
             for deferred in deferreds:
                 if code == 200:
+                    if retry_last_ts:
+                        # this host is alive! reset retry schedule
+                        yield self.store.set_destination_retry_timings(
+                            destination, 0, 0
+                        )
                     deferred.callback(None)
                 else:
+                    self.set_retrying(destination, retry_interval)
                     deferred.errback(RuntimeError("Got status %d" % code))
 
                 # Ensures we don't continue until all callbacks on that
@@ -861,11 +910,15 @@ class _TransactionQueue(object):
             logger.debug("TX [%s] Yielded to callbacks", destination)
 
         except Exception as e:
-            logger.error("TX Problem in _attempt_transaction")
-
             # We capture this here as there as nothing actually listens
             # for this finishing functions deferred.
-            logger.exception(e)
+            logger.warn(
+                "TX [%s] Problem in _attempt_transaction: %s",
+                destination,
+                e,
+            )
+
+            self.set_retrying(destination, retry_interval)
 
             for deferred in deferreds:
                 if not deferred.called:
@@ -877,3 +930,22 @@ class _TransactionQueue(object):
 
             # Check to see if there is anything else to send.
             self._attempt_new_transaction(destination)
+
+    @defer.inlineCallbacks
+    def set_retrying(self, destination, retry_interval):
+        # track that this destination is having problems and we should
+        # give it a chance to recover before trying it again
+
+        if retry_interval:
+            retry_interval *= 2
+            # plateau at hourly retries for now
+            if retry_interval >= 60 * 60 * 1000:
+                retry_interval = 60 * 60 * 1000
+        else:
+            retry_interval = 2000  # try again at first after 2 seconds
+
+        yield self.store.set_destination_retry_timings(
+            destination,
+            int(self._clock.time_msec()),
+            retry_interval
+        )

synapse/federation/transport/__init__.py

@@ -0,0 +1,62 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014, 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""The transport layer is responsible for both sending transactions to remote
+home servers and receiving a variety of requests from other home servers.
+
+By default this is done over HTTPS (and all home servers are required to
+support HTTPS), however individual pairings of servers may decide to
+communicate over a different (albeit still reliable) protocol.
+"""
+
+from .server import TransportLayerServer
+from .client import TransportLayerClient
+
+
+class TransportLayer(TransportLayerServer, TransportLayerClient):
+    """This is a basic implementation of the transport layer that translates
+    transactions and other requests to/from HTTP.
+
+    Attributes:
+        server_name (str): Local home server host
+
+        server (synapse.http.server.HttpServer): the http server to
+                register listeners on
+
+        client (synapse.http.client.HttpClient): the http client used to
+                send requests
+
+        request_handler (TransportRequestHandler): The handler to fire when we
+            receive requests for data.
+
+        received_handler (TransportReceivedHandler): The handler to fire when
+            we receive data.
+    """
+
+    def __init__(self, homeserver, server_name, server, client):
+        """
+        Args:
+            server_name (str): Local home server host
+            server (synapse.protocol.http.HttpServer): the http server to
+                register listeners on
+            client (synapse.protocol.http.HttpClient): the http client used to
+                send requests
+        """
+        self.keyring = homeserver.get_keyring()
+        self.server_name = server_name
+        self.server = server
+        self.client = client
+        self.request_handler = None
+        self.received_handler = None

synapse/federation/transport/client.py

@@ -0,0 +1,215 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014, 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from twisted.internet import defer
+
+from synapse.api.urls import FEDERATION_PREFIX as PREFIX
+from synapse.util.logutils import log_function
+
+import logging
+import json
+
+
+logger = logging.getLogger(__name__)
+
+
+class TransportLayerClient(object):
+    """Sends federation HTTP requests to other servers"""
+
+    @log_function
+    def get_room_state(self, destination, room_id, event_id):
+        """ Requests all state for a given room from the given server at the
+        given event.
+
+        Args:
+            destination (str): The host name of the remote home server we want
+                to get the state from.
+            context (str): The name of the context we want the state of
+            event_id (str): The event we want the context at.
+
+        Returns:
+            Deferred: Results in a dict received from the remote homeserver.
+        """
+        logger.debug("get_room_state dest=%s, room=%s",
+                     destination, room_id)
+
+        path = PREFIX + "/state/%s/" % room_id
+        return self.client.get_json(
+            destination, path=path, args={"event_id": event_id},
+        )
+
+    @log_function
+    def get_event(self, destination, event_id):
+        """ Requests the pdu with give id and origin from the given server.
+
+        Args:
+            destination (str): The host name of the remote home server we want
+                to get the state from.
+            event_id (str): The id of the event being requested.
+
+        Returns:
+            Deferred: Results in a dict received from the remote homeserver.
+        """
+        logger.debug("get_pdu dest=%s, event_id=%s",
+                     destination, event_id)
+
+        path = PREFIX + "/event/%s/" % (event_id, )
+        return self.client.get_json(destination, path=path)
+
+    @log_function
+    def backfill(self, destination, room_id, event_tuples, limit):
+        """ Requests `limit` previous PDUs in a given context before list of
+        PDUs.
+
+        Args:
+            dest (str)
+            room_id (str)
+            event_tuples (list)
+            limt (int)
+
+        Returns:
+            Deferred: Results in a dict received from the remote homeserver.
+        """
+        logger.debug(
+            "backfill dest=%s, room_id=%s, event_tuples=%s, limit=%s",
+            destination, room_id, repr(event_tuples), str(limit)
+        )
+
+        if not event_tuples:
+            # TODO: raise?
+            return
+
+        path = PREFIX + "/backfill/%s/" % (room_id,)
+
+        args = {
+            "v": event_tuples,
+            "limit": [str(limit)],
+        }
+
+        return self.client.get_json(
+            destination,
+            path=path,
+            args=args,
+        )
+
+    @defer.inlineCallbacks
+    @log_function
+    def send_transaction(self, transaction, json_data_callback=None):
+        """ Sends the given Transaction to its destination
+
+        Args:
+            transaction (Transaction)
+
+        Returns:
+            Deferred: Results of the deferred is a tuple in the form of
+            (response_code, response_body) where the response_body is a
+            python dict decoded from json
+        """
+        logger.debug(
+            "send_data dest=%s, txid=%s",
+            transaction.destination, transaction.transaction_id
+        )
+
+        if transaction.destination == self.server_name:
+            raise RuntimeError("Transport layer cannot send to itself!")
+
+        # FIXME: This is only used by the tests. The actual json sent is
+        # generated by the json_data_callback.
+        json_data = transaction.get_dict()
+
+        code, response = yield self.client.put_json(
+            transaction.destination,
+            path=PREFIX + "/send/%s/" % transaction.transaction_id,
+            data=json_data,
+            json_data_callback=json_data_callback,
+        )
+
+        logger.debug(
+            "send_data dest=%s, txid=%s, got response: %d",
+            transaction.destination, transaction.transaction_id, code
+        )
+
+        defer.returnValue((code, response))
+
+    @defer.inlineCallbacks
+    @log_function
+    def make_query(self, destination, query_type, args, retry_on_dns_fail):
+        path = PREFIX + "/query/%s" % query_type
+
+        response = yield self.client.get_json(
+            destination=destination,
+            path=path,
+            args=args,
+            retry_on_dns_fail=retry_on_dns_fail,
+        )
+
+        defer.returnValue(response)
+
+    @defer.inlineCallbacks
+    @log_function
+    def make_join(self, destination, room_id, user_id, retry_on_dns_fail=True):
+        path = PREFIX + "/make_join/%s/%s" % (room_id, user_id)
+
+        response = yield self.client.get_json(
+            destination=destination,
+            path=path,
+            retry_on_dns_fail=retry_on_dns_fail,
+        )
+
+        defer.returnValue(response)
+
+    @defer.inlineCallbacks
+    @log_function
+    def send_join(self, destination, room_id, event_id, content):
+        path = PREFIX + "/send_join/%s/%s" % (room_id, event_id)
+
+        code, content = yield self.client.put_json(
+            destination=destination,
+            path=path,
+            data=content,
+        )
+
+        if not 200 <= code < 300:
+            raise RuntimeError("Got %d from send_join", code)
+
+        defer.returnValue(json.loads(content))
+
+    @defer.inlineCallbacks
+    @log_function
+    def send_invite(self, destination, room_id, event_id, content):
+        path = PREFIX + "/invite/%s/%s" % (room_id, event_id)
+
+        code, content = yield self.client.put_json(
+            destination=destination,
+            path=path,
+            data=content,
+        )
+
+        if not 200 <= code < 300:
+            raise RuntimeError("Got %d from send_invite", code)
+
+        defer.returnValue(json.loads(content))
+
+    @defer.inlineCallbacks
+    @log_function
+    def get_event_auth(self, destination, room_id, event_id):
+        path = PREFIX + "/event_auth/%s/%s" % (room_id, event_id)
+
+        response = yield self.client.get_json(
+            destination=destination,
+            path=path,
+        )
+
+        defer.returnValue(response)

synapse/federation/transport/server.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,14 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""The transport layer is responsible for both sending transactions to remote
-home servers and receiving a variety of requests from other home servers.
-
-Typically, this is done over HTTP (and all home servers are required to
-support HTTP), however individual pairings of servers may decide to communicate
-over a different (albeit still reliable) protocol.
-"""
-
 from twisted.internet import defer
 
 from synapse.api.urls import FEDERATION_PREFIX as PREFIX
@@ -35,241 +27,8 @@ import re
 logger = logging.getLogger(__name__)
 
 
-class TransportLayer(object):
-    """This is a basic implementation of the transport layer that translates
-    transactions and other requests to/from HTTP.
-
-    Attributes:
-        server_name (str): Local home server host
-
-        server (synapse.http.server.HttpServer): the http server to
-                register listeners on
-
-        client (synapse.http.client.HttpClient): the http client used to
-                send requests
-
-        request_handler (TransportRequestHandler): The handler to fire when we
-            receive requests for data.
-
-        received_handler (TransportReceivedHandler): The handler to fire when
-            we receive data.
-    """
-
-    def __init__(self, homeserver, server_name, server, client):
-        """
-        Args:
-            server_name (str): Local home server host
-            server (synapse.protocol.http.HttpServer): the http server to
-                register listeners on
-            client (synapse.protocol.http.HttpClient): the http client used to
-                send requests
-        """
-        self.keyring = homeserver.get_keyring()
-        self.server_name = server_name
-        self.server = server
-        self.client = client
-        self.request_handler = None
-        self.received_handler = None
-
-    @log_function
-    def get_context_state(self, destination, context, event_id=None):
-        """ Requests all state for a given context (i.e. room) from the
-        given server.
-
-        Args:
-            destination (str): The host name of the remote home server we want
-                to get the state from.
-            context (str): The name of the context we want the state of
-
-        Returns:
-            Deferred: Results in a dict received from the remote homeserver.
-        """
-        logger.debug("get_context_state dest=%s, context=%s",
-                     destination, context)
-
-        subpath = "/state/%s/" % context
-
-        args = {}
-        if event_id:
-            args["event_id"] = event_id
-
-        return self._do_request_for_transaction(
-            destination, subpath, args=args
-        )
-
-    @log_function
-    def get_event(self, destination, event_id):
-        """ Requests the pdu with give id and origin from the given server.
-
-        Args:
-            destination (str): The host name of the remote home server we want
-                to get the state from.
-            event_id (str): The id of the event being requested.
-
-        Returns:
-            Deferred: Results in a dict received from the remote homeserver.
-        """
-        logger.debug("get_pdu dest=%s, event_id=%s",
-                     destination, event_id)
-
-        subpath = "/event/%s/" % (event_id, )
-
-        return self._do_request_for_transaction(destination, subpath)
-
-    @log_function
-    def backfill(self, dest, context, event_tuples, limit):
-        """ Requests `limit` previous PDUs in a given context before list of
-        PDUs.
-
-        Args:
-            dest (str)
-            context (str)
-            event_tuples (list)
-            limt (int)
-
-        Returns:
-            Deferred: Results in a dict received from the remote homeserver.
-        """
-        logger.debug(
-            "backfill dest=%s, context=%s, event_tuples=%s, limit=%s",
-            dest, context, repr(event_tuples), str(limit)
-        )
-
-        if not event_tuples:
-            # TODO: raise?
-            return
-
-        subpath = "/backfill/%s/" % (context,)
-
-        args = {
-            "v": event_tuples,
-            "limit": [str(limit)],
-        }
-
-        return self._do_request_for_transaction(
-            dest,
-            subpath,
-            args=args,
-        )
-
-    @defer.inlineCallbacks
-    @log_function
-    def send_transaction(self, transaction, json_data_callback=None):
-        """ Sends the given Transaction to it's destination
-
-        Args:
-            transaction (Transaction)
-
-        Returns:
-            Deferred: Results of the deferred is a tuple in the form of
-            (response_code, response_body) where the response_body is a
-            python dict decoded from json
-        """
-        logger.debug(
-            "send_data dest=%s, txid=%s",
-            transaction.destination, transaction.transaction_id
-        )
-
-        if transaction.destination == self.server_name:
-            raise RuntimeError("Transport layer cannot send to itself!")
-
-        # FIXME: This is only used by the tests. The actual json sent is
-        # generated by the json_data_callback.
-        json_data = transaction.get_dict()
-
-        code, response = yield self.client.put_json(
-            transaction.destination,
-            path=PREFIX + "/send/%s/" % transaction.transaction_id,
-            data=json_data,
-            json_data_callback=json_data_callback,
-        )
-
-        logger.debug(
-            "send_data dest=%s, txid=%s, got response: %d",
-            transaction.destination, transaction.transaction_id, code
-        )
-
-        defer.returnValue((code, response))
-
-    @defer.inlineCallbacks
-    @log_function
-    def make_query(self, destination, query_type, args, retry_on_dns_fail):
-        path = PREFIX + "/query/%s" % query_type
-
-        response = yield self.client.get_json(
-            destination=destination,
-            path=path,
-            args=args,
-            retry_on_dns_fail=retry_on_dns_fail,
-        )
-
-        defer.returnValue(response)
-
-    @defer.inlineCallbacks
-    @log_function
-    def make_join(self, destination, context, user_id, retry_on_dns_fail=True):
-        path = PREFIX + "/make_join/%s/%s" % (context, user_id,)
-
-        response = yield self.client.get_json(
-            destination=destination,
-            path=path,
-            retry_on_dns_fail=retry_on_dns_fail,
-        )
-
-        defer.returnValue(response)
-
-    @defer.inlineCallbacks
-    @log_function
-    def send_join(self, destination, context, event_id, content):
-        path = PREFIX + "/send_join/%s/%s" % (
-            context,
-            event_id,
-        )
-
-        code, content = yield self.client.put_json(
-            destination=destination,
-            path=path,
-            data=content,
-        )
-
-        if not 200 <= code < 300:
-            raise RuntimeError("Got %d from send_join", code)
-
-        defer.returnValue(json.loads(content))
-
-    @defer.inlineCallbacks
-    @log_function
-    def send_invite(self, destination, context, event_id, content):
-        path = PREFIX + "/invite/%s/%s" % (
-            context,
-            event_id,
-        )
-
-        code, content = yield self.client.put_json(
-            destination=destination,
-            path=path,
-            data=content,
-        )
-
-        if not 200 <= code < 300:
-            raise RuntimeError("Got %d from send_invite", code)
-
-        defer.returnValue(json.loads(content))
-
-    @defer.inlineCallbacks
-    @log_function
-    def get_event_auth(self, destination, context, event_id):
-        path = PREFIX + "/event_auth/%s/%s" % (
-            context,
-            event_id,
-        )
-
-        response = yield self.client.get_json(
-            destination=destination,
-            path=path,
-        )
-
-        defer.returnValue(response)
+class TransportLayerServer(object):
+    """Handles incoming federation HTTP requests"""
 
     @defer.inlineCallbacks
     def _authenticate_request(self, request):
@@ -373,8 +132,6 @@ class TransportLayer(object):
         """
         self.request_handler = handler
 
-        # TODO(markjh): Namespace the federation URI paths
-
         # This is for when someone asks us for everything since version X
         self.server.register_path(
             "GET",
@@ -528,34 +285,6 @@ class TransportLayer(object):
 
         defer.returnValue((code, response))
 
-    @defer.inlineCallbacks
-    @log_function
-    def _do_request_for_transaction(self, destination, subpath, args={}):
-        """
-        Args:
-            destination (str)
-            path (str)
-            args (dict): This is parsed directly to the HttpClient.
-
-        Returns:
-            Deferred: Results in a dict.
-        """
-
-        data = yield self.client.get_json(
-            destination,
-            path=PREFIX + subpath,
-            args=args,
-        )
-
-        # Add certain keys to the JSON, ready for decoding as a Transaction
-        data.update(
-            origin=destination,
-            destination=self.server_name,
-            transaction_id=None
-        )
-
-        defer.returnValue(data)
-
     @log_function
     def _on_backfill_request(self, origin, context, v_list, limits):
         if not limits:

synapse/federation/units.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/_base.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,11 +15,10 @@
 
 from twisted.internet import defer
 
-from synapse.api.errors import LimitExceededError
+from synapse.api.errors import LimitExceededError, SynapseError
 from synapse.util.async import run_on_reactor
 from synapse.crypto.event_signing import add_hashes_and_signatures
-from synapse.api.events.room import RoomMemberEvent
-from synapse.api.constants import Membership
+from synapse.api.constants import Membership, EventTypes
 
 import logging
 
@@ -31,10 +30,8 @@ class BaseHandler(object):
 
     def __init__(self, hs):
         self.store = hs.get_datastore()
-        self.event_factory = hs.get_event_factory()
         self.auth = hs.get_auth()
         self.notifier = hs.get_notifier()
-        self.room_lock = hs.get_room_lock_manager()
         self.state_handler = hs.get_state_handler()
         self.distributor = hs.get_distributor()
         self.ratelimiter = hs.get_ratelimiter()
@@ -44,6 +41,8 @@ class BaseHandler(object):
         self.signing_key = hs.config.signing_key[0]
         self.server_name = hs.hostname
 
+        self.event_builder_factory = hs.get_event_builder_factory()
+
     def ratelimit(self, user_id):
         time_now = self.clock.time()
         allowed, time_allowed = self.ratelimiter.send_message(
@@ -57,62 +56,93 @@ class BaseHandler(object):
             )
 
     @defer.inlineCallbacks
-    def _on_new_room_event(self, event, snapshot, extra_destinations=[],
-                           extra_users=[], suppress_auth=False,
-                           do_invite_host=None):
+    def _create_new_client_event(self, builder):
         yield run_on_reactor()
 
-        snapshot.fill_out_prev_events(event)
-
-        yield self.state_handler.annotate_event_with_state(event)
-
-        yield self.auth.add_auth_events(event)
-
-        logger.debug("Signing event...")
-
-        add_hashes_and_signatures(
-            event, self.server_name, self.signing_key
+        latest_ret = yield self.store.get_latest_events_in_room(
+            builder.room_id,
         )
 
-        logger.debug("Signed event.")
+        if latest_ret:
+            depth = max([d for _, _, d in latest_ret]) + 1
+        else:
+            depth = 1
+
+        prev_events = [(e, h) for e, h, _ in latest_ret]
+
+        builder.prev_events = prev_events
+        builder.depth = depth
+
+        state_handler = self.state_handler
+
+        context = yield state_handler.compute_event_context(builder)
+
+        if builder.is_state():
+            builder.prev_state = context.prev_state_events
+
+        yield self.auth.add_auth_events(builder, context)
+
+        add_hashes_and_signatures(
+            builder, self.server_name, self.signing_key
+        )
+
+        event = builder.build()
+
+        logger.debug(
+            "Created event %s with auth_events: %s, current state: %s",
+            event.event_id, context.auth_events, context.current_state,
+        )
+
+        defer.returnValue(
+            (event, context,)
+        )
+
+    @defer.inlineCallbacks
+    def handle_new_client_event(self, event, context, extra_destinations=[],
+                                extra_users=[], suppress_auth=False):
+        yield run_on_reactor()
+
+        # We now need to go and hit out to wherever we need to hit out to.
 
         if not suppress_auth:
-            logger.debug("Authing...")
-            self.auth.check(event, auth_events=event.old_state_events)
-            logger.debug("Authed")
-        else:
-            logger.debug("Suppressed auth.")
+            self.auth.check(event, auth_events=context.auth_events)
 
-        if do_invite_host:
-            federation_handler = self.hs.get_handlers().federation_handler
-            invite_event = yield federation_handler.send_invite(
-                do_invite_host,
-                event
-            )
+        yield self.store.persist_event(event, context=context)
 
-            # FIXME: We need to check if the remote changed anything else
-            event.signatures = invite_event.signatures
+        federation_handler = self.hs.get_handlers().federation_handler
 
-        yield self.store.persist_event(event)
+        if event.type == EventTypes.Member:
+            if event.content["membership"] == Membership.INVITE:
+                invitee = self.hs.parse_userid(event.state_key)
+                if not self.hs.is_mine(invitee):
+                    # TODO: Can we add signature from remote server in a nicer
+                    # way? If we have been invited by a remote server, we need
+                    # to get them to sign the event.
+                    returned_invite = yield federation_handler.send_invite(
+                        invitee.domain,
+                        event,
+                    )
+
+                    # TODO: Make sure the signatures actually are correct.
+                    event.signatures.update(
+                        returned_invite.signatures
+                    )
 
         destinations = set(extra_destinations)
-        # Send a PDU to all hosts who have joined the room.
-
-        for k, s in event.state_events.items():
+        for k, s in context.current_state.items():
             try:
-                if k[0] == RoomMemberEvent.TYPE:
+                if k[0] == EventTypes.Member:
                     if s.content["membership"] == Membership.JOIN:
                         destinations.add(
                             self.hs.parse_userid(s.state_key).domain
                         )
-            except:
+            except SynapseError:
                 logger.warn(
                     "Failed to get destination from event %s", s.event_id
                 )
 
-        event.destinations = list(destinations)
-
         yield self.notifier.on_new_room_event(event, extra_users=extra_users)
 
-        federation_handler = self.hs.get_handlers().federation_handler
-        yield federation_handler.handle_new_event(event, snapshot)
+        yield federation_handler.handle_new_event(
+            event, destinations=destinations,
+        )

synapse/handlers/admin.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/directory.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,7 +18,7 @@ from twisted.internet import defer
 from ._base import BaseHandler
 
 from synapse.api.errors import SynapseError, Codes, CodeMessageException
-from synapse.api.events.room import RoomAliasesEvent
+from synapse.api.constants import EventTypes
 
 import logging
 
@@ -40,7 +40,7 @@ class DirectoryHandler(BaseHandler):
 
         # TODO(erikj): Do auth.
 
-        if not room_alias.is_mine:
+        if not self.hs.is_mine(room_alias):
             raise SynapseError(400, "Room alias must be local")
             # TODO(erikj): Change this.
 
@@ -64,7 +64,7 @@ class DirectoryHandler(BaseHandler):
     def delete_association(self, user_id, room_alias):
         # TODO Check if server admin
 
-        if not room_alias.is_mine:
+        if not self.hs.is_mine(room_alias):
             raise SynapseError(400, "Room alias must be local")
 
         room_id = yield self.store.delete_room_alias(room_alias)
@@ -75,7 +75,7 @@ class DirectoryHandler(BaseHandler):
     @defer.inlineCallbacks
     def get_association(self, room_alias):
         room_id = None
-        if room_alias.is_mine:
+        if self.hs.is_mine(room_alias):
             result = yield self.store.get_association_from_room_alias(
                 room_alias
             )
@@ -123,7 +123,7 @@ class DirectoryHandler(BaseHandler):
     @defer.inlineCallbacks
     def on_directory_query(self, args):
         room_alias = self.hs.parse_roomalias(args["room_alias"])
-        if not room_alias.is_mine:
+        if not self.hs.is_mine(room_alias):
             raise SynapseError(
                 400, "Room Alias is not hosted on this Home Server"
             )
@@ -148,16 +148,11 @@ class DirectoryHandler(BaseHandler):
     def send_room_alias_update_event(self, user_id, room_id):
         aliases = yield self.store.get_aliases_for_room(room_id)
 
-        event = self.event_factory.create_event(
-            etype=RoomAliasesEvent.TYPE,
-            state_key=self.hs.hostname,
-            room_id=room_id,
-            user_id=user_id,
-            content={"aliases": aliases},
-        )
-
-        snapshot = yield self.store.snapshot_room(event)
-
-        yield self._on_new_room_event(
-            event, snapshot, extra_users=[user_id], suppress_auth=True
-        )
+        msg_handler = self.hs.get_handlers().message_handler
+        yield msg_handler.create_and_send_event({
+            "type": EventTypes.Aliases,
+            "state_key": self.hs.hostname,
+            "room_id": room_id,
+            "sender": user_id,
+            "content": {"aliases": aliases},
+        }, ratelimit=False)

synapse/handlers/events.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -46,7 +46,8 @@ class EventStreamHandler(BaseHandler):
 
     @defer.inlineCallbacks
     @log_function
-    def get_stream(self, auth_user_id, pagin_config, timeout=0):
+    def get_stream(self, auth_user_id, pagin_config, timeout=0,
+                   as_client_event=True):
         auth_user = self.hs.parse_userid(auth_user_id)
 
         try:
@@ -69,16 +70,16 @@ class EventStreamHandler(BaseHandler):
                 pagin_config.from_token = None
 
             rm_handler = self.hs.get_handlers().room_member_handler
-            logger.debug("BETA")
             room_ids = yield rm_handler.get_rooms_for_user(auth_user)
 
-            logger.debug("ALPHA")
             with PreserveLoggingContext():
                 events, tokens = yield self.notifier.get_events_for(
                     auth_user, room_ids, pagin_config, timeout
                 )
 
-            chunks = [self.hs.serialize_event(e) for e in events]
+            chunks = [
+                self.hs.serialize_event(e, as_client_event) for e in events
+            ]
 
             chunk = {
                 "chunk": chunks,

synapse/handlers/federation.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,12 +17,11 @@
 
 from ._base import BaseHandler
 
-from synapse.api.events.utils import prune_event
+from synapse.events.utils import prune_event
 from synapse.api.errors import (
     AuthError, FederationError, SynapseError, StoreError,
 )
-from synapse.api.events.room import RoomMemberEvent, RoomCreateEvent
-from synapse.api.constants import Membership
+from synapse.api.constants import EventTypes, Membership
 from synapse.util.logutils import log_function
 from synapse.util.async import run_on_reactor
 from synapse.crypto.event_signing import (
@@ -76,14 +75,14 @@ class FederationHandler(BaseHandler):
 
     @log_function
     @defer.inlineCallbacks
-    def handle_new_event(self, event, snapshot):
+    def handle_new_event(self, event, destinations):
         """ Takes in an event from the client to server side, that has already
         been authed and handled by the state module, and sends it to any
         remote home servers that may be interested.
 
         Args:
-            event
-            snapshot (.storage.Snapshot): THe snapshot the event happened after
+            event: The event to send
+            destinations: A list of destinations to send it to
 
         Returns:
             Deferred: Resolved when it has successfully been queued for
@@ -92,16 +91,12 @@ class FederationHandler(BaseHandler):
 
         yield run_on_reactor()
 
-        pdu = event
-
-        if not hasattr(pdu, "destinations") or not pdu.destinations:
-            pdu.destinations = []
-
-        yield self.replication_layer.send_pdu(pdu)
+        self.replication_layer.send_pdu(event, destinations)
 
     @log_function
     @defer.inlineCallbacks
-    def on_receive_pdu(self, origin, pdu, backfilled, state=None):
+    def on_receive_pdu(self, origin, pdu, backfilled, state=None,
+                       auth_chain=None):
         """ Called by the ReplicationLayer when we have a new pdu. We need to
         do auth checks and put it through the StateHandler.
         """
@@ -140,7 +135,7 @@ class FederationHandler(BaseHandler):
         if not check_event_content_hash(event):
             logger.warn(
                 "Event content has been tampered, redacting %s, %s",
-                event.event_id, encode_canonical_json(event.get_full_dict())
+                event.event_id, encode_canonical_json(event.get_dict())
             )
             event = redacted_event
 
@@ -153,43 +148,44 @@ class FederationHandler(BaseHandler):
             event.room_id,
             self.server_name
         )
-        if not is_in_room and not event.outlier:
+        if not is_in_room and not event.internal_metadata.outlier:
             logger.debug("Got event for room we're not in.")
 
-            replication_layer = self.replication_layer
-            auth_chain = yield replication_layer.get_event_auth(
-                origin,
-                context=event.room_id,
-                event_id=event.event_id,
-            )
-
-            for e in auth_chain:
-                e.outlier = True
-                try:
-                    yield self._handle_new_event(e, fetch_missing=False)
-                except:
-                    logger.exception(
-                        "Failed to parse auth event %s",
-                        e.event_id,
-                    )
+            replication = self.replication_layer
 
             if not state:
-                state = yield replication_layer.get_state_for_context(
+                state, auth_chain = yield replication.get_state_for_room(
+                    origin, context=event.room_id, event_id=event.event_id,
+                )
+
+            if not auth_chain:
+                auth_chain = yield replication.get_event_auth(
                     origin,
                     context=event.room_id,
                     event_id=event.event_id,
                 )
 
+            for e in auth_chain:
+                e.internal_metadata.outlier = True
+                try:
+                    yield self._handle_new_event(e, fetch_auth_from=origin)
+                except:
+                    logger.exception(
+                        "Failed to handle auth event %s",
+                        e.event_id,
+                    )
+
             current_state = state
 
         if state:
             for e in state:
-                e.outlier = True
+                logging.info("A :) %r", e)
+                e.internal_metadata.outlier = True
                 try:
                     yield self._handle_new_event(e)
                 except:
                     logger.exception(
-                        "Failed to parse state event %s",
+                        "Failed to handle state event %s",
                         e.event_id,
                     )
 
@@ -208,6 +204,13 @@ class FederationHandler(BaseHandler):
                 affected=event.event_id,
             )
 
+        # if we're receiving valid events from an origin,
+        # it's probably a good idea to mark it as not in retry-state
+        # for sending (although this is a bit of a leap)
+        retry_timings = yield self.store.get_destination_retry_timings(origin)
+        if (retry_timings and retry_timings.retry_last_ts):
+            self.store.set_destination_retry_timings(origin, 0, 0)
+
         room = yield self.store.get_room(event.room_id)
 
         if not room:
@@ -222,7 +225,7 @@ class FederationHandler(BaseHandler):
 
         if not backfilled:
             extra_users = []
-            if event.type == RoomMemberEvent.TYPE:
+            if event.type == EventTypes.Member:
                 target_user_id = event.state_key
                 target_user = self.hs.parse_userid(target_user_id)
                 extra_users.append(target_user)
@@ -231,7 +234,7 @@ class FederationHandler(BaseHandler):
                 event, extra_users=extra_users
             )
 
-        if event.type == RoomMemberEvent.TYPE:
+        if event.type == EventTypes.Member:
             if event.membership == Membership.JOIN:
                 user = self.hs.parse_userid(event.state_key)
                 yield self.distributor.fire(
@@ -258,11 +261,15 @@ class FederationHandler(BaseHandler):
             event = pdu
 
             # FIXME (erikj): Not sure this actually works :/
-            yield self.state_handler.annotate_event_with_state(event)
+            context = yield self.state_handler.compute_event_context(event)
 
-            events.append(event)
+            events.append((event, context))
 
-            yield self.store.persist_event(event, backfilled=True)
+            yield self.store.persist_event(
+                event,
+                context=context,
+                backfilled=True
+            )
 
         defer.returnValue(events)
 
@@ -274,18 +281,16 @@ class FederationHandler(BaseHandler):
         """
         pdu = yield self.replication_layer.send_invite(
             destination=target_host,
-            context=event.room_id,
+            room_id=event.room_id,
             event_id=event.event_id,
             pdu=event
         )
 
-
-
         defer.returnValue(pdu)
 
     @defer.inlineCallbacks
     def on_event_auth(self, event_id):
-        auth = yield self.store.get_auth_chain(event_id)
+        auth = yield self.store.get_auth_chain([event_id])
 
         for event in auth:
             event.signatures.update(
@@ -325,42 +330,55 @@ class FederationHandler(BaseHandler):
         event = pdu
 
         # We should assert some things.
-        assert(event.type == RoomMemberEvent.TYPE)
+        # FIXME: Do this in a nicer way
+        assert(event.type == EventTypes.Member)
         assert(event.user_id == joinee)
         assert(event.state_key == joinee)
         assert(event.room_id == room_id)
 
-        event.outlier = False
+        event.internal_metadata.outlier = False
 
         self.room_queues[room_id] = []
 
+        builder = self.event_builder_factory.new(
+            event.get_pdu_json()
+        )
+
+        handled_events = set()
+
         try:
-            event.event_id = self.event_factory.create_event_id()
-            event.origin = self.hs.hostname
-            event.content = content
+            builder.event_id = self.event_builder_factory.create_event_id()
+            builder.origin = self.hs.hostname
+            builder.content = content
 
             if not hasattr(event, "signatures"):
-                event.signatures = {}
+                builder.signatures = {}
 
             add_hashes_and_signatures(
-                event,
+                builder,
                 self.hs.hostname,
                 self.hs.config.signing_key[0],
             )
 
+            new_event = builder.build()
+
             ret = yield self.replication_layer.send_join(
                 target_host,
-                event
+                new_event
             )
 
             state = ret["state"]
             auth_chain = ret["auth_chain"]
             auth_chain.sort(key=lambda e: e.depth)
 
+            handled_events.update([s.event_id for s in state])
+            handled_events.update([a.event_id for a in auth_chain])
+            handled_events.add(new_event.event_id)
+
             logger.debug("do_invite_join auth_chain: %s", auth_chain)
             logger.debug("do_invite_join state: %s", state)
 
-            logger.debug("do_invite_join event: %s", event)
+            logger.debug("do_invite_join event: %s", new_event)
 
             try:
                 yield self.store.store_room(
@@ -373,37 +391,36 @@ class FederationHandler(BaseHandler):
                 pass
 
             for e in auth_chain:
-                e.outlier = True
+                e.internal_metadata.outlier = True
                 try:
-                    yield self._handle_new_event(e, fetch_missing=False)
+                    yield self._handle_new_event(e)
                 except:
                     logger.exception(
-                        "Failed to parse auth event %s",
+                        "Failed to handle auth event %s",
                         e.event_id,
                     )
 
             for e in state:
                 # FIXME: Auth these.
-                e.outlier = True
+                e.internal_metadata.outlier = True
                 try:
                     yield self._handle_new_event(
-                        e,
-                        fetch_missing=True
+                        e, fetch_auth_from=target_host
                     )
                 except:
                     logger.exception(
-                        "Failed to parse state event %s",
+                        "Failed to handle state event %s",
                         e.event_id,
                     )
 
             yield self._handle_new_event(
-                event,
+                new_event,
                 state=state,
                 current_state=state,
             )
 
             yield self.notifier.on_new_room_event(
-                event, extra_users=[joinee]
+                new_event, extra_users=[joinee]
             )
 
             logger.debug("Finished joining %s to %s", joinee, room_id)
@@ -412,6 +429,9 @@ class FederationHandler(BaseHandler):
             del self.room_queues[room_id]
 
             for p, origin in room_queue:
+                if p.event_id in handled_events:
+                    continue
+
                 try:
                     self.on_receive_pdu(origin, p, backfilled=False)
                 except:
@@ -421,25 +441,24 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     @log_function
-    def on_make_join_request(self, context, user_id):
+    def on_make_join_request(self, room_id, user_id):
         """ We've received a /make_join/ request, so we create a partial
         join event for the room and return that. We don *not* persist or
         process it until the other server has signed it and sent it back.
         """
-        event = self.event_factory.create_event(
-            etype=RoomMemberEvent.TYPE,
-            content={"membership": Membership.JOIN},
-            room_id=context,
-            user_id=user_id,
-            state_key=user_id,
+        builder = self.event_builder_factory.new({
+            "type": EventTypes.Member,
+            "content": {"membership": Membership.JOIN},
+            "room_id": room_id,
+            "sender": user_id,
+            "state_key": user_id,
+        })
+
+        event, context = yield self._create_new_client_event(
+            builder=builder,
         )
 
-        snapshot = yield self.store.snapshot_room(event)
-        snapshot.fill_out_prev_events(event)
-
-        yield self.state_handler.annotate_event_with_state(event)
-        yield self.auth.add_auth_events(event)
-        self.auth.check(event, auth_events=event.old_state_events)
+        self.auth.check(event, auth_events=context.auth_events)
 
         pdu = event
 
@@ -453,12 +472,24 @@ class FederationHandler(BaseHandler):
         """
         event = pdu
 
-        event.outlier = False
+        logger.debug(
+            "on_send_join_request: Got event: %s, signatures: %s",
+            event.event_id,
+            event.signatures,
+        )
 
-        yield self._handle_new_event(event)
+        event.internal_metadata.outlier = False
+
+        context = yield self._handle_new_event(event)
+
+        logger.debug(
+            "on_send_join_request: After _handle_new_event: %s, sigs: %s",
+            event.event_id,
+            event.signatures,
+        )
 
         extra_users = []
-        if event.type == RoomMemberEvent.TYPE:
+        if event.type == EventTypes.Member:
             target_user_id = event.state_key
             target_user = self.hs.parse_userid(target_user_id)
             extra_users.append(target_user)
@@ -467,7 +498,7 @@ class FederationHandler(BaseHandler):
             event, extra_users=extra_users
         )
 
-        if event.type == RoomMemberEvent.TYPE:
+        if event.type == EventTypes.Member:
             if event.content["membership"] == Membership.JOIN:
                 user = self.hs.parse_userid(event.state_key)
                 yield self.distributor.fire(
@@ -478,9 +509,9 @@ class FederationHandler(BaseHandler):
 
         destinations = set()
 
-        for k, s in event.state_events.items():
+        for k, s in context.current_state.items():
             try:
-                if k[0] == RoomMemberEvent.TYPE:
+                if k[0] == EventTypes.Member:
                     if s.content["membership"] == Membership.JOIN:
                         destinations.add(
                             self.hs.parse_userid(s.state_key).domain
@@ -490,14 +521,21 @@ class FederationHandler(BaseHandler):
                     "Failed to get destination from event %s", s.event_id
                 )
 
-        new_pdu.destinations = list(destinations)
+        logger.debug(
+            "on_send_join_request: Sending event: %s, signatures: %s",
+            event.event_id,
+            event.signatures,
+        )
 
-        yield self.replication_layer.send_pdu(new_pdu)
+        self.replication_layer.send_pdu(new_pdu, destinations)
 
-        auth_chain = yield self.store.get_auth_chain(event.event_id)
+        state_ids = [e.event_id for e in context.current_state.values()]
+        auth_chain = yield self.store.get_auth_chain(set(
+            [event.event_id] + state_ids
+        ))
 
         defer.returnValue({
-            "state": event.state_events.values(),
+            "state": context.current_state.values(),
             "auth_chain": auth_chain,
         })
 
@@ -509,7 +547,7 @@ class FederationHandler(BaseHandler):
         """
         event = pdu
 
-        event.outlier = True
+        event.internal_metadata.outlier = True
 
         event.signatures.update(
             compute_event_signature(
@@ -519,10 +557,11 @@ class FederationHandler(BaseHandler):
             )
         )
 
-        yield self.state_handler.annotate_event_with_state(event)
+        context = yield self.state_handler.compute_event_context(event)
 
         yield self.store.persist_event(
             event,
+            context=context,
             backfilled=False,
         )
 
@@ -552,13 +591,13 @@ class FederationHandler(BaseHandler):
             }
 
             event = yield self.store.get_event(event_id)
-            if hasattr(event, "state_key"):
+            if event and event.is_state():
                 # Get previous state
-                if hasattr(event, "replaces_state") and event.replaces_state:
-                    prev_event = yield self.store.get_event(
-                        event.replaces_state
-                    )
-                    results[(event.type, event.state_key)] = prev_event
+                if "replaces_state" in event.unsigned:
+                    prev_id = event.unsigned["replaces_state"]
+                    if prev_id != event.event_id:
+                        prev_event = yield self.store.get_event(prev_id)
+                        results[(event.type, event.state_key)] = prev_event
                 else:
                     del results[(event.type, event.state_key)]
 
@@ -578,13 +617,13 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     @log_function
-    def on_backfill_request(self, origin, context, pdu_list, limit):
-        in_room = yield self.auth.check_host_in_room(context, origin)
+    def on_backfill_request(self, origin, room_id, pdu_list, limit):
+        in_room = yield self.auth.check_host_in_room(room_id, origin)
         if not in_room:
             raise AuthError(403, "Host not in room.")
 
         events = yield self.store.get_backfill_events(
-            context,
+            room_id,
             pdu_list,
             limit
         )
@@ -643,75 +682,88 @@ class FederationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def _handle_new_event(self, event, state=None, backfilled=False,
-                          current_state=None, fetch_missing=True):
-        is_new_state = yield self.state_handler.annotate_event_with_state(
-            event,
-            old_state=state
+                          current_state=None, fetch_auth_from=None):
+
+        logger.debug(
+            "_handle_new_event: Before annotate: %s, sigs: %s",
+            event.event_id, event.signatures,
         )
 
-        if event.old_state_events:
-            known_ids = set(
-                [s.event_id for s in event.old_state_events.values()]
-            )
-            for e_id, _ in event.auth_events:
-                if e_id not in known_ids:
-                    e = yield self.store.get_event(
-                        e_id,
-                        allow_none=True,
+        context = yield self.state_handler.compute_event_context(
+            event, old_state=state
+        )
+
+        logger.debug(
+            "_handle_new_event: Before auth fetch: %s, sigs: %s",
+            event.event_id, event.signatures,
+        )
+
+        is_new_state = not event.internal_metadata.is_outlier()
+
+        known_ids = set(
+            [s.event_id for s in context.auth_events.values()]
+        )
+
+        for e_id, _ in event.auth_events:
+            if e_id not in known_ids:
+                e = yield self.store.get_event(e_id, allow_none=True)
+
+                if not e and fetch_auth_from is not None:
+                    # Grab the auth_chain over federation if we are missing
+                    # auth events.
+                    auth_chain = yield self.replication_layer.get_event_auth(
+                        fetch_auth_from, event.event_id, event.room_id
                     )
-
-                    if not e:
-                        # TODO: Do some conflict res to make sure that we're
-                        # not the ones who are wrong.
-                        logger.info(
-                            "Rejecting %s as %s not in %s",
-                            event.event_id, e_id, known_ids,
-                        )
-                        raise AuthError(403, "Auth events are stale")
-
-            auth_events = event.old_state_events
-        else:
-            # We need to get the auth events from somewhere.
-
-            # TODO: Don't just hit the DBs?
-
-            auth_events = {}
-            for e_id, _ in event.auth_events:
-                e = yield self.store.get_event(
-                    e_id,
-                    allow_none=True,
-                )
+                    for auth_event in auth_chain:
+                        yield self._handle_new_event(auth_event)
+                    e = yield self.store.get_event(e_id, allow_none=True)
 
                 if not e:
-                    e = yield self.replication_layer.get_pdu(
-                        event.origin, e_id, outlier=True
+                    # TODO: Do some conflict res to make sure that we're
+                    # not the ones who are wrong.
+                    logger.info(
+                        "Rejecting %s as %s not in db or %s",
+                        event.event_id, e_id, known_ids,
                     )
+                    # FIXME: How does raising AuthError work with federation?
+                    raise AuthError(403, "Cannot find auth event")
 
-                    if e and fetch_missing:
-                        try:
-                            yield self.on_receive_pdu(event.origin, e, False)
-                        except:
-                            logger.exception(
-                                "Failed to parse auth event %s",
-                                e_id,
-                            )
+                context.auth_events[(e.type, e.state_key)] = e
 
-                if not e:
-                    logger.warn("Can't find auth event %s.", e_id)
+        logger.debug(
+            "_handle_new_event: Before hack: %s, sigs: %s",
+            event.event_id, event.signatures,
+        )
 
-                auth_events[(e.type, e.state_key)] = e
+        if event.type == EventTypes.Member and not event.auth_events:
+            if len(event.prev_events) == 1:
+                c = yield self.store.get_event(event.prev_events[0][0])
+                if c.type == EventTypes.Create:
+                    context.auth_events[(c.type, c.state_key)] = c
 
-            if event.type == RoomMemberEvent.TYPE and not event.auth_events:
-                if len(event.prev_events) == 1:
-                    c = yield self.store.get_event(event.prev_events[0][0])
-                    if c.type == RoomCreateEvent.TYPE:
-                        auth_events[(c.type, c.state_key)] = c
+        logger.debug(
+            "_handle_new_event: Before auth check: %s, sigs: %s",
+            event.event_id, event.signatures,
+        )
 
-        self.auth.check(event, auth_events=auth_events)
+        self.auth.check(event, auth_events=context.auth_events)
+
+        logger.debug(
+            "_handle_new_event: Before persist_event: %s, sigs: %s",
+            event.event_id, event.signatures,
+        )
 
         yield self.store.persist_event(
             event,
+            context=context,
             backfilled=backfilled,
             is_new_state=(is_new_state and not backfilled),
             current_state=current_state,
         )
+
+        logger.debug(
+            "_handle_new_event: After persist_event: %s, sigs: %s",
+            event.event_id, event.signatures,
+        )
+
+        defer.returnValue(context)

synapse/handlers/login.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/message.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,10 +15,12 @@
 
 from twisted.internet import defer
 
-from synapse.api.constants import Membership
+from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import RoomError
 from synapse.streams.config import PaginationConfig
+from synapse.events.validator import EventValidator
 from synapse.util.logcontext import PreserveLoggingContext
+
 from ._base import BaseHandler
 
 import logging
@@ -32,7 +34,7 @@ class MessageHandler(BaseHandler):
         super(MessageHandler, self).__init__(hs)
         self.hs = hs
         self.clock = hs.get_clock()
-        self.event_factory = hs.get_event_factory()
+        self.validator = EventValidator()
 
     @defer.inlineCallbacks
     def get_message(self, msg_id=None, room_id=None, sender_id=None,
@@ -63,38 +65,9 @@ class MessageHandler(BaseHandler):
 
         defer.returnValue(None)
 
-    @defer.inlineCallbacks
-    def send_message(self, event=None, suppress_auth=False):
-        """ Send a message.
-
-        Args:
-            event : The message event to store.
-            suppress_auth (bool) : True to suppress auth for this message. This
-            is primarily so the home server can inject messages into rooms at
-            will.
-        Raises:
-            SynapseError if something went wrong.
-        """
-
-        self.ratelimit(event.user_id)
-        # TODO(paul): Why does 'event' not have a 'user' object?
-        user = self.hs.parse_userid(event.user_id)
-        assert user.is_mine, "User must be our own: %s" % (user,)
-
-        snapshot = yield self.store.snapshot_room(event)
-
-        yield self._on_new_room_event(
-            event, snapshot, suppress_auth=suppress_auth
-        )
-
-        with PreserveLoggingContext():
-            self.hs.get_handlers().presence_handler.bump_presence_active_time(
-                user
-            )
-
     @defer.inlineCallbacks
     def get_messages(self, user_id=None, room_id=None, pagin_config=None,
-                     feedback=False):
+                     feedback=False, as_client_event=True):
         """Get messages in a room.
 
         Args:
@@ -103,6 +76,7 @@ class MessageHandler(BaseHandler):
             pagin_config (synapse.api.streams.PaginationConfig): The pagination
             config rules to apply, if any.
             feedback (bool): True to get compressed feedback with the messages
+            as_client_event (bool): True to get events in client-server format.
         Returns:
             dict: Pagination API results
         """
@@ -126,7 +100,9 @@ class MessageHandler(BaseHandler):
         )
 
         chunk = {
-            "chunk": [self.hs.serialize_event(e) for e in events],
+            "chunk": [
+                self.hs.serialize_event(e, as_client_event) for e in events
+            ],
             "start": pagin_config.from_token.to_string(),
             "end": next_token.to_string(),
         }
@@ -134,19 +110,59 @@ class MessageHandler(BaseHandler):
         defer.returnValue(chunk)
 
     @defer.inlineCallbacks
-    def store_room_data(self, event=None):
-        """ Stores data for a room.
+    def create_and_send_event(self, event_dict, ratelimit=True):
+        """ Given a dict from a client, create and handle a new event.
+
+        Creates an FrozenEvent object, filling out auth_events, prev_events,
+        etc.
+
+        Adds display names to Join membership events.
+
+        Persists and notifies local clients and federation.
 
         Args:
-            event : The room path event
-            stamp_event (bool) : True to stamp event content with server keys.
-        Raises:
-            SynapseError if something went wrong.
+            event_dict (dict): An entire event
         """
+        builder = self.event_builder_factory.new(event_dict)
 
-        snapshot = yield self.store.snapshot_room(event)
+        self.validator.validate_new(builder)
 
-        yield self._on_new_room_event(event, snapshot)
+        if ratelimit:
+            self.ratelimit(builder.user_id)
+        # TODO(paul): Why does 'event' not have a 'user' object?
+        user = self.hs.parse_userid(builder.user_id)
+        assert self.hs.is_mine(user), "User must be our own: %s" % (user,)
+
+        if builder.type == EventTypes.Member:
+            membership = builder.content.get("membership", None)
+            if membership == Membership.JOIN:
+                joinee = self.hs.parse_userid(builder.state_key)
+                # If event doesn't include a display name, add one.
+                yield self.distributor.fire(
+                    "collect_presencelike_data",
+                    joinee,
+                    builder.content
+                )
+
+        event, context = yield self._create_new_client_event(
+            builder=builder,
+        )
+
+        if event.type == EventTypes.Member:
+            member_handler = self.hs.get_handlers().room_member_handler
+            yield member_handler.change_membership(event, context)
+        else:
+            yield self.handle_new_client_event(
+                event=event,
+                context=context,
+            )
+
+        if event.type == EventTypes.Message:
+            presence = self.hs.get_handlers().presence_handler
+            with PreserveLoggingContext():
+                presence.bump_presence_active_time(user)
+
+        defer.returnValue(event)
 
     @defer.inlineCallbacks
     def get_room_data(self, user_id=None, room_id=None,
@@ -180,13 +196,6 @@ class MessageHandler(BaseHandler):
             defer.returnValue(fb)
         defer.returnValue(None)
 
-    @defer.inlineCallbacks
-    def send_feedback(self, event):
-        snapshot = yield self.store.snapshot_room(event)
-
-        # store message in db
-        yield self._on_new_room_event(event, snapshot)
-
     @defer.inlineCallbacks
     def get_state_events(self, user_id, room_id):
         """Retrieve all state events for a given room.
@@ -205,7 +214,7 @@ class MessageHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def snapshot_all_rooms(self, user_id=None, pagin_config=None,
-                           feedback=False):
+                           feedback=False, as_client_event=True):
         """Retrieve a snapshot of all rooms the user is invited or has joined.
 
         This snapshot may include messages for all rooms where the user is
@@ -216,6 +225,7 @@ class MessageHandler(BaseHandler):
             pagin_config (synapse.api.streams.PaginationConfig): The pagination
             config used to determine how many messages *PER ROOM* to return.
             feedback (bool): True to get feedback along with these messages.
+            as_client_event (bool): True to get events in client-server format.
         Returns:
             A list of dicts with "room_id" and "membership" keys for all rooms
             the user is currently invited or joined in on. Rooms where the user
@@ -257,7 +267,7 @@ class MessageHandler(BaseHandler):
             }
 
             if event.membership == Membership.INVITE:
-                d["inviter"] = event.user_id
+                d["inviter"] = event.sender
 
             rooms_ret.append(d)
 
@@ -274,7 +284,10 @@ class MessageHandler(BaseHandler):
                 end_token = now_token.copy_and_replace("room_key", token[1])
 
                 d["messages"] = {
-                    "chunk": [self.hs.serialize_event(m) for m in messages],
+                    "chunk": [
+                        self.hs.serialize_event(m, as_client_event)
+                        for m in messages
+                    ],
                     "start": start_token.to_string(),
                     "end": end_token.to_string(),
                 }

synapse/handlers/presence.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -147,7 +147,7 @@ class PresenceHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def is_presence_visible(self, observer_user, observed_user):
-        assert(observed_user.is_mine)
+        assert(self.hs.is_mine(observed_user))
 
         if observer_user == observed_user:
             defer.returnValue(True)
@@ -165,7 +165,7 @@ class PresenceHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def get_state(self, target_user, auth_user, as_event=False):
-        if target_user.is_mine:
+        if self.hs.is_mine(target_user):
             visible = yield self.is_presence_visible(
                 observer_user=auth_user,
                 observed_user=target_user
@@ -212,7 +212,7 @@ class PresenceHandler(BaseHandler):
         # TODO (erikj): Turn this back on. Why did we end up sending EDUs
         # everywhere?
 
-        if not target_user.is_mine:
+        if not self.hs.is_mine(target_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         if target_user != auth_user:
@@ -291,7 +291,7 @@ class PresenceHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def user_joined_room(self, user, room_id):
-        if user.is_mine:
+        if self.hs.is_mine(user):
             statuscache = self._get_or_make_usercache(user)
 
             # No actual update but we need to bump the serial anyway for the
@@ -309,7 +309,7 @@ class PresenceHandler(BaseHandler):
         rm_handler = self.homeserver.get_handlers().room_member_handler
         curr_users = yield rm_handler.get_room_members(room_id)
 
-        for local_user in [c for c in curr_users if c.is_mine]:
+        for local_user in [c for c in curr_users if self.hs.is_mine(c)]:
             self.push_update_to_local_and_remote(
                 observed_user=local_user,
                 users_to_push=[user],
@@ -318,14 +318,14 @@ class PresenceHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def send_invite(self, observer_user, observed_user):
-        if not observer_user.is_mine:
+        if not self.hs.is_mine(observer_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         yield self.store.add_presence_list_pending(
             observer_user.localpart, observed_user.to_string()
         )
 
-        if observed_user.is_mine:
+        if self.hs.is_mine(observed_user):
             yield self.invite_presence(observed_user, observer_user)
         else:
             yield self.federation.send_edu(
@@ -339,7 +339,7 @@ class PresenceHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def _should_accept_invite(self, observed_user, observer_user):
-        if not observed_user.is_mine:
+        if not self.hs.is_mine(observed_user):
             defer.returnValue(False)
 
         row = yield self.store.has_presence_state(observed_user.localpart)
@@ -359,7 +359,7 @@ class PresenceHandler(BaseHandler):
                 observed_user.localpart, observer_user.to_string()
             )
 
-        if observer_user.is_mine:
+        if self.hs.is_mine(observer_user):
             if accept:
                 yield self.accept_presence(observed_user, observer_user)
             else:
@@ -396,7 +396,7 @@ class PresenceHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def drop(self, observed_user, observer_user):
-        if not observer_user.is_mine:
+        if not self.hs.is_mine(observer_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         yield self.store.del_presence_list(
@@ -410,7 +410,7 @@ class PresenceHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def get_presence_list(self, observer_user, accepted=None):
-        if not observer_user.is_mine:
+        if not self.hs.is_mine(observer_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         presence = yield self.store.get_presence_list(
@@ -465,7 +465,7 @@ class PresenceHandler(BaseHandler):
         )
 
         for target_user in target_users:
-            if target_user.is_mine:
+            if self.hs.is_mine(target_user):
                 self._start_polling_local(user, target_user)
 
                 # We want to tell the person that just came online
@@ -477,7 +477,7 @@ class PresenceHandler(BaseHandler):
                 )
 
         deferreds = []
-        remote_users = [u for u in target_users if not u.is_mine]
+        remote_users = [u for u in target_users if not self.hs.is_mine(u)]
         remoteusers_by_domain = partition(remote_users, lambda u: u.domain)
         # Only poll for people in our get_presence_list
         for domain in remoteusers_by_domain:
@@ -520,7 +520,7 @@ class PresenceHandler(BaseHandler):
     def stop_polling_presence(self, user, target_user=None):
         logger.debug("Stop polling for presence from %s", user)
 
-        if not target_user or target_user.is_mine:
+        if not target_user or self.hs.is_mine(target_user):
             self._stop_polling_local(user, target_user=target_user)
 
         deferreds = []
@@ -579,7 +579,7 @@ class PresenceHandler(BaseHandler):
     @defer.inlineCallbacks
     @log_function
     def push_presence(self, user, statuscache):
-        assert(user.is_mine)
+        assert(self.hs.is_mine(user))
 
         logger.debug("Pushing presence update from %s", user)
 
@@ -659,10 +659,6 @@ class PresenceHandler(BaseHandler):
             if room_ids:
                 logger.debug(" | %d interested room IDs %r", len(room_ids), room_ids)
 
-            if not observers and not room_ids:
-                logger.debug(" | no interested observers or room IDs")
-                continue
-
             state = dict(push)
             del state["user_id"]
 
@@ -683,6 +679,10 @@ class PresenceHandler(BaseHandler):
             self._user_cachemap_latest_serial += 1
             statuscache.update(state, serial=self._user_cachemap_latest_serial)
 
+            if not observers and not room_ids:
+                logger.debug(" | no interested observers or room IDs")
+                continue
+
             self.push_update_to_clients(
                 observed_user=user,
                 users_to_push=observers,
@@ -696,7 +696,7 @@ class PresenceHandler(BaseHandler):
         for poll in content.get("poll", []):
             user = self.hs.parse_userid(poll)
 
-            if not user.is_mine:
+            if not self.hs.is_mine(user):
                 continue
 
             # TODO(paul) permissions checks
@@ -711,7 +711,7 @@ class PresenceHandler(BaseHandler):
         for unpoll in content.get("unpoll", []):
             user = self.hs.parse_userid(unpoll)
 
-            if not user.is_mine:
+            if not self.hs.is_mine(user):
                 continue
 
             if user in self._remote_sendmap:
@@ -730,7 +730,7 @@ class PresenceHandler(BaseHandler):
 
         localusers, remoteusers = partitionbool(
             users_to_push,
-            lambda u: u.is_mine
+            lambda u: self.hs.is_mine(u)
         )
 
         localusers = set(localusers)
@@ -788,7 +788,7 @@ class PresenceEventSource(object):
                 [u.to_string() for u in observer_user, observed_user])):
             defer.returnValue(True)
 
-        if observed_user.is_mine:
+        if self.hs.is_mine(observed_user):
             pushmap = presence._local_pushmap
 
             defer.returnValue(
@@ -804,6 +804,7 @@ class PresenceEventSource(object):
             )
 
     @defer.inlineCallbacks
+    @log_function
     def get_new_events_for_user(self, user, from_key, limit):
         from_key = int(from_key)
 
@@ -816,7 +817,8 @@ class PresenceEventSource(object):
         # TODO(paul): use a DeferredList ? How to limit concurrency.
         for observed_user in cachemap.keys():
             cached = cachemap[observed_user]
-            if not (from_key < cached.serial):
+
+            if cached.serial <= from_key:
                 continue
 
             if (yield self.is_visible(observer_user, observed_user)):

synapse/handlers/profile.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,7 +16,7 @@
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError, AuthError, CodeMessageException
-from synapse.api.constants import Membership
+from synapse.api.constants import EventTypes, Membership
 from synapse.util.logcontext import PreserveLoggingContext
 
 from ._base import BaseHandler
@@ -51,7 +51,7 @@ class ProfileHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def get_displayname(self, target_user):
-        if target_user.is_mine:
+        if self.hs.is_mine(target_user):
             displayname = yield self.store.get_profile_displayname(
                 target_user.localpart
             )
@@ -81,7 +81,7 @@ class ProfileHandler(BaseHandler):
     def set_displayname(self, target_user, auth_user, new_displayname):
         """target_user is the user whose displayname is to be changed;
         auth_user is the user attempting to make this change."""
-        if not target_user.is_mine:
+        if not self.hs.is_mine(target_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         if target_user != auth_user:
@@ -101,7 +101,7 @@ class ProfileHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def get_avatar_url(self, target_user):
-        if target_user.is_mine:
+        if self.hs.is_mine(target_user):
             avatar_url = yield self.store.get_profile_avatar_url(
                 target_user.localpart
             )
@@ -130,7 +130,7 @@ class ProfileHandler(BaseHandler):
     def set_avatar_url(self, target_user, auth_user, new_avatar_url):
         """target_user is the user whose avatar_url is to be changed;
         auth_user is the user attempting to make this change."""
-        if not target_user.is_mine:
+        if not self.hs.is_mine(target_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         if target_user != auth_user:
@@ -150,7 +150,7 @@ class ProfileHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def collect_presencelike_data(self, user, state):
-        if not user.is_mine:
+        if not self.hs.is_mine(user):
             defer.returnValue(None)
 
         with PreserveLoggingContext():
@@ -170,7 +170,7 @@ class ProfileHandler(BaseHandler):
     @defer.inlineCallbacks
     def on_profile_query(self, args):
         user = self.hs.parse_userid(args["user_id"])
-        if not user.is_mine:
+        if not self.hs.is_mine(user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         just_field = args.get("field", None)
@@ -191,33 +191,30 @@ class ProfileHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def _update_join_states(self, user):
-        if not user.is_mine:
+        if not self.hs.is_mine(user):
             return
 
+        self.ratelimit(user.to_string())
+
         joins = yield self.store.get_rooms_for_user_where_membership_is(
             user.to_string(),
             [Membership.JOIN],
         )
 
         for j in joins:
-            snapshot = yield self.store.snapshot_room(j)
-
             content = {
-                "membership": j.content["membership"],
+                "membership": Membership.JOIN,
             }
 
             yield self.distributor.fire(
                 "collect_presencelike_data", user, content
             )
 
-            new_event = self.event_factory.create_event(
-                etype=j.type,
-                room_id=j.room_id,
-                state_key=j.state_key,
-                content=content,
-                user_id=j.state_key,
-            )
-
-            yield self._on_new_room_event(
-                new_event, snapshot, suppress_auth=True
-            )
+            msg_handler = self.hs.get_handlers().message_handler
+            yield msg_handler.create_and_send_event({
+                "type": EventTypes.Member,
+                "room_id": j.room_id,
+                "state_key": user.to_string(),
+                "content": content,
+                "sender": user.to_string()
+            }, ratelimit=False)

synapse/handlers/register.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -22,6 +22,7 @@ from synapse.api.errors import (
 )
 from ._base import BaseHandler
 import synapse.util.stringutils as stringutils
+from synapse.util.async import run_on_reactor
 from synapse.http.client import SimpleHttpClient
 from synapse.http.client import CaptchaServerHttpClient
 
@@ -54,12 +55,13 @@ class RegistrationHandler(BaseHandler):
         Raises:
             RegistrationError if there was a problem registering.
         """
+        yield run_on_reactor()
         password_hash = None
         if password:
             password_hash = bcrypt.hashpw(password, bcrypt.gensalt())
 
         if localpart:
-            user = UserID(localpart, self.hs.hostname, True)
+            user = UserID(localpart, self.hs.hostname)
             user_id = user.to_string()
 
             token = self._generate_token(user_id)
@@ -78,7 +80,7 @@ class RegistrationHandler(BaseHandler):
             while not user_id and not token:
                 try:
                     localpart = self._generate_user_id()
-                    user = UserID(localpart, self.hs.hostname, True)
+                    user = UserID(localpart, self.hs.hostname)
                     user_id = user.to_string()
 
                     token = self._generate_token(user_id)

synapse/handlers/room.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,12 +17,8 @@
 from twisted.internet import defer
 
 from synapse.types import UserID, RoomAlias, RoomID
-from synapse.api.constants import Membership, JoinRules
+from synapse.api.constants import EventTypes, Membership, JoinRules
 from synapse.api.errors import StoreError, SynapseError
-from synapse.api.events.room import (
-    RoomMemberEvent, RoomCreateEvent, RoomPowerLevelsEvent,
-    RoomTopicEvent, RoomNameEvent, RoomJoinRulesEvent,
-)
 from synapse.util import stringutils
 from synapse.util.async import run_on_reactor
 from ._base import BaseHandler
@@ -52,9 +48,9 @@ class RoomCreationHandler(BaseHandler):
         self.ratelimit(user_id)
 
         if "room_alias_name" in config:
-            room_alias = RoomAlias.create_local(
+            room_alias = RoomAlias.create(
                 config["room_alias_name"],
-                self.hs
+                self.hs.hostname,
             )
             mapping = yield self.store.get_association_from_room_alias(
                 room_alias
@@ -76,8 +72,8 @@ class RoomCreationHandler(BaseHandler):
 
         if room_id:
             # Ensure room_id is the correct type
-            room_id_obj = RoomID.from_string(room_id, self.hs)
-            if not room_id_obj.is_mine:
+            room_id_obj = RoomID.from_string(room_id)
+            if not self.hs.is_mine(room_id_obj):
                 raise SynapseError(400, "Room id must be local")
 
             yield self.store.store_room(
@@ -93,7 +89,10 @@ class RoomCreationHandler(BaseHandler):
             while attempts < 5:
                 try:
                     random_string = stringutils.random_string(18)
-                    gen_room_id = RoomID.create_local(random_string, self.hs)
+                    gen_room_id = RoomID.create(
+                        random_string,
+                        self.hs.hostname,
+                    )
                     yield self.store.store_room(
                         room_id=gen_room_id.to_string(),
                         room_creator_user_id=user_id,
@@ -120,59 +119,39 @@ class RoomCreationHandler(BaseHandler):
             user, room_id, is_public=is_public
         )
 
-        room_member_handler = self.hs.get_handlers().room_member_handler
-
-        @defer.inlineCallbacks
-        def handle_event(event):
-            snapshot = yield self.store.snapshot_room(event)
-
-            logger.debug("Event: %s", event)
-
-            if event.type == RoomMemberEvent.TYPE:
-                yield room_member_handler.change_membership(
-                    event,
-                    do_auth=True
-                )
-            else:
-                yield self._on_new_room_event(
-                    event, snapshot, extra_users=[user], suppress_auth=True
-                )
+        msg_handler = self.hs.get_handlers().message_handler
 
         for event in creation_events:
-            yield handle_event(event)
+            yield msg_handler.create_and_send_event(event)
 
         if "name" in config:
             name = config["name"]
-            name_event = self.event_factory.create_event(
-                etype=RoomNameEvent.TYPE,
-                room_id=room_id,
-                user_id=user_id,
-                content={"name": name},
-            )
-
-            yield handle_event(name_event)
+            yield msg_handler.create_and_send_event({
+                "type": EventTypes.Name,
+                "room_id": room_id,
+                "sender": user_id,
+                "state_key": "",
+                "content": {"name": name},
+            })
 
         if "topic" in config:
             topic = config["topic"]
-            topic_event = self.event_factory.create_event(
-                etype=RoomTopicEvent.TYPE,
-                room_id=room_id,
-                user_id=user_id,
-                content={"topic": topic},
-            )
+            yield msg_handler.create_and_send_event({
+                "type": EventTypes.Topic,
+                "room_id": room_id,
+                "sender": user_id,
+                "state_key": "",
+                "content": {"topic": topic},
+            })
 
-            yield handle_event(topic_event)
-
-        content = {"membership": Membership.INVITE}
         for invitee in invite_list:
-            invite_event = self.event_factory.create_event(
-                etype=RoomMemberEvent.TYPE,
-                state_key=invitee,
-                room_id=room_id,
-                user_id=user_id,
-                content=content
-            )
-            yield handle_event(invite_event)
+            yield msg_handler.create_and_send_event({
+                "type": EventTypes.Member,
+                "state_key": invitee,
+                "room_id": room_id,
+                "sender": user_id,
+                "content": {"membership": Membership.INVITE},
+            })
 
         result = {"room_id": room_id}
 
@@ -189,40 +168,44 @@ class RoomCreationHandler(BaseHandler):
 
         event_keys = {
             "room_id": room_id,
-            "user_id": creator_id,
+            "sender": creator_id,
+            "state_key": "",
         }
 
-        def create(etype, **content):
-            return self.event_factory.create_event(
-                etype=etype,
-                content=content,
-                **event_keys
-            )
+        def create(etype, content, **kwargs):
+            e = {
+                "type": etype,
+                "content": content,
+            }
+
+            e.update(event_keys)
+            e.update(kwargs)
+
+            return e
 
         creation_event = create(
-            etype=RoomCreateEvent.TYPE,
-            creator=creator.to_string(),
+            etype=EventTypes.Create,
+            content={"creator": creator.to_string()},
         )
 
-        join_event = self.event_factory.create_event(
-            etype=RoomMemberEvent.TYPE,
+        join_event = create(
+            etype=EventTypes.Member,
             state_key=creator_id,
             content={
                 "membership": Membership.JOIN,
             },
-            **event_keys
         )
 
-        power_levels_event = self.event_factory.create_event(
-            etype=RoomPowerLevelsEvent.TYPE,
+        power_levels_event = create(
+            etype=EventTypes.PowerLevels,
             content={
                 "users": {
                     creator.to_string(): 100,
                 },
                 "users_default": 0,
                 "events": {
-                    RoomNameEvent.TYPE: 100,
-                    RoomPowerLevelsEvent.TYPE: 100,
+                    EventTypes.Name: 100,
+                    EventTypes.PowerLevels: 100,
                 },
                 "events_default": 0,
                 "state_default": 50,
@@ -230,13 +213,12 @@ class RoomCreationHandler(BaseHandler):
                 "kick": 50,
                 "redact": 50
             },
-            **event_keys
         )
 
         join_rule = JoinRules.PUBLIC if is_public else JoinRules.INVITE
         join_rules_event = create(
-            etype=RoomJoinRulesEvent.TYPE,
-            join_rule=join_rule,
+            etype=EventTypes.JoinRules,
+            content={"join_rule": join_rule},
         )
 
         return [
@@ -260,16 +242,15 @@ class RoomMemberHandler(BaseHandler):
 
         self.distributor = hs.get_distributor()
         self.distributor.declare("user_joined_room")
+        self.distributor.declare("user_left_room")
 
     @defer.inlineCallbacks
-    def get_room_members(self, room_id, membership=Membership.JOIN):
+    def get_room_members(self, room_id):
         hs = self.hs
 
-        memberships = yield self.store.get_room_members(
-            room_id=room_id, membership=membership
-        )
+        users = yield self.store.get_users_in_room(room_id)
 
-        defer.returnValue([hs.parse_userid(m.user_id) for m in memberships])
+        defer.returnValue([hs.parse_userid(u) for u in users])
 
     @defer.inlineCallbacks
     def fetch_room_distributions_into(self, room_id, localusers=None,
@@ -287,7 +268,7 @@ class RoomMemberHandler(BaseHandler):
             if ignore_user is not None and member == ignore_user:
                 continue
 
-            if member.is_mine:
+            if self.hs.is_mine(member):
                 if localusers is not None:
                     localusers.add(member)
             else:
@@ -348,7 +329,7 @@ class RoomMemberHandler(BaseHandler):
         defer.returnValue(member)
 
     @defer.inlineCallbacks
-    def change_membership(self, event=None, do_auth=True):
+    def change_membership(self, event, context, do_auth=True):
         """ Change the membership status of a user in a room.
 
         Args:
@@ -358,11 +339,9 @@ class RoomMemberHandler(BaseHandler):
         """
         target_user_id = event.state_key
 
-        snapshot = yield self.store.snapshot_room(event)
-
-        ## TODO(markjh): get prev state from snapshot.
-        prev_state = yield self.store.get_room_member(
-            target_user_id, event.room_id
+        prev_state = context.current_state.get(
+            (EventTypes.Member, target_user_id),
+            None
         )
 
         room_id = event.room_id
@@ -371,10 +350,11 @@ class RoomMemberHandler(BaseHandler):
         # if this HS is not currently in the room, i.e. we have to do the
         # invite/join dance.
         if event.membership == Membership.JOIN:
-            yield self._do_join(event, snapshot, do_auth=do_auth)
+            yield self._do_join(event, context, do_auth=do_auth)
         else:
             # This is not a JOIN, so we can handle it normally.
 
+            # FIXME: This isn't idempotency.
             if prev_state and prev_state.membership == event.membership:
                 # double same action, treat this event as a NOOP.
                 defer.returnValue({})
@@ -383,10 +363,16 @@ class RoomMemberHandler(BaseHandler):
             yield self._do_local_membership_update(
                 event,
                 membership=event.content["membership"],
-                snapshot=snapshot,
+                context=context,
                 do_auth=do_auth,
             )
 
+            if prev_state and prev_state.membership == Membership.JOIN:
+                user = self.hs.parse_userid(event.user_id)
+                self.distributor.fire(
+                    "user_left_room", user=user, room_id=event.room_id
+                )
+
         defer.returnValue({"room_id": room_id})
 
     @defer.inlineCallbacks
@@ -404,33 +390,32 @@ class RoomMemberHandler(BaseHandler):
 
         host = hosts[0]
 
-        content.update({"membership": Membership.JOIN})
-        new_event = self.event_factory.create_event(
-            etype=RoomMemberEvent.TYPE,
-            state_key=joinee.to_string(),
-            room_id=room_id,
-            user_id=joinee.to_string(),
-            membership=Membership.JOIN,
-            content=content,
+        # If event doesn't include a display name, add one.
+        yield self.distributor.fire(
+            "collect_presencelike_data", joinee, content
         )
 
-        snapshot = yield self.store.snapshot_room(new_event)
+        content.update({"membership": Membership.JOIN})
+        builder = self.event_builder_factory.new({
+            "type": EventTypes.Member,
+            "state_key": joinee.to_string(),
+            "room_id": room_id,
+            "sender": joinee.to_string(),
+            "membership": Membership.JOIN,
+            "content": content,
+        })
+        event, context = yield self._create_new_client_event(builder)
 
-        yield self._do_join(new_event, snapshot, room_host=host, do_auth=True)
+        yield self._do_join(event, context, room_host=host, do_auth=True)
 
         defer.returnValue({"room_id": room_id})
 
     @defer.inlineCallbacks
-    def _do_join(self, event, snapshot, room_host=None, do_auth=True):
+    def _do_join(self, event, context, room_host=None, do_auth=True):
         joinee = self.hs.parse_userid(event.state_key)
         # room_id = RoomID.from_string(event.room_id, self.hs)
         room_id = event.room_id
 
-        # If event doesn't include a display name, add one.
-        yield self.distributor.fire(
-            "collect_presencelike_data", joinee, event.content
-        )
-
         # XXX: We don't do an auth check if we are doing an invite
         # join dance for now, since we're kinda implicitly checking
         # that we are allowed to join when we decide whether or not we
@@ -440,10 +425,22 @@ class RoomMemberHandler(BaseHandler):
             event.room_id,
             self.hs.hostname
         )
+        if not is_host_in_room:
+            # is *anyone* in the room?
+            room_member_keys = [
+                v for (k, v) in context.current_state.keys() if (
+                    k == "m.room.member"
+                )
+            ]
+            if len(room_member_keys) == 0:
+                # has the room been created so we can join it?
+                create_event = context.current_state.get(("m.room.create", ""))
+                if create_event:
+                    is_host_in_room = True
 
         if is_host_in_room:
             should_do_dance = False
-        elif room_host:
+        elif room_host:  # TODO: Shouldn't this be remote_room_host?
             should_do_dance = True
         else:
             # TODO(markjh): get prev_state from snapshot
@@ -452,31 +449,30 @@ class RoomMemberHandler(BaseHandler):
             )
 
             if prev_state and prev_state.membership == Membership.INVITE:
-                room = yield self.store.get_room(room_id)
-                inviter = UserID.from_string(
-                    prev_state.user_id, self.hs
-                )
+                inviter = UserID.from_string(prev_state.user_id)
 
-                should_do_dance = not inviter.is_mine and not room
+                should_do_dance = not self.hs.is_mine(inviter)
                 room_host = inviter.domain
             else:
-                should_do_dance = False
+                # return the same error as join_room_alias does
+                raise SynapseError(404, "No known servers")
 
-        have_joined = False
         if should_do_dance:
             handler = self.hs.get_handlers().federation_handler
-            have_joined = yield handler.do_invite_join(
-                room_host, room_id, event.user_id, event.content, snapshot
+            yield handler.do_invite_join(
+                room_host,
+                room_id,
+                event.user_id,
+                event.get_dict()["content"],  # FIXME To get a non-frozen dict
+                context
             )
-
-        # We want to do the _do_update inside the room lock.
-        if not have_joined:
+        else:
             logger.debug("Doing normal join")
 
             yield self._do_local_membership_update(
                 event,
                 membership=event.content["membership"],
-                snapshot=snapshot,
+                context=context,
                 do_auth=do_auth,
             )
 
@@ -501,10 +497,10 @@ class RoomMemberHandler(BaseHandler):
         if prev_state and prev_state.membership == Membership.INVITE:
             room = yield self.store.get_room(room_id)
             inviter = UserID.from_string(
-                prev_state.sender, self.hs
+                prev_state.sender
             )
 
-            is_remote_invite_join = not inviter.is_mine and not room
+            is_remote_invite_join = not self.hs.is_mine(inviter) and not room
             room_host = inviter.domain
         else:
             is_remote_invite_join = False
@@ -526,25 +522,17 @@ class RoomMemberHandler(BaseHandler):
         defer.returnValue(room_ids)
 
     @defer.inlineCallbacks
-    def _do_local_membership_update(self, event, membership, snapshot,
+    def _do_local_membership_update(self, event, membership, context,
                                     do_auth):
         yield run_on_reactor()
 
-        # If we're inviting someone, then we should also send it to that
-        # HS.
-        target_user_id = event.state_key
-        target_user = self.hs.parse_userid(target_user_id)
-        if membership == Membership.INVITE and not target_user.is_mine:
-            do_invite_host = target_user.domain
-        else:
-            do_invite_host = None
+        target_user = self.hs.parse_userid(event.state_key)
 
-        yield self._on_new_room_event(
+        yield self.handle_new_client_event(
             event,
-            snapshot,
+            context,
             extra_users=[target_user],
             suppress_auth=(not do_auth),
-            do_invite_host=do_invite_host,
         )
 
 
@@ -554,11 +542,10 @@ class RoomListHandler(BaseHandler):
     def get_public_room_list(self):
         chunk = yield self.store.get_rooms(is_public=True)
         for room in chunk:
-            joined_members = yield self.store.get_room_members(
+            joined_users = yield self.store.get_users_in_room(
                 room_id=room["room_id"],
-                membership=Membership.JOIN
             )
-            room["num_joined_members"] = len(joined_members)
+            room["num_joined_members"] = len(joined_users)
         # FIXME (erikj): START is no longer a valid value
         defer.returnValue({"start": "START", "end": "END", "chunk": chunk})
 

synapse/handlers/typing.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -43,22 +43,56 @@ class TypingNotificationHandler(BaseHandler):
 
         self.federation.register_edu_handler("m.typing", self._recv_edu)
 
-        self._member_typing_until = {}
+        hs.get_distributor().observe("user_left_room", self.user_left_room)
+
+        self._member_typing_until = {}  # clock time we expect to stop
+        self._member_typing_timer = {}  # deferreds to manage theabove
+
+        # map room IDs to serial numbers
+        self._room_serials = {}
+        self._latest_room_serial = 0
+        # map room IDs to sets of users currently typing
+        self._room_typing = {}
+
+    def tearDown(self):
+        """Cancels all the pending timers.
+        Normally this shouldn't be needed, but it's required from unit tests
+        to avoid a "Reactor was unclean" warning."""
+        for t in self._member_typing_timer.values():
+            self.clock.cancel_call_later(t)
 
     @defer.inlineCallbacks
     def started_typing(self, target_user, auth_user, room_id, timeout):
-        if not target_user.is_mine:
+        if not self.hs.is_mine(target_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         if target_user != auth_user:
             raise AuthError(400, "Cannot set another user's typing state")
 
+        yield self.auth.check_joined_room(room_id, target_user.to_string())
+
+        logger.debug(
+            "%s has started typing in %s", target_user.to_string(), room_id
+        )
+
         until = self.clock.time_msec() + timeout
         member = RoomMember(room_id=room_id, user=target_user)
 
         was_present = member in self._member_typing_until
 
+        if member in self._member_typing_timer:
+            self.clock.cancel_call_later(self._member_typing_timer[member])
+
+        def _cb():
+            logger.debug(
+                "%s has timed out in %s", target_user.to_string(), room_id
+            )
+            self._stopped_typing(member)
+
         self._member_typing_until[member] = until
+        self._member_typing_timer[member] = self.clock.call_later(
+            timeout / 1000.0, _cb
+        )
 
         if was_present:
             # No point sending another notification
@@ -72,24 +106,51 @@ class TypingNotificationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def stopped_typing(self, target_user, auth_user, room_id):
-        if not target_user.is_mine:
+        if not self.hs.is_mine(target_user):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         if target_user != auth_user:
             raise AuthError(400, "Cannot set another user's typing state")
 
+        yield self.auth.check_joined_room(room_id, target_user.to_string())
+
+        logger.debug(
+            "%s has stopped typing in %s", target_user.to_string(), room_id
+        )
+
         member = RoomMember(room_id=room_id, user=target_user)
 
+        if member in self._member_typing_timer:
+            self.clock.cancel_call_later(self._member_typing_timer[member])
+            del self._member_typing_timer[member]
+
+        yield self._stopped_typing(member)
+
+    @defer.inlineCallbacks
+    def user_left_room(self, user, room_id):
+        if self.hs.is_mine(user):
+            member = RoomMember(room_id=room_id, user=user)
+            yield self._stopped_typing(member)
+
+    @defer.inlineCallbacks
+    def _stopped_typing(self, member):
         if member not in self._member_typing_until:
             # No point
             defer.returnValue(None)
 
         yield self._push_update(
-            room_id=room_id,
-            user=target_user,
+            room_id=member.room_id,
+            user=member.user,
             typing=False,
         )
 
+        del self._member_typing_until[member]
+
+        if member in self._member_typing_timer:
+            # Don't cancel it - either it already expired, or the real
+            # stopped_typing() will cancel it
+            del self._member_typing_timer[member]
+
     @defer.inlineCallbacks
     def _push_update(self, room_id, user, typing):
         localusers = set()
@@ -97,16 +158,14 @@ class TypingNotificationHandler(BaseHandler):
 
         rm_handler = self.homeserver.get_handlers().room_member_handler
         yield rm_handler.fetch_room_distributions_into(
-            room_id, localusers=localusers, remotedomains=remotedomains,
-            ignore_user=user
+            room_id, localusers=localusers, remotedomains=remotedomains
         )
 
-        for u in localusers:
-            self.push_update_to_clients(
+        if localusers:
+            self._push_update_local(
                 room_id=room_id,
-                observer_user=u,
-                observed_user=user,
-                typing=typing,
+                user=user,
+                typing=typing
             )
 
         deferreds = []
@@ -135,29 +194,67 @@ class TypingNotificationHandler(BaseHandler):
             room_id, localusers=localusers
         )
 
-        for u in localusers:
-            self.push_update_to_clients(
+        if localusers:
+            self._push_update_local(
                 room_id=room_id,
-                observer_user=u,
-                observed_user=user,
+                user=user,
                 typing=content["typing"]
             )
 
-    def push_update_to_clients(self, room_id, observer_user, observed_user,
-                               typing):
-        # TODO(paul) steal this from presence.py
-        pass
+    def _push_update_local(self, room_id, user, typing):
+        if room_id not in self._room_serials:
+            self._room_serials[room_id] = 0
+            self._room_typing[room_id] = set()
+
+        room_set = self._room_typing[room_id]
+        if typing:
+            room_set.add(user)
+        elif user in room_set:
+            room_set.remove(user)
+
+        self._latest_room_serial += 1
+        self._room_serials[room_id] = self._latest_room_serial
+
+        self.notifier.on_new_user_event(rooms=[room_id])
 
 
 class TypingNotificationEventSource(object):
     def __init__(self, hs):
         self.hs = hs
+        self._handler = None
+
+    def handler(self):
+        # Avoid cyclic dependency in handler setup
+        if not self._handler:
+            self._handler = self.hs.get_handlers().typing_notification_handler
+        return self._handler
+
+    def _make_event_for(self, room_id):
+        typing = self.handler()._room_typing[room_id]
+        return {
+            "type": "m.typing",
+            "room_id": room_id,
+            "content": {
+                "user_ids": [u.to_string() for u in typing],
+            },
+        }
 
     def get_new_events_for_user(self, user, from_key, limit):
-        return ([], from_key)
+        from_key = int(from_key)
+        handler = self.handler()
+
+        events = []
+        for room_id in handler._room_serials:
+            if handler._room_serials[room_id] <= from_key:
+                continue
+
+            # TODO: check if user is in room
+            events.append(self._make_event_for(room_id))
+
+        return (events, handler._latest_room_serial)
 
     def get_current_key(self):
-        return 0
+        return self.handler()._latest_room_serial
 
     def get_pagination_rows(self, user, pagination_config, key):
         return ([], pagination_config.from_key)

synapse/http/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/http/agent_name.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,3 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from synapse import __version__
+
+AGENT_NAME = ("Synapse/%s" % (__version__,)).encode("ascii")

synapse/http/client.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,6 +14,7 @@
 # limitations under the License.
 
 
+from synapse.http.agent_name import AGENT_NAME
 from twisted.internet import defer, reactor
 from twisted.web.client import (
     Agent, readBody, FileBodyProducer, PartialDownloadError
@@ -51,7 +52,8 @@ class SimpleHttpClient(object):
             "POST",
             uri.encode("ascii"),
             headers=Headers({
-                "Content-Type": ["application/x-www-form-urlencoded"]
+                b"Content-Type": [b"application/x-www-form-urlencoded"],
+                b"User-Agent": [AGENT_NAME],
             }),
             bodyProducer=FileBodyProducer(StringIO(query_bytes))
         )
@@ -86,6 +88,9 @@ class SimpleHttpClient(object):
         response = yield self.agent.request(
             "GET",
             uri.encode("ascii"),
+            headers=Headers({
+                b"User-Agent": [AGENT_NAME],
+            })
         )
 
         body = yield readBody(response)
@@ -108,7 +113,8 @@ class CaptchaServerHttpClient(SimpleHttpClient):
             url.encode("ascii"),
             bodyProducer=FileBodyProducer(StringIO(query_bytes)),
             headers=Headers({
-                "Content-Type": ["application/x-www-form-urlencoded"]
+                b"Content-Type": [b"application/x-www-form-urlencoded"],
+                b"User-Agent": [AGENT_NAME],
             })
         )