JSON-RPC Example ================ .. contents:: :author: Ian Bicking Introduction ------------ This is an example of how to write a web service using WebOb. The example shows how to create a `JSON-RPC `_ endpoint using WebOb and the `simplejson `_ JSON library. This also shows how to use WebOb as a client library using `WSGIProxy `_. While this example presents JSON-RPC, this is not an endorsement of JSON-RPC. In fact I don't like JSON-RPC. It's unnecessarily un-RESTful, and modelled too closely on `XML-RPC `_. Code ---- The finished code for this is available in `docs/json-example-code/jsonrpc.py `_ -- you can run that file as a script to try it out, or import it. Concepts -------- JSON-RPC wraps an object, allowing you to call methods on that object and get the return values. It also provides a way to get error responses. The `specification `_ goes into the details (though in a vague sort of way). Here's the basics: * All access goes through a POST to a single URL. * The POST contains a JSON body that looks like:: {"method": "methodName", "id": "arbitrary-something", "params": [arg1, arg2, ...]} * The ``id`` parameter is just a convenience for the client to keep track of which response goes with which request. This makes asynchronous calls (like an XMLHttpRequest) easier. We just send the exact same id back as we get, we never look at it. * The response is JSON. A successful response looks like:: {"result": the_result, "error": null, "id": "arbitrary-something"} * The error response looks like:: {"result": null, "error": {"name": "JSONRPCError", "code": (number 100-999), "message": "Some Error Occurred", "error": "whatever you want\n(a traceback?)"}, "id": "arbitrary-something"} * It doesn't seem to indicate if an error response should have a 200 response or a 500 response. So as not to be completely stupid about HTTP, we choose a 500 resonse, as giving an error with a 200 response is irresponsible. Infrastructure -------------- To make this easier to test, we'll set up a bit of infrastructure. This will open up a server (using :py:mod:`wsgiref.simple_server`) and serve up our application (note that *creating* the application is left out to start with): .. code-block:: python import sys def main(args=None): import optparse from wsgiref import simple_server parser = optparse.OptionParser( usage="%prog [OPTIONS] MODULE:EXPRESSION") parser.add_option( '-p', '--port', default='8080', help='Port to serve on (default 8080)') parser.add_option( '-H', '--host', default='127.0.0.1', help='Host to serve on (default localhost; 0.0.0.0 to make public)') if args is None: args = sys.argv[1:] options, args = parser.parse_args() if not args or len(args) > 1: print 'You must give a single object reference' parser.print_help() sys.exit(2) app = make_app(args[0]) server = simple_server.make_server( options.host, int(options.port), app) print 'Serving on http://%s:%s' % (options.host, options.port) server.serve_forever() if __name__ == '__main__': main() I won't describe this much. It starts a server, serving up just the app created by ``make_app(args[0])``. ``make_app`` will have to load up the object and wrap it in our WSGI/WebOb wrapper. We'll be calling that wrapper ``JSONRPC(obj)``, so here's how it'll go: .. code-block:: python def make_app(expr): module, expression = expr.split(':', 1) __import__(module) module = sys.modules[module] obj = eval(expression, module.__dict__) return JsonRpcApp(obj) We use ``__import__(module)`` to import the module, but its return value is wonky. We can find the thing it imported in ``sys.modules`` (a dictionary of all the loaded modules). Then we evaluate the second part of the expression in the namespace of the module. This lets you do something like ``smtplib:SMTP('localhost')`` to get a fully instantiated SMTP object. That's all the infrastructure we'll need for the server side. Now we just have to implement ``JsonRpcApp``. The Application Wrapper ----------------------- Note that I'm calling this an "application" because that's the terminology WSGI uses. Everything that gets *called* is an "application", and anything that calls an application is called a "server". The instantiation of the server is already figured out: .. code-block:: python class JsonRpcApp(object): def __init__(self, obj): self.obj = obj def __call__(self, environ, start_response): ... the WSGI interface ... So the server is an instance bound to the particular object being exposed, and ``__call__`` implements the WSGI interface. We'll start with a simple outline of the WSGI interface, using a kind of standard WebOb setup: .. code-block:: python from webob import Request, Response from webob import exc class JsonRpcApp(object): ... def __call__(self, environ, start_response): req = Request(environ) try: resp = self.process(req) except ValueError, e: resp = exc.HTTPBadRequest(str(e)) except exc.HTTPException, e: resp = e return resp(environ, start_response) We first create a request object. The request object just wraps the WSGI environment. Then we create the response object in the ``process`` method (which we still have to write). We also do some exception catching. We'll turn any ``ValueError`` into a ``400 Bad Request`` response. We'll also let ``process`` raise any ``web.exc.HTTPException`` exception. There's an exception defined in that module for all the HTTP error responses, like ``405 Method Not Allowed``. These exceptions are themselves WSGI applications (as is ``webob.Response``), and so we call them like WSGI applications and return the result. The ``process`` method ---------------------- The ``process`` method of course is where all the fancy stuff happens. We'll start with just the most minimal implementation, with no error checking or handling: .. code-block:: python from simplejson import loads, dumps class JsonRpcApp(object): ... def process(self, req): json = loads(req.body) method = json['method'] params = json['params'] id = json['id'] method = getattr(self.obj, method) result = method(*params) resp = Response( content_type='application/json', body=dumps(dict(result=result, error=None, id=id))) return resp As long as the request is properly formed and the method doesn't raise any exceptions, you are pretty much set. But of course that's not a reasonable expectation. There's a whole bunch of things that can go wrong. For instance, it has to be a POST method: .. code-block:: python if not req.method == 'POST': raise exc.HTTPMethodNotAllowed( "Only POST allowed", allowed='POST') And maybe the request body doesn't contain valid JSON: .. code-block:: python try: json = loads(req.body) except ValueError, e: raise ValueError('Bad JSON: %s' % e) And maybe all the keys aren't in the dictionary: .. code-block:: python try: method = json['method'] params = json['params'] id = json['id'] except KeyError, e: raise ValueError( "JSON body missing parameter: %s" % e) And maybe it's trying to acces a private method (a method that starts with ``_``) -- that's not just a bad request, we'll call that case ``403 Forbidden``. .. code-block:: python if method.startswith('_'): raise exc.HTTPForbidden( "Bad method name %s: must not start with _" % method) And maybe ``json['params']`` isn't a list: .. code-block:: python if not isinstance(params, list): raise ValueError( "Bad params %r: must be a list" % params) And maybe the method doesn't exist: .. code-block:: python try: method = getattr(self.obj, method) except AttributeError: raise ValueError( "No such method %s" % method) The last case is the error we actually can expect: that the method raises some exception. .. code-block:: python try: result = method(*params) except: tb = traceback.format_exc() exc_value = sys.exc_info()[1] error_value = dict( name='JSONRPCError', code=100, message=str(exc_value), error=tb) return Response( status=500, content_type='application/json', body=dumps(dict(result=None, error=error_value, id=id))) That's a complete server. The Complete Code ----------------- Since we showed all the error handling in pieces, here's the complete code: .. code-block:: python from webob import Request, Response from webob import exc from simplejson import loads, dumps import traceback import sys class JsonRpcApp(object): """ Serve the given object via json-rpc (http://json-rpc.org/) """ def __init__(self, obj): self.obj = obj def __call__(self, environ, start_response): req = Request(environ) try: resp = self.process(req) except ValueError, e: resp = exc.HTTPBadRequest(str(e)) except exc.HTTPException, e: resp = e return resp(environ, start_response) def process(self, req): if not req.method == 'POST': raise exc.HTTPMethodNotAllowed( "Only POST allowed", allowed='POST') try: json = loads(req.body) except ValueError, e: raise ValueError('Bad JSON: %s' % e) try: method = json['method'] params = json['params'] id = json['id'] except KeyError, e: raise ValueError( "JSON body missing parameter: %s" % e) if method.startswith('_'): raise exc.HTTPForbidden( "Bad method name %s: must not start with _" % method) if not isinstance(params, list): raise ValueError( "Bad params %r: must be a list" % params) try: method = getattr(self.obj, method) except AttributeError: raise ValueError( "No such method %s" % method) try: result = method(*params) except: text = traceback.format_exc() exc_value = sys.exc_info()[1] error_value = dict( name='JSONRPCError', code=100, message=str(exc_value), error=text) return Response( status=500, content_type='application/json', body=dumps(dict(result=None, error=error_value, id=id))) return Response( content_type='application/json', body=dumps(dict(result=result, error=None, id=id))) The Client ---------- It would be nice to have a client to test out our server. Using `WSGIProxy`_ we can use WebOb Request and Response to do actual HTTP connections. The basic idea is that you can create a blank Request: .. code-block:: python >>> from webob import Request >>> req = Request.blank('http://python.org') Then you can send that request to an application: .. code-block:: python >>> from wsgiproxy.exactproxy import proxy_exact_request >>> resp = req.get_response(proxy_exact_request) This particular application (``proxy_exact_request``) sends the request over HTTP: .. code-block:: python >>> resp.content_type 'text/html' >>> resp.body[:10] '`_ and you can run it with `docs/json-example-code/test_jsonrpc.py `_, which looks like: .. code-block:: python if __name__ == '__main__': import doctest doctest.testfile('test_jsonrpc.txt') As you can see, it's just a stub to run the doctest. We'll need a simple object to expose. We'll make it real simple: .. code-block:: python >>> class Divider(object): ... def divide(self, a, b): ... return a / b Then we'll get the app setup: .. code-block:: python >>> from jsonrpc import * >>> app = JsonRpcApp(Divider()) And attach the client *directly* to it: .. code-block:: python >>> proxy = ServerProxy('http://localhost:8080', proxy=app) Because we gave the app itself as the proxy, the URL doesn't actually matter. Now, if you are used to testing you might ask: is this kosher? That is, we are shortcircuiting HTTP entirely. Is this a realistic test? One thing you might be worried about in this case is that there are more shared objects than you'd have with HTTP. That is, everything over HTTP is serialized to headers and bodies. Without HTTP, we can send stuff around that can't go over HTTP. This *could* happen, but we're mostly protected because the only thing the application's share is the WSGI ``environ``. Even though we use a ``webob.Request`` object on both side, it's not the *same* request object, and all the state is studiously kept in the environment. We *could* share things in the environment that couldn't go over HTTP. For instance, we could set ``environ['jsonrpc.request_value'] = dict(...)``, and avoid ``simplejson.dumps`` and ``simplejson.loads``. We *could* do that, and if we did then it is possible our test would work even though the libraries were broken over HTTP. But of course inspection shows we *don't* do that. A little discipline is required to resist playing clever tricks (or else you can play those tricks and do more testing). Generally it works well. So, now we have a proxy, lets use it: .. code-block:: python >>> proxy.divide(10, 4) 2 >>> proxy.divide(10, 4.0) 2.5 Lastly, we'll test a couple error conditions. First a method error: .. code-block:: python >>> proxy.divide(10, 0) # doctest: +ELLIPSIS Traceback (most recent call last): ... Fault: Method error calling http://localhost:8080: integer division or modulo by zero Traceback (most recent call last): File ... result = method(*params) File ... return a / b ZeroDivisionError: integer division or modulo by zero It's hard to actually predict this exception, because the test of the exception itself contains the traceback from the underlying call, with filenames and line numbers that aren't stable. We use ``# doctest: +ELLIPSIS`` so that we can replace text we don't care about with ``...``. This is actually figured out through copy-and-paste, and visual inspection to make sure it looks sensible. The other exception can be: .. code-block:: python >>> proxy.add(1, 1) Traceback (most recent call last): ... ProxyError: Error from JSON-RPC client http://localhost:8080: 400 Bad Request Here the exception isn't a JSON-RPC method exception, but a more basic ProxyError exception. Conclusion ---------- Hopefully this will give you ideas about how to implement web services of different kinds using WebOb. I hope you also can appreciate the elegance of the symmetry of the request and response objects, and the client and server for the protocol. Many of these techniques would be better used with a `RESTful `_ service, so do think about that direction if you are implementing your own protocol.