Can I Build an Online Radio With Django Framework

Building a Broadcast Platform in Python

Published Mar 20, 2018 Last updated Sep 16, 2018

Abstract - In this article, nosotros describe a evolution solution for information-streaming services. In contrast to Big-Data paid platforms, nosotros show how a lean solution can be built with a simple, notwithstanding effective, iii-tiered compages that results in a highly customizable software production. Nosotros prove code samples in Python that apply various state-of-the-art tools, such equally Django, Django-channels, and websockets.

Edit - The piece of work described in this commodity is now alive at katiedj.com

Keywords: Python, Django, MVP, Websockets

Introduction

Scientists and software researchers often face a common situation when their work has brought some interesting results in terms of data analytics that they would like to share with the customs.

In fact, open platforms, such equally GitHub, allow users to publish software in render for visibility and (perhaps more importantly) feedback. Moreover, sharing datasets is a well known method for improving data quality as well as software analysis.

On the other manus, information technology is often the case that the source of information cannot be made public — for example, data can be generated by sensors (in this instance nosotros will speak of "real" data), or by a proprietary software system that generates a real fourth dimension log (in this example we volition speak of "synthetic" data).

One of the most interesting scenarios arises when a start-up has, almost unintentionally, built a lean product while building a larger one, and it becomes possible to get some initial revenue by selling the one-time. Selling information technology direct, though, would compromise the bigger start-up plans, potentially revealing secrets to competitors.

Thus, a solution is to build a broadcast arrangement that keeps private the way data are generated, but sells access to the circulate itself.

The above reasoning is the main motivation for our work. This article is going to idea-dense. Therefore, we get-go with a content overview make reading easier.

There will be iii sections, each describing one part of the architecture (see besides the effigy below):

• The broadcast system. We like to think well-nigh it as a radio that sends messages to whoever is listening to the correct channels. However, to improve modularization of the entire arrangement, the radio is likewise congenital in order to receive, from an external organisation, the information information technology volition then broadcast. In our application, nosotros built this module in Python, using Django and Django-channels.
• The listener(s). They would be the clients, if nosotros were talking virtually a standard customer-server architecture — they are the agents listening to the radio channels. We congenital this function in Python, using the websockets module.
• The data generator organisation. It is the private module whose implementation details are non shared with the listeners of the radio. It produces new data with an undisclosed software and so sends it over to the radio in a format that tin can exist shared (sold, peradventure). This role can be built in any language, or can even be a real system of sensors, or whatsoever "internet of things" device.

Before going into the details of the implementation, we would like to emphasize some of the good points of such an compages:

• The clear separation between the Data Generator and the Radio (i.e., the circulate server), makes them highly flexible and customizable.
• Every bit a consequence, different people can work on the two modules: information technology is easy to imagine data scientists working on the Information Generator and spider web developers working on the Radio.
• As a further consequence, the compages is completely language independent: we said nosotros built everything in Python, but in fact information technology is entirely possible to build the Data Generator in R or C++ (maybe if it is a high-computation arrangement), and the Circulate in Ruddy or Java, but to name a few alternatives. The two modules but need to concur on the format of the exchanged data, which is likely to be JSON or XML.
• As a concern model, we said the nearly natural would exist to sell the access to public channels to receive the data. In fact, even a more subtle model makes sense: selling the access to the private API that publishes the data.

Allow us now go along the article by discussing more technical details about the implementation of the different modules.

The Broadcast Server (Radio)

The chief objectives of the server must be to receive data from the Information Generator and to publish them in lodge to be retrieved by clients. Therefore, these requirements translate into:

• An API endpoint that is used past the Data Generator each time a new sample is ready.
• Public channels where the clients register in order to get the new information once they are published.

It is worth noticing that the 2 higher up points are based on very different behaviors: on one hand (commencement point), an API is used, which means it must be chosen past the other side. On the other hand (2nd point), information is simply sent to the circulate, not thing who is listening to it.

Why such a divergence? Just put, it would exist very expensive to build an endpoint that may be used by the listener, especially if there are a lot of connected listeners, then a standard Residual API approach is non the best design option — nosotros used Websocket for it.

On the contrary, the Data Generator is a controlled organization, therefore it is costless to invoke a RESTful endpoint, whenever a new sample is ready to be consumed.

We used Python to build this module, and pure Django ii.0 for the API. Nosotros have a Django-app chosen publisher, whose urls.py file is very simple:

          # urls.py from django.urls import path from . import views  app_name = 'publisher'  urlpatterns = [     path('publish', views.PublisherView.as_view(), name='main'), ]                  

Notice that we utilize the function path, new in Django 2.0. This mode, the endpoint is exposed to the /publish path. Let usa now evidence the Django View that receives data from the Data Generator and publish to the channels.

          # views.py course PublisherView(View):      def post(self, request):         effort:             key = request.META['HTTP_API_KEY']             if not key in settings.API_KEYS:                 return JsonResponse(                     {'error': 'API-Cardinal non valid'}, status=400)         except:             return JsonResponse(                 {'error': 'API-KEY missing in headers'}, status=400)         try:             torso = json.loads(request.body.decode('utf-viii'))         except:             render JsonResponse(                 {'mistake': 'Mail service data is not JSON'}, status=400)         try:             group = body['network']         except:             return JsonResponse(                 {'error': '*network* key missing'}, status=400)         if group non in settings.API_KEYS[key]:             render JsonResponse(                 {'fault': 'You cannot broadcast to this channel'}, condition=403)         if not 'information' in torso:             return JsonResponse(                 {'error': '*data* cardinal missing'}, condition=400)         if not isinstance(body['data'], dict):             return JsonResponse(                 {'error': 'Tin can only circulate json information'}, status=400)         # all tests are OK         Group(group).send({'text': json.dumps(body['data'])})         render JsonResponse({'message': 'OK'}, condition=200)                  

Permit us also annotate the behavior of this View step by step:

1. The endpoint is enabled for Mail service requests.
2. The header of the request must contain a field chosen API-Cardinal. Moreover, this field must correspond to one of the enabled KEYs (stored in settings.API_KEYS). This a very standard way to protect an endpoint. Basically, only users who have got a secret personal alphanumeric primal tin can actually utilize the API.
3. The body of the request must exist in well-formed JSON format.
4. The body must contain, in the network field, the name of a valid group, that is, the name of the channel to which data will be sent.
5. The name of such a channels must be among the channels that are enabled for this item API-Key. This is to avoid that an authorized user sending information to a channel she should not be sending it to.
6. The data to exist broadcasted must be in JSON format.
7. Finally, later on all tests are passed, information is sent over to the channel. Notice that we utilize the Group object of the Django-channels package.

Allow us now "connect the dots" and show the module that handles connections, that are the channels. As we have already mentioned, nosotros used the Django-channels package, which basically implements the Websocket protocol in a very Django-ish fashion. In fact, using Django-channels, it becomes extremely easy to handle unproblematic listener connections. Hither is the entire module (very few lines!):

          # consumers.py from channels.generic import websockets  grade MyBroadcast(websockets.WebsocketConsumer):      http_user = Truthful     strict_ordering = False      def connection_groups(self, **kwargs):         """         Chosen to return the list of groups to automatically add/remove         this connection to/from.         """         return [self.channel_name]      def receive(self, text=None, bytes=None, **kwargs):         # Not talking!         laissez passer  class SampleBroadcast(MyBroadcast):     channel_name = "sample_net"                  

To give more insight, every bracket of MyBroadcast simply needs to create the proper name of its ain channel (sample_net in the in a higher place lawmaking), and the rest is automatically handled by the great chore done inside the Django-channels package.

The Broadcast Listener

Even though the most interesting part of the architecture is the Broadcast Server, a Radio is useless when nobody listens to it. Therefore, in our concept, it is as well of import to provide ready-to-use client modules that users can simply download and run to get the data.

Ideally, these modules would be published (for example in GitHub) and then experienced users tin customize the behavior according to their needs, submit merge requests, and contribute farther.

Also, ideally, nosotros would provide client modules in several languages (Python, R, MatLab, etc.) and so that each user can employ her favorite environment for information analysis.

Let us now bear witness how to write a similar module in Python. Nosotros used the neat package websockets, built on top of the standard library asyncio.

          # radio_listener.py class WSClient():      def __init__(cocky, url):         self.url = url         # constant values, but tin can be passed every bit params         self.reply_timeout = 10         self.ping_timeout = v         cocky.sleep_time = 5      async def listen_forever(cocky):         while Truthful:             logger.debug('Creating new connexion...')             try:                 async with websockets.connect(cocky.url) as ws:                     while True:                         try:                             reply = look asyncio.wait_for(ws.recv(), timeout=self.reply_timeout)                         except (asyncio.TimeoutError, websockets.exceptions.ConnectionClosed):                             attempt:                                 pong = look ws.ping()                                 await asyncio.wait_for(pong, timeout=self.ping_timeout)                                 logger.debug('Ping OK, keeping connection live...')                                 continue                             except:                                 logger.debug('Ping mistake - retrying connection in {} sec (Ctrl-C to quit)'.format(cocky.sleep_time))                                 expect asyncio.slumber(cocky.sleep_time)                                 break                         # Here exercise something with the data                         logger.debug('Got information > {}'.format(answer))             except socket.gaierror:                 logger.debug('Socket error - retrying connexion in {} sec (Ctrl-C to quit)'.format(self.sleep_time))                 await asyncio.slumber(self.sleep_time)                	continue             except ConnectionRefusedError:                 logger.debug('Nobody seems to listen to this URL')                 logger.debug('Exiting...')                 interruption                  

The above code defines a class WSClient and uses standard techniques of asynchronous programming and connectedness retry when dealing with web applications.

In short, information technology waits for data from the server in an space loop, and every time the connection fails (or seems to fail), outset sends a ping message, and so restarts a new connection. As per good programming practise, almost every upshot is logged (using standard Python library logging, we intentionally skipped details about it).

The Data Generator

Although nosotros kept the Information Generator equally the last section in this article, this is very probable the core of the application. Even more than so, it is probably something that was built for a totally different reason, and only afterwards information technology was realized that more people could be interested in using the same information.

As we mentioned before, the Data Generator tin can be practically anything: a arrangement of sensors retrieving data from a system, a network of mobile phones sending data most GPS locations, and/or a software built by scientists that computes some complex mathematical model.

In our case, the Information Generator was a Python software capable of simulating vehicles' behavior, predicting turnings at the intersections, taking into account traffic lights, and eventually computing average densities of vehicles in each street of big towns.

Basically, it is a road traffic simulator. The overall broadcast idea is actually based on the fact that we would like to share the actual data and non the mathematics behind the model. Interested readers tin find a (very) comprehensive discussion in this manuscript (do not be mislead past the comprehend page: the manuscript is in English).

A simple arroyo for periodically publishing the data computed past the generator is the post-obit:

          import requests, json import hush-hush  def publish(net_name, jdata):     url = surreptitious.URL     headers = {'content-type': 'application/json'}     headers['API-KEY'] = secret.API_KEYS[net_name]     payload = {'network': net_name}     payload['data'] = jdata     resp = requests.post(         url,         information=json.dumps(payload),         headers=headers)     return resp.status_code  def radio_publish():     # initialize "net" object     …     #     while True:         jdata = net.dump_state()         try:             resp = publish(net.name, jdata)             time.sleep(cyberspace.step)         except:             break         internet.adjacent()     logger.debug('Terminal published data is > {}'.format(jdata))     return jdata                  

These two functions should be very clear when thinking back to the Circulate Server API. Notice that, within the radio_publish method, there is an infinite loop that first calls net.dump_state to retrieve the final computed data sample, and then it publishes it, and so calls net.next, to invite the computation a new sample. Different, more complex, paradigms are likewise possible, of course, such as event-based programming.

Conclusions and outlook

In this article, we have shown the journey towards building a data-streaming platform. Nosotros have expressed applied motivations past ways of existent-world business examples, and nosotros have also shown the backbone of such a streaming platform built in Python.

We have underlined several times the flexibility given past our design choices: the net separations between the three modules allows fast prototyping and development too every bit compartmentalization of team resource distribution.

Finally, numerous paths remain worth investigating further: what is the best infrastructure to back up deployment of such an architecture, what is the best advice prototype between the Data Generator and the Broadcast Server, and, perhaps more than interestingly, how would the open-source community receive and contribute to this type of design?

brinsonfaturaved.blogspot.com

Source: https://www.codementor.io/@pietrograndinetti/building-a-broadcast-platform-in-python-hp4oejaaz

Share this post