docs_old/dev-docs/architecture.rst
============
Architecture
============
.. contents::
:depth: 2
High level overview
===================
EJ adopts a monolithic architecture with the main server that controls business
logic and some additional services that complement the application. The
recommended way to deploy EJ is to run each service in a separate container in
order to provide a good level of isolation. The aforementioned services
are organized in a simple 3-layer architecture described bellow:
.. image:: img/ej_arch.svg
Nginx
Web traffic should not be handled directly by the application service.
Gunicorn is not efficient to serve static files and an additional reverse
proxy can also be used to enforce more strict security policies and more
efficient caching. EJ adopts Nginx. It serves static files and redirect
dynamic routes to the application service described bellow. The Nginx
container must share some volumes with the Django application in order to
find static files.
Django Application
EJ is written mostly in Python and uses the Django web framework.
The main application service is responsible for all dynamic routes, which
are generated by Django using the templating language Jinja2. The
Django application should be run with a WSGI compatible service such as
Gunicorn. This is the default approach adopted in the application container.
The default task uses a number of workers equal to the number of CPU cores,
which is usually the recommended configuration.
SQL Database
EJ does not use any database-specific functionality or raw SQL commands.
This means that it can run in any database supported by Django such as
Postgres SQL, MariaDB, Sqlite3, etc. We recommend Postgres (v10.0), which is used
in the default installation. The database connection is controlled by
the DJANGO_DB_URL environment variable in the main application container.
Frontend
........
EJ Frontend is implemented using the Jinja2 templating language and uses
progressive enhancement to include styles via CSS and custom behaviors with
JavaScript. The following presents a brief overview of the technologies used
in each of those layers:
CSS
CSS is implemented with Sass using a ITCSS-inspired architecture (Inverse
Triangle CSS). The CSS module is implemented using the `Mendeleev.css`_
framework for atomic CSS and can be easily customized using themes. The
CSS assets are statically compiled and served by Nginx. Sass compilation
requires libsass, which is bundled into the Python dependencies of the
application.
JavaScript/TypeScript
EJ does not adopt any traditional JavaScript framework, but instead relies on
progressive enhancement to add optional functionalities. EJ uses Unpoly_ in
conjunction to jQuery_ to provide the core functionality. EJ-specific components
are created using TypeScript and enhance tags annotated with the "is-component"
attribute with extra behaviors and functionalities. TypeScript compilation
requires the Node Package Manager (NPM) and Parcel_.
.. _Mendeleev.css: https://www.npmjs.com/package/mendeleev.css
.. _Unpoly: https://unpoly.com
.. _jQuery: https://jquery.com
.. _Parcel: https://parceljs.org
Django application
==================
Django splits a web system into modules called "apps" which implement reusable
database models, routes and functionalities. This section describes all "apps"
implemented in EJ.
EJ project
----------
The `ej` module is not properly an app, but a regular Python package used to
coordinate apps by defining settings, common functionality and loading static
assets like Javascript, CSS, images, themes etc. The following is an overview
of the main sub-packages and modules:
``ej.all``
Useful namespace to be used in a interactive section as ``from ej.all import *``.
It imports models and managers of all ej apps and examples in the global
namespace.
``ej.components``
Similarly to ``ej.roles``, this module defines renderers for reusable UI
elements. The difference between the two modules is that components can have
a more complicated structure and may not be directly associated with some
known Python data type.
``ej.contrib``
Location to include ad-hoc migrations for specific deployments. Most users
and developers should never touch this.
``ej.fixes``
Monkey patch third part modules that have known issues with EJ or any of
its dependencies.
``ej.forms``
Base form classes that are used in other EJ apps. Forms are derived from
django.forms.
``ej.jinja2``
EJ uses Jinja2 as the default templating language. This module configures the
Jinja2 environment and set global functions and filters.
``ej.roles``
Functions that define Hyperpython roles. Roles are mappings of
``(type, name) -> HTML`` that define how a certain object should be rendered
in a given context or role. This module defines many reusable UI elements
as Python functions.
``ej.routes``
Define some global view functions such as the home page that do not have
functionality tied to any app.
``ej.services``
Helper functions to initialize connections to external services such as
Postgres SQL database and redis (if enabled).
``ej.settings``
Django settings module. Defines configuration using Django Boogie's
configuration framework in which configuration is defined in reusable classes
instead of a flat Python module.
``ej/templates/jinja2``
Contains templates available globally. The global ``base.jinja2`` template
defines the base page HTML structure (navigation bars, meta information, etc)
that is shared among most pages in the site.
``ej.testing``
Helper tools used in testing across apps.
``ej.tests``
Global tests. Most tests are implemented in app-specific test folders.
``ej.urls``
URL mapping for the project. Most URLs are included from an app's own
``routes.py``.
``ej.utils``
Utility functions module.
``ej.wsgi``
Django wrapper for the WSGI interface.
Applications
------------
The listing bellow describes all apps implemented inside EJ source tree.
``ej_conversations``
.. image:: orm/ej_conversations.svg
:target: ../_images/ej_conversations.svg
This is the main application and defines models for conversations, comments,
and votes. The ej_applications app implements the UI for creating, configuring
and interacting with conversations.
``ej_users``
.. image:: orm/ej_users.svg
:target: ../_images/ej_users.svg
This app defines the main User model for EJ and all routes related to
authentication and account management (e.g., reset passwords, cancel account,
etc). EJ can be used with Django's regular users, although this is not
encouraged.
``ej_profiles``
.. image:: orm/ej_profiles.svg
:target: ../_images/ej_profiles.svg
Implements profile management UI and defines a model that store profile
information. This app can be easily modified to include extra profile fields
or to remove unwanted fields for some particular installation.
``ej_clusters``
.. image:: orm/ej_clusters.svg
:target: ../_images/ej_clusters.svg
Implements the mathematical routines to classify users into opinion groups.
The ej_clusters.math module implements our modified K-means algorithm that
takes into account "opinion stereotypes" and also provides interfaces to
manage those stereotypes and the resulting clusters.
``ej_dataviz``
.. image:: orm/ej_dataviz.svg
:target: ../_images/ej_dataviz.svg
Implements routines to visualize data about conversations. It generates
structured reports and export data to spreadsheet-compatible formats. This
module also implements visualization techniques such as Word Cloud and
Scatter Maps of user opinions.
``ej_boards``
.. image:: orm/ej_boards.svg
:target: ../_images/ej_boards.svg
The boards app allow regular users to have their own "board" or "timeline"
of conversations. The default conversation feed in "/conversations/" can
only be managed by users with special permissions.
Third party Apps
----------------
``boogie.apps.fragments``
The Boogie fragments app implements configurable text or HTML fragments. This
allows a greater level of configurability by allowing administrative users
to customize parts of the platform without using any code.
``rules``
Django-rules_ implements a mechanism to define business logic rules by
registering simple predicate functions. This package nicely integrates with
Django's own permission mechanisms. The business rules relevant to each
EJ application are implemented into the respective "rules.py module of each
Django app and can be overridden by third party apps or modules.
``Django taggit``
Django-taggit_ is a Django application that implements tags to arbitrary
models. It is used to support tagging of EJ conversations.
``rest_framework``
The Django-Rest-Framework_ (DRF) is a powerful toolkit to develop REST Web APIs.
EJ uses DRF through the rest_api module of Django-Boogie.
``allauth, allauth.account, allauth.socialaccount``
The `allauth`_ project implement authentication and authorization workflows
and integration with third party OAuth providers such as Google, Twitter and
Facebook.
.. _Django-rules: https://github.com/dfunckt/django-rules
.. _allauth: https://www.intenct.nl/projects/django-allauth/
.. _Django-taggit: https://github.com/jazzband/django-taggit
.. _Django-Rest-Framework: https://www.django-rest-framework.org
App structure
=============
EJ uses Django Boogie adopts an architecture that may be slightly different
from a typical Django app. One important goal of the architecture is to make
lightweight view functions and models. This is accomplished by either moving
functionality to the Boogie framework itself or to splitting functionality
into different modules.
A typical EJ App has the following structure:
``<app>.admin``
Django admin classes and functions.
``<app>.api``
Defines fields and API routes for the models defined by the app. Normally,
functionalities implemented in this module simply supplement the main API
declarations that are created using the ``@rest_api`` decorator directly
on models.
``<app>.apps``
Django AppConfig mechanism. EJ apps usually should override the ``ready()``
method of the app config and import the api, roles and rules modules.
``<app>.enums``
This modules defines any enum type that is eventually used by models. Enums
are usually imported into the model base namespace, so they should have no
dependency on models.
``<app>.forms``
Django forms defined for the app. Usually forms should inherit from ej.forms
instead of using django.forms directly.
``<app>.math``
All math functions should be defined in this module. Complicated mathematical
transformations should be implemented as ScikitLearn transformations or
pipelines.
``<app>.managers``
In Django, model classes defines row logic and managers and querysets
implements table logic. All methods that query or create models or filter
querysets should be implemented in the "managers" module
``<app>.models``
Just like in regular Django apps, this module defines the models for the
app. Models should avoid implementing business logic inside them and ideally
should restrict to database actions such as querying, validation, etc.
``<app>.mommy_recipes``
EJ uses Model Mommy to create random fixtures for tests. This module should
define a class that derives from ``ej.testing.EjRecipes`` and implements
fixtures for each model defined in the app. This is only used in tests.
``<app>.routes``
Regular Django apps have a views.py and a urls.py files. Django Boogies
encourages to join both files into a single routes.py that defines view
functions and maps them to routes using decorators.
``<app>.rules``
Business rules are implemented as regular functions inside this module.
This helps avoiding the "fat-models" anti-pattern that is common in Django
projects. The rules module can define both permissions (which are user-centric
predicate functions) and regular "values", which can return non-boolean
values (e.g., number of comments user still has in conversation).
``<app>.roles``
Hyperpython roles are simple functions that render objects in given contexts.
For instance, we can register a "card" role to the Conversation class that
renders the input conversation as a card in a listing view. Role functions
must be associated with a type and a name describing is role and must return
a Hyperpython html structure.
In cases that Jinja2 is more convenient than Hyperpython, the
``ej.roles.with_template`` decorator can be used to associate the role with
a Jinja template.
``<app>.tests``
App's unit tests.
``<app>.validators``
Implement the validation functions used in model fields or form fields inside
the app.
Templates
---------
Templates reside inside the ``<app>/jinja2/`` folder. We use Django best practices
and save app-specific templates inside ``jinja2/<app-name>/<template-name>.jinja2``.
Templates names usually mirror the names of view functions in the ``routes.py``
file. For instance, a edit view for some conversation would be declared as::
urlpatterns = Router(template='ej_conversations/{name}.jinja2')
@urlpatterns.route("/<model:conversation>/edit/")
def edit(request, conversation):
... # Implementation
This view function is automatically associated with the ``ej_conversations/edit.jinja2``
template, unless specified otherwise.
Most templates inherit from a base template at ``src/ej/templates/jinja2/base.jinja2``.
This template imports navigation elements such as menus and toolbars.