A list is a basic data structure that types the view that displays the artifacts on several lines. A line can be configured to display some categories and hide some other information.

A table is a data structure that can type a view. A table displays the artifacts in a matrix with one dimension (column) or optionally with two dimensions (columns + rows). Columns and rows are defined with the available categories (user, status, interval and additional catalogs defined in the organization). When the categories are used to list the columns or the rows, their values are automatically associated to the artifact regarding its position inside the table.

When the artifact 's fields values corresponding to the category used in the columns and the rows are not defined, artifacts are displayed outside the table. A zone dedicated to that purpose is visible and is part of the table type.

When an interval category is used to define the columns or the rows, the artifact date is automatically computed with the associated value that has been obtained relatively to the starting date of the view.

Reactivity architecture allows the system to face an unpredictable load of data when the users, the data sources or even the resources of some parties inside the information system scale. This is why all the core architecture heavily uses the reactive streams where each service acts as a consumer and/or as a producer, emphasizing an event-driven model in all layers.

The event driven system is ready to route messages from source to sink in both live and batch contexts. This need comes from the necessity to cover messaging based communication between the users and also asynchronous streams processing to prepare the data for some views. Since everything is message, it’s mandatory to expect that a message is going to be consumed by different consumers for different purpose. This case is typically well illustrating by RabbitMQ documentation when talking about direct exchange routing.

We can see that there is as many consumer with different purpose as many routing keys are declared. But it’s important to notice that two consumers for the same purpose can be registered, and only one of them will consume the message.

Reactivity architecture expects a various number of streams and transformations of those streams. The transformation phase requires messages to be delivered in the same order they are publishing. This means also that if a consumer fails to acknowledge a message consumption, it must be able to easily retry to process the same message before treating a new one.

Transformation can be also aggregations to satisfy some user views requirements. This means that a stream can be composed of different streams.

There is no reason to exclude some system interactions from the asynchronous messaging model. Even a click from a user can be considered as a stream source.

This consideration can be wrong in a few cases where technical constraints make more realistic a different technique.

Routing messages with guarantee of delivery, ordering preservation, acknowledgement mechanism and high performance must be addressed by a distributed streaming platform. That streaming platform will allow all producers and consumers to register themselves. This is that platform that will receive all messages from producers and push them to consumers.

The streaming platform will interact with producers and consumers in respect of microservice architecture definition. This mainly means that a consumer or a producer is considered as a component serving one purpose, exposing a clear API and easy to replace in the information system.

A microservice is autonomous as much as possible which is possible when control of data is decentralized. Consumers and producers can take advantage of this by embedding their own database, which can lead to a duplication of data. A consistency check mechanism should be considered to regularly make sure that a database is not incorrectly synchronized with others. In case of unexpected desynchronization (typically a state that is expected to be synchronized regarding the messages that are delivered and those who are not), a re-initialization of that data should be possible. Message streaming should be used as much as possible to replicate a data from scratch, including transformations process.

Reactivity has a core concept but producing and consuming data can take different forms. This is why Reactivity must be extensible by providing a plugable API allowing extensions to register to the system. An extension can cover different capabilities.

Technologies selected in respect of architecture principles must be open source. Selection process must have a particular attention to the license to avoid patent or viral clause in the open source license. Apache 2.0 and MIT licenses are the typical open source license recommended for the technologies used by Reactivity.

The latest version of Java is the version 8. This version will be used in the core of Reactivity as it’s well maintained by Oracle, provides new features (especially around the Stream API) and is required by different technologies selected in the next section.

The goal of Reactivity is also to take advantage from the features introduced in next Java releases and be upgraded to Java 9 when available.

Progressive web apps are a set of technologies and concepts designed to give the best possible user experience, whatever the device. The main concepts will be implemented in Reactivity wep-application:

In order to leverage those concepts, Reactivity will use Polymer, in its 2.0 version (preview for the moment). Polymer is based on the web-components and provides some. There are several advantages to relying on web components, and especially two that have a definite interest compared to Reactivity:

Moreover, as Reactivity must load fast, the code-splitting pattern, and the PRPL pattern are two solutions.

Apache Kafka is the central platform of Reactivity that takes messages sent by producers and delivers them to consumers. Kafka provides the key following benefits that make it a perfect solution for Reactivity requirements:

WEB technologies such as Javascript describe object structures in JSON format. This format is also used in document oriented databases. Each artefact can be represented in a single document that will be loaded by the database when requested by the user in a particular view. Some transformation results can be also represented in a document, which makes document database adapted for Reactivity requirements.

Couchbase is document oriented database and distributed by design that meets the scalability requirements of Reactivity. This database also provides additional interesting products like embedded database for mobile that can be leveraged to cover new features. This is why our primary choice goes for Couchbase server to store the data of Reactivity.

Since a JSON document data model can be easily extended by addibg more properties to it, Couchbase also offers the opportunity to dynamically add extensions to Reactivity.

Front-end architecture will balance the requests coming from the web clients to different instances. This will be achieved by Traefik, a moden reverse proxy dealing with microservices architecture. A ZooKeeper support is provided out of the box, allowing to detect all available back-ends and remove them in case of failure.

Different components will interract with different roles:

ZooKeeper is used as a centralized system to resolve all microservices. It’s deployed in a highly available cluster where nodes replicate their configuration to each others. Microservices register themselves to ZooKeeper and retrieve dependencies through it.

The streaming platform implemented with Kafka offers a REST API that can be consumed to produce new artifacts. When a new event is received by Kafka, it keeps it until it has been delivered and acknowledge by a consumer. Kafka is deployed as a cluster of multiple distributed nodes. All nodes register to ZooKeeper.

Couchbase is the document oriented database that stores data of Reactivity. This distributed system is deployed as a cluster where nodes register to ZooKeeper. Asynchronous driver is available to read and write documents to the database.

In this architecture, Traefik is in charge of balancing requests coming from web application to:

Traefik will use ZooKeeper to retrieve the correct backends instances. Other components will retrieve their dependent services also via ZooKeeper but a client load balancing approach will be used:

The version is a value indicating the version of the module that produces the message. This version can be used by the consumer to know what kind of assertion can be made on the data. By default, new version must be backward compatible which means that a consumer should be able to deal with an event flagged with an older version.

Version format follows the semver protocol.

The snapshot field indicates if the version is under development. Snapshots are not always backward compatible, an the backend will never send snapshot documents when the application version is a release. Therefore, snapshot will be always false in production.

The event describes the nature of the message. The possible values are an enumeration specific to the type of data contained inside the message.

No field in the message indicates the type of data because it corresponds to the topic they are published to. In the core module, messages can be published to the following topic:

This statement is true only when the client produce and consumes data directly from/to the topic (through Kafka). However, there is the particular case of a WEB application subscribing to a source of events comming from a suspended SSE connection from the server.

In this simple diagram we can see that the Broadcaster is consuming different topics from Kafka. When a message is received, it’s pushed to the web application that established a SSE connection. The only chance for the web application to understand the type of data represented by a received message is to find topic attribute in the JSON.

The data contains the payload of the event provided by the producer. The structure of the data depends on the topic where is has been published, the event and version fields values. The consumer will use those informations to know what structure is expected in the data value.

timestamp and id are not defined by the producer but are assigned by the streaming platform. They allow to identify the message inside Reactivity system and to know when it has been published by the consumer.

Therefore, a producer can generate a message that simply have this structure:

Since the id is not generated by the producer, it’s not possible for him to recognize a produced message when it’s received through another channel. The tag allows him to address this issue. The producer can associate a random value that looks like an unique UID to the tag attribute, allowing him to implement a kind of acknowledgment for the produced message. This is particularly handy for web applications that receive a message previously produced. In fact, the web application acts as a producer who expects to receive its own messages sent from the broadcaster.

In this figure, we see that the message A produced by the web application is received from the broadcaster. With the tag attribute whose value won’t have been changed, the web application will be able to recognized it and to treat it differently comparing to the messages produced by other instances. For example, the web application can perform an optimistic change on the UI as soon as the message is sent, giving a better feeling of performance to the user. When the message is received, the web application can simply ignore it. The web application can also raise an error if the message is never received (timeout), which means that the message has possibly been discarded for any reason.

All messages related to the User data type are published to the user topic.

All messages related to the Organization data type are published to the organization topic.

All messages related to the Artifact data type are published to the artifact topic.

Loads all the organizations for the current user.

You can subscribe to a particular organization to receive the view and the artifacts corresponding to those views.

When a view is loaded, you also receive a subset of the associated artifacts according to the configured Period. You can use endpoints to load more artifacts.

Before installing the specific components related to Reactivity, make sure you have installed the following tools:

Core components are hosted in this repository. You can fork it and then clone it. The branching model follows the GitHub flow. You will find for each feature under development a dedicated branch.

The reactivity repository is currently organized with the following structure:

If docker is installed on your OS, you can use it to install quickly Couchbase server. This also provides an environment with command line tools like travis or CloudFoundry already installed.

Prompt a shell and move to the directory where the repository has been cloned. Run the following commands:

First we are going to launch the backend. You need to build two artifacts with maven. You can achieve this through your Java IDE which has to resolve dependencies and compile code for the two following artifacts:

With IntelliJ, it’s easy to import both modules with File > New > Project From Existing Source and then select Maven as the external model to be used.

Once imported, you can execute the main class io.reactivity.core.broadcaster.Application of the broadcaster project to start the server. Make sure you have a local instance of Couchbase which is running. The application will try to connect to it on its default ports.

The application is launched with Spring Boot and uses Netty by default. You can alternatively use tomcat by enabling the tomcat profile in maven. By default the server listens the port 8080, you should be able to see a test page by browsing http://localhost:8080/.

The application stores documents in the artifact bucket of Couchbase by default. You can configure a different bucket and Couchbase server location with the following properties:

Those two properties can be specified in several ways and are resolved with Spring Boot. You can see the different options offered by the framework here.

With IntelliJ, a possible option is to execute the main class with the properties declared as parameter. You can achieve this by editing the Run/Debug Configuration of the main Application and specifying the following information in Program arguments:

--reactivity.couchbase.nodes=[my host goes here] --reactivity.couchbase.bucket=[my bucket goes here]

The frontend web application manages the dependencies with npm and is packaged with gulp. Simply move to the front directory, open a command line and run the following commands:

npm install npm start

This will install the dependencies and then start a HTTP server listening the port 3000. You should be able to browse http://localhost:3000 at this moment. The HTTP server is also configured to redirect traffic from http://localhost:3000/api to http://localhost:8080/. This allows the frontend web application to naturally consume the backend application previously started.

You can refer to the README.md inside the front directory for more details.

Before installing the specific components related to Reactivity, make sure you have installed the following tools:

Core components are hosted in this repository. You can fork it and then clone it. Identify the branch of the tag corresponding to the version of the code that you want to deploy and checkout it.

The reactivity repository is currently organized with the following structure:

You have a manifest.yml file that you can customize according to your need. You should have a particular attention to the routes that are configured for each module. Adapt them according to the domains registered inside your CloudFoundry instance.

First, move to the core/java-lib directory and run mvn clean install -Dmaven.test.skip. Then, move to the core/broadcaster directory and run the same command. This will generate a JAR file in the core/broadcaster/target directory to be deployed.

Finally, move to the front directory and run npm run dist. This will package the static files to be deployed in a front/dist directory.

Now we can upload the artifacts to CloudFoundry. Use cf login to identify yourself with the remote server (see more details here).

Then push the apps with cf push from the root directory. This will use the manifest.yml and uploads the two artifacts.

The backend will try to connect to Couchbase when it starts. You can define in CloudFoundry the following environment variables to specify the location of your server.