Architecture contains UI Component, Logic Component, Storage Component, Common Component, Test Driver Component, E2E Component, Client Component.
TEAMMATES is a Web application that runs on Google App Engine (GAE).
Given above is an overview of the main components.
The UI Browser seen by users consists of Web pages containing HTML, CSS for styling and JavaScript for client-side interactions such as sorting, input validation, etc..
This UI is a single HTML page generated by Angular framework.
The initial page request is sent to the server over HTTP, and requests for data are sent asynchronously with AJAX.
In the UI Server the entry point for the application back end logic is designed as a REST-ful controller.
The main logic of the application is in POJOs (Plain Old Java Objects).
The storage layer of the application uses the persistence framework provided by GAE Datastore, a NoSQL database.
The following explains the use of the Test Driver.
TEAMMATES makes heavy use of automated regression testing.
Test data is transmitted using JSON format.
TestNG is used for Java testing (all levels) and Jest for JavaScript unit-testing.
HttpUnit is used to set up a simulated web server in servlet-level tests, where an actual web server is not required.
The E2E end-to-end component is used to interact with the application as a whole with Web browsers.
Its primary function is for E2E tests.
Selenium Java is used to automate E2E testing with actual Web browsers.
The Client component can connect to the back end directly without using a Web browser.
It is used for administrative purposes, e.g. migrating data to a new schema.
The Common component contains utility code (data transfer objects, helper classes, etc.) used across the application.
The diagram below shows how the code in each component is organized into packages and the dependencies between them.
logic, ui.website, ui.controller represent an application of Model-View-Controller pattern.
ui.website is not a real package.
It is a conceptual package representing the front-end of the application.
The diagram below shows the object structure of the UI component.
ui.website is not a Java package.
It is written in Angular framework and consists of HTML, SCSS, and TypeScript files.
The framework will build those files into HTML, CSS and JavaScript files ready to be used by standard Web browsers.
The UI component is the first stop for 99% of all requests that are received by the application.
Such a request will go through the following steps.
First, request received by the GAE server.
Second, custom filters are applied according to the order specified in web.xml, e.g. OriginCheckFilter.
Third, request forwarded to a Servlet object as specified in web.xml for further processing, depending on the type of request.
There are two general types of requests.
The user-invoked requests and the automated (GAE server-invoked) requests, which are processed differently.
User-invoked requests are all requests made by the users of the application, typically from the Web browser (i.e. by navigating to a particular URL of the application).
The request will be processed as in the image.
The initial request for the web page will be processed as follows.
First, request forwarded to WebPageServlet.
Second, WebPageServlet returns the built single web page (index.html).
Third, the browser will render the page and execute the page scripts, most of the time requiring AJAX requests to the server.
Subsequent AJAX requests sent to the server will be processed as follows.
First, request forwarded to the WebApiServlet.
Second, WebApiServlet uses the ActionFactory to generate the matching Action object, e.g. InstructorHomePageAction.
Third, WebApiServlet executes the action.
Fourth, the Action object checks the access rights of the user.
If the action is allowed, it will be performed, interacting with the Logic component as necessary.
Fifth, the Action packages and processes the result into an ActionResult object.
The most common format is JsonResult (requests for obtaining data or processing existing data), and other formats are defined as necessary, e.g. FileDownloadResult (e.g. downloading feedback session report) and ImageResult (e.g. profile pictures).
Sixth, WebApiServlet sends the result back to the browser which will then process it on the front-end.
Requests for static asset files (e.g. CSS, JS files, images) are served directly without going through web.xml configuration at all.
The Web API is protected by two layers of access control check.
These are origin check, authentication and authorization check.
Origin check mitigates CSRF attack.
Authentication and authorization check checks if the logged in user (or lack thereof) has sufficient privileges to trigger the API's actions.
Special keys (csrf key and backdoor key) can be used to bypass each of the checks, typically for testing purpose.
Those keys are strings known only to the person who deployed the application (typically, the administrator).
Automated requests are all requests sent automatically by the GAE server during specific periods of time.
This type of request will be processed as follows.
First, the source of the request will be checked for administrator privilege.
If this privilege is absent (e.g. non-administrator users trying to invoke the automated actions), the request will be dropped and a 403 Forbidden status will be returned.
Second, requests generated by the GAE server are equipped with this privilege.
Third, administrators can manually invoke these requests.
This is particularly useful in testing the actions associated with those requests.
Fourth, request forwarded to the AutomatedServlet.
Fifth, automatedServlet uses the AutomatedActionFactory to generate the matching AutomatedAction object, e.g. CompileLogsAction.
Sixth, automatedServlet executes the action.
Seventh, the corresponding AutomatedAction will be performed, interacting with the Logic component as necessary.
GAE server sends such automated requests through two different configurations.
These configurations are cron jobs or task queue workers.
Cron jobs are jobs that are automatically scheduled for a specified period of time, e.g. scheduling feedback session opening reminders.
It is configured in cron.xml.
Task queue workers are hybrids of user-invoked and GAE-invoked in that they are queued by users (i.e. users request for the tasks to be added to queue), but executed by GAE (i.e. GAE determines when and which tasks in the queue are executed at any point of time).
This is typically used for tasks that may take a long time to finish and can exceed the 1 minute standard request processing limit imposed by GAE.
It is configured in queue.xml as well as the TaskQueue nested class of the Const class.
Since the high-level workflow of processing a request is same for any request (differing by the two request types only), we use the Template Method pattern to abstract the process flow into the Action and AutomatedAction classes.
The Logic component handles the business logic of TEAMMATES.
In particular, it is responsible for the following.
Managing relationships between entities, e.g. cascade logic for create/update/delete.
Managing transactions, e.g. ensuring atomicity of a transaction.
Sanitizing input values received from the UI component.
Providing a mechanism for checking access control rights.
Connecting to GAE-provided or third-party APIs, e.g. for adding tasks to the task queue and for sending emails with third-party providers.
Package overview contains logic.api, logic.core.
logic.api provides the API of the component to be accessed by the UI.
logic.core contains the core logic of the system.
Logic API is represented by the classes Logic, GateKeeper, EmailGenerator, EmailSender, TaskQueuer.
Logic is a Facade class which connects to the several Logic classes to handle the logic related to various types of data and to access data from the Storage component.
GateKeeper checks access rights of a user for a given action.
EmailGenerator generates emails to be sent.
EmailSender sends email with the provider chosen based on the build configuration.
It connects to the email provider by using the appropriate Service class.
TaskQueuer adds tasks to the task queue.
It connects to GAE's task queue API.
To access control the following information are presented.
Although this component provides methods to perform access control, the API itself is not access controlled.
The UI is expected to check access control (using GateKeeper class) before calling a method in the Logic.
To API for creating entities the following information are presented.
Null parameters causes an assertion failure.
Invalid parameters throws InvalidParametersException.
Entity already exists throws EntityAlreadyExistsException (escalated from Storage level).
To API for retrieving entities the following information are presented.
Attempting to retrieve objects using null parameters causes an assertion failure.
Entity not found returns null if the target entity not found.
This way, read operations can be used easily for checking the existence of an entity.
To API for updating entities the following information are presented.
Update is done using UpdateOptions inside every Attributes.
The UpdateOptions will specify what is used to identify the entity to update and what will be updated.
Entity not found throws EntityDoesNotExistException.
Invalid parameters throws InvalidParametersException.
To API for deleting entities the following information are presented.
The following explains the FailDeleteSilentlyPolicy.
In general, delete operation do not throw exceptions if the target entity does not exist.
This is because if it does not exist, it is as good as deleted.
The following explains the Cascade policy.
When a parent entity is deleted, entities that have referential integrity with the deleted entity should also be deleted.
Refer to the API for the cascade logic.
The Storage component performs CRUD (Create, Read, Update, Delete) operations on data entities individually.
It contains minimal logic beyond what is directly relevant to CRUD operations.
In particular, it is reponsible for the following.
Validating data inside entities before creating/updating them, to ensure they are in a valid state.
Hiding the complexities of datastore from the Logic component.
All GQL queries are to be contained inside the Storage component.
Hiding the persistable objects.
Classes in the storage.entity package are not visible outside this component to hide information specific to data persistence.
Instead, a corresponding non-persistent data transfer object named Attributes (e.g., CourseAttributes is the data transfer object for Course entities) object is returned.
These datatransfer classes are in common.datatransfer package, to be explained later.
The Storage component does not perform any cascade delete/create operations.
Cascade logic is handled by the Logic component.
Package overview contains storage.api, storage.entity, storage.search.
storage.api provides the API of the component to be accessed by the logic component.
storage.entity contains classes that represent persistable entities.
storage.search contains classes for dealing with searching and indexing.
Note that the navigability of the association links between entity objects appear to be in the reverse direction of what we see in a normal OOP design.
This is because we want to keep the data schema flexible so that new entity types can be added later with minimal modifications to existing elements.
Represented by the Db classes.
These classes act as the bridge to the GAE Datastore.
Add and Delete operations try to wait until data is persisted in the datastore before returning.
This is not enough to compensate for eventual consistency involving multiple servers in the GAE production environment.
However, it is expected to avoid test failures caused by eventual consistency in dev server and reduce such problems in the live server.
Eventual consistency here means it takes some time for a database operation to propagate across all serves of the Google's distributed datastore.
As a result, the data may be in an inconsistent states for short periods of time although things should become consistent eventually.
For example, an object we deleted may appear to still exist for a short while.
Implementation of Transaction Control has been minimized due to limitations of GAE environment and the nature of our data schema.
To API for creating the following information are presented.
Attempt to create an entity that already exists throws EntityAlreadyExistsException.
Attempt to create an entity with invalid data throws InvalidParametersException.
To API for retrieving the following information are presented.
Attempt to retrieve an entity that does not exist returns null.
To API for updating the following information are presented.
Attempt to update an entity that does not exist throws EntityDoesNotExistException.
Attempt to update an entity with invalid data throws InvalidParametersException.
To API for deleting the following information are presented.
Attempt to delete an entity that does not exist fails silently.
The Common component contains common utilities used across TEAMMATES.
Package overview contains common.util, common.exceptions, common.datatransfer.
common.util contains utility classes.
common.exceptions contains custom exceptions.
common.datatransfer contains data transfer objects.
common.datatransfer package contains lightweight data transfer object classes for transferring data among components.
They can be combined in various ways to transfer structured data between components.
Given below are three examples.
Test Driver can use the DataBundle in this manner to send an arbitrary number of objects to be persisted in the database.
This structure can be used to transfer data of a course (e.g., when constructing the home page for an instructor).
This structure can be used to send results of a feedback session (e.g., when showing a feedback session report to an instructor).
Some of these classes are methodless and thus more of a data structure rather than a class.
These classes use public variables for data for easy access.
This component automates the testing of TEAMMATES.
Package overview contains test.driver, test.cases and subpackages.
test.driver contains infrastructure and helpers needed for running the tests.
test.cases contains test cases.
Sub-packages contains x.testdriver, x.datatransfer, x.util, x.logic, x.storage, x.search, x.webapi, x.automated.
x.testdriver contains component test cases for testing the test driver infrastructure and helpers.
x.datatransfer contains component test cases for testing the datatransfer objects from the Common component.
x.util contains component test cases for testing the utility classes from the Common component.
x.logic contains component test cases for testing the Logic component.
x.storage contains component test cases for testing the Storage component.
x.search contains component test cases for testing the search functions.
x.webapi contains system test cases for testing the user-invoked actions.
x.automated contains system test cases for testing the system-automated actions (manually invoked during testing).
Some Component tests are pure unit tests (i.e. testing one component in isolation) while others are integration tests that test units as well as integration of units with each other.
Front-end files (particularly TypeScript) are tested separately with Jest.
The test cases are found in x.spec.ts files.
This is how TEAMMATES testing maps to standard types of testing.
The E2E component has no knowledge of the internal workings of the application and can only interact either with Web browser as a whole application or REST API calls for the back-end logic.
Its primary function is for E2E tests and L&P (Load & Performance) tests.
Package overview contains e2e.util, e2e.pageobjects, e2e.cases, x.util, x.e2e, x.lnp.
e2e.util contains helpers needed for running E2E tests.
e2e.pageobjects contains abstractions of the pages as they appear on a Browser (i.e. SUTs).
e2e.cases contains test cases.
x.util contains component test cases for testing the test helpers.
x.e2e contains system test cases for testing the application as a whole.
x.lnp contains load and performance tests (experimental).
The Client component contains scripts that can connect directly to the application back-end for administrative purposes, such as migrating data to a new schema and calculating statistics.
Package overview contains client.util, client.remoteapi, client.scripts.
client.util contains helpers needed for client scripts.
client.remoteapi classes needed to connect to the back end directly.
client.scripts scripts that deal with the back end data for administrative purposes.