portal-webapp/docs/src/ToolsLibraries.tex

\chapter{Tools \& Libraries}
\label{toolslibraries}

This chapter describes the tools and libraries used in the \portal development environment. Both where to get the necessary components and how to set them up are covered here. 
This chapter is only of interest to \portal developers. If you are a \portal player or administrator you do not need to read this chapter. Instead the details for setting up the 
components of a working \portal system are described in Chapter~\ref{deployment} on Deployment.

The following list reflects the tools and libraries used by the \portal developers.

\begin{itemize}
	\item Eclipse \EclipseVersion
	\item NetBeans \NetBeansVersion
	\item Java \JavaVersion
	\item Maven \MavenVersion
	\item Ant \AntVersion
	\item AngularJs \AngularJsVersion
	\item Spring \SpringVersion
	\item Hibernate \HibernateVersion
	\item Quartz \QuartzVerson
	\item JSch \JSchVersion
	\item JGit \JGitVersion
	\item Node.js \NodeVersion
	\item Yarn \YarnVersion
	\item Flyway \FlywayVersion
	\item Tomcat \TomcatVersion
\end{itemize}

\section{Eclipse}
Eclipse\cite{eclipse} provides IDEs and platforms for nearly every language and architecture such as Java, C/C++, JavaScript and PHP IDEs. It is built on extensible platforms for 
creating desktop, Web and cloud IDEs. These platforms deliver the most extensive collection of add-on tools available for software developers.

The \portal uses Eclipse freely to develop both the client and server side code. Eclipse is not needed to build the application. The offical version of Eclipse used by the \portal 
project is \EclipseVersion. Other nearby versions are likely to work as long as it supports Java \JavaVersion

\section{NetBeans}
NetBeans\cite{netbeans} is an integrated development environment (IDE) for Java. NetBeans allows applications to be developed from a set of modular software components called modules. 
NetBeans runs on Microsoft Windows, macOS, Linux and Solaris. In addition to Java development, it has extensions for other languages like PHP, C, C++ and HTML5, Javadoc and Javascript. 
Applications based on NetBeans, including the NetBeans IDE, can be extended by third party developers.

The \portal uses NetBeans to generate the database entity classes for use as it is more robust than what Eclipse provides. See Chapter~\ref{development} section~\ref{netbeans} for 
more information 

\section{Java}
Java is the language the server is written in \cite{java}.

\portal uses extensive features that are provided in version \JavaVersion, therefore; previous versions of java will not be able to compile the server code.

\section{Maven}
Maven is utilized for the build process \cite{maven}. It dynamically downloads the required libraries as specified in the \filename{pom.xml} 
files as to reduce the size of the overall projects. The downloaded libraries are stored within the users home directory under \filename{.m2/repository}. 
First maven will look inside the local repository for the required library and if its not found, it will download it from the internet automatically. With that being said, 
an active internet connection is required to build the web application for the first time. It is possible to configure the \filename{pom.xml} files for offline builds but requires 
the libraries to be pre staged in the local repository or a service such as Nexus.

To build the web application, be in the root project folder and issue the commaond \command{mvn install}. This will start the download process and run through various tasks to build 
and verify the application. If any part of the process fails to complete (such as a failed java compile or invalid javascript) the build will be terminated and will not continue until 
the errors are corrected.

The \portal utilizes Maven version \MavenVersion for its build process. It is possible to use older versions of Maven as long as it is version 3.0+

\section{Ant}
Ant\cite{ant} is only utilized to copy the war file to the tomcat directory if tomcat is not currently running. The maven command \command{antrun:run@warcopy} is the command that will 
invoke the ant process. \portal utilizes ant version \AntVersion but older versions of Ant will work since the process of copying files has not changed

\section{AngularJs}
AngularJs\cite{angularjs} is extensively used inside the client html code along with various other 3rd party javascript libraries. The list of 3rd party libraries used in the client
can be found in \filename{assets.js}. It is not necessary to download the AngularJs library as it will be downloaded automatically along with the other vendor libraries during the build 
process. Yarn is utilized to do this task and place references to the files inside the \filename{index.html} during build. AngularJs is specifically called out because of how much it does 
inside the client and developers should be familiar with it. \portal utlizes version \AngularJsVersion of AngularJs as specified in the \filename{package.json} file. 
 
\section{Spring}
Spring Framework is the workhorse library of the server\cite{spring}. It drives every aspect of the server, from client authentication, page permissions and database transactions. 
It is responsible for converting json strings passed from the client to java objects in which can be utilized in the spring services. Spring operates on the Model View Controller aspect 
(MVC) in which the client requests information and spring returns the model to be displayed on the client page.

The typical workflow for spring can be shown in Figure~\ref{fig:spring-flow}

\begin{figure}[htbp]
	\centering
	\scalebox{0.6}{\includegraphics*{figures/spring-flow.png}}
	\caption{Spring Workflow}
	\label{fig:spring-flow}
\end{figure}

Spring security is utilized to make sure users do not access portions of the website they are not permitted to view. This is done through permission rules setup in the code and through 
an authorization token which is assigned to the user during login. The authentication token tells the security provider which part of the website the user has access to. 
The authentication token is populated with information that is retrieved from the permissions tables based on the users group permissions.

Spring websocket is utilized for the server to pass information back to the client asynchronously. Every time a user logins in they are automatically subscribed to the system websocket 
channel and their own private channel so the user can receive personal websocket messages and system wide websocket messages. This allows for the client to update the users screen when 
other users change something on the website. Without websockets the user would not know what has changed by other users unless they refresh their browser. During login, the website 
checks to see if websockets is allowed (some web site providers might block websockets). If the check passes normal processes can continue. If it fails, the site will fall back into a 
degredated state in which the user will not be able to get all the changes as specified above.

Spring CRUD Repository is utilized for querying the database. Developers do not need to worry about creating SQL statements as the repository will take care of that for them. 
It allows for create, read, update, and delete operations on the table the repository references. This is done through JPQL (java persistence query language) and Spring annotations. 

Developers should consult the Spring JPA guide if more spring query methods are needed during development. The guide can be found at 
\url{https://docs.spring.io/spring-data/jpa/docs/current/reference/html/}

\portal utilizes Spring Framework version \SpringVersion. The developer does not need to worry about downloading this library is it will automatically be downloaded through maven during 
build time.

\section{Hibernate}
Hibernate ORM\cite{hibernate} enables developers to more easily write applications whose data outlives the application process. As an Object/Relational Mapping (ORM) framework, 
Hibernate is concerned with data persistence as it applies to relational databases (via JDBC). Hibernate maps database tables to Java objects called Java Persistence Objects (JPO). 
This allows developers to create normal objects and act on them like a regular object but the changes will then persist to the database once the object has been saved. \portal utilizes 
Hibernate through the use of Spring which uses Hibernate in the background for persistence. The JPO objects are also referenced in the Spring CRUD Repositories. These objects are the 
objects the developer will use when doing CRUD operations (Create/Read/Update/Delete).

\portal utilizes Hibernate ORM version \HibernateVersion. The developer does not need to worry about downloading this library is it will automatically be downloaded through maven 
during build time.

\section{Quartz Scheduler}
Quartz Scheduler\cite{quartz} is a richly featured, open source job scheduling library that can be integrated within virtually any Java application - from the smallest stand-alone 
application to the largest e-commerce system. Quartz can be used to create simple or complex schedules for executing tens, hundreds, or even tens-of-thousands of jobs; jobs whose tasks 
are defined as standard Java components that may execute virtually anything you may program them to do. The Quartz Scheduler includes many enterprise-class features, such as support for 
JTA transactions and clustering.

Quartz is freely usable, licensed under the Apache 2.0 license.  

Sample uses of job scheduling with Quartz:
\begin{itemize}
  \item \bold{Driving Process Workflow:} As a new order is initially placed, schedule a Job to fire in exactly 2 hours, that will check the status of that order, and trigger a 
  warning notification if an order confirmation message has not yet been received for the order, as well as changing the order's status to ``awaiting intervention''.
  \item \bold{System Maintenance:} Schedule a job to dump the contents of a database into an XML file every business day (all weekdays except holidays) at 11:30 PM.
  \item Providing reminder services within an application.  
\end{itemize}

\portal utilizes Quartz Scheduler version \QuartzVerson to schedule all the tasks to be completed. The developer does not need to worry about downloading this library is it will automatically be downloaded through maven 
during build time.

\section{JSch}
JSch\cite{jsch} is a pure Java implementation of SSH2.

JSch allows you to connect to an sshd server and use port forwarding, X11 forwarding, file transfer, etc., and you can integrate its functionality into your own Java programs. 
JSch is licensed under BSD style license. 

\portal utilizes JSch version \JSchVersion. The developer does not need to worry about downloading this library is it will automatically be downloaded through maven during build time.

\section{JGit}
JGit\cite{jgit} i is a pure Java implementation of the Git version control system. Git is a distributed SCM, which means every developer has a full copy of all history of every revision 
of the code, making queries against the history very fast and versatile. 

\portal utilizes JGit version \JGitVersion. The developer does not need to worry about downloading this library is it will automatically be downloaded through maven during build time.

\section{Node.js}

Node.js\cite{nodejs} is a JavaScript runtime built on Chrome's V8 JavaScript engine. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient. 

Node.js is used to assemble and verify the client javascript and html. Node.js stores the 3rd party vendor libraries in the \filename{portal\_client} project folder under 
\filename{node\_modules} folders. Typically when using Node.js, there is a Node.js server running that serves up these 3rd party libraries to the client, but \portal does not use a
node.js server to do this. Instead during compile time, the process looks at the \filename{assets.js} file to see which 3rd party libraries are needed and appends these files to the 
\filename{index.html} file for refence by the client.

Once the \filename{index.html} file has been written it is copied to the rest of the client code, then utilizing \command{jshint}, the javascript is checked for any errors. It uses 
'strict' checking to make sure there is proper syntax in the javascript such as semi-colons in the right place and proper closing tags. If the javascript check fails the whole build will
fail and a message will tell the developer where the javascript error is.  

Node.js is also used for live reloading of the client code. Utilizing \command{grunt}, the developer can make local changes to the client code and see the changes live on the website. 
This is done through a proxy service that points to the local content instead of the deployed content (as long as the deployed content is on the same machine as the developer is on). 
This allows the developer to make changes and not have to rebuild the whole application to see the client side changes (See Section~\ref{grunt} for more details).

\portal utilizes version \NodeVersion\ of Node.js. It is possible to use older versions of Node.js but it not recommended for security concerns.

\section{Yarn}
\label{yarn2}
  
Yarn\cite{yarn} is a package manager program that allows for easy download of 3rd party javascript/typescript libraries. It caches every package so it never needs to download it again. 
Yarn is used as a replacement for node.js package manager (npm) and utilizes the same \filename{package.json} file as npm along with installing the packages in the same directory that 
npm (\filename{portal\_client/node\_modules}).

This allows for developers to switch easily from npm to yarn. During build time, maven will automatically download yarn locally to the project so it can use the local process to download 
the 3rd party libraries as specified in the \filename{package.json} file.

Yarn needs to be installed on the developers home path to be utilizes by the developer to install new packages. To install new packages issue the command
 \command{yarn add <package-name> --save}. This will download the request package and save the package to the \filename{package.json} file for future reference.

See~\ref{yarn} for more details.

\portal utilizes version \YarnVersion of yarn. No older version can be used at this time.

\section{Flyway}
Flyway\cite{flyway} is an open source database migration tool. It strongly favors simplicity and convention over configuration. It is based around 7 basic commands: Migrate, Clean, 
Info, Validate, Undo, Baseline and Repair. Migrations can be written in SQL (database-specific syntax (such as MySQL, MSSQL, Oracle) is supported) or Java (for advanced data transformations
or dealing with LOBs). It has a Command-line client, a Java API (also works on Android) for migrating the database on application startup and a Maven plugin.

\portal utilizes the sql syntax migration along with java implementaiton and maven plugin. It is not necessary to download flyway as maven will download the required lbraries automatically
during build time.

Flyway allows for easy database migrations between versions from older versions of the software to new version (but not from newer to older). This allows for every deployment of the web
application to perform the migraiton if needed to keep the database up to date. Flyway looks into the specified migration folder to execute migration scripts. These scripts can range from
table creation, to adding data and modifying existing tables/data. \portal utilizes Flyway at two different times. The first time is during build flyway is executed to do the required
migrations. The second time is during applcation startup to perform any migrations that need to happen on a production enviornment. 

Flyway creates a table within the application table schema to keep track of which migrations have been applied. It also creates a checksum for each file. During migration, 
flyway will check the checksum of the file against a database entry. If no entry exists the migration is performed and an entry is entered into the flyway table. If the checksum
matches, that migration is skipped as its already been applied. If the checksum does not match the entry, an error will be thrown and the build/startup will fail. This is to ensure that 
no previous migration files have been modified. All new modifications to existing migrations should be performed in another migration script.

The flyway maven plugin allows developers to perform clean, migrate, repair, info, and validate commands. Below are the maven commands that can be executed for various flyway processes

\begin{itemize}
  \item \command{mvn flyway:clean}\newline
  Clean drops all objects in the configured schemas. Clean is a great help in development and test. It will effectively give you a fresh start, by wiping your configured schemas 
  completely clean. All objects (tables, views, procedures, \ldots) will be dropped. Needless to say: \bold{do not use against your production DB!}
  \item \command{mvn flyway:migrate}\newline
  Migrate migrates the schema to the latest version. Flyway will create the schema history table automatically if it doesn�t exist. Migrate is the centerpiece of the Flyway workflow. 
  It will scan the filesystem or your classpath for available migrations. It will compare them to the migrations that have been applied to the database. If any difference is found, it 
  will migrate the database to close the gap. Migrate should preferably be executed on application startup to avoid any incompatibilities between the database and the expectations 
  of the code.
  \item \command{mvn flyway:repair}\newline
  Repair repairs the schema history table. Repair is your tool to fix issues with the schema history table. It has two main uses:
  \begin{itemize}
    \item Remove failed migration entries (only for databases that do NOT support DDL transactions)
    \item Realign the checksums, descriptions and types of the applied migrations with the ones of the available migrations
  \end{itemize}
  \item \command{mvn flyway:info}\newline
  Info prints the details and status information about all the migrations. Info lets developers know where they stand. At a glance they will see which migrations have already been 
  applied, which other ones are still pending, when they were executed and whether they were successful or not.
  \item \command{mvn flyway:validate}\newline
  Validate validates the applied migrations against the available ones. Validate helps developers verify that the migrations applied to the database match the ones available locally. 
  This is very useful to detect accidental changes that may prevent you from reliably recreating the schema.
\end{itemize}

\portal utilizes version \FlywayVersion of flyway. The developer does not need to worry about downloading this library is it will automatically be downloaded through maven during build time.

\section{Tomcat}
\label{tomcat2}
Apache Tomcat\cite{tomcat}, often referred to as Tomcat Server, is an open-source Java Servlet Container developed by the Apache Software Foundation (ASF). Tomcat implements several 
Java EE specifications including Java Servlet, JavaServer Pages (JSP), Java EL, and WebSocket, and provides a "pure Java" HTTP web server environment in which Java code can run.

Tomcat is developed and maintained by an open community of developers under the auspices of the Apache Software Foundation, released under the Apache License 2.0 license, and is 
open-source software.

While \portal was developed to run on tomcat, it is possible to be ran on other servers that support \filename{.war} files such as JBOSS, TomEE, and Wildfly.

Once tomcat is installed on the host server, the only configuration that needs to change is adding a user to the manager-script role. This user should be specified in 
\filename{conf/tomcat-users.xml} file under the tomcat home directory. This user and the corresponding password should be the same one as reflected in the \filename{settings.xml} 
file (see Reference~\ref{sample-settings.xml} and Section~\ref{settings.xml}). This will allow the application to redeploy to tomcat without having to shutdown deploy and startup tomcat 
after a successful build. 

The developer just has to issues the command \command{mvn tomcat7:redeploy} to deploy the application to tomcat.

\portal utilizes version \TomcatVersion of Tomcat. Older or newer versions of tomcat can be used but it is not recommend to go any lower than tomcat 7