Development Environment

We recommend developing SkillTree in a *nix environment.

Prerequisites

• modern *nix environment
• JDK 17+, we suggest Open JDK (opens new window)
• Git (opens new window) version 2.23+
• Node.js (opens new window) v12+ and npm (opens new window) 6+
• Maven (opens new window) 3.6.2+
• PostgreSQL (opens new window) 12+

Development Overview

The following sections will cover:

1. skills-service project development
2. skills-client project development

TIP

More than likely your development will be scoped to the skills-service project but, although rare, there are situations when you'll need to modify the skills-client as well. If you are making improvements to the SkillTree dashboard or its Skills Display views then your development will be scoped within the skills-service project. If you have not yet, please take few minutes to review the architecture section.

The tech stack is large but to get started you will need to be familiar (but more likely experienced) with these core technologies:

• Java and Groovy
• Spring Framework (especially Spring Boot (opens new window))
• Web stack: Javascript, html, css
• Vue.js (opens new window)
• cypress.io (opens new window)
• Build tools: maven, npm & webpack

Each sub-section follows the same flow:

1. Get, build, and test
2. cypress.io (opens new window) end-to-end tests
3. Day-to-day development
4. Test checklist

Skills-service development

The skills-service project encapsulates code for the SkillTree dashboard, REST apis and Skills Display views.

Get, build, and test

First things first, fork and checkout the code (please see the Contribution Steps), build it and run all of the unit and integration tests. Then we'll discuss development and testing steps.

git clone <github>:skills-service.git

To build the project, run the unit tests and the backend integration tests:

cd skills-service
mvn install

Since this process runs all of the service integration tests it will take a while (5 to 20 minutes depending on your hardware).

SkillTree uses maven (opens new window) and npm (opens new window) for its dependency management and to facilitate the build lifecycle. Let's get familiar with the project's layout:

skills-service
└───service
│   │   pom.xml
|   └───target
|   |   skills-service-<version>.jar
|   |   ....
└───dashboard
|   │   pom.xml
|   │   package.json
|   |   ....
└───client-display
|   │   pom.xml
|   │   package.json
|   |   ....
└───e2e-tests
|   │   package.json
|   |   ....
└─── pom.xml

The runtime artifact is a spring boot (opens new window) application and will be created in service/skills-service-<version>.jar; we ran mvn install to generate this artifact and the following sequence of steps were performed:

• Build dashboard web application: npm run build in dashboard project
• Build client display web application: npm run build in client-display project
• Compile Java and Groovy classes in the service project: mvn compile
• Copy the built dashboard app to service/src/main/resources/public so the spring boot app can host the dashboard web-application
• Copy the built client-display app to service/src/main/resources/public/static/clientPortal so the spring boot app can host the client-display web-application
• Generate runtime artifact: mvn package in the service project

Of course this doesn't cover the entire build cycle so please familiarize yourself with all of the pom.xml and package.json files. Now that you have the runtime artifact, you can start the SkillTree dashboard and service using the following command:

java -jar service/target/skills-service-<version>.jar

The app will run on http://localhost:8080 (opens new window), visit it to create an account, new project, subjects, skills, etc. To learn more about the features of the SkillTree dashboard, please visit the Dashboard Guide.

The SkillTree application will be started using default properties. Some things to note:

• Spring datasource properties will need to be configured for connecting to an available PostgreSQL database using the following properties:

spring.datasource.url=jdbc:postgresql://localhost:5432/skills
spring.datasource.username=<username>
spring.datasource.password=<password>

• By default the app runs in Password Auth Mode which is the common use-case.

Cypress.io end-to-end tests

SkillTree utilizes the cypress.io (opens new window) framework to perform end-to-end tests and specifically verify features of the dashboard and client-display web applications. Cypress tests are located in the e2e-tests project:

skills-service
|   ....
└───e2e-tests
|   │   package.json
|   └───cypress
|   |   └───integration
|   |   <dashboard test name>_spec.js
|   |   |   └───client-display
|   |   |   |   <client-display test name>_spec.js   

The end-to-end tests stand-up the skills-service, the dashboard, and the client display web applications and then execute numerous tests against these web-applications to mimic users' actions.

To run cypress end-to-end tests (assumes that you have built the skills-service.jar via the previous step):

cd skills-service/e2e-tests
npm install

# start servers in the background: skills-service and client-display
npm run cyServices:start
# run cypress integration tests
npm run cy:run
# kill background servers
npm run cyServices:kill 

Buckle down as these tests will take a while to run, and depending on your system specs you should expect anywhere from 5 to 20 minutes.

TIP

Please note that ports 8080, 8083, 1026, and 1081 have to be available on your system

npm run cyServices:start command starts:

• The programmatic service and the dashboard by utilizing the already built jar; will run on port 8080
• the client-display using webpack dev server; will run on port 8083
• A mock smtp server; will run on ports 1026 and 1081

Now that you can build and run integration tests let's discuss day-to-day development setup for the skills-service project.

Day-to-day development

Most IDEs (Intellij, Eclipse, etc...) provide first-class support for maven projects so the very first step would be to import the skills-service project into your favorite IDE.

Please note that code additions will generally fall into these categories:

1. Enhancing the skills-service programmatic REST API
2. Making changes to the web-based dashboard application or the client-display web-based application

skills-service programmatic REST API

The code for the API can be found under skills-service/service which follows standard maven conventions.

The programmatic API tests reside in service/src/test/java and you can run all of the tests via

cd skills-service/service
mvn test

You can also run any of the tests using your favorite IDE.

TIP

SkillTree's overall testing strategy is to implement black-box integration tests and supplement with unit tests whenever an integration test is not possible.

Generally, programmatic API service development is facilitated via the integration tests .

Skills Service integration tests stand-up the skills-service application and then execute various endpoints, validating the result. These integration tests reside in the skills.intTests package and extend the DefaultIntSpec.groovy class.

DefaultIntSpec.groovy is annotated with the @SpringBootTest (opens new window) annotation which facilitates running the Spring Boot application, exposing endpoints on a random port.

A few things to note:

• You should interact with the skills-service using the test client skills.intTests.utils.SkillsService class
  • Available via skills.intTests.utils.DefaultIntSpec#skillsService
  • SkillsService class represents an authenticated dashboard user
• Runtime port can be retrieved via skills.intTests.utils.DefaultIntSpec#localPort but should rarely be used directly; instead please use skills.intTests.utils.DefaultIntSpec#skillsService instead
• DefaultIntSpec setup and cleanup methods purge data from DB between each test case
• Use the skills.intTests.utils.DefaultIntSpec#createService method if you need a new SkillsService instance
  • For example to represent another dashboard user

There are hundreds of tests in the skills.intTests package, please feel free to explore.

Web-Based Dashboard and/or Client Display

Steps to develop the web-based dashboard are:

1. Stand up the service (programmatic API)
2. Bring up the webpack dev server
3. Use browser and cypress tests to drive development

To stand up the service you can execute skills.SpringBootApp in the service project from your IDE. If that's not an option you can always build a jar and run it from the command line:

java -jar service/target/skills-service-<version>.jar

The service will run on port 8080.

Next is to start the webpack dev server in the dashboard project:

cd dashboard
npm run serve

This webpack dev server will run on port 8082 and will make requests for data to the service running on port 8080.

Generally development is facilitated by writing cypress tests:

cd e2e-tests
npm run cy:open:dev

You can then start adding tests under e2e-tests/cypress/integration to an existing file or by creating a new file.

If you are making changes to the client display OR want to run all of the cypress integration tests, then you will need to start the client-display webpack dev server:

cd client-display
npm run serve

The dev server will run on port 8083 and will make requests for data to the service running on port 8080.

Test Checklist

TIP

SkillTree's overall testing strategy is to implement black-box integration tests and supplement with unit tests whenever an integration test is not possible.

We like tests (especially integration tests), so please make sure you thoroughly test your code.
Prior to making a Pull Request make sure that ALL tests pass:

1. Run all service tests
2. Run all Cypress tests

Provide the following properties to run the service tests against PostgreSQL:

spring.datasource.url=jdbc:postgresql://localhost:5432/skills
spring.datasource.username=<username>
spring.datasource.password=<password>

On the command line it would look something like this:

mvn --batch-mode test -Dspring.datasource.url=jdbc:postgresql://postgres:5432/skills -Dspring.datasource.username=user -Dspring.datasource.password=pass

To run all the cypress tests:

cd skills-service/e2e-tests
npm install

# start servers in the background: skills-service and client-display to test
npm run cyServices:start
# run cypress integration tests
npm run cy:run
# kill background servers
npm run cyServices:kill 

Skills-client development

Skills-client JS libraries provide skills event reporting utilities and a thin iFrame-based wrapper for the Client Display.

Is this the right project for me?

Please keep in mind that the majority of the SkillTree features are encapsulated in the skills-service project.

If you are making improvements to the SkillTree dashboard or its Skills Display views then your development will be scoped within the skills-service project. I f you have not yet, please take few minutes to review the architecture section.

If you are not sure whether you need to make skill-client changes, contact the SkillTree core development team.

Get, build and test

First things first, fork and checkout the code (please see the Contribution Steps), build it and run all of the unit and integration tests. Then we'll discuss the development and testing steps.

git clone <github>:skills-client.git
cd skills-client

skills-client is an assembly of projects that build JS client libraries AND integration tests that enable comprehensive testing of these libraries:

• skills-client-js: client app implementation, please see Pure JS Integration Guide to understand how these artifacts are utilized by a SkillTree integrator
  • Majority of client code resides here, framework specific libraries are minimal wrappers around this client implementation
  • Provides support for Skill Event Reporting, Skills Display as well as Skills Configuration
• skills-client-integration: integration tests that enable comprehensive testing of skills-client-js and all of the framework specific libraries.
  • A set of web-applications that integrate and exercise these skills client libraries
  • Cypress tests that utilize these applications to end-to-end test functionality of the skills client libraries

SkillTree uses maven (opens new window) and npm (opens new window) for its dependency management and to facilitate the build lifecycle. Let's get familiar with the projects layout:

skills-client
└───skills-client-js
|   │   package.json
|   |   ....
└───skills-client-integration
│   │   pom.xml

To build client libraries and run their unit tests:

cd skills-client-js
npm install
npm run build

To build integration test artifacts:

cd skills-client-integration
mvn install

Several actions were performed in the mvn install command:

• client libraries were built
• client libraries' unit tests were executed
• integration apps were built (these apps utilize client libs)
• This step does not execute end-to-end tests.

End-to-end cypress.io (opens new window) tests then exercise these integration applications in order to thoroughly test our client libraries. The next section will break this process down in detail.

Cypress.io end-to-end tests

There are a number of challenges when integration testing client libraries. These JS libraries themselves don't present an executable artifact since the purposes of them is to be utilized by the external web applications.

To enable end-to-end testing of skills-client-js, we developed a set of web-applications that integrate and exercise these libraries. These applications mimic real world integration and usage scenarios. cypress.io (opens new window) is then used to execute numerous end-to-end tests against these web applications, ensuring that these libraries properly work in an integrated environment.

The code for the web applications that mimics real world usage and end-to-end cypress.io (opens new window) tests resides in the skills-client/skills-client-integration project. Let's take a look at its directory structure:

skills-client
└───skills-client-integration
│   │   pom.xml
|   └───skills-int-client-js
|   │   │   package.json
|   │   │   pom.xml
|   |   └───target
|   └───skills-int-e2e-test
|   │   │   package.json
|   │   │   pom.xml
|   |   └───target
|   |   |   skills-int-e2e-test-<version>.jar
|   └───skills-int-service
|   │   │   pom.xml
|   |   └───target
|   |   |   skills-int-service-<version>.jar

Here is an explanation for each project:

• skills-int-client-js: Web application that depends on the skills-client-js artifact and mimics its real world usage
• skills-int-service: Pulls together the web application (skills-int-client-js) and exposes a web server to serve these applications. Cypress.io (opens new window) tests execute tests against this service.
• skills-int-e2e-test: Cypress.io (opens new window) tests that utilize skills-int-service to perform thorough end-to-end tests of the integrated libraries.

To run cypress.io (opens new window) tests you will need to

1. Start backend service
   • skills-service: REST data endpoints
   • serves Skills Display application
2. Start integration web applications (skills-int-service)
3. Run cypress tests

Please visit the architecture section to further explore the relationship between these services.

We are going to assume that you are versed in skills-service installation and development, but if not please visit the skills-service development section. Commands below assume:

• skills-service-XXX.jar was generated and that skills-service and skills-client projects reside in the same parent directory.
• skills-int-service--XXX.jar was generated via earlier steps.

To run cypress end-to-end tests:

cd skills-client/skills-client-integration/skills-int-e2e-test
npm install

# will start skills-service and skills-int-service processes
npm run cyServices:start

# run cypress integration tests
npm run cy:run

# kill background servers (skills-service and skills-int-service)
npm run cyServices:kill 

Buckle down as these tests will take a while to run, and depending on your system specs you should expect anywhere from 2 to 15 minutes.

TIP

Please note that ports 8080 and 8090 have to be available on your system

Now that you can build and run the integration tests let's discuss day-to-day development setup for the skill-client project.

Day-to-day development

Most IDEs (Intellij, Eclipse, etc...) provide first-class support for maven projects so the very first step would be to import the skills-client project into your favorite IDE.

Let's first understand JS dependency chain.

The Diagram above depicts JS dependencies that can be traced using package.json files in each project.
Because we are authoring code across multiple projects in the dependency chain, a traditional npm-publish (opens new window) approach falls short. With the npm-publish (opens new window) approach, in order to test the change, the artifact would have to be built and pushed to the npm repository. Then all of the projects that depend on this resource would have to be updated to point to it (package.json). And then any follow-on projects that depend on the updated projects need to be modified as well. That is a large amount of overhead to test a potentially trivial change.

SkillsTree opted to utilize the npm link (opens new window) mechanism to work around this issue. Npm link enables dependency on the local version of the code and will not require a new artifact version to be generated and published.
This way we can make changes to several projects in the chain then build and test them locally before committing. It also reduces the number of versions we publish and reduces the risk of publishing JS libs with critical bugs.

We've created a script that facilitates npm links for all the connections depicted in the above diagram:

cd skills-client/skills-client-integration/skills-int-e2e-test
npm run dev:setupNpmLinks

Now that links are established you will be able to make changes to multiple JS projects then build them and see those changes persist. For example skills-int-client-js depends on skills-client-js, we can make a code change and see it right away:

Terminal 1:

cd skills-client/skills-client-integration/skills-int-client-js
npm run serve
# view dev server on http://localhost:8091/

Then make a code change in skills-client/skills-client-js, followed by in

Terminal 2:

cd skills-client/skills-client-js
npm run build

The change is automatically reflected in the dev server hosted on http://localhost:8091/

What happened is that npm run build in skills-client-js created js artifacts in its dist directory and skills-int-client-js points to that directory using the npm-link mechanism.

Here is the summary of the client lib and its corresponding integration app.

client lib	corresponding integration app
skills-client-js	skills-int-client-js

TIP

Please note that you can start the development server from an integration app by executing npm run serve.

Exception to the Rule

skills-int-client-js has a unique setup and requires that npm install be run for its dependencies to be picked up!

This integration apps mimics a pure JS application without the use of webpack (uses <script> tag) so it packages all of its dependencies in the assets directory. npm install will copy dependencies from the node_modules directory into its assets folder.

Now that we covered the JS dependencies challenges, as an example, let's walk through the steps of modifying code in the skills-client-js project and then adding cypress.io (opens new window) integration tests:

• start the skills-service daemon
  • if you have the project setup locally you can start it from the IDE OR
  • build the project and start it from the command line npm run cyServices:start:skills-service in the skills-int-e2e-test project
• start the skills-int-service daemon
  • start from your IDE OR
  • build the project and start it from the command line:

cd skills-client/skills-client-integration/skills-int-e2e-test
npm run integration:start

• start the development server of the corresponding integration app

cd skills-client/skills-client-integration/skills-int-client-js
npm run serve

• navigate to skills-int-e2e-test and start the cypress dev console:

cd skills-client/skills-client-integration/skills-int-e2e-test
npm run cy:open:dev:js

• run the test that corresponds to the integration app you are working on (in this case that would be ``js.specs.js```).
• write new cypress.io (opens new window) tests

TIP

The SkillTree development process focuses on providing full and comprehensive tests. As such, we encourage users to start any modifications by creating a Cypress test. Starting via a test-first approach is also necessary to ensure that the required user configured for the skills-int-service application is created.

If you attempt to access the examples application outside of a cypress test, you will notice that the Client Display and Level components will not properly render, and the network requests for each component will receive a 500 error.

The Cypress.io (opens new window) dev console command depends on which integration app you are using/testing:

integration app	command
skills-int-client-js	npm run cy:open:dev:js

Test Checklist

Once you are done developing and writing tests then please follow these steps to run all the tests:

1. To test the latest/updated code make sure npm links are up-to-date: (1) remove npm links (2) recreate
2. Build and run unit tests
3. Execute all cypress.io (opens new window) integration tests

To re-create the links:

cd skills-client/skills-client-integration/skills-int-e2e-test
npm run dev:removeNpmLinks
npm run dev:setupNpmLinks

Build and run unit tests and then execute cypress.io (opens new window) integration tests:

# build
cd skills-client/skills-client-integration
mvn install

cd skills-int-e2e-test
npm prune 
npm install

# will start skills-service and skills-int-service processes
npm run cyServices:start

# run cypress integration tests
npm run cy:run

# kill background servers
npm run cyServices:kill