Set Your Ubuntu 14.04 Image Set-up Properly from the Start

For VirtualBox Users

If working on a VirtualBox Machine, you may have notice that you can not change the resolution of your screen. Use the following instructions to fix this issue.

Virtualbox updates

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install dkms
sudo apt-get install build-essential

If the previous code does not change the resolution, use the following instead:

Virtualbox updates

sudo apt-get upgrade
sudo apt-get install virtualbox-guest-additions-iso
sudo apt-get install virtualbox-guest-utils

While in your VirtualBox Machine, bring your mouse to the top of the screen and choose devices
1. Choose Insert Guest Additions CD Image
A dialog Box will appear. Click: Run, then enter your password
When prompted, click Enter/return on your keyboard
Now restart your VirtuaBox machine
1. Your screen resolution should now be fixed. If not rerun all the steps above one more time.
When your VirtualBox machine comes back up, right click on the disk icon sitting in your Dock, and choose eject.

Starting on your task

If you don't already have git installed on your computer, run the command: sudo apt-get install git
Go to the following URL: https://github.com/dmc-uilabs
Create clone of the dmcdb repository, restservices repository, and frontend repository(for yaml)
1. A Jira account should be created for you, and your GitHub Account should have Access to the dmc-uilabs repositories. If you do not have access, reach-out.
Create create a new branch in each of these related to your task.
1. As you work on your branch, be sure to perform a git pull on your branch to make sure it is up to date with newly merged code.

Installing everything for your restservices testing environment

Installing Java

Check to see if java is already installed

java -version

If java is not installed, run the following commands

sudo apt-get update
sudo apt-get upgrade
sudo add-apt-repository ppa:openjdk-r/ppa
sudo apt-get update
sudo apt-get install openjdk-8-jdk
sudo apt-get update
close and reopen terminal
sudo update-alternatives --config java

Ensure that there is a file named 'java-8-openjdk-amd64

ls /usr/lib/jvm/

Edit the .bashrc file

cd ~

nano .bashrc
when in terminal

or

nano .zshrc
When is zshell

Add the following new lines to the bottom of the file (those not referencing JAVA will be used later)
1. 1. export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  2. export PSQLUSER=gforge
  3. export PSQLPASS=gforge
  4. export PSQLDBNAME=gforge
2. Save your changes
Close and reopen the terminal
1. Run the command: echo $JAVA_HOME
2. You should get back the location you added to the .bashrc file
Installing Tomcat
1. If you have not installed Java yet, install it first before installing tomcat

Installing Tomcat

sudo apt-get install tomcat7
sudo apt-get update

Close and reopen terminal, then run:

Starting Tomcat

sudo service tomcat7 start

If you get an error along the lines of set JAVA_HOME or no JDK installed, do the following:

Editing Tomcat File

sudo nano /etc/default/tomcat7

In this file where you see #JAVA_HOME, erase the entire line and replace it with:

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Close and reopen the terminal, then re-run the command the command to start tomcat7

Wait a few moments, and you should get an [OK] back

Go to a browser and go to the following URL: localhost:8080. This should say It works!

Stopping Tomcat

Now you can run: sudo service tomcat7 stop

Installing pgadmin

Install pgAdmin

sudo apt-get install pgadmin3

Installing Maven

Installing Maven

sudo apt-get install maven

If you have issues running maven or mvn, the following commands can at time be useful.

Solutions to Maven not building Properly

mvn clean
mvn dependency:resolve

Installing Eclipse Mars 1

1. If working with a 64 bit linux machine, the following URL will take you to a location from which you can download this IDE:
  1. http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/mars/1/eclipse-java-mars-1-linux-gtk-x86_64.tar.gz
2. Once downloaded, go to your downloads folder, right click the newly downloaded eclipse folder, then choose extract here.
3. Change directories to the new eclipse folder that was just created.
4. Open eclipse application
  1. When selecting a workspace, click OK to set it as default
  2. After the application is finished loading, click on Workbench in the top right-hand corner
5. Next click, Window → Show View → Project Explorer
  1. This lets you view your project structure on the left hand side.
6. **For information on setting up your Eclipse environment for development, click here.

Installing and Configuring Postgres

In order to set up postgres to run locally, you will need to first have a database installed on your local machine
If you have not already done so, got to the following URL and close the dmcdb repository: https://github.com/dmc-uilabs
In a terminal, navigate to the dmcdb folder.
If you already had the repository downloaded, make sure to do a git pull before proceeding.

Configuring a local PostSQL Database:

The following instructions will set up a local Postgres database for developers to test REST services against.
All of these commands are done on the command line. Lines with # are comments, lines with $ are shell commands, and lines with no prefix are postgres shell commands.

#Use to install postgres (if update command appears to have issues, that is okay. Just make sure postgres was installed)

$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list'
$ wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install postgresql-9.3 postgresql-contrib-9.3

#Check if postgres is install

$ psql --version

#login into postgres shell with default user postgres

$ sudo -u postgres psql postgres

#inside postgres shell, use the following command and follow prompt to set password for user postgres. A good password to set is just 'postgres'

\password postgres

#next, create a new user called gforge (make sure you are still in postgres shell)

CREATE USER gforge WITH PASSWORD 'gforge';

#check if new user was created

\du

#exit out of postgres shell

\q

#create a new db called gforge

$ sudo -u postgres createdb -O gforge gforge

#check if DB gforge was created (should have owner set to user gforge)

$ sudo -u postgres psql -c "\l"

#set your current db to gforge

$ sudo -u postgres psql postgres

#type connect command inside the postgres shell

\connect gforge

#outside of postgres shell, use text editor nano to update the following file. For postgres version, don't include anything past the second decimal (eg only use 9.3 or 9.4)

\q
$ sudo nano /etc/postgresql/{your postgres version goes here}/main/pg_hba.conf

#In this file, search for 'postgres'. You should find a line that has the following:

Local

All

Postgres

Peer

#In this file, change the word 'peer' to 'md5'. Save the changes and navigate to the folder with the stable version of the dmcdb

Starting your local database

Flyway: The DMC uses Flyway, which is a database migration program used for tracking changes to the database. Every time an update, change, or a new file is made in the database flayway is used to keep track of these individual changes, making it easy to revert changes made by developers.

Once you have configured your PostSQL Database run the following commands to start a new database instance:

This will drop the database called gforge.

sudo -u postgres psql -c "DROP DATABASE gforge"

Restart postgresql

sudo service postgresql restart

This will create a new database called gforge.

psql -U postgres -c "CREATE DATABASE gforge WITH OWNER gforge;"

This will list off the running database instances. the first one on the list should be called gforge.

sudo -u postgres psql -c "\l"

This will fill the database with the latest tables that have been created.

./flyway clean migrate info -configFile=conf/core/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge  -user=gforge -password=gforge

This will fill the database table with the latest sample data.

./flyway migrate info -configFile=conf/data/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge  -user=gforge -password=gforge -locations=filesystem:./sql/data/dev2

The database is now running and can be viewed by pgAdmin and can be used to run restservices locally.

Installing everything for frontend testing environment

Navigate to the front end repo and run the following. Installing Node and Bower to allow use of gulp serve and gulp build

sudo apt-get update
cd ~/DMC/frontend
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.2/install.sh | bash

//if not using zshell, only run .bashrc
source ~/.bashrc
source ~/.zshrc
nvm install v5.1.1

//Check to see that your node version is 5.1.1
node -v

//If you have issue running the commands below, you may need to close all your terminal windows, return to your frontend folder and try them once more.
sudo apt-get install npm
npm install
npm install nodejs
npm install -g bower
bower install
npm install gulp
npm install babel-register
sudo ./post-install.sh

//Using Yarn to Serve and Build Frontend
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install yarn

//Download the following file and un zip in the root of frontend directory
https://s3.amazonaws.com/dmc-build-aritifacts/0.1.5/bower_components.tar.gz

//Serving Front end
//in one terminal tab run the following in the following directory: /dmcfront/jsonserver
node server.js
//in another tab run the following in the root of the frontend directory
gulp serve

//building the frontend for testing
//if you already have a dist folder in the root of your frontend directory to remove the dist folder
rm -rf /dist
//To build the dist folder, run the following in the root of the frontend directory:
gulp build

Work Flow

The following are the stages your task for the most part should go through

Checking the .yaml file
1. If you have not yet done so, clone the frontend repository from github.
  1. Change directories to the following: frontend/restapi
  2. Open the file called: DMCInterfaceSwagger.yaml
2. Check and make sure that the .yaml file has information about your endpoint already there.
  1. If it is missing, you will need to update your endpoint to reflect your code after it is completed.
Working on the data base
1. Follow the instructions listed above for creating a local database and connect to the database using pgAdmin (make sure you are on your own branch).
2. Open the folder structure as such: Server Groups → Servers → <Local Database running with postgres> → Databases → gforge → Schemas → public → Tables
3. Locate the table(s) that will be required for your endpoint(s) to function
  1. If your table(s) is missing or has missing or incorrect column names or types, or does not have existing sample data, you will need to do the following
    1. You will need to generate the code required to create your table. To do this do the following
      1. In pgadmin, right click the tables folder and choose new table.
      2. Under properties, fill out the table name (table should use lowercase characters only)(the norm for table names with more than one word should look like this: multiple_word_table_name).
      3. Under columns, click add to create a new column. Fill out the name field (do not use camel case here, follow the same convention as used in the step above) and chose a type for the field such as: text, date, integer, or serial (generally used for ids).
        When creating an id column that should generate an id on its own, do the following
        Create the column with a name and the type of serial.
        Click on the constraints tab.
        Make sure Primary Key is selected from the drop-down menu.
        Click Add.
        In the new window, click on the columns tab
        In the column drop-down menu, choose the name of the column.
    2. Find the table that you created in the tables folder in pgadmin.
    3. Click on the table name and then click on the table icon (7^th icon from the top left of the window).
      1. This is a view of what your table will look like
    4. At the bottom of the pgadmin window, there is a pane with code. Copy this code.
    5. With postgres still running, run the following command:
      1. This command will show you the full list of migrations for table data
        
        ./flyway clean migrate info -configFile=conf/core/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge -user=gforge -password=gforge
      2. This command will show you the full list of migrations for sample data
        
        ./flyway info -configFile=conf/data/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge -user=gforge -password=gforge -locations=filesystem:./sql/data/dev2
    6. Thus the lists generated by the commands above to name your database table files and database sample data files such that they are one migration number higher. This will be covered again momentarily.
    7. You will now update 3 files in the database. File names and locations are as follows:
      1. dmcdb/sql/core/tables
        Create a new table file.
        If the highest version listed from the ./flyway info -configFile=conf/core/flyway.conf command was 1.0081, the new file should be named V1_0082__<table_name>.psql
        V1 is the version number
        0081 is the migration number
        Edit the file and paste the code that you copied in the step above here.
        Cut the line that says CONSTRAINT table_name_PKEY PRIMARY KEY (id)
        erase the lines: )WITH ( UIDS=FALSE
        Save the file.
      2. /dmcdb/sql/core/constraints/
        Create a new primary_key_constraints file.
        Just like the new table file make sure the new file has a migration number that is one higher than the table name file.
        If the table file you just created is called V1_0082__example_table.psql, the new constrains file should be named V1_0083__primary_key_constrains.psql
        __primary_key_constraints.psql should never change on the version or migration number should.
        Add the following lines to the file:
        
        - -Name: table_name_pkey; Type: CONSTRAINT; Schema: public; Owner: gforge; Tablespace: - ALTER TABLE ONLY <table_name> ADD <paste what you cut earlier here>;
        
        Save the file.
      3. dmcdb/data/dev2/
        Create a new sample data file.
        If the highest version listed from the ./flyway info -configFile=conf/data/flyway.conf command was 1.0014, the new file should be named V1_0015__<sample_data_name>.psql
        V1 is the version number
        0015 is the migration number
        Add three lines of sample data here. Use the following as an example for syntax
        insert into table_name (column_name one, column_name_two) values ('text', 'textAlso');
        Explanation:
        table_name: use the name of your table here in its place.
        The column names should be listed in order separated by commas.
        The sample data must be listed in the same order as you list the column names.
        Text data should be quoted using apostrophes and separated by commas. Numbers should not be quoted at all.
        The id that should be auto generated should not be listed in the column name or the sample data.
    1. Rerun postgres commands to drop database, create a database, check that the data base was created, add the tables to the database, and add the sample data.
      1. Remember, you can not use the above commands if you are connected to the database with pgadmin, you will need to close pgAdmin, run the commands and then reopen pgAdmin..
      2. At the bottom of this page there is a bash script for dropping your database, creating a new database, checking that the database exists, filling the tables and filling the sample data.
    2. Check to make sure you can open your table and that it has sample data in it. Also try to add another line of information. pgAdmin should handle generating the serial id, and the next line.
    3. Once you've confirmed the information in the database is correct and updated you can now start working on your code.
Working on your code
1. Make sure that if you have not yet cloned the restservices repository from GitHub, that you do so now.
2. Also, make sure you create a new branch from the master branch.
3. Open Eclipse
4. Refresh Eclipse

Testing your code with swagger

Manual

Depricated Instructions

Once you have code you would like to test, start drop and rebuild your local database.

Next, fine the Config.java file located in the following directory:
restservices/src/main/java/org/dmc/services/

Change the first 6 lines to be the following:
public static final String DB_PORT = "5432";
public static final String DB_IP = "127.0.0.1";
public static final String DB_HOST = "jdbc:postgresql://" + DB_IP;
public static final String DB_NAME = "gforge";
public static final String DB_USER = "gforge";
public static final String DB_PASS = "gforge";

Save the changes in this file.

Next, add the following to the bottom of your /etc/profile file by using the command:sudo nano /etc/profile
Run the command: source /etc/profile
export DBip="127.0.0.1"
export DBport=5432
export DBuser="gforge"
export DBpass="gforge"
export SOLR_BASE_URL="http://<SOLR IP Address>/solr"
export ActiveMQ_URL="<ActiveMQ IP Address>"
export ActiveMQ_Port=61616
export ActiveMQ_User="active"
export ActiveMQ_Password="active"

Next update the following file restservices/src/main/java/org/dmc/services/Application.javaAll references to DB under the @Beam should be replaced with the following:

ds.setUser(Config.DB_USER);
ds.setPassword(Config.DB_PASS);
ds.setServerName(Config.DB_IP);
ds.setPortNumber(Integer.parseInt(Config.DB_PORT));
ds.setDatabaseName(Config.DB_NAME);

Finally
 on debian operating systems update the file /etc/default/tomcat7 (also 
add this to the /etc/tomcat7/tomcat.conf file as well)
Add the following to the bottom of the file:


DBip="127.0.0.1"
DBport=5432
DBuser="gforge"
DBpass="gforge"
SOLR_BASE_URL="http://<SOLR IP Address>/solr"

ActiveMQ_URL="A<ctiveMQ IP Address>"
ActiveMQ_Port=61616
ActiveMQ_User="active"

Note
 that the above system environments do include IP addresses from live 
SOLR and ActiveMQ machines. If the above addresses do not work you will 
need to get these IP address from a DMC developer

Next, go to the restservices directory.
Run the command: sudo service tomcat stop

Next, run the command: mvn package -P swagger

When the .war file build is complete, go to the following directory:
/var/lib/tomcat7/webapps
If there are any files in the webapps folder, delete them. The command to do this is:
sudo -r *

Next, go to the following directory
restservices/target/

Rename the file named: dmc-site-services-0.1.0-swagger.war to rest.war; you can use the following command to do so:
mv dmc-site-services-0.1.0-swagger.war rest.war

Move the rest.war file to the /var/lib/tomcat7/webapps/ folder using the following command:
sudo mv rest.war /var/lib/tomcat7/webapps/

Now start tomcat using the following command:
sudo service tomcat7 start


Go to the following URL in a browser: localhost:8080/rest/services/2 or http://localhost:8080/rest/swagger-ui.html.
This may take some time to load. Wait a few moments.

Automatic
```
public static final String DB_PORT = "5432";
public static final String DB_IP = "127.0.0.1";
public static final String DB_HOST = "jdbc:postgresql://" + DB_IP;
public static final String DB_NAME = "gforge";
public static final String DB_USER = "gforge";
public static final String DB_PASS = "gforge";
```
1. Save the changes in this file.
2. Next, make sure th following is found in one of the the files listed below
  1. sudo nano /etc/profile, ~/.bashrc, or ~/.zshrc
    1. Run the command: source /etc/profile
      export DBip="127.0.0.1" export DBport=5432 export DBuser="gforge" export DBpass="gforge" export SOLR_BASE_URL="http://<SOLR IP Address>/solr" export ActiveMQ_URL="<ActiveMQ IP Address>" export ActiveMQ_Port=61616 export ActiveMQ_User="active" export ActiveMQ_Password="active"
3. Next update the following file restservices/src/main/java/org/dmc/services/Application.java
  1. All references to DB under the @Beam should be replaced with the following:
  2. ds.setUser(Config.DB_USER); ds.setPassword(Config.DB_PASS); ds.setServerName(Config.DB_IP); ds.setPortNumber(Integer.parseInt(Config.DB_PORT)); ds.setDatabaseName(Config.DB_NAME);
4. Save the changes in this file.
5. Run the command: sudo service tomcat7 stop
6. Go to the following directory:
  1. /var/lib/tomcat7/webapps
7. If there are any files in the webapps folder, delete them. The command to do this is:
  1. sudo -r *
8. Next, go to the restservices directory.
  1. Run the command: mvn spring-boot:run -P swagger
  2. Compilation now happens
9. When completed, open a web browser and got to the following URL:
  1. http://localhost:8080/swagger-ui.html
10. You will need to then write tests for the code you have written.
11. Once you believe your tests are complete, you can test them by first closing all terminals you have.
12. Open a new terminal.
  1. Make sure postgres is running an instance of your database.
13. Change directories to your rest services folder
14. Then write the following command
  1. mvn -Dtest=<TestName> test
    1. DO NOT include the triangle brackets in your command
  2. Make sure your environment variable are set or are included in your Config.java file so that your local database is being referenced.

Submission/Code Review (for database changes and resetservice changes)
1. To submit your cod for code review use the following commands:
2. Git status
  1. Shows you what files you have made changes to in the repository
3. git add -A
  1. -A add all files that have changes, by giving the full file path in its place you can add specific files
4. Git status
  1. this checks which files you've added to the staging area.
5. Git commit -m “information about commit”
  1. -m lets you add information about the commit.
  2. “” are used for capturing comments.
6. Git push origin BranchName
  1. In place of branch name include your actual branch name.
  2. This will push your changes to GitHub.
7. Go to GitHub, and navigate to your branch
  1. Make sure to sync your branch with the master branch
  2. Make a pull request and add a few reviewers.
8. Updating the .yaml file
  1. After your changes have been merged, find the .yaml file mentioned in the steps above and edit it as well. Make sure to add tags live and has tests to your endpoint. Submit this for code review.

Frontend-JSON Implimentation on Local Machine

In the DMC frontend folder, navigate to app/scripts/common/factory/data.factory.js
1. Comment out line 7.
  1. //localhost = $window.apiUrl ? $window.apiUrl + '/' : localhost;
2. On line 8 change port 8080 to 3000 like shown below
  1. var urlSocketServer = 'http://localhost:3000/rest';
  2. This makes your frontend now receive data from your REST services, instead of from the sample JSONserver
Open two terminal windows, one in the /frontend directory, and one in the /frontend/jsonserver directory.
1. In the /frontend directory run the command: gulp serve
2. In the /fronted/jsonserver directory run the command: node server.js
In your internet browser, navigate to localhost:9000 and your frontend code should now be communicating with the JSON server.

Full-Stack Local Implementation (Frontend-Restservices-Database)

Documentation on this is coming soon.

Useful Bash Scripts

Starting a new database

#!/bin/bash

echo "Changing Directories"
cd dmcdb
echo "Droping the current database"
sudo -u postgres psql -c "DROP DATABASE gforge"
echo "creating a new database"
sudo -u postgres createdb -O gforge gforge
echo "Confirm new database (gforge) was created"
sudo -u postgres psql -c "\l"
echo "creating tables"
./flyway clean migrate info -configFile=conf/core/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge  -user=gforge -password=gforge
echo "filling tables with sample data"
./flyway info -configFile=conf/data/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge  -user=gforge -password=gforge -locations=filesystem:./sql/data/dev2
./flyway migrate info -configFile=conf/data/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge  -user=gforge -password=gforge -locations=filesystem:./sql/data/dev2
echo "-----------------open pgadmin-----------------------"

If restoring a database the following command is used

psql -U gforge gforge < azurePortal20170523.sql

You can use the commands below to stop postgres from running and subsequently disconnect application that are accessing your database.

sudo service postgresql stop
sudo service postgresql start

Running Rest Service Locally

#!/bin/bash
echo "Stopping Tomcat."
sudo service tomcat7 stop
echo " "
sleep 2
echo "Moving to restservices directory."
echo " "
cd ~/Desktop/dmc/restservices
echo "Starting mvn spring-boot:run -P swagger"
echo " "
sleep 3
firefox localhost:8080/swagger-ui.html
echo "When complete go to localhost:8080/swagger-ui.html in a browser."
mvn spring-boot:run -P swagger