Set Your Ubuntu 14.04 Image Set-up Properly from the Start
For VirtualBox Users
If working on a VirtualBox Machine, you may have notice that you can not change the resolution of your screen. Use the following instructions to fix this issue.
sudo apt-get update sudo apt-get upgrade sudo apt-get install dkms sudo apt-get install build-essential
If the previous code does not change the resolution, use the following instead:
sudo apt-get upgrade sudo apt-get install virtualbox-guest-additions-iso sudo apt-get install virtualbox-guest-utils
While in your VirtualBox Machine, bring your mouse to the top of the screen and choose devices
Choose Insert Guest Additions CD Image
A dialog Box will appear. Click: Run, then enter your password
When prompted, click Enter/return on your keyboard
Now restart your VirtuaBox machine
Your screen resolution should now be fixed. If not rerun all the steps above one more time.
When your VirtualBox machine comes back up, right click on the disk icon sitting in your Dock, and choose eject.
Starting on your task
If you don't already have git installed on your computer, run the command: sudo apt-get install git
Go to the following URL: https://github.com/dmc-uilabs
Create clone of the dmcdb repository, restservices repository, and frontend repository(for yaml)
A Jira account should be created for you, and your GitHub Account should have Access to the dmc-uilabs repositories. If you do not have access, reach-out.
Create create a new branch in each of these related to your task.
As you work on your branch, be sure to perform a git pull on your branch to make sure it is up to date with newly merged code.
Installing everything for your restservices testing environment
Installing Java
java -version
sudo apt-get update sudo apt-get upgrade sudo add-apt-repository ppa:openjdk-r/ppa sudo apt-get update sudo apt-get install openjdk-8-jdk sudo apt-get update close and reopen terminal sudo update-alternatives --config java
ls /usr/lib/jvm/
cd ~ nano .bashrc when in terminal or nano .zshrc When is zshell
Add the following new lines to the bottom of the file (those not referencing JAVA will be used later)
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
- export PSQLUSER=gforge
- export PSQLPASS=gforge
- export PSQLDBNAME=gforge
Save your changes
Close and reopen the terminal
Run the command: echo $JAVA_HOME
You should get back the location you added to the .bashrc file
Installing Tomcat
If you have not installed Java yet, install it first before installing tomcat
sudo apt-get install tomcat7 sudo apt-get update
Close and reopen terminal, then run:
sudo service tomcat7 start
If you get an error along the lines of set JAVA_HOME or no JDK installed, do the following:
sudo nano /etc/default/tomcat7
In this file where you see #JAVA_HOME, erase the entire line and replace it with:
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
Close and reopen the terminal, then re-run the command the command to start tomcat7
Wait a few moments, and you should get an [OK] back
Go to a browser and go to the following URL: localhost:8080. This should say It works!
Now you can run: sudo service tomcat7 stop
Installing pgadmin
sudo apt-get install pgadmin3
Installing Maven
sudo apt-get install maven
If you have issues running maven or mvn, the following commands can at time be useful.
mvn clean mvn dependency:resolve
Installing Eclipse Mars 1
If working with a 64 bit linux machine, the following URL will take you to a location from which you can download this IDE:
Once downloaded, go to your downloads folder, right click the newly downloaded eclipse folder, then choose extract here.
Change directories to the new eclipse folder that was just created.
Open eclipse application
When selecting a workspace, click OK to set it as default
After the application is finished loading, click on Workbench in the top right-hand corner
Next click, Window → Show View → Project Explorer
This lets you view your project structure on the left hand side.
- **For information on setting up your Eclipse environment for development, click here.
Installing and Configuring Postgres
In order to set up postgres to run locally, you will need to first have a database installed on your local machine
If you have not already done so, got to the following URL and close the dmcdb repository: https://github.com/dmc-uilabs
In a terminal, navigate to the dmcdb folder.
If you already had the repository downloaded, make sure to do a git pull before proceeding.
Configuring a local PostSQL Database:
The following instructions will set up a local Postgres database for developers to test REST services against.
All of these commands are done on the command line. Lines with # are comments, lines with $ are shell commands, and lines with no prefix are postgres shell commands.
$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list' $ wget -q https://www.postgresql.org/media/keys/ACCC4CF8.asc -O - | sudo apt-key add - $ sudo apt-get update $ sudo apt-get install postgresql-9.3 postgresql-contrib-9.3
$ psql --version
$ sudo -u postgres psql postgres
\password postgres
CREATE USER gforge WITH PASSWORD 'gforge';
\du
\q
$ sudo -u postgres createdb -O gforge gforge
$ sudo -u postgres psql -c "\l"
$ sudo -u postgres psql postgres
\connect gforge
\q $ sudo nano /etc/postgresql/{your postgres version goes here}/main/pg_hba.conf
#In this file, search for 'postgres'. You should find a line that has the following:
Local | All | Postgres | Peer |
#In this file, change the word 'peer' to 'md5'. Save the changes and navigate to the folder with the stable version of the dmcdb
Starting your local database
Flyway: The DMC uses Flyway, which is a database migration program used for tracking changes to the database. Every time an update, change, or a new file is made in the database flayway is used to keep track of these individual changes, making it easy to revert changes made by developers.
Once you have configured your PostSQL Database run the following commands to start a new database instance:
sudo -u postgres psql -c "DROP DATABASE gforge"
sudo service postgresql restart
psql -U postgres -c "CREATE DATABASE gforge WITH OWNER gforge;"
sudo -u postgres psql -c "\l"
./flyway clean migrate info -configFile=conf/core/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge -user=gforge -password=gforge
./flyway migrate info -configFile=conf/data/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge -user=gforge -password=gforge -locations=filesystem:./sql/data/dev2
The database is now running and can be viewed by pgAdmin and can be used to run restservices locally.
Installing everything for frontend testing environment
sudo apt-get update cd ~/DMC/frontend curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.2/install.sh | bash //if not using zshell, only run .bashrc source ~/.bashrc source ~/.zshrc nvm install v5.1.1 //Check to see that your node version is 5.1.1 node -v //If you have issue running the commands below, you may need to close all your terminal windows, return to your frontend folder and try them once more. sudo apt-get install npm npm install npm install nodejs npm install -g bower bower install npm install gulp npm install babel-register sudo ./post-install.sh //Using Yarn to Serve and Build Frontend curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list sudo apt-get update && sudo apt-get install yarn //Download the following file and un zip in the root of frontend directory https://s3.amazonaws.com/dmc-build-aritifacts/0.1.5/bower_components.tar.gz //Serving Front end //in one terminal tab run the following in the following directory: /dmcfront/jsonserver node server.js //in another tab run the following in the root of the frontend directory gulp serve //building the frontend for testing //if you already have a dist folder in the root of your frontend directory to remove the dist folder rm -rf /dist //To build the dist folder, run the following in the root of the frontend directory: gulp build
Work Flow
The following are the stages your task for the most part should go through
Checking the .yaml file
If you have not yet done so, clone the frontend repository from github.
Change directories to the following: frontend/restapi
Open the file called: DMCInterfaceSwagger.yaml
Check and make sure that the .yaml file has information about your endpoint already there.
If it is missing, you will need to update your endpoint to reflect your code after it is completed.
Working on the data base
Follow the instructions listed above for creating a local database and connect to the database using pgAdmin (make sure you are on your own branch).
Open the folder structure as such: Server Groups → Servers → <Local Database running with postgres> → Databases → gforge → Schemas → public → Tables
Locate the table(s) that will be required for your endpoint(s) to function
If your table(s) is missing or has missing or incorrect column names or types, or does not have existing sample data, you will need to do the following
You will need to generate the code required to create your table. To do this do the following
In pgadmin, right click the tables folder and choose new table.
Under properties, fill out the table name (table should use lowercase characters only)(the norm for table names with more than one word should look like this: multiple_word_table_name).
Under columns, click add to create a new column. Fill out the name field (do not use camel case here, follow the same convention as used in the step above) and chose a type for the field such as: text, date, integer, or serial (generally used for ids).
When creating an id column that should generate an id on its own, do the following
Create the column with a name and the type of serial.
Click on the constraints tab.
Make sure Primary Key is selected from the drop-down menu.
Click Add.
In the new window, click on the columns tab
In the column drop-down menu, choose the name of the column.
Find the table that you created in the tables folder in pgadmin.
Click on the table name and then click on the table icon (7th icon from the top left of the window).
This is a view of what your table will look like
At the bottom of the pgadmin window, there is a pane with code. Copy this code.
- With postgres still running, run the following command:
- This command will show you the full list of migrations for table data
./flyway clean migrate info -configFile=conf/core/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge -user=gforge -password=gforge
- This command will show you the full list of migrations for sample data
./flyway info -configFile=conf/data/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge -user=gforge -password=gforge -locations=filesystem:./sql/data/dev2
- Thus the lists generated by the commands above to name your database table files and database sample data files such that they are one migration number higher. This will be covered again momentarily.
You will now update 3 files in the database. File names and locations are as follows:
dmcdb/sql/core/tables
Create a new table file.
- If the highest version listed from the ./flyway info -configFile=conf/core/flyway.conf command was 1.0081, the new file should be named V1_0082__<table_name>.psql
- V1 is the version number
- 0081 is the migration number
- If the highest version listed from the ./flyway info -configFile=conf/core/flyway.conf command was 1.0081, the new file should be named V1_0082__<table_name>.psql
Edit the file and paste the code that you copied in the step above here.
Cut the line that says CONSTRAINT table_name_PKEY PRIMARY KEY (id)
erase the lines: )WITH ( UIDS=FALSE
Save the file.
/dmcdb/sql/core/constraints/
Create a new primary_key_constraints file.
- Just like the new table file make sure the new file has a migration number that is one higher than the table name file.
- If the table file you just created is called V1_0082__example_table.psql, the new constrains file should be named V1_0083__primary_key_constrains.psql
- __primary_key_constraints.psql should never change on the version or migration number should.
- Just like the new table file make sure the new file has a migration number that is one higher than the table name file.
Add the following lines to the file:
- -Name: table_name_pkey; Type: CONSTRAINT; Schema: public; Owner: gforge; Tablespace: - ALTER TABLE ONLY <table_name> ADD <paste what you cut earlier here>;
Save the file.
dmcdb/data/dev2/
Create a new sample data file.
- If the highest version listed from the ./flyway info -configFile=conf/data/flyway.conf command was 1.0014, the new file should be named V1_0015__<sample_data_name>.psql
- V1 is the version number
- 0015 is the migration number
- If the highest version listed from the ./flyway info -configFile=conf/data/flyway.conf command was 1.0014, the new file should be named V1_0015__<sample_data_name>.psql
Add three lines of sample data here. Use the following as an example for syntax
insert into table_name (column_name one, column_name_two) values ('text', 'textAlso');
Explanation:
table_name: use the name of your table here in its place.
The column names should be listed in order separated by commas.
The sample data must be listed in the same order as you list the column names.
Text data should be quoted using apostrophes and separated by commas. Numbers should not be quoted at all.
The id that should be auto generated should not be listed in the column name or the sample data.
Rerun postgres commands to drop database, create a database, check that the data base was created, add the tables to the database, and add the sample data.
Remember, you can not use the above commands if you are connected to the database with pgadmin, you will need to close pgAdmin, run the commands and then reopen pgAdmin..
- At the bottom of this page there is a bash script for dropping your database, creating a new database, checking that the database exists, filling the tables and filling the sample data.
Check to make sure you can open your table and that it has sample data in it. Also try to add another line of information. pgAdmin should handle generating the serial id, and the next line.
Once you've confirmed the information in the database is correct and updated you can now start working on your code.
Working on your code
Make sure that if you have not yet cloned the restservices repository from GitHub, that you do so now.
Also, make sure you create a new branch from the master branch.
Open Eclipse
Refresh Eclipse
Testing your code with swagger
Manual
Automatic
Once you have code you would like to test, start drop and rebuild your local database.
Next, fine the Config.java file located in the following directory.
restservices/src/main/java/org/dmc/services/
Change the first 6 lines to be the following.
public static final String DB_PORT = "5432"; public static final String DB_IP = "127.0.0.1"; public static final String DB_HOST = "jdbc:postgresql://" + DB_IP; public static final String DB_NAME = "gforge"; public static final String DB_USER = "gforge"; public static final String DB_PASS = "gforge";
Save the changes in this file.
- Next, make sure th following is found in one of the the files listed below
- sudo nano /etc/profile, ~/.bashrc, or ~/.zshrc
Run the command: source /etc/profile
export DBip="127.0.0.1" export DBport=5432 export DBuser="gforge" export DBpass="gforge" export SOLR_BASE_URL="http://<SOLR IP Address>/solr" export ActiveMQ_URL="<ActiveMQ IP Address>" export ActiveMQ_Port=61616 export ActiveMQ_User="active" export ActiveMQ_Password="active"
- sudo nano /etc/profile, ~/.bashrc, or ~/.zshrc
- Next update the following file restservices/src/main/java/org/dmc/services/Application.java
- All references to DB under the @Beam should be replaced with the following:
ds.setUser(Config.DB_USER); ds.setPassword(Config.DB_PASS); ds.setServerName(Config.DB_IP); ds.setPortNumber(Integer.parseInt(Config.DB_PORT)); ds.setDatabaseName(Config.DB_NAME);
Save the changes in this file.
Run the command: sudo service tomcat7 stop
Go to the following directory:
/var/lib/tomcat7/webapps
If there are any files in the webapps folder, delete them. The command to do this is:
sudo -r *
Next, go to the restservices directory.
Run the command: mvn spring-boot:run -P swagger
Compilation now happens
When completed, open a web browser and got to the following URL:
You will need to then write tests for the code you have written.
Once you believe your tests are complete, you can test them by first closing all terminals you have.
Open a new terminal.
Make sure postgres is running an instance of your database.
Change directories to your rest services folder
Then write the following command
mvn -Dtest=<TestName> test
DO NOT include the triangle brackets in your command
Make sure your environment variable are set or are included in your Config.java file so that your local database is being referenced.
Submission/Code Review (for database changes and resetservice changes)
To submit your cod for code review use the following commands:
Git status
Shows you what files you have made changes to in the repository
git add -A
-A add all files that have changes, by giving the full file path in its place you can add specific files
Git status
this checks which files you've added to the staging area.
Git commit -m “information about commit”
-m lets you add information about the commit.
“” are used for capturing comments.
Git push origin BranchName
In place of branch name include your actual branch name.
This will push your changes to GitHub.
Go to GitHub, and navigate to your branch
Make sure to sync your branch with the master branch
Make a pull request and add a few reviewers.
Updating the .yaml file
After your changes have been merged, find the .yaml file mentioned in the steps above and edit it as well. Make sure to add tags live and has tests to your endpoint. Submit this for code review.
Frontend-JSON Implimentation on Local Machine
- In the DMC frontend folder, navigate to app/scripts/common/factory/data.factory.js
- Comment out line 7.
- //localhost = $window.apiUrl ? $window.apiUrl + '/' : localhost;
- On line 8 change port 8080 to 3000 like shown below
- var urlSocketServer = 'http://localhost:3000/rest';
- This makes your frontend now receive data from your REST services, instead of from the sample JSONserver
- Comment out line 7.
- Open two terminal windows, one in the /frontend directory, and one in the /frontend/jsonserver directory.
- In the /frontend directory run the command: gulp serve
- In the /fronted/jsonserver directory run the command: node server.js
- In your internet browser, navigate to localhost:9000 and your frontend code should now be communicating with the JSON server.
Full-Stack Local Implementation (Frontend-Restservices-Database)
Documentation on this is coming soon.
Useful Bash Scripts
#!/bin/bash echo "Changing Directories" cd dmcdb echo "Droping the current database" sudo -u postgres psql -c "DROP DATABASE gforge" echo "creating a new database" sudo -u postgres createdb -O gforge gforge echo "Confirm new database (gforge) was created" sudo -u postgres psql -c "\l" echo "creating tables" ./flyway clean migrate info -configFile=conf/core/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge -user=gforge -password=gforge echo "filling tables with sample data" ./flyway info -configFile=conf/data/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge -user=gforge -password=gforge -locations=filesystem:./sql/data/dev2 ./flyway migrate info -configFile=conf/data/flyway.conf -url=jdbc:postgresql://localhost:5432/gforge -user=gforge -password=gforge -locations=filesystem:./sql/data/dev2 echo "-----------------open pgadmin-----------------------"
If restoring a database the following command is used
psql -U gforge gforge < azurePortal20170523.sql
You can use the commands below to stop postgres from running and subsequently disconnect application that are accessing your database.
sudo service postgresql stop sudo service postgresql start
#!/bin/bash echo "Stopping Tomcat." sudo service tomcat7 stop echo " " sleep 2 echo "Moving to restservices directory." echo " " cd ~/Desktop/dmc/restservices echo "Starting mvn spring-boot:run -P swagger" echo " " sleep 3 firefox localhost:8080/swagger-ui.html echo "When complete go to localhost:8080/swagger-ui.html in a browser." mvn spring-boot:run -P swagger