Skip to content
This repository has been archived by the owner on May 27, 2024. It is now read-only.

Automated Deployment

jmfield2 edited this page Nov 30, 2016 · 7 revisions

.The structure of the automated deployment is based on Jenkins, and Chef.

#Jenkins# The development machine (with Jenkins) automatically builds: usf-mobullity/mobullityrebase, usf-mobullity/usf-gui (geocoder), and bullrunner-gtfs-realtime-generator/master every night at midnight if there were SCM changes.

After the build finishes, there are build hooks defined to run the following batch files (on the development machine) depending on the repository built:

NOTE: The last line is "cmd1 && cmd2" because on separate lines cmd2 was not executed - aka, the file was updated in the local c:/chef-repo but NOT uploaded to the server for use by chef.

c:/OTPFILES/chef-geocoder.bat:
REM Copy geocoder WAR from jenkins
copy "C:\Program Files (x86)\Jenkins\jobs\OTP Geocoder\workspace\opentripplanner-geocoder\target\opentripplanner-geocoder.war" c:\chef-repo\cookbooks\geocoder_tomcat\files\default\otp-geocoder.war

cd c:\chef-repo

REM Bump cookbook version, Upload cookbook (XXX and freeze?)
C:\opscode\chef\bin\knife spork bump geocoder_tomcat && C:\opscode\chef\bin\knife cookbook upload geocoder_tomcat

This copies the newly built WAR and uploads it into a new revision of the geocoder_tomcat cookbook in Chef for deployment by the servers.

C:/OTPFiles/chef-otp.bat:
REM copy newly compiled JAR from Jenkins workspace into the chef cookbook
copy "C:\Program Files (x86)\Jenkins\jobs\Mobullity\workspace\target\otp.jar" c:\chef-repo\cookbooks\otp\files\default\otp.jar

cd c:\chef-repo

REM First, BUMP the cookbook patch version & Second, UPLOAD the changes (the file) to the chef repo server
C:\opscode\chef\bin\knife spork bump otp && c:\opscode\chef\bin\knife cookbook upload otp

XXX

C:/OTPFiles/chef-gtfs.bat:
REM Copy JAR from jenkins workspace
copy "C:\Program Files (x86)\Jenkins\jobs\GTFS-RT\workspace\target\cutr-gtfs-realtime-bullrunner-0.9.0-SNAPSHOT.jar" c:\chef-repo\cookbooks\gtfsrealtime\files\default\cutr-gtfs-realtime-bullrunner-0.9.0-SNAPSHOT.jar

cd c:\chef-repo

REM Bump cookbook version
C:\opscode\chef\bin\knife spork bump gtfsrealtime && c:\opscode\chef\bin\knife cookbook upload gtfsrealtime

XXX

Service Management

Non-sucking Service Manager (NSSM) is configured on the 3 mobullity servers to manage the following Java services: mobullityOTP, mobullityGTFS and allows for the use of the Windows services control panel to start and stop the processes. The command-line configuration options are stored inside the NSSM wrapper and can be accessed by running:

nssm.exe edit mobullityOTP

OTP: -Xmx2G -jar c:\otpfiles\otp.jar -g c:\otpfiles -s --port 80 GTFS: -jar c:\bullrunner-gtfs-realtime-generator\target\cutr-gtfs-realtime-bullrunner-0.9.0-SNAPSHOT.jar --tripUpdatesUrl=http://localhost:8088/trip-updates --vehiclePositionsUrl=http://localhost:8088/vehicle-positions NOTE: GTFS must be in startup directory \target\ because of hard-coded ../myGTFS/ path.

Tomcat7 is used to host the old-style geocoder WAR on the following paths:

Mobullity1: c:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps Mobullity2: Mobullity3:

Chef Dependencies

knife spork plugin windows

Chef Environment

Development

Environment

This is defined in chef_environment per node (knife node edit), and knife environment [list|edit]

This will let you set e.g. default cookbook versions:

  "cookbook_versions": {
    "geocoder_tomcat": "= 1.0.0",
    "gtfsrealtime": "= 1.0.0",
    "otp": "= 1.0.4"
  },

Chef Attributes

GeocoderTomcat:

attributes/default.rb: default['tomcat']['path'] = 'C:/Program Files (x86)/Apache Software Foundation/Tomcat 7.0/'

This is overridden by node with: knife node edit as follows:

  "normal": {
    "tomcat": {
      "path": "c:/Program Files/Apache Software Foundation/Tomcat 7.0/"
    },

Chef Cookbooks

Dependencies: Windows, chef_handler

OTP:

Manage Graph.properties config for OTP deployment for consistent settings and restart service.

cookbook_file "c:/OTPfiles/Graph.properties" do
	source "Graph.properties"
	action :create
	notifies :restart, "windows_service[mobullityOTP]"
end

To make a change and distribute to chef, simply open the file in c:\chef-repo\cookbooks\otp\files, save, and after optionally bumping the version (in cookbooks\otp\metadata.rb or with knife spork bump) you upload the new file to chef with: knife cookbooks upload otp.

This block defines the commands used to build a new graph using the OSM and GTFS data in the working directory. It must be scheduled independently of chef (via e.g schtasks) so that the server does not go down every time a change is detected.

execute "osm_build" do
	#notifies :stop, "windows_service[mobullityOTP]", :immediately
	
	command "java -jar otp.jar -g . -b ."
	creates "C:/OTPFiles/Graph.obj"
	cwd "C:/OTPFiles"

	action :nothing
	notifies :start, "windows_service[mobullityOTP]"
end

These blocks manage the HART, PSTA, and Bullrunner GTFS files from the remote sites shown and updates the local file if any changes are found. This also schedules a graph rebuild.

remote_file "C:/OTPFiles/bullrunner-gtfs.zip" do
	source "https://github.com/CUTR-at-USF/bullrunner-gtfs-realtime-generator/raw/master/bullrunner-gtfs.zip"
	action :create
	
	notifies :run, "execute[osm_build]"
end

remote_file "C:/OTPFiles/hart.zip" do
	source "http://www.gohart.org/google/google_transit.zip"
	action :create
	
	notifies :run, "execute[osm_build]"
end

remote_file "C:/OTPFiles/psta.zip" do
	source "http://www.psta.net/latest/google_transit.zip"
	action :create
	
	notifies :run, "execute[osm_build]"
end

Makes sure the Tampa, FL extract from https://s3.amazonaws.com/metro-extracts.mapzen.com/tampa_florida.osm.pbf is being used on the server and copies it to map.osm.pbf if any changes occur. If this happens it will run the osm_build block to schedule a graph rebuild.

The OSM File is not automatically polled since changes would happen at least daily, but instead must be manually pulled from the metro-extracts URL and uploaded to chef the same way as Graph.properties.

cookbook_file "C:/OTPFiles/map.osm.pbf" do
	source "tampa_florida.osm.pbf"
	action :create
	
	notifies :run, "execute[osm_build]"
end

These two blocks copy any newly built JAR files from files/otp.jar into OTPFiles/otp.jar.stage, stops the service, and then runs copy_jar which copies this file to otp.jar and restarts the service. This will happen every time Jenkins builds the project and pushes the new JAR to chef when the build hook executes.

execute "copy_jar" do

	command "copy /y otp.jar.stage otp.jar"	
	cwd "C:/OTPFiles"

	action :nothing
	notifies :start, "windows_service[mobullityOTP]"
end

cookbook_file "C:/OTPFiles/otp.jar.stage" do

	source "otp.jar"
	action :create
	
	# XXX rebuild graph?
	
	notifies :stop, "windows_service[mobullityOTP]", :immediately
	notifies :run, "execute[copy_jar]", :immediately
	
	#notifies :start, "windows_service[mobullityOTP]"
end

Finally, this block makes chef ensure the OTP service is running every time it runs (30 minute intervals).

windows_service "mobullityOTP" do
	action :start
	startup_type :automatic
	
    supports :status => true, :restart => true, :start => true
end

files/: tampa_florida.osm.pbf otp.jar Graph.properties

Geocoder_Tomcat:

Sync the Tomcat/Geocoder configuration files from chef and restart Tomcat. This allows the path to tomcat to change based on the installation and is configured by node attribute.

knife node edit NODE-NAME

cookbook_file "#{node["tomcat"]["path"]}/conf/context.xml" do
	source "context.xml"
	action :create
	notifies :restart, "windows_service[Tomcat7]"
end

cookbook_file "#{node["tomcat"]["path"]}/conf/server.xml" do
	source "server.xml"
	action :create
	notifies :restart, "windows_service[Tomcat7]"
end

cookbook_file "#{node["tomcat"]["path"]}/conf/web.xml" do
	source "web.xml"
	action :create
	notifies :restart, "windows_service[Tomcat7]"
end

cookbook_file "#{node["tomcat"]["path"]}/conf/tomcat-users.xml" do
	source "tomcat-users.xml"
	action :create
	notifies :restart, "windows_service[Tomcat7]"
end

cookbook_file "#{node["tomcat"]["path"]}/webapps/otp-geocoder/WEB-INF/web.xml" do
	source "geocoder-web.xml"
	action :create
	notifies :restart, "windows_service[Tomcat7]"
end

Using either the default attribute for this cookbook, or the node attribute, copy the WAR from chef to the tomcat webapps dir, and restart tomcat.

cookbook_file "#{node["tomcat"]["path"]}/webapps/otp-geocoder.war" do
	source "otp-geocoder.war"
	action :create
	
	notifies :restart, "windows_service[Tomcat7]"
end
windows_service "Tomcat7" do
	action :start
	startup_type :automatic
end

files/: otp-geocoder.war context.xml geocoder-web.xml server.xml tomcat-users.xml web.xml

GTFSRealtime:

This copies new bullrunner GTFS data to the working directory and extracts the zipfile (as expected by the server) to myGTFS.

windows_zipfile "c:/bullrunner-gtfs-realtime-generator/myGTFS" do
	source "c:/bullrunner-gtfs-realtime-generator/bullrunner-gtfs.zip"
	action :nothing
	notifies :restart, "windows_service[mobullityGTFS]"
end

cookbook_file "c:/bullrunner-gtfs-realtime-generator/bullrunner-gtfs.zip" do
	source "bullrunner-gtfs.zip"
	action :create
	
	notifies :unzip, "windows_zipfile[c:/bullrunner-gtfs-realtime-generator/myGTFS]"
end

As above, copy any new gtfsrealtime jars to a staging area, stop the service, copy it over, and restart.

execute "copy_jar" do

	command "copy /y cutr-gtfs-realtime-bullrunner-0.9.0-SNAPSHOT.jar.stage cutr-gtfs-realtime-bullrunner-0.9.0-SNAPSHOT.jar"
	cwd "C:/bullrunner-gtfs-realtime-generator/target"

	action :nothing
	notifies :start, "windows_service[mobullityGTFS]"
end

cookbook_file "c:/bullrunner-gtfs-realtime-generator/target/cutr-gtfs-realtime-bullrunner-0.9.0-SNAPSHOT.jar.stage" do
	
	source "cutr-gtfs-realtime-bullrunner-0.9.0-SNAPSHOT.jar"
	action :create

	notifies :stop, "windows_service[mobullityGTFS]", :immediately
	notifies :run, "execute[copy_jar]", :immediately
	
end

As above, just keep the service running.

windows_service "mobullityGTFS" do
	action :start
	startup_type :automatic
end

files/: cutr-gtfs-realtime-bullrunner-0.9.0-SNAPSHOT.jar

Steps to Install/Provision Servers w/ Chef:

Install the following software manually:

  1. Oracle Java 1.8 64bit
  2. Tomcat 64bit service installer
  3. http://nssm.cc (the non-sucking service manager) (OTP, GTFS)
  4. Chef-client https://downloads.chef.io/chef-client/windows/

Install NSSM for OTP,GTFS:

  • Download package from website, extract win64\nssm.exe to desktop and in command prompt run:
nssm install mobullityotp java -jar c:\otpfiles\otp.jar -g c:\otpfiles -s --port 80
nssm set mobullityotp AppDirectory c:\otpfiles # for startup dir

nssm install mobullitygtfs java -jar c:\bullrunner-gtfs-realtime-generator\target\cutr-gtfs-realtime-generator-0.9.0-SNAPSHOT.jar --tripUpdatesUrl=http://localhost:8088/trip-updates --vehiclePositionsUrl=http://localhost:8088/vehicle-positions

"Service "mobullitygtfs" installed successfully!"

Individual settings can be set via commandline:

nssm set mobullitygtfs AppDirectory c:\bullrunner-gtfs-realtime-generator

"Set parameter "AppDirectory" for service "mobullitygtfs"."

Chef

We use Opscode Chef (Hosted) on the free tier which allows monitoring of 5 nodes.

  1. Install the client (instructions @ https://docs.chef.io/release/12-5/install_dk.html)
  2. Setup the client and node information and keys: knife client create, knife node create
  3. Add chef permission for client: (on manage.chef.io or knife download/upload /acls/nodes)
  • chef-client -c c:\chef-repo.chef\knife.rb -N mobullity2
  • copy new .rb config to c:\chef\client.rb for chef-windows-service to find

Install knife-spork plugin using embedded ruby: c:\opscode\chef\embedded\bin\gem install knife-spork

NOTE: Chef stores cookbook files @ Amazon S3 (knife cookbook show otp 1.0.4) Also, you may need to install the knife 'spork' plugin (used by development build hooks)

After this, you just have to specify which environment this node is in (production, development), and make sure the chef-client service is running.

Every 30 minutes (the default), chef-client will run and process the cookbook recipes specified in the environment. Upgrading or downgrading versions and files as necessary.