Importance of seeing Apache way in a Hackathon with DataVis

[kanchai123]

I'm recently found the is cool Bitergia (http://bitergia.com/) to see as a tool that we can analyze data to evaluate community efforts by getting, providing and analyzing Software Development metrics and information about FLOSS projects and communities for managing and improving them . After hosting a hackathon, we expect to see the marvel of growing these session ideas carried away to perform as active communities. We will be able to find the meritocracy, code review activities and active mentorship among the communities. This will also help to understand community interests based on the topics.

[kanchai123]

Start with Ubuntu 14.04 image - shortest path to working software
Install prerequisites

MySQL
git - clone and upgrade to >= 2.1.0
Python: python-setuptools python-pysqlite2 python-storm python-launchpadlib python-beautifulsoup python-feedparser python-configglue python-mysqldb javascript-common python-testresources python-egenix-mxdatetime mysql-server python-mysqldb-dbg python-distribute python-dateparser python-dateutil
Build tools: gcc mysql-devel gcc-c++ libmysqlclient-dev
Add deb http://cran.cnr.berkeley.edu/bin/linux/ubuntu/ precise/ to /etc/apt/sources.list, authorise, and install R >= 3.0 from that repo
Clone, make, and install these VizGrimoire packages:
mkdir /usr/src/viztools; cd /usr/src/viztools
git clone https://github.com/MetricsGrimoire/RepositoryHandler.git
git clone https://github.com/MetricsGrimoire/CVSAnalY.git
git clone https://github.com/MetricsGrimoire/Bicho.git
git clone https://github.com/MetricsGrimoire/MailingListStats.git
git clone https://github.com/MetricsGrimoire/CMetrics.git
Make and install this:
git clone https://github.com/MetricsGrimoire/Automator.git
Create a text file with the following format:
[ProjectName]
source="url","url"
trackers="url","url"
...
[ProjectName2]
...
Now that internal GitHub is private you will need an api key to access repositories. Create one here

Repo urls are of the form: https://MyPersonalAPIKey:x-oauth-basic@OrgName/RepoName
Issue urls are of the form: https:///api/v3/repos/OrgName/RepoName/issues. Bicho will insert the username/API key to the issues url when querying (after configuration)
Set up the projects with create_projects.py in the Automator directory: python create_projects.py -d /path/to/projects -p your_project_configs_file

This will create a number of directories under your project dir, and clone the GrimoireLib tools to each project (designated with the square bracket sections in your config file)
Patch Bicho to allow for local GitHub access

Allow floating point rate limits config.dff
Enable Enterprise GitHub connections github.diff
Add the following config to each project's conf/main.conf file

[generic] add db_user and db_password for all databases
[generic] Make sure that each tool (cvsanaly, bicho, etc) has a "db_tool" entry. Missing entries usually means an issue with the config file used for create_projects.py.
[bicho] section change backend = github and add backend_user = and backend_password = entries
Run analysis from Automator: python launch.py --dir /path/to/projects/ProjectName

Generate html in VizGrimoireJS directory

Copy all files from VizGrimoireJS/browser to your web server

Copy /path/to/projects/ProjectName/json/* to the data/json/ directory under your web server directory where you copied the VizGrimoireJS html files

Run your web server and evaluate the data

What has and hasn't worked

Commit and committer data seems to work well. Trends over time are visible and easy to understand.

Several of the other aspects of the analysis do not seem to work. Several of the R scripts fail out with error messages due to bugs dealing with the database such as when an array is expected but the database only has a single value to return.

Examples

A full install of two projects is available from the demo directory. Copy all the files from that directory and serve from a webserver. Example screenshots can be found in the examples directory.

Overview screen
Source code metrics
Issues metrics
Suggestions on working w/ Bitergia

Some things that could be part of working with Bitergia staff

Fix the database access bug
when doing the SQL queries sometimes an array is returned, sometimes just an item if there is only one row result. The solution is to change it to always return and array, but in some places of the code, an array is not expected. We should change this last case, but it is not done yet.
Get all the R scripts working
Improve the integration with Enterprise GitHub: Parse Enterprise GH headers, add Enterprise url munging when referencing Enterprise repos, etc.
Highlight code review trends and activities - both PR and non-PR based (merge without PR?)
Research integrating mailing list stats with an Exchange email / DLManager email list
Research other interesting things that can be learned with our data

[pzimdars]

@kanchai123 We can deploy an Ubuntu 14.04 image and give you access to install the above. Would that work?