Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run Distributed Release Audit Tool (DRAT) on all codefest generated code and report out on license statistics #9

Open
chrismattmann opened this issue Oct 7, 2014 · 6 comments
Open

Run Distributed Release Audit Tool (DRAT) on all codefest generated code and report out on license statistics #9

chrismattmann opened this issue Oct 7, 2014 · 6 comments
Labels
data science proposed session

Comments

@chrismattmann
Copy link
Collaborator

DRAT (https://github.com/chrismattmann/drat/) is a release audit tool that takes Apache RAT and turns it into a Map Reduce style system for large and heterogeneous code bases where RAT falls flat on its face. RAT is unable to easily differentiate between different file MIME types and tries to do license analysis on e.g., binary files unless specified through complex white lists and black lists. DRAT on the other hand, improves upon RAT by taking Apache Tika, partitioning the code base by MIME type, constructing a Solr4 catalog of the code, and then farming out a large Map Reduce style job wherein which the Mapper is a N-sized (configurable, set initially to 100) set of files of the same MIME type, partitioned across machines using Apache OODT, and the reducer is the RAT log aggregator that combines each Mapped RAT job's intermediate RAT log output.

DRAT has been run on the DARPA XDATA code base (~50K thousand files, 10s of M of lines of code), and the Computational Infrastructure for Geodynamics (CIG) (~500K thousand files, 100s of M of lines of code) and scales well, is easy to use and the software can be run on a single machine with an existing OS or ran using Vagrant and vagrant up as a virtual machine.

This task will involve deploying DRAT, and then running it across the code bases to perform a license analysis and to report out on the results at the end of the hackathon. Patches and improvements to DRAT are welcomed as well.

Continuation of issue from Open Science Codefest: NCEAS/open-science-codefest#27
@lewismc
Copy link
Collaborator

lewismc commented Apr 23, 2015

The following codebases were analyzed using DRAT

https://github.com/NSF-Polar-Cyberinfrastructure/datavis-hackathon.git
https://github.com/NSF-Polar-Cyberinfrastructure/issue-79.git
https://github.com/NSF-Polar-Cyberinfrastructure/issue-15.git
https://github.com/NSF-Polar-Cyberinfrastructure/issue-3.git
https://github.com/NSF-Polar-Cyberinfrastructure/issue-43.git
https://github.com/alb0/3d-greenland.git

Reports are available at the following ZIP

https://www.dropbox.com/s/w5pn6fp8tms9m60/nsf_polar_hackathon.zip?dl=0

Advice on how to interpret the report can be found at https://github.com/chrismattmann/drat/wiki/Interacting-with-DRAT

Notes Binaries Archives Standards Apache Generated Unknown
0 0 0 109 17 0 92

@lewismc
Copy link
Collaborator

lewismc commented Apr 23, 2015

Unapproved licenses

27 Javascript
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/Detector.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/KML.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/Tween.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/bootstrap.js_04232015_1231
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/issues.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.fileupload-process.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.fileupload-ui.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.fileupload.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.flexslider.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.js_04232015_1231
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.main.eng.min.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.min.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.ui.widget.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/jquery.xdr-transport.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/kml-layer.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/leaflet-0.6.4_leaflet.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/leaflet.filelayer.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/leaflet.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/main.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/map.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/ol.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/pictures.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/side_menu.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/three.min.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/togeojson.js
  /usr/local/drat/deploy/data/jobs/rat/1429817469424/input/update-script.js

14 CSS
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/OpenLayer_style.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/bootstrap-glyphicons.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/bootstrap.css_04232015_1231
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/flexslider.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/jquery.fileupload-ui.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/jquery.fileupload.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/leaflet-0.6.4_leaflet.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/leaflet-0.6.4_leaflet.ie.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/leaflet.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/ol.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/promo-min.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/side_menu.css
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/side_menu.css_04232015_1231
  /usr/local/drat/deploy/data/jobs/rat/1429817469787/input/style.css

26 Python
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/ajax.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/clip_geotiff_by_shp.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/config.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/conversion.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/data_management.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/extract_shp_table.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/extract_shp_table.py_04232015_1231
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/forms.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/gtif_to_tile.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/manage.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/metadata.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/models.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/nc_dict.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/netcdf_dict.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/open_file.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/opendap.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/re-project.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/settings.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/shp_dict.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/spatial_analysis.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/tif_dict.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/urls.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/urls.py_04232015_1231
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/views.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/what_file.py
  /usr/local/drat/deploy/data/jobs/rat/1429817471306/input/wsgi.py

23 HTML

  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/buffer_shp.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/clip_geotiff_by_shp.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/color_table_geotiff.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/coord_to_point_shp.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/extract_shp_table.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/footer.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/geotiff_resolution.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/geotiff_to_kml.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/header.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/index.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/index.html_04232015_1231
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/ncdump_header.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/ncdump_whole_netcdf.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/netcdf_to_geojson.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/netcdf_to_geotiff.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/point_inside_shapefile.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/re_project_geotiff.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/re_project_shapefile.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/shp_to_json.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/shp_to_kml.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/shp_to_tif.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/tif_to_point_json.html
  /usr/local/drat/deploy/data/jobs/rat/1429817469678/input/tif_to_point_shp.html

1 x-text
  /usr/local/drat/deploy/data/jobs/rat/1429817470196/input/report.tex

1 x-sh
  /usr/local/drat/deploy/data/jobs/rat/1429817469962/input/parse.sh

@lewismc
Copy link
Collaborator

lewismc commented Apr 23, 2015

@r4space @chrismattmann check this out

@lewismc
Copy link
Collaborator

lewismc commented Apr 23, 2015

@chrismattmann feel free to close off if you feel the task has been completed. The remainder of the work is to get ALv2.0 over everything else.

@lewismc
Copy link
Collaborator

lewismc commented Apr 23, 2015

Some more stats folks. Using cloc I managed to obtain the following which is rather nice

lmcgibbn@LMC-032857 ~/Desktop $ ./cloc-1.62.pl nsf_polar_hackathon.zip
     143 text files.
     135 unique files.
      25 files ignored.

http://cloc.sourceforge.net v 1.62  T=0.72 s (164.3 files/s, 44677.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
CSS                             15           2135            134          12528
Javascript                      32           1151           1291           8739
Python                          24            265             49           1245
HTML                            25            172             70           1116
XML                              4            411           1513            627
Teamcenter met                   8              0              0            488
Bourne Shell                    10             12              0            148
-------------------------------------------------------------------------------
SUM:                           118           4146           3057          24891
-------------------------------------------------------------------------------

@chrismattmann
Copy link
Collaborator Author

Thanks @lewismc this is great! Let's leave this open as I want all sessions to remain showing on the website. Thank you this is perfect!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
data science proposed session
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chrismattmann @lewismc