Merge pull request #3100 from nexB/add-license-dump-option

Add license dump option

.gitignore

@@ -95,3 +95,20 @@ tcl
 /00-*.txt
 /z-todo-licenses-*
 
+# Extra ignores from licensedb
+*.pyc
+*.db
+.installed.cfg
+parts
+develop-eggs
+eggs
+downloads
+.settings
+TAGS
+Procfile
+local.cfg
+geckodriver.log
+var
+.metaflow
+selenium
+/dist/

CHANGELOG.rst

@@ -53,15 +53,30 @@ License detection:
   detected license only once. This data can contain the reference license text
   as an option.
 
-- We can now detect licenses using custom license texts and license rules.
-  These can be provided as a one off in a directory or packaged as a plugin
-  for consistent reuse and deployment.
-
 - There is a new "scancode-reindex-licenses" command that replace the
   "scancode --reindex-licenses" command line option which has been
   removed. This new command supports simpler reindexing using custom
   license texts and license rules contributed by plugins or stored in an
-  additional directory. 
+  additional directory. The "--reindex-licenses-for-all-languages" CLI option
+  is also moved to the "scancode-reindex-licenses" command as an option
+  "--all-languages".
+
+- We can now detect licenses using custom license texts and license rules.
+  These can be provided as a one off in a directory or packaged as a plugin
+  for consistent reuse and deployment. There is an option "--additional-directory"
+  with the "scancode-reindex-licenses" command and also a new "--only-builtin"
+  option to only use the builtin licenses to build the cache.
+
+- Scancode LICENSE and RULE files now also contain their data as YAML frontmatter,
+  which previously used to be in their respective YAML files. This reduces number of
+  files in those directories, 'rules' and 'licenses' to half. Git line history is
+  preserved for the files.
+
+- A new command line option "--get-license-data" is added to dump license data in
+  JSON, YAML and HTML formats, and also generates a local index and a static website
+  to view the data. This will essentially be an API/way to get scancode license data
+  as opposed to just reading the files. 
+
 
 Package detection:
 ~~~~~~~~~~~~~~~~~~~~~

requirements.txt

@@ -56,6 +56,7 @@ pygmars==0.7.0
 Pygments==2.12.0
 pymaven-patch==0.3.0
 pyparsing==3.0.9
+python-frontmatter==1.0.0
 pytz==2022.1
 PyYAML==6.0
 rdflib==6.2.0

setup.cfg

@@ -101,6 +101,7 @@ install_requires =
     pygmars >= 0.7.0
     pygments
     pymaven_patch >= 0.2.8
+    python-frontmatter >= 1.0.0
     requests >= 2.7.0
     saneyaml >= 0.5.2
     spdx_tools >= 0.7.0a3

src/licensedcode/data/inactive/rules/free-unknown_url_glc_90.RULE

@@ -1 +1,12 @@
+---
+license_expression: free-unknown
+relevance: 100
+notes: |
+    License test derived from a file of the BSD-licensed repository at:
+    https://raw.githubusercontent.com/google/licensecheck/v0.3.1/testdata/ccurls.t1
+    originally expected to be detected as LiLiQ-P-1.1
+    with coverage of 0
+    This is a URL test extracted from ccurls.t1.
+---
+
 http://opensource.org/licenses/liliq-p-1.1

src/licensedcode/data/inactive/rules/free-unknown_url_glc_91.RULE

@@ -1 +1,12 @@
+---
+license_expression: free-unknown
+relevance: 100
+notes: |
+    License test derived from a file of the BSD-licensed repository at:
+    https://raw.githubusercontent.com/google/licensecheck/v0.3.1/testdata/ccurls.t1
+    originally expected to be detected as LiLiQ-P-1.1
+    with coverage of 0
+    This is a URL test extracted from ccurls.t1.
+---
+
 https://opensource.org/licenses/liliq-p-1.1

src/licensedcode/data/inactive/rules/free-unknown_url_glc_92.RULE

@@ -1 +1,12 @@
+---
+license_expression: free-unknown
+relevance: 100
+notes: |
+    License test derived from a file of the BSD-licensed repository at:
+    https://raw.githubusercontent.com/google/licensecheck/v0.3.1/testdata/ccurls.t1
+    originally expected to be detected as LiLiQ-R-1.1
+    with coverage of 0
+    This is a URL test extracted from ccurls.t1.
+---
+
 http://opensource.org/licenses/liliq-r-1.1

src/licensedcode/data/inactive/rules/free-unknown_url_glc_93.RULE

@@ -1 +1,12 @@
+---
+license_expression: free-unknown
+relevance: 100
+notes: |
+    License test derived from a file of the BSD-licensed repository at:
+    https://raw.githubusercontent.com/google/licensecheck/v0.3.1/testdata/ccurls.t1
+    originally expected to be detected as LiLiQ-R-1.1
+    with coverage of 0
+    This is a URL test extracted from ccurls.t1.
+---
+
 https://opensource.org/licenses/liliq-r-1.1

src/licensedcode/data/inactive/rules/free-unknown_url_glc_94.RULE

@@ -1 +1,12 @@
+---
+license_expression: free-unknown
+relevance: 100
+notes: |
+    License test derived from a file of the BSD-licensed repository at:
+    https://raw.githubusercontent.com/google/licensecheck/v0.3.1/testdata/ccurls.t1
+    originally expected to be detected as LiLiQ-Rplus-1.1
+    with coverage of 0
+    This is a URL test extracted from ccurls.t1.
+---
+
 http://opensource.org/licenses/liliq-rplus-1.1

src/licensedcode/data/inactive/rules/free-unknown_url_glc_95.RULE

@@ -1 +1,12 @@
+---
+license_expression: free-unknown
+relevance: 100
+notes: |
+    License test derived from a file of the BSD-licensed repository at:
+    https://raw.githubusercontent.com/google/licensecheck/v0.3.1/testdata/ccurls.t1
+    originally expected to be detected as LiLiQ-Rplus-1.1
+    with coverage of 0
+    This is a URL test extracted from ccurls.t1.
+---
+
 https://opensource.org/licenses/liliq-rplus-1.1

src/licensedcode/data/inactive/rules/unknown_use_subject_to_licence_terms.RULE

@@ -1 +1,7 @@
+---
+license_expression: unknown
+is_license_reference: yes
+notes: mostly used by Oracle in the JDK and Linux
+---
+
 Use is subject to licence term

src/licensedcode/data/inactive/rules/unknown_use_subject_to_licence_terms2.RULE

@@ -1 +1,7 @@
+---
+license_expression: unknown
+is_license_reference: yes
+notes: mostly used by Oracle in the JDK and Linux
+---
+
 Use is subject to licence terms.

src/licensedcode/data/inactive/rules/unknown_use_subject_to_license_terms.RULE

@@ -1 +1,7 @@
+---
+license_expression: unknown
+is_license_reference: yes
+notes: mostly used by Oracle in the JDK and Linux
+---
+
 Use is subject to license term

src/licensedcode/data/inactive/rules/unknown_use_subject_to_license_terms2.RULE

@@ -1 +1,7 @@
+---
+license_expression: unknown
+is_license_reference: yes
+notes: mostly used by Oracle in the JDK and Linux
+---
+
 Use is subject to license terms.

src/licensedcode/data/licenses/389-exception.LICENSE

@@ -1,3 +1,49 @@
+---
+key: 389-exception
+short_name: 389 Directory Server Exception to GPL 2.0
+name: 389 Directory Server Exception to GPL 2.0
+category: Copyleft Limited
+owner: Fedora
+homepage_url: https://spdx.org/licenses/389-exception.html
+is_exception: yes
+spdx_license_key: 389-exception
+text_urls:
+    - http://www.directory.fedora.redhat.com/wiki/GPL_Exception_License_Text
+other_urls:
+    - http://directory.fedoraproject.org/wiki/GPL_Exception_License_Text
+    - http://www.gnu.org/licenses/gpl-2.0.txt
+    - https://web.archive.org/web/20080828121337/http://directory.fedoraproject.org/wiki/GPL_Exception_License_Text
+standard_notice: |
+    This Program is free software; you can redistribute it and/or modify it
+    under the terms of the GNU General Public License as published by the Free
+    Software Foundation; version 2 of the License.
+    This Program is distributed in the hope that it will be useful, but WITHOUT
+    ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+    FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+    more details.
+    You should have received a copy of the GNU General Public License along
+    with this Program; if not, write to the Free Software Foundation, Inc., 59
+    Temple Place, Suite 330, Boston, MA 02111-1307 USA.
+    In addition, as a special exception, Red Hat, Inc. gives You the additional
+    right to link the code of this Program with code not covered under the GNU
+    General Public License ("Non-GPL Code") and to distribute linked
+    combinations including the two, subject to the limitations in this
+    paragraph. Non-GPL Code permitted under this exception must only link to
+    the code of this Program through those well defined interfaces identified
+    in the file named EXCEPTION found in the source code files (the "Approved
+    Interfaces"). The files of Non-GPL Code may instantiate templates or use
+    macros or inline functions from the Approved Interfaces without causing the
+    resulting work to be covered by the GNU General Public License. Only Red
+    Hat, Inc. may make changes or additions to the list of Approved Interfaces.
+    You must obey the GNU General Public License in all respects for all of the
+    Program code and other code used in conjunction with the Program except the
+    Non-GPL Code covered by this exception. If you modify this file, you may
+    extend this exception to your version of the file, but you are not
+    obligated to do so. If you do not wish to provide this exception without
+    modification, you must delete this exception statement from your version
+    and license this file solely under the GPL without exception.
+---
+
 In addition, as a special exception, Red Hat, Inc. gives You the additional
 right to link the code of this Program with code not covered under the GNU
 General Public License ("Non-GPL Code") and to distribute linked combinations

src/licensedcode/data/licenses/3com-microcode.LICENSE

@@ -1,3 +1,13 @@
+---
+key: 3com-microcode
+short_name: 3Com Microcode
+name: 3Com Microcode
+category: Permissive
+owner: HP - Hewlett Packard
+notes: this is a Linux firmware BSD-like license with some weird extra terms
+spdx_license_key: LicenseRef-scancode-3com-microcode
+---
+
 
 Redistribution and use in source and binary forms of the 
 microcode software are permitted provided that the following conditions
@@ -26,4 +36,4 @@ MICROCODE SOFTWARE WILL NOT CREATE OR GIVE GROUNDS FOR A LICENSE BY
 IMPLICATION, ESTOPPEL, OR OTHERWISE IN ANY INTELLECTUAL PROPERTY RIGHTS
 (PATENT, COPYRIGHT, TRADE SECRET, MASK WORK, OR OTHER PROPRIETARY RIGHT)
 EMBODIED IN ANY OTHER 3COM HARDWARE OR SOFTWARE EITHER SOLELY OR IN
-COMBINATION WITH THE MICROCODE SOFTWARE
+COMBINATION WITH THE MICROCODE SOFTWARE