Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cljdoc-analyzer can fail mysteriously #34

Open
lread opened this issue Dec 23, 2020 · 3 comments
Open

Cljdoc-analyzer can fail mysteriously #34

lread opened this issue Dec 23, 2020 · 3 comments

Comments

@lread
Copy link
Member

lread commented Dec 23, 2020

This issue originated from a diagnosis session on Slack with @mccraigmccraig.

The ingestion of alia on cljdoc was failing with less than helpful error messaging.

Here's what was showing on circleci:

WARNING: When invoking clojure.main, use -M
2020-12-22 18:10:17,140 INFO  cljdoc-analyzer.cljdoc-main - args:
{:project "cc.qbits/alia",
 :version "5.0.0-alpha1",
 :jarpath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.jar",
 :pompath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.pom",
 :repos
 {"clojars" {:url "https://repo.clojars.org/"},
  "central" {:url "https://repo.maven.apache.org/maven2/"}}}

2020-12-22 18:10:17,155 INFO  cljdoc-analyzer.runner - args:
{:project "cc.qbits/alia",
 :version "5.0.0-alpha1",
 :jarpath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.jar",
 :pompath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.pom",
 :extra-repos nil,
 :namespaces :all,
 :exclude-with [:no-doc :skip-wiki],
 :output-filename
 "/tmp/cljdoc/analysis-out/cljdoc-edn/cc.qbits/alia/5.0.0-alpha1/cljdoc.edn"}

2020-12-22 18:10:17,160 INFO  cljdoc-analyzer.runner - Downloading https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.jar
2020-12-22 18:10:17,722 ERROR cljdoc-analyzer.runner - nil
2020-12-22 18:10:17,722 ERROR cljdoc-analyzer.runner - STDOUT
 nil
2020-12-22 18:10:17,722 ERROR cljdoc-analyzer.runner - STDERR
 nil

We managed to reproduce this failure locally and after some trial and error we noticed that alia's lein project was generating a pom with dependencyManagement that included org.clojure/clojure. The dependencies section later included, as you might expect, org.clojure/clojure without any specific version.

Here's a minimal project.clj to illustrate. I've included an additional arbritray dependency to demonstrate a point.

(defproject lread/repro-odd-cljdoc-failure "1.0.0"
  :description "Trying to reproduce cljdoc failure"

  ;; it seems to be caused by include clojure as a managed dependency
  :managed-dependencies [[org.clojure/clojure "1.10.1"]
                         [version-clj/version-clj "0.1.2"]]

  ;; and referencing it without a version
  :dependencies [[org.clojure/clojure]
                 [version-clj/version-clj]])

We'll include a dummy namespace under src/my_stuff/api.clj just so we have something to analyze:

(ns my-stuff.api
  (:require [version-clj.core :refer [version-compare]]))

(defn oh-hi []
  ;; just referencing a lib for the sake of referncing a lib
  (version-compare "1.0.0" "1.0.0"))

Steps to reproduce:

1. install locally
lein install

2. run cljdoc-analyzer
I ran from source:

clojure -m cljdoc-analyzer.main analyze \
  --project  lread/repro-odd-cljdoc-failure --version 1.0.0 \
  --output-filename "/tmp/blarp.edn" \
  --exclude-with :no-doc \
  --exclude-with :skip-wiki

result

WARNING: When invoking clojure.main, use -M
2020-12-23 15:03:56,682 INFO  cljdoc-analyzer.runner - args:
{:project "lread/repro-odd-cljdoc-failure",
 :version "1.0.0",
 :exclude-with [:no-doc :skip-wiki],
 :output-filename "/tmp/erp.edn",
 :jarpath
 "/Users/lee/.m2/repository/lread/repro-odd-cljdoc-failure/1.0.0/repro-odd-cljdoc-failure-1.0.0.jar",
 :pompath
 "/Users/lee/.m2/repository/lread/repro-odd-cljdoc-failure/1.0.0/repro-odd-cljdoc-failure-1.0.0.pom",
 :extra-repos {}}

2020-12-23 15:03:56,800 ERROR cljdoc-analyzer.runner - nil
2020-12-23 15:03:56,800 ERROR cljdoc-analyzer.runner - STDOUT
 nil
2020-12-23 15:03:56,800 ERROR cljdoc-analyzer.runner - STDERR
 nil

work-around
If we put explicitly specified a version for org.clojure/clojure under dependencies things analyze just fine.

  :dependencies [[org.clojure/clojure "1.10.1"]
                 [version-clj/version-clj]])

And rerun lein install
Followed by cljdoc-analyzer (see step 2 above).
Then the analysis succeeds.
Notice that version-clj/version-clj could remain maven version free.

thoughts
We don't know why the ingest failed, but we do know:

  • dependencyManagent followed by versionless dependencies do work for deps other than org.clojure/clojure
  • the error message that cljdoc-analyzer/cljdoc produced was less than spectacular.

We can at least report the symptom, which I suspect might be "no dependencies found".
@lread
Copy link
Member Author

lread commented Oct 31, 2021

Same symptom has occurred for https://github.com/logicblocks/salutem for @tobyclemson.

If I change cljdoc-analyzer to log full exception, for salutem I see:

2021-10-30 21:22:43,860 ERROR cljdoc-analyzer.runner - Unable to analyze
java.lang.NullPointerException: null
	at java.base/java.util.regex.Matcher.getTextLength(Matcher.java:1770)
	at java.base/java.util.regex.Matcher.reset(Matcher.java:416)
	at java.base/java.util.regex.Matcher.<init>(Matcher.java:253)
	at java.base/java.util.regex.Pattern.matcher(Pattern.java:1133)
	at clojure.core$re_matcher.invokeStatic(core.clj:4845)
	at clojure.core$re_matcher.invoke(core.clj:4838)
	at version_clj.split$split_once.invokeStatic(split.cljc:24)
	at version_clj.split$split_once.invoke(split.cljc:21)
	at version_clj.split$split_known_qualifier.invokeStatic(split.cljc:116)
	at version_clj.split$split_known_qualifier.invoke(split.cljc:113)
	at version_clj.split$split_version_and_qualifier.invokeStatic(split.cljc:127)
	at version_clj.split$split_version_and_qualifier.invoke(split.cljc:123)
	at version_clj.split$version__GT_seq.invokeStatic(split.cljc:142)
	at version_clj.split$version__GT_seq.doInvoke(split.cljc:139)
	at clojure.lang.RestFn.invoke(RestFn.java:410)
	at version_clj.compare$version_compare.invokeStatic(compare.cljc:95)
	at version_clj.compare$version_compare.doInvoke(compare.cljc:90)
	at clojure.lang.RestFn.invoke(RestFn.java:442)
	at version_clj.core$version_compare.invokeStatic(core.cljc:28)
	at version_clj.core$version_compare.doInvoke(core.cljc:23)
	at clojure.lang.RestFn.invoke(RestFn.java:425)
	at cljdoc_analyzer.deps$ensure_recent_ish$choose_version__12671.invoke(deps.clj:18)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.core$apply.invokeStatic(core.clj:669)
	at clojure.core$update_in$up__6870.invoke(core.clj:6174)
	at clojure.core$update_in$up__6870.invoke(core.clj:6173)
	at clojure.core$update_in.invokeStatic(core.clj:6175)
	at clojure.core$update_in.doInvoke(core.clj:6161)
	at clojure.lang.RestFn.invoke(RestFn.java:467)
	at cljdoc_analyzer.deps$ensure_recent_ish$fn__12674.invoke(deps.clj:21)
	at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
	at clojure.core.protocols$fn__8162.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__8162.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__8110$G__8105__8123.invoke(protocols.clj:13)
	at clojure.core$reduce.invokeStatic(core.clj:6830)
	at clojure.core$reduce.invoke(core.clj:6812)
	at cljdoc_analyzer.deps$ensure_recent_ish.invokeStatic(deps.clj:19)
	at cljdoc_analyzer.deps$ensure_recent_ish.invoke(deps.clj:10)
	at cljdoc_analyzer.deps$deps.invokeStatic(deps.clj:94)
	at cljdoc_analyzer.deps$deps.invoke(deps.clj:82)
	at cljdoc_analyzer.deps$resolved_deps.invokeStatic(deps.clj:106)
	at cljdoc_analyzer.deps$resolved_deps.invoke(deps.clj:97)
	at cljdoc_analyzer.runner$get_metadata_STAR_.invokeStatic(runner.clj:178)
	at cljdoc_analyzer.runner$get_metadata_STAR_.invoke(runner.clj:165)
	at cljdoc_analyzer.runner$get_metadata.invokeStatic(runner.clj:211)
	at cljdoc_analyzer.runner$get_metadata.invoke(runner.clj:198)
	at cljdoc_analyzer.runner$analyze_BANG_.invokeStatic(runner.clj:233)
	at cljdoc_analyzer.runner$analyze_BANG_.invoke(runner.clj:216)
	at cljdoc_analyzer.main$analyze.invokeStatic(main.clj:21)
	at cljdoc_analyzer.main$analyze.invoke(main.clj:17)
	at cli_matic.core$invoke_subcmd.invokeStatic(core.cljc:546)
	at cli_matic.core$invoke_subcmd.invoke(core.cljc:525)
	at cli_matic.core$run_cmd_STAR_.invokeStatic(core.cljc:589)
	at cli_matic.core$run_cmd_STAR_.invoke(core.cljc:560)
	at cli_matic.core$run_cmd.invokeStatic(core.cljc:601)
	at cli_matic.core$run_cmd.invoke(core.cljc:591)
	at cljdoc_analyzer.main$_main.invokeStatic(main.clj:60)
	at cljdoc_analyzer.main$_main.doInvoke(main.clj:59)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.core$apply.invokeStatic(core.clj:667)
	at clojure.main$main_opt.invokeStatic(main.clj:514)
	at clojure.main$main_opt.invoke(main.clj:510)
	at clojure.main$main.invokeStatic(main.clj:664)
	at clojure.main$main.doInvoke(main.clj:616)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.main.main(main.java:40)

And this seems like cljdoc-analyzer having no concept of a pom's <depencyManagement> section. It just grabs the version-less dependency and does at test against that for a minimum version.

So two things:

  1. Why shouldn't cljdoc-analyzer understand <depencyManagement>? There might be a good reason, I just don't remember any.
  2. Why not log full exception on error? Were we worried about leaking maybe a secret or something?

@SevereOverfl0w
Copy link
Member

I don't remember how this bit of code works, but I do remember thinking that poking at the POM directly was a recipe for trouble.

There are some Apache Maven APIs for dealing with POMs, maybe those can give you a "resolved version" or something.

@lread
Copy link
Member Author

lread commented Nov 4, 2021

Thanks @SevereOverfl0w, yeah, I think you are right.
We are already using clojure tools deps, it probably already can give us what cljdoc-analyzer needs to know. Maybe.
And I guess we should question if cljdoc-analyzer really needs to know what it thinks it needs to know here.

lread added a commit to lread/cljdoc-analyzer that referenced this issue Feb 14, 2022
@lread
Move shared code to new cljdoc-shared project
2709c58 
Of note:
- Unit tests also moved, entirely gutting current unit test suite here.
  That's not a problem, just a note.
- Bumped most deps to current
- Jsoup no longer directly needed here, removed.
- Turfed pom.xml files, we aren't using them, so why maintain them?
  Can always add back if needed.
- Made more use of existing babashka/fs dep to replace some of our file
  utility fns
- The shared code brought in some pom behavior from cljdoc that looks at
  dependencyManagement, this addresses the specific symptoms described
  under cljdoc#34, but I'll leave this issue open to later address lack of logging
  concern.

Closes cljdoc#9
@lread lread mentioned this issue Feb 14, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lread @SevereOverfl0w