generateWikiConfig won't work for some languages #159

claeyzre · 2018-06-08T09:12:50Z

Hi,

I am trying to parse wiki-text from many wikipedias. For some languages the configs output from generateWikiConfig work fine. But I end up with that kind of stacktraces when I try for example for Japanese. I am using "org.sweble.wikitext" % "swc-engine" % "3.1.7". Checking the issues, related bugs should have been fixed in the 3.1.7.

java.lang.IllegalArgumentException: The name `sub' was already registered by the alias `sub' when trying to register it for alias `img_sub'.
	at org.sweble.wikitext.engine.config.WikiConfigImpl.addI18nAlias(WikiConfigImpl.java:386)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.addi18NAliases(LanguageConfigGenerator.java:204)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:112)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:78)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:63)
	at PageBuilder$.apply(PageBuilder.scala:59)
	at Xml2FasttextDS$.xml2Dataset(Xml2FasttextDS.scala:27)
	at Xml2FasttextDS$.main(Xml2FasttextDS.scala:139)
	at Xml2FasttextDS.main(Xml2FasttextDS.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sbt.Run.invokeMain(Run.scala:67)
	at sbt.Run.run0(Run.scala:61)
	at sbt.Run.sbt$Run$$execute$1(Run.scala:51)
	at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Logger$$anon$4.apply(Logger.scala:84)
	at sbt.TrapExit$App.run(TrapExit.scala:248)
	at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalArgumentException: The name `名前空間:' was already registered by the alias `namespace' when trying to register it for alias `ns'.
	at org.sweble.wikitext.engine.config.WikiConfigImpl.addI18nAlias(WikiConfigImpl.java:386)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.addi18NAliases(LanguageConfigGenerator.java:204)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:112)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:78)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:63)
	at PageBuilder$.apply(PageBuilder.scala:59)
	at Xml2FasttextDS$.xml2Dataset(Xml2FasttextDS.scala:27)
	at Xml2FasttextDS$.main(Xml2FasttextDS.scala:139)
	at Xml2FasttextDS.main(Xml2FasttextDS.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sbt.Run.invokeMain(Run.scala:67)
	at sbt.Run.run0(Run.scala:61)
	at sbt.Run.sbt$Run$$execute$1(Run.scala:51)
	at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Logger$$anon$4.apply(Logger.scala:84)
	at sbt.TrapExit$App.run(TrapExit.scala:248)
	at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalArgumentException: The name `noerror' was already registered by the alias `defaultsort_noerror' when trying to register it for alias `displaytitle_noerror'.
	at org.sweble.wikitext.engine.config.WikiConfigImpl.addI18nAlias(WikiConfigImpl.java:386)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.addi18NAliases(LanguageConfigGenerator.java:204)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:112)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:78)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:63)
	at PageBuilder$.apply(PageBuilder.scala:59)
	at Xml2FasttextDS$.xml2Dataset(Xml2FasttextDS.scala:27)
	at Xml2FasttextDS$.main(Xml2FasttextDS.scala:139)
	at Xml2FasttextDS.main(Xml2FasttextDS.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sbt.Run.invokeMain(Run.scala:67)
	at sbt.Run.run0(Run.scala:61)
	at sbt.Run.sbt$Run$$execute$1(Run.scala:51)
	at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Logger$$anon$4.apply(Logger.scala:84)
	at sbt.TrapExit$App.run(TrapExit.scala:248)
	at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalArgumentException: The name `noreplace' was already registered by the alias `defaultsort_noreplace' when trying to register it for alias `displaytitle_noreplace'.
	at org.sweble.wikitext.engine.config.WikiConfigImpl.addI18nAlias(WikiConfigImpl.java:386)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.addi18NAliases(LanguageConfigGenerator.java:204)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:112)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:78)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:63)
	at PageBuilder$.apply(PageBuilder.scala:59)
	at Xml2FasttextDS$.xml2Dataset(Xml2FasttextDS.scala:27)
	at Xml2FasttextDS$.main(Xml2FasttextDS.scala:139)
	at Xml2FasttextDS.main(Xml2FasttextDS.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sbt.Run.invokeMain(Run.scala:67)
	at sbt.Run.run0(Run.scala:61)
	at sbt.Run.sbt$Run$$execute$1(Run.scala:51)
	at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Logger$$anon$4.apply(Logger.scala:84)
	at sbt.TrapExit$App.run(TrapExit.scala:248)
	at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalArgumentException: No alias registered for parser function `ns'.
	at org.sweble.wikitext.engine.config.WikiConfigImpl.addParserFunction(WikiConfigImpl.java:449)
	at org.sweble.wikitext.engine.config.WikiConfigImpl.addParserFunctionGroup(WikiConfigImpl.java:430)
	at org.sweble.wikitext.engine.utils.DefaultConfig.addParserFunctions(DefaultConfig.java:573)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:114)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:78)
	at org.sweble.wikitext.engine.utils.LanguageConfigGenerator.generateWikiConfig(LanguageConfigGenerator.java:63)
	at PageBuilder$.apply(PageBuilder.scala:59)
	at Xml2FasttextDS$.xml2Dataset(Xml2FasttextDS.scala:27)
	at Xml2FasttextDS$.main(Xml2FasttextDS.scala:139)
	at Xml2FasttextDS.main(Xml2FasttextDS.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sbt.Run.invokeMain(Run.scala:67)
	at sbt.Run.run0(Run.scala:61)
	at sbt.Run.sbt$Run$$execute$1(Run.scala:51)
	at sbt.Run$$anonfun$run$1.apply$mcV$sp(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Run$$anonfun$run$1.apply(Run.scala:55)
	at sbt.Logger$$anon$4.apply(Logger.scala:84)
	at sbt.TrapExit$App.run(TrapExit.scala:248)
	at java.lang.Thread.run(Thread.java:745)
[success] Total time: 1 s, completed Jun 8, 2018 11:00:56 AM```

Can you tell me if that's possible to generate Japanese WikiConfig with this method or should I do it manually ?

Thanks for your help,

The text was updated successfully, but these errors were encountered:

mawiesne · 2018-06-22T06:39:18Z

I can confirm the output of the stacktrace for other languages than Japanese: German/French... are also affected. It seems that Swebble in version 3.1.7 (or below) has issues at WikiConfigImpl.addI18nAlias(WikiConfigImpl.java:386). The code that prints out these errors to the standard console even has a documented TODO stating

// TODO resolve conflicts problem

see: LanguageConfigGenerator.addi18NAliases(LanguageConfigGenerator.java:204) and below.

Besides the actual problem with multiple registrations ("was already registered by the alias") , it would be better to use a logging framework here, as otherwise it will not be captured within log files, given a server environment (e.g., application containers...).

Can @ferschke or @reckart have a look at this? Or: contact Samy Ateia, samyateia [at] hotmail.de

reckart · 2018-06-22T08:21:34Z

@mawiesne I'm trying to help out with PRs and build infrastructure etc a bit and I could even try running a release, but unfortunately, I have no spare resources to actually work on the code.

tgalery · 2018-06-22T08:50:59Z

fyi I took a look at the code and this is not an error per se. The language configurator tries to associate some aliases for some specific tags and prints the stacktrace if the tag is already in use. The problem is in Sweble sweble/sweble-wikitext#72 and not something that lies in the code of dkpro. That being said, you should be able to create a japanese config even with these stacktraces being printed.

claeyzre · 2018-06-22T08:54:21Z

On some languages, the stacktraces are just printed and the config seems correct and the parsing goes on. On some other languages like Japanese, this leads to not just a print but in a real exception and no config are created.

mawiesne · 2018-06-22T09:10:35Z

@reckart Thanks for your support. Just want to raise awareness that both #159 and #160 should be tackled before a release of 1.2.0 is conducted. IMHO #160 is a blocker atm. For #159 I'd say it could be compensated (by ignoring it) even though @claeyzre might have a point that things can fail later during runtime. @claeyzre Can you pls add a full stacktrace of the exceptions you encounter "later on" for a Japanese language/environment.

claeyzre · 2018-06-22T09:13:36Z

@mawiesne The stacktrace is the one I gave you. The thing is that sometimes it's just printed and my program carries on. Sometimes it stops directly after the stacktrace being printed.

To reproduce, you can execute this (Scala) code:

import org.sweble.wikitext.engine.config.WikiConfig
import org.sweble.wikitext.engine.WtEngineImpl
import org.sweble.wikitext.engine.utils.LanguageConfigGenerator

val config = LanguageConfigGenerator.generateWikiConfig(language)
val engine = new WtEngineImpl(config)

with language being a prefix like 'ja'.

mawiesne · 2018-07-17T13:43:47Z

@tgalery Could you look into this issue, providing a workaround or fix?

tgalery · 2018-07-19T09:40:02Z

Will try to reproduce the bug first. But probably this would be a fix in sweble.

mawiesne · 2018-07-19T09:50:45Z

Sounds reasonable. If you could provide a test case that demonstrates this scenario that would be of great help.

mawiesne · 2018-07-19T09:52:16Z

@reckart Why can't I assign @tgalery for this issue? Is he missing some rights?

reckart · 2018-07-19T12:14:46Z

@tgalery I have added you to the JWPL dev team so @mawiesne can assign issues to you. This also allows you to create PRs directly in the JWPL repo. However, the settings (should) prevent pushes directly to the master branch - PRs are required and need to be approved as usual.

reckart · 2018-07-19T12:14:58Z

@tgalery oh, and you can approve PRs by others now.

tgalery · 2018-07-25T15:11:40Z

FYI, I've confirmed the bug. For some language, i.e. German, we can create the config, whereas for some others, like Japanese, we cannot. I still have the feeling that the best place to fix it would be in sweble, so I will create an issue there and reference it here. If things get nasty too quickly, I might just handle the exception and return the English config for the languages that are throwing the error.

tgalery · 2018-07-25T15:17:23Z

and here's some note to self:

scala> lastException.getClass.getCanonicalName
res5: String = java.lang.IllegalArgumentException

scala> lastException.getCause
res6: Throwable = null

mawiesne · 2018-08-02T18:37:49Z

@tgalery Could you cross-link the issue in Sweble in here?

tgalery · 2018-08-03T12:34:49Z

Sure, the issue is this sweble/sweble-wikitext#72 I've been swamped this week, but should crack on it next week.

tgalery · 2018-09-28T12:10:18Z

Can someone test PR #202 to confirm this issue is gone ?

mawiesne added 🐛Bug 🆘 help wanted labels Jun 22, 2018

mawiesne added this to the 1.2.0 milestone Jun 22, 2018

tgalery mentioned this issue Jun 22, 2018

Conflicts within language configurator sweble/sweble-wikitext#72

Open

mawiesne added the major label Jul 3, 2018

mawiesne assigned tgalery Jul 19, 2018

tgalery mentioned this issue Jul 25, 2018

Revision Machine tests failing when executed on machines with non-CEST compatible timezones #195

Closed

tgalery mentioned this issue Aug 29, 2018

Don't throw exepction when an alias is registered sweble/sweble-wikitext#73

Merged

tgalery mentioned this issue Sep 28, 2018

upgrading sweble #202

Merged

daxenberger closed this as completed in #202 Oct 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

generateWikiConfig won't work for some languages #159

generateWikiConfig won't work for some languages #159

claeyzre commented Jun 8, 2018 •

edited by reckart

Loading

mawiesne commented Jun 22, 2018

reckart commented Jun 22, 2018

tgalery commented Jun 22, 2018

claeyzre commented Jun 22, 2018

mawiesne commented Jun 22, 2018 •

edited

Loading

claeyzre commented Jun 22, 2018 •

edited

Loading

mawiesne commented Jul 17, 2018 •

edited

Loading

tgalery commented Jul 19, 2018

mawiesne commented Jul 19, 2018

mawiesne commented Jul 19, 2018

reckart commented Jul 19, 2018

reckart commented Jul 19, 2018

tgalery commented Jul 25, 2018

tgalery commented Jul 25, 2018

mawiesne commented Aug 2, 2018

tgalery commented Aug 3, 2018

tgalery commented Sep 28, 2018

generateWikiConfig won't work for some languages #159

generateWikiConfig won't work for some languages #159

Comments

claeyzre commented Jun 8, 2018 • edited by reckart Loading

mawiesne commented Jun 22, 2018

reckart commented Jun 22, 2018

tgalery commented Jun 22, 2018

claeyzre commented Jun 22, 2018

mawiesne commented Jun 22, 2018 • edited Loading

claeyzre commented Jun 22, 2018 • edited Loading

mawiesne commented Jul 17, 2018 • edited Loading

tgalery commented Jul 19, 2018

mawiesne commented Jul 19, 2018

mawiesne commented Jul 19, 2018

reckart commented Jul 19, 2018

reckart commented Jul 19, 2018

tgalery commented Jul 25, 2018

tgalery commented Jul 25, 2018

mawiesne commented Aug 2, 2018

tgalery commented Aug 3, 2018

tgalery commented Sep 28, 2018

claeyzre commented Jun 8, 2018 •

edited by reckart

Loading

mawiesne commented Jun 22, 2018 •

edited

Loading

claeyzre commented Jun 22, 2018 •

edited

Loading

mawiesne commented Jul 17, 2018 •

edited

Loading