Poetry related question

[sinhalaquiz]

I'm trying to test a fix for #1739 but my changes don't seem to be getting picked up when I run ./scripts/exportYamlCourse.sh sinhala-from-english

My changes are below:

I suspect there are some poetry commands that I need to invoke, but can't work it out. Any suggestions?

Thank you.

diff --git a/apps/librelingo_utils/librelingo_utils/__init__.py b/apps/librelingo_utils/librelingo_utils/__init__.py
index 7d300f137..d75e5bb5a 100644
--- a/apps/librelingo_utils/librelingo_utils/__init__.py
+++ b/apps/librelingo_utils/librelingo_utils/__init__.py
@@ -6,4 +6,4 @@ to create Python software that works with LibreLingo courses.
 
 from librelingo_utils.utils import *
 
-__version__ = "2.5.0"
+__version__ = "2.6.0"
diff --git a/apps/librelingo_utils/librelingo_utils/utils.py b/apps/librelingo_utils/librelingo_utils/utils.py
index 3796d654d..60a4f5237 100644
--- a/apps/librelingo_utils/librelingo_utils/utils.py
+++ b/apps/librelingo_utils/librelingo_utils/utils.py
@@ -1,4 +1,4 @@
-import re
+import regex
 import hashlib
 from librelingo_types.data_types import Phrase, Word
 
@@ -19,7 +19,7 @@ def get_words_from_phrase(phrase):
     # Regex matches all spaces not between { and }
     return [
         w.strip(" {}")
-        for w in re.split("( |\\{.*?\\}|'.*?')", phrase)
+        for w in regex.split("( |\\{.*?\\}|'.*?')", phrase)
         if w.strip(" {}")
     ]
 
@@ -38,8 +38,8 @@ def clean_word(word):
     """
     Remove punctuation and other special characters from a word.
     """
-    MATCH_NON_WORD_CHARACTERS_BEGINNING = re.compile("^[^\\w']+")
-    MATCH_NON_WORD_CHARACTERS_END = re.compile("[^\\w']+$")
+    MATCH_NON_WORD_CHARACTERS_BEGINNING = regex.compile("^[^\\w']+")
+    MATCH_NON_WORD_CHARACTERS_END = regex.compile("[^\\w']+$")
     return MATCH_NON_WORD_CHARACTERS_BEGINNING.sub(
         "", MATCH_NON_WORD_CHARACTERS_END.sub("", word)
     )
diff --git a/apps/librelingo_utils/poetry.lock b/apps/librelingo_utils/poetry.lock
index f8e2f3f1d..b492de20d 100644
--- a/apps/librelingo_utils/poetry.lock
+++ b/apps/librelingo_utils/poetry.lock
@@ -225,6 +225,14 @@ category = "dev"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
 
+[[package]]
+name = "regex"
+version = "2021.10.23"
+description = "Alternative regular expression module, to replace re."
+category = "main"
+optional = false
+python-versions = "*"
+
 [[package]]
 name = "requests"
 version = "2.26.0"
@@ -286,7 +294,7 @@ watchmedo = ["PyYAML (>=3.10)", "argh (>=0.24.1)"]
 [metadata]
 lock-version = "1.1"
 python-versions = "^3.8"
-content-hash = "ed35d759689cb9bd4c9cf5af25c349eee81b615bdf0aeeb44219aa101a459bb7"
+content-hash = "640dbed4972738b9b05f8dba100b47e8e3b22611498c9be01e24bfd4cf9bb5ec"
 
 [metadata.files]
 certifi = [
@@ -395,6 +403,44 @@ pyyaml = [
     {file = "PyYAML-5.4.1-cp39-cp39-win_amd64.whl", hash = "sha256:c20cfa2d49991c8b4147af39859b167664f2ad4561704ee74c1de03318e898db"},
     {file = "PyYAML-5.4.1.tar.gz", hash = "sha256:607774cbba28732bfa802b54baa7484215f530991055bb562efbed5b2f20a45e"},
 ]
+regex = [
+    {file = "regex-2021.10.23-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:45b65d6a275a478ac2cbd7fdbf7cc93c1982d613de4574b56fd6972ceadb8395"},
+    {file = "regex-2021.10.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:74d071dbe4b53c602edd87a7476ab23015a991374ddb228d941929ad7c8c922e"},
+    {file = "regex-2021.10.23-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:34d870f9f27f2161709054d73646fc9aca49480617a65533fc2b4611c518e455"},
+    {file = "regex-2021.10.23-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:2fb698037c35109d3c2e30f2beb499e5ebae6e4bb8ff2e60c50b9a805a716f79"},
+    {file = "regex-2021.10.23-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:cb46b542133999580ffb691baf67410306833ee1e4f58ed06b6a7aaf4e046952"},
+    {file = "regex-2021.10.23-cp310-cp310-win32.whl", hash = "sha256:5e9c9e0ce92f27cef79e28e877c6b6988c48b16942258f3bc55d39b5f911df4f"},
+    {file = "regex-2021.10.23-cp310-cp310-win_amd64.whl", hash = "sha256:ab7c5684ff3538b67df3f93d66bd3369b749087871ae3786e70ef39e601345b0"},
+    {file = "regex-2021.10.23-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:de557502c3bec8e634246588a94e82f1ee1b9dfcfdc453267c4fb652ff531570"},
+    {file = "regex-2021.10.23-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ee684f139c91e69fe09b8e83d18b4d63bf87d9440c1eb2eeb52ee851883b1b29"},
+    {file = "regex-2021.10.23-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:5095a411c8479e715784a0c9236568ae72509450ee2226b649083730f3fadfc6"},
+    {file = "regex-2021.10.23-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:7b568809dca44cb75c8ebb260844ea98252c8c88396f9d203f5094e50a70355f"},
+    {file = "regex-2021.10.23-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:eb672217f7bd640411cfc69756ce721d00ae600814708d35c930930f18e8029f"},
+    {file = "regex-2021.10.23-cp36-cp36m-win32.whl", hash = "sha256:a7a986c45d1099a5de766a15de7bee3840b1e0e1a344430926af08e5297cf666"},
+    {file = "regex-2021.10.23-cp36-cp36m-win_amd64.whl", hash = "sha256:6d7722136c6ed75caf84e1788df36397efdc5dbadab95e59c2bba82d4d808a4c"},
+    {file = "regex-2021.10.23-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:9f665677e46c5a4d288ece12fdedf4f4204a422bb28ff05f0e6b08b7447796d1"},
+    {file = "regex-2021.10.23-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:450dc27483548214314640c89a0f275dbc557968ed088da40bde7ef8fb52829e"},
+    {file = "regex-2021.10.23-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:129472cd06062fb13e7b4670a102951a3e655e9b91634432cfbdb7810af9d710"},
+    {file = "regex-2021.10.23-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a940ca7e7189d23da2bfbb38973832813eab6bd83f3bf89a977668c2f813deae"},
+    {file = "regex-2021.10.23-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:530fc2bbb3dc1ebb17f70f7b234f90a1dd43b1b489ea38cea7be95fb21cdb5c7"},
+    {file = "regex-2021.10.23-cp37-cp37m-win32.whl", hash = "sha256:ded0c4a3eee56b57fcb2315e40812b173cafe79d2f992d50015f4387445737fa"},
+    {file = "regex-2021.10.23-cp37-cp37m-win_amd64.whl", hash = "sha256:391703a2abf8013d95bae39145d26b4e21531ab82e22f26cd3a181ee2644c234"},
+    {file = "regex-2021.10.23-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:be04739a27be55631069b348dda0c81d8ea9822b5da10b8019b789e42d1fe452"},
+    {file = "regex-2021.10.23-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:13ec99df95003f56edcd307db44f06fbeb708c4ccdcf940478067dd62353181e"},
+    {file = "regex-2021.10.23-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:8d1cdcda6bd16268316d5db1038965acf948f2a6f43acc2e0b1641ceab443623"},
+    {file = "regex-2021.10.23-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0c186691a7995ef1db61205e00545bf161fb7b59cdb8c1201c89b333141c438a"},
+    {file = "regex-2021.10.23-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:2b20f544cbbeffe171911f6ce90388ad36fe3fad26b7c7a35d4762817e9ea69c"},
+    {file = "regex-2021.10.23-cp38-cp38-win32.whl", hash = "sha256:c0938ddd60cc04e8f1faf7a14a166ac939aac703745bfcd8e8f20322a7373019"},
+    {file = "regex-2021.10.23-cp38-cp38-win_amd64.whl", hash = "sha256:56f0c81c44638dfd0e2367df1a331b4ddf2e771366c4b9c5d9a473de75e3e1c7"},
+    {file = "regex-2021.10.23-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:80bb5d2e92b2258188e7dcae5b188c7bf868eafdf800ea6edd0fbfc029984a88"},
+    {file = "regex-2021.10.23-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e1dae12321b31059a1a72aaa0e6ba30156fe7e633355e445451e4021b8e122b6"},
+    {file = "regex-2021.10.23-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_12_i686.manylinux2010_i686.whl", hash = "sha256:1f2b59c28afc53973d22e7bc18428721ee8ca6079becf1b36571c42627321c65"},
+    {file = "regex-2021.10.23-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d134757a37d8640f3c0abb41f5e68b7cf66c644f54ef1cb0573b7ea1c63e1509"},
+    {file = "regex-2021.10.23-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:0dcc0e71118be8c69252c207630faf13ca5e1b8583d57012aae191e7d6d28b84"},
+    {file = "regex-2021.10.23-cp39-cp39-win32.whl", hash = "sha256:a30513828180264294953cecd942202dfda64e85195ae36c265daf4052af0464"},
+    {file = "regex-2021.10.23-cp39-cp39-win_amd64.whl", hash = "sha256:0f7552429dd39f70057ac5d0e897e5bfe211629652399a21671e53f2a9693a4e"},
+    {file = "regex-2021.10.23.tar.gz", hash = "sha256:f3f9a91d3cc5e5b0ddf1043c0ae5fa4852f18a1c0050318baf5fc7930ecc1f9c"},
+]
 requests = [
     {file = "requests-2.26.0-py2.py3-none-any.whl", hash = "sha256:6c1246513ecd5ecd4528a0906f910e8f0f9c6b8ec72030dc9fd154dc1a6efd24"},
     {file = "requests-2.26.0.tar.gz", hash = "sha256:b8aa58f8cf793ffd8782d3d8cb19e66ef36f7aba4353eec859e74678b01b07a7"},
diff --git a/apps/librelingo_utils/pyproject.toml b/apps/librelingo_utils/pyproject.toml
index af353dec1..87d8b746e 100644
--- a/apps/librelingo_utils/pyproject.toml
+++ b/apps/librelingo_utils/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "librelingo_utils"
-version = "2.5.0"
+version = "2.6.0"
 description = "Utilities to be used in LibreLingo-related-packages"
 authors = ["Dániel Kántor <git@daniel-kantor.com>"]
 license = "AGPLv3"
@@ -9,6 +9,7 @@ readme = "README.md"
 [tool.poetry.dependencies]
 python = "^3.8"
 librelingo-types = "^3.0.0"
+regex = "^2021.10.23"
 
 [tool.poetry.dev-dependencies]
 pydoc-markdown = "^3.10.0"

[sinhalaquiz]

I'm not very familiar with poetry and there's some detail that's eluding me.

As an experiment, I placed the following on the top of apps/librelingo_utils/librelingo_utils/utils.py

raise "Poetry test"

My changes are effective only if I run the export cli tool outside the librelingo_json_export directory.

From the top level directory:

/src/LibreLingo$ poetry run python apps/librelingo_json_export/librelingo_json_export/cli.py 
Traceback (most recent call last):
  File "apps/librelingo_json_export/librelingo_json_export/cli.py", line 5, in <module>
    from librelingo_json_export.export import export_course
  File "/home/sinhalaquiz/src/LibreLingo/apps/librelingo_json_export/librelingo_json_export/__init__.py", line 5, in <module>
    import librelingo_json_export.export
  File "/home/sinhalaquiz/src/LibreLingo/apps/librelingo_json_export/librelingo_json_export/export.py", line 5, in <module>
    from .skills import _get_skill_data
  File "/home/sinhalaquiz/src/LibreLingo/apps/librelingo_json_export/librelingo_json_export/skills.py", line 1, in <module>
    from librelingo_utils import get_opaque_id, calculate_number_of_levels
  File "/home/sinhalaquiz/src/LibreLingo/apps/librelingo_utils/librelingo_utils/__init__.py", line 7, in <module>
    from librelingo_utils.utils import *
  File "/home/sinhalaquiz/src/LibreLingo/apps/librelingo_utils/librelingo_utils/utils.py", line 1, in <module>
    raise "Poetry test"
TypeError: exceptions must derive from BaseException

When invoked from the export tool directory (as done by the export script):

~/src/LibreLingo/apps/librelingo_json_export$ poetry run python librelingo_json_export/cli.py 
Usage: cli.py [OPTIONS] INPUT_PATH OUTPUT_PATH
Try 'cli.py --help' for help.

Any suggestions?

[kantord]

My changes are effective only if I run the export cli tool outside the librelingo_json_export directory.

yeah, this is because the way poetry is set up in the repo. Basically each package is a separate poetry project, and there's an overarching project that encompasses them. The overarching project is meant only for development purposes, and it links the dependencies from your local project folder instead of installing them from pip. This is why your changes are effective when you run it in the root project folder.

The reason why it doesn't work in the librelingo_json_export folder is because that project is using the latest version from pip, instead of using it from your local dev directory. This is because otherwise when we publish it, it would be broken when installed from pip.

I guess the best would be if poetry had a feature which would use the local files, but only in development. But as far as I know there's no feature that could do this 🤔

[sinhalaquiz]

Ok got it. Thanks.

So I'm basically trying to create an MR for #1739. What do you suggest I do?

1. I assume that testing it from the top level directory is sufficient?
2. The version if the utils package in pip is 2.5. Do I bump it up or keep it at 2.5?
3. librelingo_json_export is referring to version 2.4 of utils shouldn't that be updated as well?