From 86e4b9d19566c0c3e203ae21b94fa4b2315eeca2 Mon Sep 17 00:00:00 2001
From: Delaram Rajaei <48606206+DelaramRajaei@users.noreply.github.com>
Date: Sat, 24 Feb 2024 16:38:47 -0500
Subject: [PATCH 01/11] Update README.md
---
README.md | 84 +++++--------------------------------------------------
1 file changed, 7 insertions(+), 77 deletions(-)
diff --git a/README.md b/README.md
index fc7d316..35f36c9 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,11 @@
# ``RePair``: A Toolkit for Query Refinement Gold Standard Generation Using Transformers
-Search engines have difficulty searching into knowledge repositories since they are not tailored to the users' information needs. User's queries are, more often than not, under-specified that also retrieve irrelevant documents. Query refinement, also known as query `reformulation`, or `suggesetion`, is the process of transforming users' queries into new `refined` versions without semantic drift to enhance the relevance of search results. Prior query refiners have been benchmarked on web retrieval datasets following `weak assumptions` that users' input queries improve gradually within a search session. To fill the gap, we contribute `RePair`, an open-source configurable toolkit to generate large-scale gold-standard benchmark datasets from a variety of domains for the task of query refinement. `RePair` takes a dataset of queries and their relevance judgements (e.g. `trec-datasets`, `msmarco` or `aol`), an information retrieval method (e.g., `bm25`), and an evaluation metric (e.g., `map`), and outputs refined versions of queries using supervised and unsupervised methods (e.g., `backtranslation`, [`T5`](https://github.com/google-research/text-to-text-transfer-transformer)), each of which with the relevance improvement guarantees.
+Web users often struggle to express their information needs clearly in short, vague queries, making it hard for search engines to find relevant results. Query refinement, which aims to improve search relevance by adjusting original queries, is crucial in addressing this challenge. However, current evaluation methods for query refinement models may not accurately reflect real-world usage patterns. We propose a novel approach using natural language backtranslation to create benchmark datasets for evaluating query refinement models. Backtranslation involves translating a query from one language to another and then translating it back, ensuring that the meaning remains consistent. We believe that backtranslation can:
+
+1. Identify missing terms in a query that are assumed to be understood due to their common usage in the original language.
+2. Include relevant synonyms from the target language to provide additional context.
+3. Clarify the meaning of ambiguous terms or phrases.
+
+We conducted extensive experiments using widely recognized TREC query sets and multiple languages. Our results, evaluated using various metrics, demonstrate the effectiveness of backtranslation in creating high-quality benchmark datasets for evaluating query refinement methods.
@@ -10,13 +16,9 @@ Search engines have difficulty searching into knowledge repositories since they
- [2. Quickstart](#2-quickstart)
* [`query_refinement`](#query_refinement)
* [`similarity`](#similarity)
- * [`pair`](#pair)
- * [`finetune`](#finetune)
- * [`predict`](#predict)
* [`search`](#search)
* [`eval`](#eval)
* [`agg, box`](#agg-box)
- * [`dense_retrieve`](#dense_retrieve)
- [3. Gold Standard Datasets](#3-gold-standard-datasets)
* [File Structure](#file-structure)
* [Settings](#settings)
@@ -154,78 +156,6 @@ These samples are taken from an ANTIQUE dataset that has been refined using a ba
| 4473331 | How can I keep my rabit indoors? | How can I keep my rabbit in the house | 0.625 | 0.571429 | 0.625 | 0.625 | 0.446324 | [0.5555555555555556, 0.5, 0.42857142857142855, 0.3333333333333333] | 1 | 1.125 | 9 | 8 | 0.7701595 |
| 1509982 | How is th Chemistry is a basic of Science? | How is chemistry a principle of science | 0.75 | 0.285714 | 0.75 | 0.75 | 0 | [0.5714285714285714, 0.16666666666666666, 0.0, 0.0] | 0.651439058 | 0.7 | 7 | 10 | 0.7796929 |
-### [`['pair']`](./src/param.py#L25)
-We create training sets based on different pairings of queries and relevant passages in the [`./data/preprocessed/{domain name}/`](./data/preprocessed/) for each domain like [`./data/preprocessed/toy.msmarco.passage/`](./data/preprocessed/toy.msmarco.passage/) for `msmarco.passage`.
-
-1. `ctx.query.docs`: context: query -> _concatenated_ relevant documents (passages)
-2. `ctx.docs.query`: context: _concatenated_ relevant documents (passages) -> query, like in [docTTTTTTQuery](https://github.com/castorini/docTTTTTquery#learning-a-new-prediction-model-t5-training-with-tensorflow)
-3. `ctx.query.doc`: context: query -> relevant document (passage)
-4. `ctx.doc.query`: context: relevant documents (passages) -> query
-
-where the context will be `userid` (personalized) or empty (context free). For instance, for `msmarco.passage` which has no contextual information, we have [`docs.query`](./data/preprocessed/toy.msmarco.passage/docs.query.passage.train.tsv) or `query.docs` since there is no context. Further, if a query has more than one relevant documents, we can either _concatenate_ all relevant documents into a single document, i.e., `doc`+`s` or _duplicate_ the (query, doc) pairs for each relevant document, i.e., `doc`.
-
-After this step, [`./data/`](./data) directory looks like:
-
-```bash
-├── data
-│ ├── preprocessed
-│ │ └── toy.msmarco.passage
-│ │ ├── docs.query.passage.test.tsv
-│ │ ├── docs.query.passage.train.tsv
-│ │ ├── queries.qrels.docs.ctx.passage.test.tsv
-│ │ └── queries.qrels.docs.ctx.passage.train.tsv
-```
-
-### [`['finetune']`](./src/param.py#L14)
-We have used [`T5`](https://github.com/google-research/text-to-text-transfer-transformer) to generate the refinements to the original queries. We can run [`T5`](https://github.com/google-research/text-to-text-transfer-transformer) on local machine (cpu/gpu), or on google cloud (tpu), which is the [`T5`](https://github.com/google-research/text-to-text-transfer-transformer) pereferance,
-> - `local machine (cpu, gpu)(linux, windows)`
-> - `google cloud (tpu)`
-
-We store the finetuned transformer in `./output/{domain name}/{transformer name}.{pairing strategy}`. For instance, for [`T5`](https://github.com/google-research/text-to-text-transfer-transformer) whose `small` version has been finetuned on a local machine for `toy.msmarco.passage`, we save the model in [`./output/toy.msmarco.passage/t5.small.local.docs.query.passage/`](./output/toy.msmarco.passage/t5.small.local.docs.query.passage/)
-
-After this step, [`./output/`](./output) looks like:
-
-```bash
-├── output
-│ ├── t5-data
-│ │ ├── pretrained_models
-│ │ │ └── small
-│ │ └── vocabs
-│ │ ├── cc_all.32000
-│ │ └── cc_en.32000
-│ └── toy.msmarco.passage
-│ └── t5.small.local.docs.query.passage
-│ ├── checkpoint
-│ ├── events.out.tfevents.1675961098.HFANI
-│ ├── graph.pbtxt
-│ ├── model.ckpt-1000000.data-00000-of-00002
-│ ├── model.ckpt-1000000.index
-│ ├── model.ckpt-1000000.meta
-│ ├── model.ckpt-1000005.data-00000-of-00002
-│ ├── model.ckpt-1000005.index
-│ ├── model.ckpt-1000005.meta
-│ ├── operative_config.gin
-```
-
-### [`['predict']`](./src/param.py#L16)
-Once a transformer has been finetuned, we feed input original queries w/ or w/o context to the model and whaterver the model generates is considered as a `potential` refined query. To have a collection of potential refined queries for the same original query, we used the [`top-k random sampling`](https://aclanthology.org/P18-1082/) as opposed to `beam search`, suggested by [`Nogueira and Lin`](https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf). So, we ran the transformer for [`nchanges`](./src/param.py#L16) times at inference and generate [`nchanges`](./src/param.py#L16) potential refined queries.
-
-We store the `i`-th potential refined query of original queries at same folder as the finetuned model, i.e., `./output/{domain name}/{transformer name}.{pairing strategy}/pred.{refinement index}-{model checkpoint}` like [`./output/toy.msmarco.passage/t5.small.local.docs.query.passage/pred.0-1000005`](./output/toy.msmarco.passage/t5.small.local.docs.query.passage/pred.0-1000005)
-
-After this step, prediction files will be added to [`./output`](./output):
-
-```bash
-├── output
-│ └── dataset_name
-│ └── t5.small.local.docs.query.passage
-│ ├── original.-1
-│ ├── pred.0-1000005
-│ ├── pred.1-1000005
-│ ├── pred.2-1000005
-│ ├── pred.3-1000005
-│ ├── pred.4-1000005
-│ │ └── refined_queries_files
-```
### [`['search']`](./src/param.py#L17)
We search the relevant documents for both the original query and each of the `potential` refined queries. We need to set an information retrieval method, called ranker, that retrieves relevant documents and ranks them based on relevance scores. We integrate [`pyserini`](https://github.com/castorini/pyserini), which provides efficient implementations of sparse and dense rankers, including `bm25` and `qld` (query likelihood with Dirichlet smoothing).
From 232e8e276e8327f61514bc8b87bb177dcc49d6a2 Mon Sep 17 00:00:00 2001
From: Delaram Rajaei <48606206+DelaramRajaei@users.noreply.github.com>
Date: Sat, 24 Feb 2024 16:39:18 -0500
Subject: [PATCH 02/11] Update README.md
---
README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 35f36c9..06e206a 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,4 @@
-# ``RePair``: A Toolkit for Query Refinement Gold Standard Generation Using Transformers
+# ``No Query Left Behind``: Query Refinement via Language Backtranslation
Web users often struggle to express their information needs clearly in short, vague queries, making it hard for search engines to find relevant results. Query refinement, which aims to improve search relevance by adjusting original queries, is crucial in addressing this challenge. However, current evaluation methods for query refinement models may not accurately reflect real-world usage patterns. We propose a novel approach using natural language backtranslation to create benchmark datasets for evaluating query refinement models. Backtranslation involves translating a query from one language to another and then translating it back, ensuring that the meaning remains consistent. We believe that backtranslation can:
1. Identify missing terms in a query that are assumed to be understood due to their common usage in the original language.
From 5ea1e414fbe3955e2f27c3ddb2f537d410b07f85 Mon Sep 17 00:00:00 2001
From: Delaram Rajaei <48606206+DelaramRajaei@users.noreply.github.com>
Date: Sat, 24 Feb 2024 16:56:05 -0500
Subject: [PATCH 03/11] Update README.md
---
README.md | 68 +------------------------------------------------------
1 file changed, 1 insertion(+), 67 deletions(-)
diff --git a/README.md b/README.md
index 06e206a..f964cbc 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,4 @@
-# ``No Query Left Behind``: Query Refinement via Language Backtranslation
+# ``No Query Left Behind``: Query Refinement via Backtranslation
Web users often struggle to express their information needs clearly in short, vague queries, making it hard for search engines to find relevant results. Query refinement, which aims to improve search relevance by adjusting original queries, is crucial in addressing this challenge. However, current evaluation methods for query refinement models may not accurately reflect real-world usage patterns. We propose a novel approach using natural language backtranslation to create benchmark datasets for evaluating query refinement models. Backtranslation involves translating a query from one language to another and then translating it back, ensuring that the meaning remains consistent. We believe that backtranslation can:
1. Identify missing terms in a query that are assumed to be understood due to their common usage in the original language.
@@ -207,50 +207,6 @@ After this step, the final structure of the output will be look like below:
```
-### [`['dense_retrieve']`](./src/param.py#L17)
-
-It is imperative for us to support dense retrieval as a ranker method for which we choose to explore `tct_colbert` the current `state-of-the-art` method.
-
-`tct colbert` is supported directly by `pyserini` , which allows us to index and search.
-
-We explain in detail how to do this for any dataset and release our dense index for `aol.title` and `aol.title.url`, for `msmarco.passage`, we used [the prebuild index](https://github.com/castorini/pyserini/blob/master/docs/experiments-tct_colbert-v2.md) provided by pyserini
-
-**Indexing**:
-Before building a dense index, we need to encode the passages either in faiss format or jsonl format. We provide a sample command that indexes `aol.title` :
-```
-python -m pyserini.encode \
- input --corpus data/raw/aol-ia/title\
- --fields text \
- --delimiter "\n" \
- --shard-id 0 \
- --shard-num 1 \
- output --embeddings data/raw/aol-ia/dense-encoder/tct_colbert.title \
- --to-faiss \
- encoder --encoder castorini/tct_colbert-v2-hnp-msmarco \
- --fields text \
- --batch 32
-```
-Now once the encoding is done, we need to index this, This indexing will take about 24 hours:
-```
- python -m pyserini.index.faiss \
- --input data/raw/aol-ia/dense-encoder/tct_colbert.title \
- --output data/raw/aol-ia/dense-index/tct_colbert.title \
- --hnsw
-```
-We chose to go with the HNSW file structure,you can explore more about it [here](https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexHNSW.html#struct-faiss-indexhnsw)
-
-[**Searching**:](./src/dal/ds.py#L53)
-We added the searching to our pipeline, since it's very similar as searching in a sparse index.
-```python
-from pyserini.search.faiss import TctColBertQueryEncoder, FaissSearcher
-encoder = TctColBertQueryEncoder('./data/raw/aol-ia/dense-encoder/tct_colbert.title')
-searcher = FaissSearcher('./data/raw/aol-ia/dense-index/tct_colbert.title',encoder)
-```
-We do not apply `tct_colbert` to our whole dataset instead we attempt to improve our predicted queries which does not meet a certain condition:
-```
-original_q_metric > refined_q_metric and 0 >= original_q_metric >= 1
-```
-
## 3. Gold Standard Datasets
@@ -260,28 +216,6 @@ original_q_metric > refined_q_metric and 0 >= original_q_metric >= 1
| `aol-ia.title` | [`./output/aol-ia/t5.base.gc.docs.query.title/bm25.map.agg.gold.tsv`](https://uwin365-my.sharepoint.com/:u:/g/personal/lakshmiy_uwindsor_ca/EVkDvYIyWyFGjEl88GAcKXABKVWSGITtOA8EEBeFAmc9Zw?e=bq3Ydd)
756 MB| [`./data/raw/aol-ia`](https://uwin365-my.sharepoint.com/:f:/g/personal/lakshmiy_uwindsor_ca/EiYYSmz-L-VHqj8x-Zl58LIBl1XKzmgI6hmZHz8rruMfeA?e=VTDsvC)
[`./data/preprocessed/aol-ia`](https://uwin365-my.sharepoint.com/:f:/g/personal/lakshmiy_uwindsor_ca/EqGqoB05KMdAn0j1frOkIV4BS2cE7bWwbSysVXtxkiSNrA?e=mf8loW) | [`./output/aol-ia/t5.base.gc.docs.query.title/`](https://uwin365-my.sharepoint.com/:f:/g/personal/lakshmiy_uwindsor_ca/EunaA74D03tMk21oGHlWFccBZBzQuhFCE7J21BRuIUu4Dw?e=XfkcoO) |
| `aol-ia.url.title` | [`./output/aol-ia/t5.base.gc.docs.query.url.title/bm25.map.agg.gold.tsv`](https://uwin365-my.sharepoint.com/:u:/g/personal/lakshmiy_uwindsor_ca/Eaf9S3WqvaBNlzu1MZhB8ZwBwFUeLkiKWumy-VNbej_Iqw?e=OwWnEy)
706 MB | [`./data/raw/aol-ia`](https://uwin365-my.sharepoint.com/:f:/g/personal/lakshmiy_uwindsor_ca/EiYYSmz-L-VHqj8x-Zl58LIBl1XKzmgI6hmZHz8rruMfeA?e=VTDsvC)
[`./data/preprocessed/aol-ia`](https://uwin365-my.sharepoint.com/:f:/g/personal/lakshmiy_uwindsor_ca/EqGqoB05KMdAn0j1frOkIV4BS2cE7bWwbSysVXtxkiSNrA?e=mf8loW) | [`./output/aol-ia/t5.base.gc.docs.query.url.title/`](https://uwin365-my.sharepoint.com/:f:/g/personal/lakshmiy_uwindsor_ca/ErGC88ga8VVMj9z49C96pgsBw3l5W6O3px680-ElvyaTWw?e=Fek7LD) |
-### File Structure
-Here is the refined queries for the two original queries in [`./output/aol-ia/t5.base.gc.docs.query.title/bm25.map.agg.gold.tsv`](https://uwin365-my.sharepoint.com/:u:/g/personal/lakshmiy_uwindsor_ca/EVkDvYIyWyFGjEl88GAcKXABKVWSGITtOA8EEBeFAmc9Zw?e=bq3Ydd) for `aol-ia.title`:
-
-```
-qid order query bm25.map
-8c418e7c9e5993 -1 rentdirect com 0.0
-8c418e7c9e5993 pred.9 hurston apartments 0.0556
-8c418e7c9e5993 pred.4 rental apartments 0.0357
-8c418e7c9e5993 pred.8 apartments nyc 0.0312
-8c418e7c9e5993 pred.6 first class apartments in nyc 0.0135
-8c418e7c9e5993 pred.2 apartments and new york 0.0068
-0cc411681d1441 -1 staple com 0.037
-0cc411681d1441 pred.8 staple pubs 1.0
-0cc411681d1441 pred.7 staple england pub 0.5
-0cc411681d1441 pred.1 staple east of england 0.1
-0cc411681d1441 pred.10 staple 0.0385
-0cc411681d1441 pred.3 staple england 0.0385
-0cc411681d1441 pred.4 staple england 0.0385
-0cc411681d1441 pred.6 staple england 0.0385
-```
-As seen, `order: -1` shows the original query with its retrieval preformance. For the rest, it shows the refined queries in decreasing retrieval performance. For instance, for the original query `query: staple com`, the retrieval performance is `bm25.map: 0.037` while the best refined query could imporove it to `bm25.map: 1.0`!
-
### Settings
`RePair` has generated gold standard query refinement datasets for `msmarco.passage` and `aol-ia` query sets using `t5.base` transformer on google cloud's tpus (`gc`) with `docs.query` pairing strategy for `bm25` ranker and `map` evaluation metric. The golden datasets along with all the artifacts including preprocessed `docs.query` pairs, model checkpoint, predicted refined queries, their search results and evaluation metric values are available at the above links. You can adjust the settings [./src/param.py](./src/param.py)
From dd3ad471c96a7ea51adf2e4af23b3b8aab53b135 Mon Sep 17 00:00:00 2001
From: Delaram Rajaei <48606206+DelaramRajaei@users.noreply.github.com>
Date: Sat, 24 Feb 2024 16:56:28 -0500
Subject: [PATCH 04/11] Add files via upload
---
similarity.pdf | Bin 0 -> 18420 bytes
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 similarity.pdf
diff --git a/similarity.pdf b/similarity.pdf
new file mode 100644
index 0000000000000000000000000000000000000000..ecbf658d907bc80d4b54ee59cc721f16fd349f00
GIT binary patch
literal 18420
zcmeHvbyOWo)-Mi$5D0F8LvTHCaCdiig1a2x;2H=7hoC`%yC%33+}$lW!GZ>d@J@2?
zoteAlu5V`ETHpKQt?9+0yQ+8Xs`~BfuD$Ejuc?&9C7D>6IZ&y_r+em6*#Rs7M-v-V
zetuM-yt#uV#0tR91^E3y1xi}mLCl>2KuJ4eh`G4AsiT=Ws-PgM3&h#n*dEnmkzos@
z&H^JUhG-yW3NN4r?}BKchGr^YipCbarO)g;rZ0w=>{B7k43iqJV1dVD@`S=+)0!
zS^uGSbx*K40I02GVqLHn
z3WWry?CfZ&W)1=90f1150YG(g4+sD#V-IbE*zaf2-_J4tgP$n=0XT@2gXcHI|25#-
zX#>$DTu(!eJ;S|TC&}e#ets_&Kq>4Xxr4VgzWvE141;I;LTpC#o5|R`Aj0dF_eai+
zH~2jEQEpdlJfm$RCt$NkOy
z+1;HhVtvKgX45|UE7Vth1|NKHH*e;)3%?E*s2h4Xp8DGwtBfZ@5X*HBq%^4lSbEhh
za#L0g(PV|EF1N2qR)hB6yI(VD?(9@Dlza9Mm*Qsfil3{(=3sxj8iy
zQ*Ld3y5!@KTES8fg#D#tu*P7bjDNVVu+C~03Bub3sM+bT$(~#0H%Li)YxD+DFQq-5
zRH2uKv+&$oagG|LQr#l8Vp!o|EjXb9^rDYbkVkUarhb-0GEAA5OG;~JClXC`$Re_y
zVmP``ta_ismOH&%sOKtP8aaq|k$+Oj_-&*jxZqJG_#hPB@=c^&02aLmQ4Y=aDw$tk
zm(sBxC!(r!T$j4jvZJA9u4dC%=vsQzl}@%8q)>S`{lRe4;Okw&MVS85Hd{zXY7V%)534IudjZQ@g^{oC
zU=E~F99=T~hkehMnlRde?r8Jh-R;+Kfivr{s-wR(0DVe#?=Tu-RM{IV6*u9p2!kGA
ze6+3}kkFV>%4(hJ@T>yxA}VGurqgRQf}?HGLM;i*pli3!~-YCzR@6I4Wxa
zS&9P|!?VaM185nt)9A3h=MwxiLTc#LL{=dfYiAIDdK8p!}|(1)|Vhto?OPH+@CXi5m4m*D2wQp1oloOVWGPj$$S3!c85mSY}C%-VFHF%s^}>0&7y%
z>q%uEvvK3$_1#?LsG!J>{#6b9O>tLUwD^T^`RK06vP#mnu55=~8MfNk?in*@61)wo
z&%?zz*!yi`*(qSCUJ%bSG$&6ci0_!IsDM{RGG!<)mR9_LllHTN^e$iR`nPNZDpJ41
zhKyI{pU%ouvZ55!j!XD&6J*%%JpF)+q+Ul-TfcKK(T~x*j>cQ1>olOEtsOO{Sp$oK
zcuSiU+9;|*oYsw1|$ss|%Nl?wn-Mk>i!#4fWm1V`0zHH=4pfajr>EV35=_VN>XzQ#z^7^ct34Gs?UN-CF!mg$1I#>^1AFZiq
zVc<47hdFwxzt?Pe{_;Fk(Yxt;+INmf7xCm<=DaHzs+IzwK)T7i%ItL0rfK(?+kxv#
z>Q?=7rCXv=o|3!G<~v|Z!Iw;dWd&3G%Ba#t#Y0kazN7=XKGf=<3Bhm|+
zY4)CtQ@lQkmca#R{V5kjPDx)oc`pw9NMdf$w^P1zgjVw>e`zd4@$Pv!3?RjB$*UAe
z-)&``qK&LZE`8s~j>hyvRR@83FP%
zdor)OgwVW;fec?hmyEyv%d{?x2Bb9ij$xwkW2=RU)HEl=v{gI#|rSqJy)
zRSUZVk2;COuMRQDJDRQenxPA|90m4D9gzoSjglX>n?9yHBE;aW-16bJzDxi>yS>ji
zm)~|$(1P8d!?|axOh!6V4D^9lI+3b%8nx{fKeNlq_{_tCwdfud?HV
zdGmNOmkoulaUD^Bowu>*%KIUvZwH3K`VQNR3$9yUqm7?l3V-VJ5#S$cgf-?cLvD?s
z3fSyF9|wtiV4T~nPRCC=kJ@Rq^}ha$JgkUJEpYu3-56u!*)B
z!cK#;)!pYG(PMAPeju9l(p>z7!v=!7JAe3#f8`wh;nT3Qvi_r^m~Ifi6y7Ne^;_<6
zT=uG@F-$FS^0F=hHb_rCUe!i9bSMsK1AXr7zhW5LWizc(-mpCh%XGjaR;%lLtaJ6t
z($tq#({7r>NQ>v5s%1AM2+JAPqrMs`EjzvcPGKup9sZ_$rQE;n;A*9>tpdbGc(ixh
zVD9F@TOmq?+VuJ6+D=-0IIU;h)3JE*y&O}4ni5Jo8pCn9Z+J&p(b)IdOy7k-JL?J)
zuq?*W)`?Xrv4)xMa}Sw~EzxD>O0*^}9Yszyt+N7=D@v25{fur3wfpL!C!M3o_gvMj
z-rUE_K}I;#(W+9YI1(Qv08faqHxdw-)_LH{jo=3`>j)4OdJq(NkgVr%tez5kD+-ql
zEY#+ztkLQQaIYeC-@G@O-vO8@(}i;An8WG@DrdftjeKf@D2f!8(L*7{3qvl3fE(;>
z15p;Cl0?Q;XUA5j+*fO;6@P`pZVD022BumgDQSvG#GPlqjPkQ-Lt}5wq?j&joO?}7
zo{{MH{+7z)oA8g1b;L6DbPn@(jg086NyWu!f`hI8J%Nj$zp(K?#svS9joCOk|7_#9
zSGGYcI6bQmIG%?DJ^uESfheV#n6s|KNuQ)jwP>bkRmg6?74l?bpwZsPd5?_@w7Mnq
z7TZZ$_?J}lTbXH4$VrHC6$j2o?Mv!b^bHmepzrxl?6jIo6}CL~>zQ0`-(9;1wzp*}
zemJ~50G1SbSIo}IZ8&SH2G0-W((Di_ZXU8mP8U?|Yh!9hF1YBjMecWKI-0xZCYv9Z
z$*M+`E$kDSd(oub{!nBZc*mnjV8er-%+M4UA_$b;uA*iD@8DJeQnakzF+P6q=F3|t
z+2GQWYakw3kM3FllaC>LX2JI(Q)NX_ufp4lz;%mwE3}2N)5DpmyQOkUnqzEa!CN7R
z3rkWV^<+`(hI%P4G;zxb6DxGWib#BBau)^(6FWUb1<^g5z_=FuIvk<)L%vAo6A-zjnHAdIckNU;@32wflN)l`E_&
zm{i(`+A?I|S|?E}(DtN#Mw)nf+^GXcakGq8G(rI1kN|d5-LZcK0+>9nPVH9$0j?_@
zd%RTy{V+!)1VHLJ)VEEK9e!=kYunR@PM#3BmF~1izp?7l_`(?9n~k-CBC`Q|E!j&`
zjyK=>@ly}-kT2clpae#<+$?uNN`A|FfT-A7esMM0T_CG;tYh)D5g2~)+Vx}wEl?GIJ2E&4mO#T=8
z|K+3hw=h^tM*L?a`ZES*5&K){$j-^l{im7}-=pbcYqj;!n1G63(wa_>*09kD?qe$-
z=x>eZ{`eobn*M@-}bl~216`BgpGoL&1v^tJ!I%*`d81vpd4IK
z;&vXx73r1W#{wRrO#)b&7d^g4Jqt0C@#W=5v
zoV~nn4znyP)KT=Y7kGgk(}^%oFpQTL+4@$XBlffG$>7bYpoJsafW=3yau!qR7G?*?DOTin>l3Y(IKPo%dF(5FCCDSGikXTIJ4OP+f05DtNVF7%1}
zR6I{I4C~UpV~X}Mc;4jDbeK>tu)JH-xRe#&j;W4xDb&7Gu#2A+I81!rWj1ff2orDG
zp^btU9&Eg>O^FxOqj#!%UY{~AAF-eN2|my%uu4TuvRLY&hkkN@!K!k?J(&2-Bh_Mv
z|C`FLh*SIx`cwDg_kG;aH$I2oKZSn+mec6aMCoxH6^~G-t=c`<>GgU?3JDmWy@52-
zxiy}Ij(}N>)|r!*a2W_@DVQW;Ci?FL23Z2OGWbn*(%y!0ClwZE-$#JoYX5m{XlB4OW51>tT^gmUUUB}L&?<)m%8SR`|S=)LEE~z2!(5bQd{hkp89>Qf=YDk$|uoZj+C%;
zzh=IyT(RYIWX=zpsY>t7QXGWjA2J7NW+q}5EHHFk9e$bexh-w0TyN1|w9xKAJI#Mm
zSLFuYcUOL7+z%UK6XkD#X_7+{Y6>n3x6XOcnVHkN9XS5#6HzcuVIon!>sG)SOIX|i
zf3dh-BiW&V{hVI<;RGBig1zQDa8_G!ZpBV!5t_<&-ek=!Bfjjl*ftzgEOkXWB%mpu
zJApId;A_YRf3RlB&59ZxGqTt{51y-~z78s$L4fB&06Gd!1>m|P&{KjthhN3kE*-5&
zbD&qQTNP*)eXhhOSssXdnsPcT*C}vlz0HFf9h{r-d0_Zuu9^E{9j!kvqgSN~@is%H
ztp*!%$?H#XZ@f=rjTq?*n^V09mx@L=ZFB_Gj*}aCB5T%tYAoi@iY6GXNCUh>${KLN
zWqI0Y5Z$cRWXWcmI&n|6gAZe!DdT$%S3PkqUK$M8@z1>hfsrNQ<&~9!-G*+~dZE;#
z?{c08ubKh=@kgc5#h5Yc<=%*65(nYK9PnQS9l`7HQ?_o<$A
zIudVR2<3FXR&vCgB&l%4ifGQ?x?ovGgU**XWZmNA9u7^8#~{=NSxfRKsVC=8xi4o2
zM)y%I1Qdt6*S023h3kbvsH^UMa0Q)yBxL#=^6Cul3Sd37Yo=
zcjm`sYJN~noLamO4bS60-D}b-Y^U;n@9x*udM+#FX{~I4;7iNQwRtXlD!QL
ztq|z9A}ZEw-nsZt*v0vd^MuHVha8PfJ8|F{s$;<01&lE3agt1y}pW2F|#MckX!#SUnL$R^RF47P4p4@ec`bv_=
zr@xEMdznkWseq1~c=5f|Nvvav=W5LjJKg6PT(*MdMs^qQ;2N7%yKLB?2iM2s6@t+0
zMvZ_t1xC)8>@KLrscF6WmNah*=ZuWlO}03h>#5qGR;
zE>H?C7*)F+1!LM;YjOfrWj3)=gf!6wd$)qcj=gR2Q<&(Z9
z;3kYep?34fzt|i(aQ8|W3CousI(1N%&p%L|+LW<3)VH_xH0Fp}eoj3RuyAgjgQ*9T
zG0SeoNmgQ7GDtKyJPmkXZk#m}s=d*;>`toCU~5^vFZjyvdIsiP8$z&1Bxr26hTU*l
zm4(8o-ND(Q;!Mb4*KN;zk3N$ZqzMLNm3AiYM(7b#AUO^&w#lVwzjLT0KFPgb7M?Yf
zH10gMphnaI;O8mnik5n4EL8?6mYnmB*PfF{w+Ltz>!WyE{HDL3)u!t5x5Z1cyF07Wp%|_K}?5P3XoWvo_01i=Rj}(Y_5Noqv-Eb&~
zok%$O%}I=ht;K|WQxaL~G2WWkjUB8`TVxL2BaL4~m7?LAt^XWi7T>@}BA>=DRVA8h
z{1R`*;WbdtYZ32i&BS3s>j{Ff3p->=esGU}@GOU4)goQIhojatrdyqJ)gA8v`*(#2
z{V@H;;Ljl9p8qLgU(js)iiNXF2sUZ#M2a8o*$F1W3PuoPOc?h&*5$ER5%WjoaAmJw
z8(za3J+se$veZQIXhLQ)qPW0GbuJO-KV)JvF#1i9HFCW;lqGks9Y3_hKhh(dSHood
z3gp;?yyEnd*km=m&7>VZoW!=2)D<9UL&)r`0O+ECnOq$Kz^!x6vVJ1A~
z5qOzVu_f|||D;T8_SJ~gH3XTR@OjKv;q~&()(^N4_Gn1|;F#Lx2VctYQM~8=1}T+7
zN&e|&F_o(Hn;Tto7q{r5Py93C5^K6l2lmH{~MBhm6@z+c+!Gzo)-cdObB`|2?I^
zl!|Hk0%>l`u7I7zJD$w?^qqynWohax$mq3pD1u_fE!@IOlaPc;J#53_H+8Ys+xS*;
zK>ApOz%g}iE3$m0vdBepwii<@DlbC5FjwJzPg%S<$Yi$7zFq7REqSk-GNC8gmP0u=
zXt+&I=)Dwv2f-o^;oA+@BRqn00GXBSUJ$={9jLmNm@NFjW0992WgZ9rqSw-INNilO
z76Z;8!W?UY6K|BGL&SxndCcyrifr~NCfloK-ew)4Tvk1I@#DlPE%3(n!F1cZw4J_4
zgvVwbm)_j{G_m&Em170bm-WG!*MkQ!C41fUu>*H4(&+@}p653u&-T+I&Yxe}=!nRl
z2y~ZaUGd1|E_PFh;-2!oY#_u-o>Vc{@c=ru+-J-Ww{}G(lRR@gyNDg&w6W?!dwz;t
z^{AC{A$0K7uB8>l4@IH|c8(Y;ZraT3qHS@15hRy&FkwN3CNRR1aV0#yU^nM}QC|w~
z&@ch@;+}!OMet-hV%0!BDd1g+Z+JiecMn{4w$uRk@D7ue
zn|@Wk>6`vu4aw<`X`V!8wt!{veHiH|^H3b7#qhqs=?(tePuwxg#nOuW&YFi#H;T>}
zi1uv{H&jB!)gEcgppIK2k#oZL9=+k%9bs|tpU@|2A^q>NS~T)EiqdDOe7DvYqopv1
z?z9tSXb44g!}6yRhq3Q5X{BZ7g7U<5E0ttq|*73mub9NI|E;o?0SRnNw9
z3tv`3;OiV7t13(dQ3n~TPc+4dZ3ZiA;!jqBx-e|B8m_3(^T81xvrx=5<10_He4e(m
zt$6z^2O<|{H6~L0_78DW1Xg9*3dT1%CTxVP{oH_Rt(wIqn
zu0Djh*@sNsEg`AVojQ_;tMJq~!SLkRb%W^|IaJt29oc^NhcGrn&-ngVa_(R!kgVJ5
z4^FSy-&IYjefnxdxlOpZ^1i$K1N$-qO&CV^oOY-H-7zdN##Ey7M;Cm@&a2Y{!c*LF
z)gLX~jxvmnX^$ZhCMxVe7zlH@&711tLCGwj89gX+fIbdrL2lG8@WM!E5}c5)d@kju
zQVrW6Kx+qQUt=XC8*0mb;lAjwksnPHA&O{3BSbY{l_G_PodM%ibf3|C>!J
z-@)>0dI{xm-4kC#^t3eyp2Az=ga%`$lbQwIPpC8w0>V)**>Cj}fC`aKZr}H#=I&d&
zyiMwV9)9ru5QO6S3J0s>}BWDI6S;-d3YP4R>-{6^0J1y
z;?t+5rgw#OZt4bjqVChLO7^pl5-XQpWYoz~S>m*H+@iF;_ZTnL+7qEqG?C`88tC=`
zhaTYxSh=J`P&XlZ_Kl40U771njucp&g_iJ&;~Z0YU-hUhhGP5bm+`kU+JX8xj6b45iy-YGPj
z)Iq&e6C0d8UZegj$7urrpJ*>XNX5bTZL;IV(X0!rLggVx5e`bJP{zCFXNvRW9tO|L
z4RO1*nt4#1*uT1SH?E<%mAyK5^7_g{{Y-rE_EPHI0FrM=QbxFQEXpjJYvb)Q
ze^ncz`Xpn;)fB;{a>sK3L#X2ygC}r91l;f<#-a{w6#?}9WBdtypRd;HXDp-f$T-@D
z-im?L$cC(SnoI7Aufimc#!DNTSP-T2evsD7VFZGt?IoQh=>_2PvnD>uzwcE;h`kue
zN{xvqgNS~h=|^27&Mcdj!eaw>pj0d}F5|se-%%{SOf=gk*hu!t>@QZVIR39!#MUq|)t{=T?vQ^3*u8^?A(BKu_zV62auxGmtcZbl*#7*>L$KBp
zb!>N{Okq$M<~vZY8yjzY$#sixc)6wy#)%|s?FOPICRS)Sf-`1x0E+q>f9`JhMj$y#
zkwRS<4lxR6aleNo{8`Ph7H(2=OQ~mxeb%cR;&pp?{
z_=vytqdi_^2aDk>%3b{h(=Ivb(8DO4(w=7)eB-WCH|bhOMkf$I}lof=+E
zyh8@G)5Kt#$Yq^;yBtZTrI#@=nsjDF!FEwTt;1RIwyss&<-M^*X*p<@*kwo@^
zk#{qL8fR7?_%$t*`vz-4zAgH7l=rzDRXs$thIOKZ6j}#cATqKPd79**jf>mvg{gu)
zbf-2V`(<`S_b7+dL8o^PsSkpgJ;vF#KAMbrA8EhmO0?bLw902s44Q1;TII`ry%Xww
zMjB#Mvy`Jja77dq)m30L#o}*A&54HgdB<2}-o%csjs&ek^pnOs5hGkgXPuEOBkWR0
z@h5Tl2rAR!3X=R~+oznpip*p~9T}%Pr;1H2+i{0y{k2h#%>K>7wA$3yG!>tZ(PY*I
zpHqk9K3L8t8DFg=H5M-uTv>fmZoxwGIyCDf;7;`H_4rUN5)>6`#;`>
zGS%xNvKG<%{#|xge#B&yC6`P3Vo#3{BuYDbB;4lq^WXJgJ^S*YR
zV%1p&bB(h`slF8dOC!z^Sk&*j*jRp
z#45$lY||vnXzmDlCuex$vlXz^UT`&?#_%YI?BiDRI`^|0=j8OCikMrY%dF`gtNCuC
zjirb^Kb8@LkKg4kkPz%;F=4j#VthS}Dx7Y(j1SzBCnCgT=pRM68Bld~>SaYu?R(2Z
z7%capqO+|U9aR6(AP@`u9;O}7BP)TF`_9H{tdX>#t}o(Kus+(SC>$(|`ISRynd)rU
za=&vL8$^wAm7^FMU$~$GhG&%O2fK&gU&zQz?(!-a3)WJ4X?>L8eR#_vEnG>~OxERO
zg{Z8C=Com=NKHGKJ#{XugZH7;eeN~Zt%8+
z%H&Z~x^q=_LDM_Qj!>E>G2y!Hf+bTn-ws>A*=+Rbx3Hh$-b<)pQzL#jJhbBT%znS$
zn<+|q4F<4j`IW>c>O1)15wULoxiV648|myz(*uY6b9JAly@dvv1M?OHq5j@qrM0vg
z9jJmKPed&xY50mDyB(K{Xua(jmJ}6LD(PCxBh%U*bL5U&QE4P~4A!*5np>>h$*#iq
z?CWo-KSEdV+C*|E4meVgT6-qPjk1wusJEs=T$`(U(!Wv0Di(vc(6vO|`|(LS7*ZRJ
z8mrmQTzGBzCe+y#VP5fmW_FQ1T_zUm#x%a=2_|$RZugFe0OtTi`idJ66Q-2;jlM+3
zaeze#xOK6Sf`oJZ=#ALnxu~3x(hgCiT3NGH#G=U_>|Tpu^?DfWhBQ2|IkQi2#J+@I
z)17~2ED+%H_~N-y7hVas(6$|S@oH*(UP0I$lc!{?tb^Iq5tKi;53D
zK3BoRe(HPo_)L$W@|;uqJm9=6L;&l3@W&S+v-5GNTYnZyJZjy#Sc>?pb=&f4-f+?0)Y@}FI&hsUT&H7hfTKSuBQk6&5)AFp%R
zmKF(w!1;ziKJNJs@@!3()F|O9WhP4pAP8c=qdD^tBs?R3yg}QQK}?eUf`*2QirRef
zU1qGdjW{#-T8&iNHZxp*gy(uA^^UJSRiQf~H-&2d6`tO=V-RlG1FJjNa7O?b_&WPt$-bbhtY41
z%|%2vlChFGh}1$Oi`Y$7B)8xy^RMoEIR!0aY+TYB``2{Ytgc#->h=y_{LqVGyePiW
zu)prE3qMmGBBlRvZ>}uYxezrLjZsFbFZaUud!U-#XSY7d7Nf3gac9aWX8FYD*r-LJ
z_XI(4PZ4G%i3&dNiGI~i^YaHh$8oa#Eb;m6=)yHj%8Ap%DHqwL>(b%VYMscro=Q&i
zLrr&`89JGW@poNY;;4ita+c(e106^zeG@;vB&+TUo|2yBp%4<`VX_2K^P4WZsVjGD
z)hcMb-FKFCPZ}>QRpc@Nw12d}HB#}=9@ZrKCj2fieR)|wemVmiRj_yI>WGKGAzl01
zMyoNOJ%zeh$-|Lj88vG9ZYqy^voo`mb-vL#Zc`MB++MbnV+tR~;_P
zYKP7~3JZQS1O+ki?HwFFv}Xq66t8p2V8IdHBH)DTZ*XL2x+meUF-a~N78tea-n_6}
zHB!JKI;!y}U45hY1ofn4=X?7O36=q9TZ6v8fq$w5!YlWVFr^R|ca3qv6idgJ8D-7*k!L%x+P`}!lBCT3IuQGU#M1yR0pg(9r*L%
zo4;&fe4YK}#J*c9@QL)GJ$-iN#FBnJPD`LFfjr_-GT<1K-EK4;a~G*k?D$~y`_-$=
z$X+#4_?S3Gx{S{Px5luSeV+*`%N`B`AHa&9?+6Q*uWi{kjD-(J8y1#hG>+KP(9j}A
zG=jamH=;v2bK*8SWzP*6%(}Z4Zr=&X{y^L4?n3_yr~c<=>wjn05G&80e$2Kt3V_E8
z5kV!2osWbuk%dM23)MfjWdDz<|3$pO&BFGlI$hQDY_U}^zi@;ZEOBVCz9it1QYlWR
zH0&S_#AB!}$E^aS^T7++O=q;5^-hQQ&i-utb!o&s(UF&
z=rjH+@%~v{r}{R!(XL_R9loz=DQsd`nhYXwTZfX0>%r8l0aV#xYNAI5+yqhJIKtAo
z^hNzF!YFcf{NikxMg9oRr4q6%CaT5TgQ|1BZ)HDr(rlL7r)GX+2RSFDG|&!ck9
zpR|7d-ja)tg&U6&_g-=MC4mWDW5BX7=Z}$k9zfmNEesxxVc55hxZwym-F2OM3xo1`
zy!dM)X%U9xO6@&2VlNIGTq^c=Jh$<@pjFWkyB9WL*-WH0AI{(QxbX}F#rOpJ3#DIh
zZybjPKc_*EA5+Z0rrs#(U9;{WXPyl2*6iMBrKKnSPD>j$!Rh%4`4hRJ;tgJ8Xt3=adyCQzxzr`6H^)o;Jx0t?o+}KF6mD8mcM84}A$qL8%7;8tt6Ds?$Z6mWxZiT+S>RND%
zup43(#9;3=>gB+RVTCMbF|eY-=T8qT+Tac&|F(8vfyrmQS#M{0-Ga;hug*ft*IZ4{O?xW%bRyQBnqK%>!3y
zr257`KZ9&PIS_w5gHRfTnX4(3aX||uVyK&IL%AmGEX*KwC?`V-A+UJwh2odpD92Z1;^*mxLNSZP>TXrOWmj%NQ#MBUjK3^q4|ax{$XT+IJq
zh`8F>nHW1b{A`Q5m9+~1`uAI#7y+8*&MwfZ0c^~y>`?Xvh?9ej3B=6#M`M3A5g>2v
zW)6V13Ir54H?cN$paa-~jUiSp=EeZG-^+6{voS-v1|@U+&S?3^<&j@}6`+!djT!{X
zU{M6Ha{c6YIJ-aqtRR-ZZQ)^MVdegJ*&nT%OAbrin7&t9!%K^_`tNXnCMpQh8{ElJ
zTq1GyP2h*D`N-U0Cc?Pos7F6bi0fpY!
zQU|zz(>4ajIq8lkFv!|@0M&DG&1TGswHRp8xu6@;E+;k|*Ejev
z=N^_cRzJ1LJH1Ule!*)W{KM%|KzHOQIp(B1q62{~-S^;P-v`-vC#2!|$>?ZId-@B1
zZZZR^A(pxa2Ci4Y&KiGSu{f2sy-Y~U>!AYXj9@24LBh!5wSH1A7aXN{9h|X_CAS9X
z3a!LI5%}pr^8E{`hVk{+2@>=Svg>|%5Et+7V?BtfL{v^^Ly`Aaptxqj7nvjv+sufwXak?3b&V$O~fMsRTE4{EfFLu^m(wKy(^{j^U
z0H8G`oN|Z~%oQtA@za+Im%p**$%lIfG=nRtm4#d%m+K#psA1C1wNa8=K3UBKY}#sfUs$u7IN^K_`_j
zj7n*1Mx17EvKYc;SrS|j?8awD6^1p4SEnNW!R*XSY+?z$r=>nSD*$mY$T|e|8Ny2O
zyxMDoF~q7;j&@p{8!yY}!OHjbC_R%MsMtou_c`xXH-hW`97-noB4ol`ZpKn2P;n3?}#w6Xo9vZ+~nnFH8Bs6Z`%9xDI@VEsvNb996N
zK))DlKgBH^0l!}0P%@m1I6#jdBnA=_7iAOUlo01(6XW3KW&??d@`y?Bh>1(EbFzvF
z0{*oKw3gq=ggl)8KA8}AuoYKW83%q1VeEJ_MMOy?+Y|*}yOYR*C0f;<$T{M8Yl{*_
za6=qn4kZ*uvH1d!AZ;Sk{Yl?WZjWy$^1?Pnjyzv|Bi~}T101+QkIzkVHs8M8@cKN8
z?27zJ^*>|i0x@=mcsN5*V`T?%@t{&sNhnF8Qi+;dT02O1SX-z&I@)P~&7jj0>Mywd
znBsnWuZtN&jO`pPp_2`i;pYPQxo!=GP{ABx3@riOF!6Jzyy`#94FB;gDPd*}fzD~}
z)=*BS6$Aoy;ROPr^Ov)^3)HC2%%+a^|G!S6zp(o=+JE5s55!Py1^K@XGd31hj=#I{
z=&1Zc9$;frTPPjVj>F>@yz<(tC-_iA7xc(yq{v+Z4H>2zCK093mWJ0Kto+<5CMXIXXW=0nEh7>486zxTkiiH1pgZ$E>y(qUj@p44};k_etsav
z+zs$sr1`f`=%Ay*{5&2XVPRl+IALIZo;-h@c=LXpB->$N{uu=-2?w)30*~J!RH$N3
V5C`
Date: Sat, 24 Feb 2024 17:05:19 -0500
Subject: [PATCH 05/11] Update README.md
---
README.md | 17 +++++------------
1 file changed, 5 insertions(+), 12 deletions(-)
diff --git a/README.md b/README.md
index f964cbc..60b2246 100644
--- a/README.md
+++ b/README.md
@@ -148,13 +148,16 @@ For additional details, please refer to this [document](./misc/Backtranslation.p
To evaluate the quality of the refined queries, metrics such as bleu, rouge, and semsim are employed. The bleu score measures the similarity between the backtranslated and original query by analyzing n-grams, while the rouge score considers the overlap of n-grams to capture essential content. Due to their simplicity and effectiveness, these metrics are widely utilized in machine translation tasks. Despite their usefulness, both scores may not accurately capture the overall meaning or fluency of the translated text due to their heavy reliance on n-grams. To address topic drift and evaluate the similarity between the original and refined queries, we additionally employ [declutr](https://aclanthology.org/2021.acl-long.72/) for query embeddings, computing cosine similarity. Declutr, a self-learning technique requiring no labeled data, minimizes the performance gap between unsupervised and supervised pretraining for universal sentence encoders during the extension of transformer-based language model training. The semsim metric, relying on cosine similarity of embeddings, proves highly effective in capturing the subtle semantic nuances of language, establishing itself as a dependable measure of the quality of backtranslated queries.
+The below images demonstrate the average token count for the original queries in English and their backtranslated versions across various languages, along with the average pairwise semantic similarities measured using 'rouge' and 'declutr'. It's evident that all languages were able to introduce new terms into the backtranslated queries while maintaining semantic coherence.
+![image](similarity.png)
+
## Example
These samples are taken from an ANTIQUE dataset that has been refined using a backtranslation refiner with the German language.
| **id** | **original** | **refined** | **rouge1** | **rouge2** | **rougeL** | **rougeLsum** | **bleu** | **precisions** | **brevity_penalty** | **length_ratio** | **translation_length** | **reference_length** | **semsim** |
|--- |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |
| 1290612 | why do anxiety and depression seem to coexist? | Why do fear and depression seem to be linked | 0.705882 | 0.533333 | 0.705882 | 0.705882353 | 0.315598 | [0.5555555555555556, 0.375, 0.2857142857142857, 0.16666666666666666] | 1 | 1 | 9 | 9 | 0.8554905 |
| 4473331 | How can I keep my rabit indoors? | How can I keep my rabbit in the house | 0.625 | 0.571429 | 0.625 | 0.625 | 0.446324 | [0.5555555555555556, 0.5, 0.42857142857142855, 0.3333333333333333] | 1 | 1.125 | 9 | 8 | 0.7701595 |
-| 1509982 | How is th Chemistry is a basic of Science? | How is chemistry a principle of science | 0.75 | 0.285714 | 0.75 | 0.75 | 0 | [0.5714285714285714, 0.16666666666666666, 0.0, 0.0] | 0.651439058 | 0.7 | 7 | 10 | 0.7796929 |
+| 1509982 | How is the Chemistry is a basic of Science? | How is chemistry a principle of science | 0.75 | 0.285714 | 0.75 | 0.75 | 0 | [0.5714285714285714, 0.16666666666666666, 0.0, 0.0] | 0.651439058 | 0.7 | 7 | 10 | 0.7796929 |
### [`['search']`](./src/param.py#L17)
@@ -217,20 +220,10 @@ After this step, the final structure of the output will be look like below:
| `aol-ia.url.title` | [`./output/aol-ia/t5.base.gc.docs.query.url.title/bm25.map.agg.gold.tsv`](https://uwin365-my.sharepoint.com/:u:/g/personal/lakshmiy_uwindsor_ca/Eaf9S3WqvaBNlzu1MZhB8ZwBwFUeLkiKWumy-VNbej_Iqw?e=OwWnEy)
706 MB | [`./data/raw/aol-ia`](https://uwin365-my.sharepoint.com/:f:/g/personal/lakshmiy_uwindsor_ca/EiYYSmz-L-VHqj8x-Zl58LIBl1XKzmgI6hmZHz8rruMfeA?e=VTDsvC)
[`./data/preprocessed/aol-ia`](https://uwin365-my.sharepoint.com/:f:/g/personal/lakshmiy_uwindsor_ca/EqGqoB05KMdAn0j1frOkIV4BS2cE7bWwbSysVXtxkiSNrA?e=mf8loW) | [`./output/aol-ia/t5.base.gc.docs.query.url.title/`](https://uwin365-my.sharepoint.com/:f:/g/personal/lakshmiy_uwindsor_ca/ErGC88ga8VVMj9z49C96pgsBw3l5W6O3px680-ElvyaTWw?e=Fek7LD) |
### Settings
-
-`RePair` has generated gold standard query refinement datasets for `msmarco.passage` and `aol-ia` query sets using `t5.base` transformer on google cloud's tpus (`gc`) with `docs.query` pairing strategy for `bm25` ranker and `map` evaluation metric. The golden datasets along with all the artifacts including preprocessed `docs.query` pairs, model checkpoint, predicted refined queries, their search results and evaluation metric values are available at the above links. You can adjust the settings [./src/param.py](./src/param.py)
-
-### Stats
-
-| query set={q} | #q | avg\|q\| | avg`bm25.map`(q) | #gold | avg\|q*\| | % | avg`bm25.map`(q*) | Δ% | #diamond (ap=1) | %q* |
-|-----------------|:---------:|:---------:|:------:|:---------:|:----------:|:---:|:------:|:--------:|:---------------:|:----------------:|
-| msmarco.passage | 502,939 | 5.9675 | 0.0862 | 414,337 | 7.4419 | 82% | 0.5704 | +562 % | 176,922 | 35% |
-| aol-ia-title | 4,459,613 | 3.5849 | 0.0252 | 2,583,023 | 3.1270 | 58% | 0.4175 | +1,556 % | 649,764 | 14% |
-| aol-ia-url-title| 4,672,506 | 3.5817 | 0.0271 | 2,421,347 | 3.5354 | 52% | 0.3997 | +1,374 % | 591,001 | 13% |
+We've created benchmark query refinement datasets for the 'trec' dataset using the 'backtranslated' refiner with both 'bm25' and 'qld' rankers, along with 'map' and 'qld' evaluation metrics.You can adjust the settings [./src/param.py](./src/param.py)
### Samples
-![image](https://user-images.githubusercontent.com/8619934/221714147-d2c212b8-8e00-47b4-8f3f-89fe805d441a.png)
## 4. Acknowledgement
We benefit from [``trec_eval``](https://github.com/usnistgov/trec_eval), [``pyserini``](https://github.com/castorini/pyserini), [``ir-dataset``](https://ir-datasets.com/), and other libraries. We would like to thank the authors of these libraries and helpful resources.
From e138e3272b087d93c141b8181c24366d4f2119e0 Mon Sep 17 00:00:00 2001
From: Delaram Rajaei <48606206+DelaramRajaei@users.noreply.github.com>
Date: Sat, 24 Feb 2024 17:05:40 -0500
Subject: [PATCH 06/11] Delete similarity.pdf
---
similarity.pdf | Bin 18420 -> 0 bytes
1 file changed, 0 insertions(+), 0 deletions(-)
delete mode 100644 similarity.pdf
diff --git a/similarity.pdf b/similarity.pdf
deleted file mode 100644
index ecbf658d907bc80d4b54ee59cc721f16fd349f00..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001
literal 18420
zcmeHvbyOWo)-Mi$5D0F8LvTHCaCdiig1a2x;2H=7hoC`%yC%33+}$lW!GZ>d@J@2?
zoteAlu5V`ETHpKQt?9+0yQ+8Xs`~BfuD$Ejuc?&9C7D>6IZ&y_r+em6*#Rs7M-v-V
zetuM-yt#uV#0tR91^E3y1xi}mLCl>2KuJ4eh`G4AsiT=Ws-PgM3&h#n*dEnmkzos@
z&H^JUhG-yW3NN4r?}BKchGr^YipCbarO)g;rZ0w=>{B7k43iqJV1dVD@`S=+)0!
zS^uGSbx*K40I02GVqLHn
z3WWry?CfZ&W)1=90f1150YG(g4+sD#V-IbE*zaf2-_J4tgP$n=0XT@2gXcHI|25#-
zX#>$DTu(!eJ;S|TC&}e#ets_&Kq>4Xxr4VgzWvE141;I;LTpC#o5|R`Aj0dF_eai+
zH~2jEQEpdlJfm$RCt$NkOy
z+1;HhVtvKgX45|UE7Vth1|NKHH*e;)3%?E*s2h4Xp8DGwtBfZ@5X*HBq%^4lSbEhh
za#L0g(PV|EF1N2qR)hB6yI(VD?(9@Dlza9Mm*Qsfil3{(=3sxj8iy
zQ*Ld3y5!@KTES8fg#D#tu*P7bjDNVVu+C~03Bub3sM+bT$(~#0H%Li)YxD+DFQq-5
zRH2uKv+&$oagG|LQr#l8Vp!o|EjXb9^rDYbkVkUarhb-0GEAA5OG;~JClXC`$Re_y
zVmP``ta_ismOH&%sOKtP8aaq|k$+Oj_-&*jxZqJG_#hPB@=c^&02aLmQ4Y=aDw$tk
zm(sBxC!(r!T$j4jvZJA9u4dC%=vsQzl}@%8q)>S`{lRe4;Okw&MVS85Hd{zXY7V%)534IudjZQ@g^{oC
zU=E~F99=T~hkehMnlRde?r8Jh-R;+Kfivr{s-wR(0DVe#?=Tu-RM{IV6*u9p2!kGA
ze6+3}kkFV>%4(hJ@T>yxA}VGurqgRQf}?HGLM;i*pli3!~-YCzR@6I4Wxa
zS&9P|!?VaM185nt)9A3h=MwxiLTc#LL{=dfYiAIDdK8p!}|(1)|Vhto?OPH+@CXi5m4m*D2wQp1oloOVWGPj$$S3!c85mSY}C%-VFHF%s^}>0&7y%
z>q%uEvvK3$_1#?LsG!J>{#6b9O>tLUwD^T^`RK06vP#mnu55=~8MfNk?in*@61)wo
z&%?zz*!yi`*(qSCUJ%bSG$&6ci0_!IsDM{RGG!<)mR9_LllHTN^e$iR`nPNZDpJ41
zhKyI{pU%ouvZ55!j!XD&6J*%%JpF)+q+Ul-TfcKK(T~x*j>cQ1>olOEtsOO{Sp$oK
zcuSiU+9;|*oYsw1|$ss|%Nl?wn-Mk>i!#4fWm1V`0zHH=4pfajr>EV35=_VN>XzQ#z^7^ct34Gs?UN-CF!mg$1I#>^1AFZiq
zVc<47hdFwxzt?Pe{_;Fk(Yxt;+INmf7xCm<=DaHzs+IzwK)T7i%ItL0rfK(?+kxv#
z>Q?=7rCXv=o|3!G<~v|Z!Iw;dWd&3G%Ba#t#Y0kazN7=XKGf=<3Bhm|+
zY4)CtQ@lQkmca#R{V5kjPDx)oc`pw9NMdf$w^P1zgjVw>e`zd4@$Pv!3?RjB$*UAe
z-)&``qK&LZE`8s~j>hyvRR@83FP%
zdor)OgwVW;fec?hmyEyv%d{?x2Bb9ij$xwkW2=RU)HEl=v{gI#|rSqJy)
zRSUZVk2;COuMRQDJDRQenxPA|90m4D9gzoSjglX>n?9yHBE;aW-16bJzDxi>yS>ji
zm)~|$(1P8d!?|axOh!6V4D^9lI+3b%8nx{fKeNlq_{_tCwdfud?HV
zdGmNOmkoulaUD^Bowu>*%KIUvZwH3K`VQNR3$9yUqm7?l3V-VJ5#S$cgf-?cLvD?s
z3fSyF9|wtiV4T~nPRCC=kJ@Rq^}ha$JgkUJEpYu3-56u!*)B
z!cK#;)!pYG(PMAPeju9l(p>z7!v=!7JAe3#f8`wh;nT3Qvi_r^m~Ifi6y7Ne^;_<6
zT=uG@F-$FS^0F=hHb_rCUe!i9bSMsK1AXr7zhW5LWizc(-mpCh%XGjaR;%lLtaJ6t
z($tq#({7r>NQ>v5s%1AM2+JAPqrMs`EjzvcPGKup9sZ_$rQE;n;A*9>tpdbGc(ixh
zVD9F@TOmq?+VuJ6+D=-0IIU;h)3JE*y&O}4ni5Jo8pCn9Z+J&p(b)IdOy7k-JL?J)
zuq?*W)`?Xrv4)xMa}Sw~EzxD>O0*^}9Yszyt+N7=D@v25{fur3wfpL!C!M3o_gvMj
z-rUE_K}I;#(W+9YI1(Qv08faqHxdw-)_LH{jo=3`>j)4OdJq(NkgVr%tez5kD+-ql
zEY#+ztkLQQaIYeC-@G@O-vO8@(}i;An8WG@DrdftjeKf@D2f!8(L*7{3qvl3fE(;>
z15p;Cl0?Q;XUA5j+*fO;6@P`pZVD022BumgDQSvG#GPlqjPkQ-Lt}5wq?j&joO?}7
zo{{MH{+7z)oA8g1b;L6DbPn@(jg086NyWu!f`hI8J%Nj$zp(K?#svS9joCOk|7_#9
zSGGYcI6bQmIG%?DJ^uESfheV#n6s|KNuQ)jwP>bkRmg6?74l?bpwZsPd5?_@w7Mnq
z7TZZ$_?J}lTbXH4$VrHC6$j2o?Mv!b^bHmepzrxl?6jIo6}CL~>zQ0`-(9;1wzp*}
zemJ~50G1SbSIo}IZ8&SH2G0-W((Di_ZXU8mP8U?|Yh!9hF1YBjMecWKI-0xZCYv9Z
z$*M+`E$kDSd(oub{!nBZc*mnjV8er-%+M4UA_$b;uA*iD@8DJeQnakzF+P6q=F3|t
z+2GQWYakw3kM3FllaC>LX2JI(Q)NX_ufp4lz;%mwE3}2N)5DpmyQOkUnqzEa!CN7R
z3rkWV^<+`(hI%P4G;zxb6DxGWib#BBau)^(6FWUb1<^g5z_=FuIvk<)L%vAo6A-zjnHAdIckNU;@32wflN)l`E_&
zm{i(`+A?I|S|?E}(DtN#Mw)nf+^GXcakGq8G(rI1kN|d5-LZcK0+>9nPVH9$0j?_@
zd%RTy{V+!)1VHLJ)VEEK9e!=kYunR@PM#3BmF~1izp?7l_`(?9n~k-CBC`Q|E!j&`
zjyK=>@ly}-kT2clpae#<+$?uNN`A|FfT-A7esMM0T_CG;tYh)D5g2~)+Vx}wEl?GIJ2E&4mO#T=8
z|K+3hw=h^tM*L?a`ZES*5&K){$j-^l{im7}-=pbcYqj;!n1G63(wa_>*09kD?qe$-
z=x>eZ{`eobn*M@-}bl~216`BgpGoL&1v^tJ!I%*`d81vpd4IK
z;&vXx73r1W#{wRrO#)b&7d^g4Jqt0C@#W=5v
zoV~nn4znyP)KT=Y7kGgk(}^%oFpQTL+4@$XBlffG$>7bYpoJsafW=3yau!qR7G?*?DOTin>l3Y(IKPo%dF(5FCCDSGikXTIJ4OP+f05DtNVF7%1}
zR6I{I4C~UpV~X}Mc;4jDbeK>tu)JH-xRe#&j;W4xDb&7Gu#2A+I81!rWj1ff2orDG
zp^btU9&Eg>O^FxOqj#!%UY{~AAF-eN2|my%uu4TuvRLY&hkkN@!K!k?J(&2-Bh_Mv
z|C`FLh*SIx`cwDg_kG;aH$I2oKZSn+mec6aMCoxH6^~G-t=c`<>GgU?3JDmWy@52-
zxiy}Ij(}N>)|r!*a2W_@DVQW;Ci?FL23Z2OGWbn*(%y!0ClwZE-$#JoYX5m{XlB4OW51>tT^gmUUUB}L&?<)m%8SR`|S=)LEE~z2!(5bQd{hkp89>Qf=YDk$|uoZj+C%;
zzh=IyT(RYIWX=zpsY>t7QXGWjA2J7NW+q}5EHHFk9e$bexh-w0TyN1|w9xKAJI#Mm
zSLFuYcUOL7+z%UK6XkD#X_7+{Y6>n3x6XOcnVHkN9XS5#6HzcuVIon!>sG)SOIX|i
zf3dh-BiW&V{hVI<;RGBig1zQDa8_G!ZpBV!5t_<&-ek=!Bfjjl*ftzgEOkXWB%mpu
zJApId;A_YRf3RlB&59ZxGqTt{51y-~z78s$L4fB&06Gd!1>m|P&{KjthhN3kE*-5&
zbD&qQTNP*)eXhhOSssXdnsPcT*C}vlz0HFf9h{r-d0_Zuu9^E{9j!kvqgSN~@is%H
ztp*!%$?H#XZ@f=rjTq?*n^V09mx@L=ZFB_Gj*}aCB5T%tYAoi@iY6GXNCUh>${KLN
zWqI0Y5Z$cRWXWcmI&n|6gAZe!DdT$%S3PkqUK$M8@z1>hfsrNQ<&~9!-G*+~dZE;#
z?{c08ubKh=@kgc5#h5Yc<=%*65(nYK9PnQS9l`7HQ?_o<$A
zIudVR2<3FXR&vCgB&l%4ifGQ?x?ovGgU**XWZmNA9u7^8#~{=NSxfRKsVC=8xi4o2
zM)y%I1Qdt6*S023h3kbvsH^UMa0Q)yBxL#=^6Cul3Sd37Yo=
zcjm`sYJN~noLamO4bS60-D}b-Y^U;n@9x*udM+#FX{~I4;7iNQwRtXlD!QL
ztq|z9A}ZEw-nsZt*v0vd^MuHVha8PfJ8|F{s$;<01&lE3agt1y}pW2F|#MckX!#SUnL$R^RF47P4p4@ec`bv_=
zr@xEMdznkWseq1~c=5f|Nvvav=W5LjJKg6PT(*MdMs^qQ;2N7%yKLB?2iM2s6@t+0
zMvZ_t1xC)8>@KLrscF6WmNah*=ZuWlO}03h>#5qGR;
zE>H?C7*)F+1!LM;YjOfrWj3)=gf!6wd$)qcj=gR2Q<&(Z9
z;3kYep?34fzt|i(aQ8|W3CousI(1N%&p%L|+LW<3)VH_xH0Fp}eoj3RuyAgjgQ*9T
zG0SeoNmgQ7GDtKyJPmkXZk#m}s=d*;>`toCU~5^vFZjyvdIsiP8$z&1Bxr26hTU*l
zm4(8o-ND(Q;!Mb4*KN;zk3N$ZqzMLNm3AiYM(7b#AUO^&w#lVwzjLT0KFPgb7M?Yf
zH10gMphnaI;O8mnik5n4EL8?6mYnmB*PfF{w+Ltz>!WyE{HDL3)u!t5x5Z1cyF07Wp%|_K}?5P3XoWvo_01i=Rj}(Y_5Noqv-Eb&~
zok%$O%}I=ht;K|WQxaL~G2WWkjUB8`TVxL2BaL4~m7?LAt^XWi7T>@}BA>=DRVA8h
z{1R`*;WbdtYZ32i&BS3s>j{Ff3p->=esGU}@GOU4)goQIhojatrdyqJ)gA8v`*(#2
z{V@H;;Ljl9p8qLgU(js)iiNXF2sUZ#M2a8o*$F1W3PuoPOc?h&*5$ER5%WjoaAmJw
z8(za3J+se$veZQIXhLQ)qPW0GbuJO-KV)JvF#1i9HFCW;lqGks9Y3_hKhh(dSHood
z3gp;?yyEnd*km=m&7>VZoW!=2)D<9UL&)r`0O+ECnOq$Kz^!x6vVJ1A~
z5qOzVu_f|||D;T8_SJ~gH3XTR@OjKv;q~&()(^N4_Gn1|;F#Lx2VctYQM~8=1}T+7
zN&e|&F_o(Hn;Tto7q{r5Py93C5^K6l2lmH{~MBhm6@z+c+!Gzo)-cdObB`|2?I^
zl!|Hk0%>l`u7I7zJD$w?^qqynWohax$mq3pD1u_fE!@IOlaPc;J#53_H+8Ys+xS*;
zK>ApOz%g}iE3$m0vdBepwii<@DlbC5FjwJzPg%S<$Yi$7zFq7REqSk-GNC8gmP0u=
zXt+&I=)Dwv2f-o^;oA+@BRqn00GXBSUJ$={9jLmNm@NFjW0992WgZ9rqSw-INNilO
z76Z;8!W?UY6K|BGL&SxndCcyrifr~NCfloK-ew)4Tvk1I@#DlPE%3(n!F1cZw4J_4
zgvVwbm)_j{G_m&Em170bm-WG!*MkQ!C41fUu>*H4(&+@}p653u&-T+I&Yxe}=!nRl
z2y~ZaUGd1|E_PFh;-2!oY#_u-o>Vc{@c=ru+-J-Ww{}G(lRR@gyNDg&w6W?!dwz;t
z^{AC{A$0K7uB8>l4@IH|c8(Y;ZraT3qHS@15hRy&FkwN3CNRR1aV0#yU^nM}QC|w~
z&@ch@;+}!OMet-hV%0!BDd1g+Z+JiecMn{4w$uRk@D7ue
zn|@Wk>6`vu4aw<`X`V!8wt!{veHiH|^H3b7#qhqs=?(tePuwxg#nOuW&YFi#H;T>}
zi1uv{H&jB!)gEcgppIK2k#oZL9=+k%9bs|tpU@|2A^q>NS~T)EiqdDOe7DvYqopv1
z?z9tSXb44g!}6yRhq3Q5X{BZ7g7U<5E0ttq|*73mub9NI|E;o?0SRnNw9
z3tv`3;OiV7t13(dQ3n~TPc+4dZ3ZiA;!jqBx-e|B8m_3(^T81xvrx=5<10_He4e(m
zt$6z^2O<|{H6~L0_78DW1Xg9*3dT1%CTxVP{oH_Rt(wIqn
zu0Djh*@sNsEg`AVojQ_;tMJq~!SLkRb%W^|IaJt29oc^NhcGrn&-ngVa_(R!kgVJ5
z4^FSy-&IYjefnxdxlOpZ^1i$K1N$-qO&CV^oOY-H-7zdN##Ey7M;Cm@&a2Y{!c*LF
z)gLX~jxvmnX^$ZhCMxVe7zlH@&711tLCGwj89gX+fIbdrL2lG8@WM!E5}c5)d@kju
zQVrW6Kx+qQUt=XC8*0mb;lAjwksnPHA&O{3BSbY{l_G_PodM%ibf3|C>!J
z-@)>0dI{xm-4kC#^t3eyp2Az=ga%`$lbQwIPpC8w0>V)**>Cj}fC`aKZr}H#=I&d&
zyiMwV9)9ru5QO6S3J0s>}BWDI6S;-d3YP4R>-{6^0J1y
z;?t+5rgw#OZt4bjqVChLO7^pl5-XQpWYoz~S>m*H+@iF;_ZTnL+7qEqG?C`88tC=`
zhaTYxSh=J`P&XlZ_Kl40U771njucp&g_iJ&;~Z0YU-hUhhGP5bm+`kU+JX8xj6b45iy-YGPj
z)Iq&e6C0d8UZegj$7urrpJ*>XNX5bTZL;IV(X0!rLggVx5e`bJP{zCFXNvRW9tO|L
z4RO1*nt4#1*uT1SH?E<%mAyK5^7_g{{Y-rE_EPHI0FrM=QbxFQEXpjJYvb)Q
ze^ncz`Xpn;)fB;{a>sK3L#X2ygC}r91l;f<#-a{w6#?}9WBdtypRd;HXDp-f$T-@D
z-im?L$cC(SnoI7Aufimc#!DNTSP-T2evsD7VFZGt?IoQh=>_2PvnD>uzwcE;h`kue
zN{xvqgNS~h=|^27&Mcdj!eaw>pj0d}F5|se-%%{SOf=gk*hu!t>@QZVIR39!#MUq|)t{=T?vQ^3*u8^?A(BKu_zV62auxGmtcZbl*#7*>L$KBp
zb!>N{Okq$M<~vZY8yjzY$#sixc)6wy#)%|s?FOPICRS)Sf-`1x0E+q>f9`JhMj$y#
zkwRS<4lxR6aleNo{8`Ph7H(2=OQ~mxeb%cR;&pp?{
z_=vytqdi_^2aDk>%3b{h(=Ivb(8DO4(w=7)eB-WCH|bhOMkf$I}lof=+E
zyh8@G)5Kt#$Yq^;yBtZTrI#@=nsjDF!FEwTt;1RIwyss&<-M^*X*p<@*kwo@^
zk#{qL8fR7?_%$t*`vz-4zAgH7l=rzDRXs$thIOKZ6j}#cATqKPd79**jf>mvg{gu)
zbf-2V`(<`S_b7+dL8o^PsSkpgJ;vF#KAMbrA8EhmO0?bLw902s44Q1;TII`ry%Xww
zMjB#Mvy`Jja77dq)m30L#o}*A&54HgdB<2}-o%csjs&ek^pnOs5hGkgXPuEOBkWR0
z@h5Tl2rAR!3X=R~+oznpip*p~9T}%Pr;1H2+i{0y{k2h#%>K>7wA$3yG!>tZ(PY*I
zpHqk9K3L8t8DFg=H5M-uTv>fmZoxwGIyCDf;7;`H_4rUN5)>6`#;`>
zGS%xNvKG<%{#|xge#B&yC6`P3Vo#3{BuYDbB;4lq^WXJgJ^S*YR
zV%1p&bB(h`slF8dOC!z^Sk&*j*jRp
z#45$lY||vnXzmDlCuex$vlXz^UT`&?#_%YI?BiDRI`^|0=j8OCikMrY%dF`gtNCuC
zjirb^Kb8@LkKg4kkPz%;F=4j#VthS}Dx7Y(j1SzBCnCgT=pRM68Bld~>SaYu?R(2Z
z7%capqO+|U9aR6(AP@`u9;O}7BP)TF`_9H{tdX>#t}o(Kus+(SC>$(|`ISRynd)rU
za=&vL8$^wAm7^FMU$~$GhG&%O2fK&gU&zQz?(!-a3)WJ4X?>L8eR#_vEnG>~OxERO
zg{Z8C=Com=NKHGKJ#{XugZH7;eeN~Zt%8+
z%H&Z~x^q=_LDM_Qj!>E>G2y!Hf+bTn-ws>A*=+Rbx3Hh$-b<)pQzL#jJhbBT%znS$
zn<+|q4F<4j`IW>c>O1)15wULoxiV648|myz(*uY6b9JAly@dvv1M?OHq5j@qrM0vg
z9jJmKPed&xY50mDyB(K{Xua(jmJ}6LD(PCxBh%U*bL5U&QE4P~4A!*5np>>h$*#iq
z?CWo-KSEdV+C*|E4meVgT6-qPjk1wusJEs=T$`(U(!Wv0Di(vc(6vO|`|(LS7*ZRJ
z8mrmQTzGBzCe+y#VP5fmW_FQ1T_zUm#x%a=2_|$RZugFe0OtTi`idJ66Q-2;jlM+3
zaeze#xOK6Sf`oJZ=#ALnxu~3x(hgCiT3NGH#G=U_>|Tpu^?DfWhBQ2|IkQi2#J+@I
z)17~2ED+%H_~N-y7hVas(6$|S@oH*(UP0I$lc!{?tb^Iq5tKi;53D
zK3BoRe(HPo_)L$W@|;uqJm9=6L;&l3@W&S+v-5GNTYnZyJZjy#Sc>?pb=&f4-f+?0)Y@}FI&hsUT&H7hfTKSuBQk6&5)AFp%R
zmKF(w!1;ziKJNJs@@!3()F|O9WhP4pAP8c=qdD^tBs?R3yg}QQK}?eUf`*2QirRef
zU1qGdjW{#-T8&iNHZxp*gy(uA^^UJSRiQf~H-&2d6`tO=V-RlG1FJjNa7O?b_&WPt$-bbhtY41
z%|%2vlChFGh}1$Oi`Y$7B)8xy^RMoEIR!0aY+TYB``2{Ytgc#->h=y_{LqVGyePiW
zu)prE3qMmGBBlRvZ>}uYxezrLjZsFbFZaUud!U-#XSY7d7Nf3gac9aWX8FYD*r-LJ
z_XI(4PZ4G%i3&dNiGI~i^YaHh$8oa#Eb;m6=)yHj%8Ap%DHqwL>(b%VYMscro=Q&i
zLrr&`89JGW@poNY;;4ita+c(e106^zeG@;vB&+TUo|2yBp%4<`VX_2K^P4WZsVjGD
z)hcMb-FKFCPZ}>QRpc@Nw12d}HB#}=9@ZrKCj2fieR)|wemVmiRj_yI>WGKGAzl01
zMyoNOJ%zeh$-|Lj88vG9ZYqy^voo`mb-vL#Zc`MB++MbnV+tR~;_P
zYKP7~3JZQS1O+ki?HwFFv}Xq66t8p2V8IdHBH)DTZ*XL2x+meUF-a~N78tea-n_6}
zHB!JKI;!y}U45hY1ofn4=X?7O36=q9TZ6v8fq$w5!YlWVFr^R|ca3qv6idgJ8D-7*k!L%x+P`}!lBCT3IuQGU#M1yR0pg(9r*L%
zo4;&fe4YK}#J*c9@QL)GJ$-iN#FBnJPD`LFfjr_-GT<1K-EK4;a~G*k?D$~y`_-$=
z$X+#4_?S3Gx{S{Px5luSeV+*`%N`B`AHa&9?+6Q*uWi{kjD-(J8y1#hG>+KP(9j}A
zG=jamH=;v2bK*8SWzP*6%(}Z4Zr=&X{y^L4?n3_yr~c<=>wjn05G&80e$2Kt3V_E8
z5kV!2osWbuk%dM23)MfjWdDz<|3$pO&BFGlI$hQDY_U}^zi@;ZEOBVCz9it1QYlWR
zH0&S_#AB!}$E^aS^T7++O=q;5^-hQQ&i-utb!o&s(UF&
z=rjH+@%~v{r}{R!(XL_R9loz=DQsd`nhYXwTZfX0>%r8l0aV#xYNAI5+yqhJIKtAo
z^hNzF!YFcf{NikxMg9oRr4q6%CaT5TgQ|1BZ)HDr(rlL7r)GX+2RSFDG|&!ck9
zpR|7d-ja)tg&U6&_g-=MC4mWDW5BX7=Z}$k9zfmNEesxxVc55hxZwym-F2OM3xo1`
zy!dM)X%U9xO6@&2VlNIGTq^c=Jh$<@pjFWkyB9WL*-WH0AI{(QxbX}F#rOpJ3#DIh
zZybjPKc_*EA5+Z0rrs#(U9;{WXPyl2*6iMBrKKnSPD>j$!Rh%4`4hRJ;tgJ8Xt3=adyCQzxzr`6H^)o;Jx0t?o+}KF6mD8mcM84}A$qL8%7;8tt6Ds?$Z6mWxZiT+S>RND%
zup43(#9;3=>gB+RVTCMbF|eY-=T8qT+Tac&|F(8vfyrmQS#M{0-Ga;hug*ft*IZ4{O?xW%bRyQBnqK%>!3y
zr257`KZ9&PIS_w5gHRfTnX4(3aX||uVyK&IL%AmGEX*KwC?`V-A+UJwh2odpD92Z1;^*mxLNSZP>TXrOWmj%NQ#MBUjK3^q4|ax{$XT+IJq
zh`8F>nHW1b{A`Q5m9+~1`uAI#7y+8*&MwfZ0c^~y>`?Xvh?9ej3B=6#M`M3A5g>2v
zW)6V13Ir54H?cN$paa-~jUiSp=EeZG-^+6{voS-v1|@U+&S?3^<&j@}6`+!djT!{X
zU{M6Ha{c6YIJ-aqtRR-ZZQ)^MVdegJ*&nT%OAbrin7&t9!%K^_`tNXnCMpQh8{ElJ
zTq1GyP2h*D`N-U0Cc?Pos7F6bi0fpY!
zQU|zz(>4ajIq8lkFv!|@0M&DG&1TGswHRp8xu6@;E+;k|*Ejev
z=N^_cRzJ1LJH1Ule!*)W{KM%|KzHOQIp(B1q62{~-S^;P-v`-vC#2!|$>?ZId-@B1
zZZZR^A(pxa2Ci4Y&KiGSu{f2sy-Y~U>!AYXj9@24LBh!5wSH1A7aXN{9h|X_CAS9X
z3a!LI5%}pr^8E{`hVk{+2@>=Svg>|%5Et+7V?BtfL{v^^Ly`Aaptxqj7nvjv+sufwXak?3b&V$O~fMsRTE4{EfFLu^m(wKy(^{j^U
z0H8G`oN|Z~%oQtA@za+Im%p**$%lIfG=nRtm4#d%m+K#psA1C1wNa8=K3UBKY}#sfUs$u7IN^K_`_j
zj7n*1Mx17EvKYc;SrS|j?8awD6^1p4SEnNW!R*XSY+?z$r=>nSD*$mY$T|e|8Ny2O
zyxMDoF~q7;j&@p{8!yY}!OHjbC_R%MsMtou_c`xXH-hW`97-noB4ol`ZpKn2P;n3?}#w6Xo9vZ+~nnFH8Bs6Z`%9xDI@VEsvNb996N
zK))DlKgBH^0l!}0P%@m1I6#jdBnA=_7iAOUlo01(6XW3KW&??d@`y?Bh>1(EbFzvF
z0{*oKw3gq=ggl)8KA8}AuoYKW83%q1VeEJ_MMOy?+Y|*}yOYR*C0f;<$T{M8Yl{*_
za6=qn4kZ*uvH1d!AZ;Sk{Yl?WZjWy$^1?Pnjyzv|Bi~}T101+QkIzkVHs8M8@cKN8
z?27zJ^*>|i0x@=mcsN5*V`T?%@t{&sNhnF8Qi+;dT02O1SX-z&I@)P~&7jj0>Mywd
znBsnWuZtN&jO`pPp_2`i;pYPQxo!=GP{ABx3@riOF!6Jzyy`#94FB;gDPd*}fzD~}
z)=*BS6$Aoy;ROPr^Ov)^3)HC2%%+a^|G!S6zp(o=+JE5s55!Py1^K@XGd31hj=#I{
z=&1Zc9$;frTPPjVj>F>@yz<(tC-_iA7xc(yq{v+Z4H>2zCK093mWJ0Kto+<5CMXIXXW=0nEh7>486zxTkiiH1pgZ$E>y(qUj@p44};k_etsav
z+zs$sr1`f`=%Ay*{5&2XVPRl+IALIZo;-h@c=LXpB->$N{uu=-2?w)30*~J!RH$N3
V5C`
Date: Sat, 24 Feb 2024 17:05:52 -0500
Subject: [PATCH 07/11] Add files via upload
---
similarity.png | Bin 0 -> 36253 bytes
1 file changed, 0 insertions(+), 0 deletions(-)
create mode 100644 similarity.png
diff --git a/similarity.png b/similarity.png
new file mode 100644
index 0000000000000000000000000000000000000000..cd92c30b29cf6af0475c9330647ba521cf19f910
GIT binary patch
literal 36253
zcmdSA^;aBS&@~!@ySoK<4}(h*Toc?Kf(-6XaCdhJ5FofagS)#9?(Y6gp7*}%yX*c3
z_lH^2-K$sku2WT~_t{k^?7M;_DiRUWhYuf6rKQA_K74=%fA|1JhyV?_^69y=5%Py<
zC#C85;R9LszaNy%q_Z&OV%c{&74i4?_mGf~nVA_mIXNgOsPpslwY4=96O*rBzrMb{
z4h#(J@9%GIZN9l(z5L&sj!k`XLGK{S9GLe31@Ba7pIp*T!j%MY
zBj1am@XHy>`b6?y4}kfPgu#ju($;0`azv@y5H2@b0aSVoak`qI@>MxSvrPuJ7~xT<
znuEJ<`}G4xjw|ag{OByM_uAU`SY9b(KP!m`oa^r_?scS)d7iX2-c5@_9xqy9CIENM
zQZ%>*0n_T4P5Na!AF)<(UPE9uSQ+;$IUQ~=cOR?2TPwxS9WP~a^4?LP9UMVj)MDt&
zyh@V!pVlnT?^(4+zEbp@)3`NXsIur8UH0)-%nQE~&3Z<~W1q^ty&K#^Y+wcTIk1Uw
zhk}jp*%ENyLpQe25m_7TYw3PE66#AK#pxP4c9Ogi``-iVFEIRuORHzBFs7`++@ld#Fr+@c;f>@vpqo@d128H?3q=P4J>2rjt2fAH-
zHi#WS$l*uD`{d>*{Qf(;mHcyZ&w1*Jp8x&+^{kOyjhM%r8!2*ixs=ZBV=M2(lda)Q>*3S;P=uZSTmAJ2m#F_9Lh~MWHRwqbBYO2;H$pdPehbW~%piMD5u8MVJHl
zo5M#|XZJ;#IzE*derCGq2X@+4?vfmWtiRRgzI?hjED&A2Q+phCT7Ry^;pAT13UEtNK(ah;d0>T#gGh|4IGf)4M7-8A2f=om?S6Zc
z&xZEFMVYysl)fwz#1)k|m!}Cm_4q4EpndnRLVYIj4d>c$gbTz%_W+RR)Z`RN*L%XL
z$Zg)t=sAYhP1uPlr5oz@4wl&{;_;$v@6Xda4%%oTC$(m=##5H`TgM*ZQ|Holi)}rB
zWR8JS?NGu+ig?T@SscZ$la_9{WsUr)FefxE;v>A(hjnGmo#J*(N&3A4RV&2fhIVmV
zDBn5Tut6J8>y~mHr&6KInT$iEBEHLfNZ1-!ir$l&8bwB7K8kYHf|63`l&)ZQn)WpC
z&sK>9T7Q>MRSH?wBf?V5GcL?P
zA++f1UK?S2!iB(M=IEl7wV624E0KoFZq+ygsu)T(9;H@a+E`9d1|n-Y#qkw}m#R3q
z0C(R(CJBnCQG1^7{a%xY9|OEFZvJQDN@Ina9K$#K=J(kUw7qkTwg^5p>>R{0EsQNI
zSg!6@a!o7fW3!?yQtU1%tW1xPD#p}WBi|6P6F+UQTSC_dF*ynyXKA};v4Jz3;^g|~
z#W9vk*i6imJ(uPwe^$Iz?2c_e6=-M*I$pRufT+Red_qCbubq)ATo2}~b$sVvOp9O9
z0D7|4?);c}#;`h~d9_P1YPf@ae>rdjmLCG_xInd)2mG3)V-&nB$j44(dU6p578mve
zpgDq2EtY|~<>IFLhO+U@1CbA2zVjZNYpIB`4eOhIZU>yxmz$@WtjKw(=NLM@dbQUN
zhRmf&-PKcd4VPa44WnVNd$FV@;`tt-pSR+s0XZ?gn`)1TTp5u$RXeRbs{#Ry*HI>9
z*uQzlYm3p5&+6^#d4e2C=;AXZpc)ms@?{Kf!<08blp9^(06njVSoP+!4+p&9Xu+_4
z3eRwI<=f%>grHwtsm!b&!hgKoa_M6?yE6+^%8A?5(tj?svK|*;{4U)Y5Va6rYPP_I
zQM{yq1NdxG0sr}f{C0xZdI!m%@+|@Pz6<}=(LRGT?HE0xJq29;@hAar
z^}dMUz)?A<>^Z5@8?~=xg=w%=a9gh-GF&!DPkCPCLWvRhc$
zy3l)?X&{jG-q;(1!9OtZUfuFj?Cc1)Q+pR2Nf=xA(EN+s$DhR7T*v_9@`l
zPlUcV_Vy;EG4IcyNN@5AhP@>dbc;w#OBUEyZ=sWcqPfTeTH(>gtnBz(S|4SK+qdXF
zdeYS+W};V(D|ie+o`@K>;|0}s(E0l(kbrFZ?>F|QtZh$NZ)VG^sgRT64W>`j3%=Rl
zf(Z@bYI1qHrtqoCisQa-9=w?bn-Ah6j#b=O?ot|MGf5?@w-jUxi@W#wlCmah0XEg@
ze$LP5aVD2_QQW<(nigAtxeKW?OVZjQU`^KXZUl1X<;cYc|3(pEJ&Q#*w}%a?Cfw&P
zagYeA&HjU)^zPuX;9ZmzZR6}+4qcPf)(%pEaV?ai+=Yp)Z5jrZ4hN2^UHTiX{@2+I
z^~l{P4CAl(rt)@4e>6+N90Y&KMl`MYFMQ;;mz%#VqW^0LY7H2vA^p)MI(dxvrPz&A
z&=0Jh>+^o`P6q0e_vVM}?)TEQ|
z6fGW%=)0mfl}<6M<`UBNFLX1NZT!`Xq#nhI`@0JzTlVwDh0ZYAYGpx#e8XCiHYewc
zng9Nw-WVHd1+))&*GA;2(Dn1lt%lg=fOF$|tVIb$?V$zI7UF+g!q2tuj#BGu;e&to&6Umt^u&ymLngwSv1fgQh9Q9IMX6NE2XJnZDI
z{1DzC67_mM0siDhG~sK&uvRY!L+60L+@!_bSKzwffBema&wq7aoGC>OX2Vojx8<)|
zw-1`^4b-5}hO2mhYz)gg*z&Rs{Yya}GMt(VqQJRWo^wD!nX`cK
zZ|0ESi}GmRR*|Y-_}2TevCo5;$D3K&?5ZoB#GM(_N4B)&@PfP+uu3KZ%>%AE12JC?
zeoZCC{PM{6aia?a3$u5uk@t)D37|Fo9f9v}^h|r(SMgVySU!4lC%800d*~t$h
zTzrHS#D^EN=dK;PRhi}#*>5-G;Zn;eZ3P497%%9~I=v@M
z(&(b^2q5y1(@U85oUS{_^HSGd0w2>}-5tlse<1#$P6YQ|M-1MCzx~}-U6H|QKeoAL
zZb`XGtd69&ozD1~pIhwq;^u$9k_m=78BwHe3t<`4+FW`T1H_
z^0n_ue5D78!&$j}W7z%Ce%5Yh)(Olf3j|RCOs#OOwD(_{dAgOd5y)PN=8$U2(ZVET
z5#qo`xwG?C!syq=w$XIXDs|KLeEkZQR&DNIDen37NA(VsauN~j=lJRz2AwqJuYN>f
z3Vz6dx&97pzBY(XDJ0pAwOSOlBi4RuVh&AF7tH>B;)0q1Jk@o1CSBASyf}0d&M`}I
zT&-!Iuo1gpZ*4a%F7G(uZt2u)e>CI!qLR9H)?JhoV#0l+c+2zi9UmBo{etFKC#H_x
z9V!hKv*;v7I?<;xZNQGOo|#h5{0qQa$Bkv%v~6Q$wX?oe4=UmPB^O3o?xcftDOJOm
zWD;(&xHR^k{Uc5(Rr$JuH(fAAk1F%VtknsF&n{pBr41TI>waM
za14+2ECeijLETOIx^(j2{MP#%@o6}
zT?e6MV`~+|E`?^qOYu_%@#T3VnaqVwb2HtqWE|GLZC0T*rIgD0Uto?=5C_TENw~(+
zA)`8IaT0Cqf@cb&+Ht1rq}qbR&1S4ORfosgs<3e4_lD`1WP~67@DS8w()AIz3xDu3
zK+si~sUb!QjBzswRQ-0i#0VAD;FZqkFc&bOp!sO
z!e=yE1%yNIf@6>AyZZ`_BJ|OX34ozvEUP~4ooQ&^stYR8@(fw?K?cCrIL#4mi|ARj
z&)7JGRgkr{)M?{?{9(-tn%S71F}d~Q8#tpuLzqKV^>xgzG;U2Ceyp4TUwujCv2%M3
zN7SBe;eM8TrN?a^1iNR7So!mm5g4Ld6_&?MCC^RW08!2Ce&A+;Wi4D;Fh63&@ejx`
zg#7gwr0x?iVcgFQO+J8H{180bO%^h?oLeKVg!MuT{L~a+t@NpNEOD*EK)JlV$EFh|
zqmW`nkH5#tU?Ize${&8kIy{Ur2A2BVRvG1WK9}n2sSA38Y)R((6JL`LhD0KdlXt=E
zCD~zdwLQ+fV50yC%<)|f+lxfs3|~Wr!8+=?Mko}e?8m?0yvsN<31ZN4@qfxoFOIJl
z3su@TXWEUm=1z`k=5pidT->=nKoEn(=Z9+pV1dNWqG(~G5j2PtxHg=+bczI+TlPG;
z6R1k!Ho3Cke}0UkJ6{irLLdV+?n$>z4pE-lE*kli8&1|(5WWZRd)L*OPu3dc$Eu(;^R@hf%}{`6);jBDzeZ_Ex4kIF7{d53GcK
zEsz415Elj+i&lwKTsDy28Lgaza@f11>#GH(+3(HETO(}255&LRnE@Pc@iG+upy~ph
zyUS1iZ|Y+$&&@8PFq9kUh8!QdE;H>jAqMD4T!HbH7~TWBDLD+*90i4pDsLRBYPiee4l;>pyW
zKIFCSn>mI|w?j7a_+zH~4?>Hf;B{~hpr3Ga(KAK(g_f)*0XSqd6kbtv
zK<%Fr5{0D@#_oSa_>zH3So4_p&83XsD@fi=+V0C@KX%(kMcDw~@Kb;AJBqe|i~rgV
z&~jyblJ6~PjkQV_Yt^c8+Nl0n_Vx5*#>(~EYceQg1KRXcllj6iH3pse`dQnl)fS4O
z)|9F4fv5AMqb_5}btqDmfuHdUOxEhWvEDDT
zup#UtSJJ;SEV97`Q3DFo5h{>}52Iin#)4LzUeMnm=cP`nyKlGvXi&~#g)0I!$TwoG
zqChfybwnRAXgQ#Cs6*&aE;)VCMMkyIFqDX4qkd$0%F{16kw1geEGQGy?=qqCWGZ7v
zKVXFr(AoA3qjrLX&7Z?gJ%b}jJ5tVVcRU{W4p^WiMAJi9rjF~L-U=y9!rmq@MMj}S
zCgV3ESs-W7$q(A%T7?lA(7qbCJzfMZXm23|@4U|m;0>6>|t7x8H*K)Tw})RHyY?n33~qg@!!LO|1{1k=2{#gAp}ymnX54SQ5g=HHT0|Ay+C
zv~z6ngC>%~iG=5+8z*(4w+;mD{3Vm;QYoj-s6s+8Mm{P7*)2Hn?f*gigjVB>vy;bS>0Je{S{>@N8mPIWq*JH%f2
zklO^!w-)zTTHmFc6H(5j)I51axnA#m?<{l`JrGJJIB
zykHost6B?f)(iHw)Ugp?55hWO*E3VA`d%r4)WzJi%5jYep0Y?uRt{({o?gb`2D?@I
zftEY_l_`Hu%?CqpT-U(!{cFc?{kmP*nhtOadpyPMg1@Y^z)f_cNr%m1=9K}U8?{s%
z39cyrbEK4mAPgD?9%T7{W_^041?*>JZ!WsYZ?1d}h^Z;m@|8Q}?@N2x@%p*!nmdEi
z!4!H^F*D%#qffXaAa$+%muHv-W|sGAj`)JD6S;>vGnTGa0e?{_S}%67a`+ua+uQ)c
z?bZBhw2AkWp@l?^;++8JVIvQTxD;-LryR?in)M)md{tP<$zKTM)u-1|E<)p9mMZlAoV+e?#DP!;9S(bmFiSjl0`|fFv{IL#z;qR
z4OPd{!y<$et%glsI$N4rkQu`?ss(I=bx~3>`Yi)M6e}94dvz}L8xH02diyFC4n8`4
z95(92IXRU$hVba1TF=nWec!swJ|$Ms-#+}$hmEf_
zjH!-cplAhgYd;~51Egb=Mm0}f21ofp#J7*})A0(yy4P+EVysv7B}N{o2xy>Z7|S}$
z1ZtmzGQS=E<-%2Z8_^1U;#1(n8;UxNf*(ebug
zi}pF3`)TObWh`Hg`Sp!B%^)QMpVAOC_r
zc)C?Al|5Ejgvn_K=pDOtV~02bIYto`jvXb&b-k!B#-)u^3#TJN6ueFH`m*)4iXV7C
z^OCC01NeS#Ya9-eZ4T|xJk;f);q5K*X2F88l-l>ek#`~iQE~JsY9*pwm3Kca;0PZN-d*HvUpBq$R5VIoM_vB7y
z_rf8wCh1evwaF*sS!b!ob`1aL)1qtGyqay&oYEaA;ad!yMS%R9qMsVP^
zU)XK8WKk{~G!2}>os4kY_F_K|rbJE9?%-8?Q8HvHp(c_)@qIjc25X=0868ibw(fp0nsLTqPmP>B{@9&kG&7g#J&afZq&bz)pN9US
z_XX=|&qfpy2GVw=v?SW1h4@oxY=Q^z(JE?LKHIZcB||``KR9B0-!sV?Tp)}^Fl{17
z#GW(L(|hH9dw2zyk{?7N=|EAdbVLD
z(qLK$DzY!Ba*u{J`V^Xp)Rf1H6BxhI(|zh{vT
z{qu3;fv5`m0{NrB2I3xCPOkgx)Y2ZALbsU9q*0g7FnLwu-hPB}Mh70(`==X))*wU2
zA1^OzWC=zZ2KY-}LzH!s>zfJKjj-q-+mWjUC0MMNN(zLE!DGMkYk{WVbMi~0XCFS1
zxN9PiZ~nNxhMD#4uR>cXgge7672S3K5{0`^NH)xU>h@li0sN{?Zw$@L*kU$J=h&&C
zF!1bLgGL0=Cj<0a(oF(Zd~Uq=?P7uj=6jk=-+RAvlB&46pT&Q9T9ElD*jiO9qe<`FU~N4=8j?g_a9?H(a&t
zzsm&KI-d1-Jy1Vb!`&Bqq(Z;IiWt1}7<*#90N~*tF5XtSGr>AjG;u6jNmelGV%u10
z^8;O&Udhxe-IZ$o&2lHb=D%@1Fu$+jPEX7>0Jwa{II&QEM3?YfjuXH
z!`<2?poCM+tAW6Zsb|%a{17?S;Hk*S7Ry93&=_k3P1&K{w5fF=9O#sM
z09E-900}!?;j8n+wX^;r9C?3KiDI1p?FF#=AvjQUVph?=!pD4D5jOczFi(be)h0mE
zYu-IOVtmAi6!F_$Ur{<|-o&e#K!;Z(xUF*5%uS3tI0c|nPp&U;8;
zvy6=-P}#@y#6!VbwRdHvdH6
z6ljue=`4x3$61YC0INp-HKGKU578B{`dFTQgE17}%QZvM`V5)I#
zcpI8*7RW*X+b;@pl_lph+wZlr0W+p_w@EiwUVt@^Blr6FkKZwrD9-xyAQYW`~?)&eH%A#^LjZl`Zw(=Z>zeg22iv7IViY5IF
z3P`eRT5~v%%VJnRW-y>mo>(|`VD;exw8e;SF@MxWPZxjdLQn{C(kC=@UP?Td5wIPevS_d7ZE1tsKZavGptv`q|UMS2TAS1^*0Hh)KCeJSukviFd_cQM_
zbq3uc#yrUOVe+NC~Hf
z5uPMz2NAU5p!=LW(5;A@rMWCveo#k-uZaP=5aUV)?e#+BSr#LM;xzio+GHP4qv+pGHw*8W)R0-hj|AK>@AO(d|c;ya=TrS
zykMsxiCMc?Cu6ZtjzuBrvk*f?A<0-tjNA-DLI(+Xn`kU+1`{Sr3v&F;6FdIWQ~rmnKjRM~J&gs+ZbpkZr+*#6?G
zW#pocf9(~=o_Xt9h&)5OPF;Yw9k60eZmw#aqu;_m2W-;uiH5|6huX6+K`3tczI9X9
zDSFIWCQr1v9rVo(WmZ9-q4I=l@PF{Tml=A>a7*>?h=$3vIm+9G?7pdIjE11WWPlHm
z67>OMcF!tJ^L6>{=rT@GkO^BHkMR)00d4%Z@VXt=rv~V}L<5sMKVPlhKr}0Ryy~F0
zmA>Q2NU%=YZxPY)FG3-%r{Zz*jKN*EVxIxj|7DA6fEi(RGtAH36)AxAu>w$Qs&^8L
zF5Fnz1^G{~b9AGZ@O$+rb}{CO<3Bgp@Z^)52@>}IW+osZ0L&+vzH`D;H5FIx;8|1`
zs%o->P!}~!Q&dX`3Vo>VX*Cj(x2CN81%8>nwnLAGC=>!f14%46js0jKlG^MYJna6C
zrtz=D6=0S0<}WLsy-O(TvJrZMsTL3_f-Dd+O5s`>C4j6T=M{#8nicEC+=KkMP=U)MhGw
zfXUS|4Zqf)w&*I8kk$^}%X_zzu3`0j(8h6ev8wkXe-GA&_KORLPcD=GPRAPJ(QdCR
zfH9VkNRG0`c6$hcF7#EwTH$pPr_L;--$eqxu~0~^B#Sp1
zvWD{;qiF&@FR95-xwIk@GbF^WA!I0hw)Z0d@B(NojVt$!xd*Fl+9wjjTKtT39k?thvC6<}SUOn3%)H)hg6d@7E
zrM3;~|5A1e0c+EkS+0_hR)|{YR?xI|6PFA+hy-i=SS~?1A&F<>*?yJHZ_eNQYT*(!
zw%fz>?;B949Fz_(tQCU6>}~iRf|kCLgWVJ>E7Quk$(GTTE5*Z1Sqa)&S%>R*_2$eJ
z$mJ*5sR^~+ySDy6pxa`ub;a?c##FVNf#KmCOJn~7d3f^lr%lJ-(H-U6k7K4rbiqt5
zPH=86HRrc}2^Z?tRd3==iRrR%@vMgw)lHh2nv~VHH0wLpUClPW2n}l5+)taJe*`Ir
z43-pg$n_SCzC?_po9Wetb`DT~|31{L1}{5`7;RpDkiRPzyYz|BX#YO(53a5b4S`zN
z&jjO{aMtS|-LGCb3F&+WQs%_npj(#}CKe}3`|#h^10m3=mv`v_OQJJ1|np>FNv6OV_DOy=Jj&MHNr
z&x)MX!Oqz7$LlZBMv>%ePep&_T%H4jqA<4bI?!quA1LeVlbVU+m&0r;o}58e2t(x$
z3mv#tEfyy%dw$iKpnZjK`(2vcFD_UUUkc$ID=oua&}C56Gaj+U@heoQ=F~E2pLhga
zW=a_(fRSJo1R%|b!}58HX8*M>ZR?t-UB$tb2Yoahm+d)N^(EVSz7ru)%@On3Qd)H1
zB>_{-nZCg-Cq($xr;A{{7j@85@plPNO&20$LXX{&Vv7ls(I>#wJ@NVr_f89O#N0z2
zzZ2R&o`ZALt06*m0MbfUDFeM~3w$Rn4T;)$4tt>RhwsqemfxxgWkCHrx@ikT^kS1Wp
zXJPCP>TbiNkr-hdR~K$lO3D^1J6;2|(>&px&A)I~r?*BS`P9~^nxKQdrSI5>tFDBj
zAy2*a`vlX<%8>{iZ~1!R8!YLH^>?MFo1h<4;(N+L#J){WA*KDrb9uGy2+>l`^@B-V
z9MZvyY3^ZK_I*j_Q@A=W4nW3caZ3}1oH@aLup4^qqJFQFA#x_p<&ksS^$5Bp(6zLP
zzBO-<$&$-A|6t1iTg$r7!rwPwMZTm6#^4`#Q}`mH0ydU#A<+V3Dc`jyaB`C6Ia&^bq6t}gPp!5S9lm7FZBsN
zerQLp9F2~PidWsq-ozkKnzMI&Om;S$DOxN-8Po?iP?VrviIo{1Hl!WCIM^6Cw>zh=
z(hWhGc;GUJL#<}kQ3|+*4`)RKNLugaGw}QW2NOb{5m>DYM{@5Rc?2#!dVoC7E&d9|
z54+*4LtI$B_ifbQqBk&b6)qgO;|I&1jI_TfLRx*YYzy6t{>^;IyFX4*lK;W1^7>_F
zG)&FkJb!etX9%+&&Z-3m_mKr|2tzfHHIAXhJPZ$#=|aKsbZlZ(Vgv5cJa+J-+W9cB
zBqixeIT8aiaa6)vW54e{1BwnD8ZE<5v&$^GRJL7lb?cTR%NmS;QIRGt%B2iYGQ64K
zVKoz;IiFKAM{&5;wJI^l%UE=N$=J$lo^hzZr5L-vJIv|Se77%=gDP|{1*$cx(R3`S
zZ>c)9kC~CJac?D2yb1F3JYr=rw1Dp#k7g*Q@qNVcOXTveVTpSS-VEs$v`E+GBEe+h
z_%uKLBl5lt~^Mul}=_*uuCr|4>!)=?Zk?k*A2
z^$`-oGUN69nJv5^$R#bferm#e$7`(mEtyoYh~a_ilAUB}*g%0U%cWtu7R#sIxofxP
zF?2@YeQ8l(TAFd|A7V@%CP?B@petN_g|5~9V~GUexq@qNqoa0;O2wZS<;jtrO~YC9
zjU%8g0<12!@b*=TMsUfAZ`t+@5zb&lT}lb=$Z^nfu8*FOqjFkgP`
zzN-(rr=mkz;Pp}<8qNPw1nGV_lk+Or#*p-(DwHwuOq^Quk9Tp!I7*#9wZQpT-Ypn+WcBB#qm
zlFLW%u;(cy1iU?{!j`vu{_5IhZG`)D{B=^uiDT^UvYC=_dda6Dx_s#8mDmJyaG+9U
z4$Ja@Ym%ZMcPCbulyLJ!`ydilHO&+6iaw4?-YHOovu9hVUyczKzM|UHEXUZH_aMa3
zSbhj>A*v5XlVS^peU;0q$)NX^Kc*%%LZSuERBk;YVG;rcY#x2y?Bb$~zrFsp5d~>U
zdX%vMZ2B5>y7}DV&V+&a9x^^?n`Be~g^NaK-~1D^x#wft>^1}I#8jd8rnS!i)ik;+
zTgleIeATN}&vpfJ+Shi_e$P4dE{tss^QEPTvTpXin9X-Q_Y<>gsJ=#HH@_zn@?Lyi
zanip6WC=Y?$q|O&&_a_2my@#ZytA-T6xF{wi4-m-@W=`z8@~+RUf9MX3{oe65(+T(zOZr~C8*4)YN$HUW8blM=w
zB!cU^){8<+DeI_ea|zqgYo*6y{Ct3m=y{T;`OxgULU34-{3Ckil#UeCxRV?$TaOZc8f9ZxuWI=8McgGwTwJw52H#%$`
z4!7Id4d29tLYC>_aIW*I5JY#_}jf|NIM!g}LTLI?rn%)q7YuM2WFdT7)KyGmlmN{?RPS@_j;2UQR@z<8k
z>p$mDUCdhFx!j1OEw&>zc#OXtdfFZ70Uu|jxYr