Better File Chunk | 更加强大的文件分块 #3550
Replies: 12 comments 16 replies
-
|
excel可以说是知识库的刚需了，转成html然后借用html分块。 |
Beta Was this translation helpful? Give feedback.
-
|
然而市面上并没有免费的 excel 分块方案。excel 转成 html 我也没找到现成方案。 要么就是用 Unstructed.io ，自己填这个 key。 如果要自己做的话比较麻烦，后续考虑成付费项吧。 |
Beta Was this translation helpful? Give feedback.
-
|
可能引入下SpreadJS？不过这个包似乎太重了。 |
Beta Was this translation helpful? Give feedback.
-
|
SpreadJS 是纯前端组件，而 excel 分块是在服务端做的。另外我看了下 SpreadJS 收费可不便宜： https://developer.mescius.com/spreadjs/pricing ，你确定个人能付得起 $999 就为了这个功能么… 现阶段可行的方案应该就只有 Unstructed.io 了 |
Beta Was this translation helpful? Give feedback.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
This comment has been hidden.
-
|
计划支持一波 Lrc/Lrcx，这样就可以做歌词文件的分析了 |
Beta Was this translation helpful? Give feedback.
-
|
可以支持一下typst么？比较新的标记语言，和latex算是竞品。 |
Beta Was this translation helpful? Give feedback.
-
|
能否给出一个示例？以及给出一下你期望的分块的模式？ |
Beta Was this translation helpful? Give feedback.
-
|
示例： == 函数极限与连续
=== 极限定义
$forall xi > 0$, $exists delta > 0$ ,当 $|x - x_(0)| < delta $时, $|f(x) - A| <= 0$ ,则 $limits(lim)_(x -> x_(0)) f(x) = A$
当$x -> 0$ 时, $ln(x + sqrt(1 + x^(2))) ~ x$, $ln(x + sqrt(1 + x^(2)))' = (1)/(sqrt(1 + x^(2)))$
=== 常见的大小关系比较
#block(fill:rgb("#f0f0f0"), inset: 10pt, radius:5pt, width: 100%)[
当 $x -> oo$ 时:
$ln^alpha x << x^beta << a^x$
$ln^alpha n << n^beta << a^n << n! << n^n$
]
#mblock[
$2/pi < sin x < x < tan x < 4/pi$
]
=== 泰勒公式
#block(fill:rgb("#f0f0f0"), inset: 10pt, radius:5pt, width: 100%)[
$sin(x) = x - (x^3)/3! + o(x^3)$
$cos x = 1 - x^2/2! + x^4/4! + o(x^(4))$
$arcsin x = x + x^3/3! + o(x^3)$
$tan x = x + x^3/3 + o(x^3)$
$arctan x = x - x^3/3 + o(x^(3))$
$ln(1 + x) = x - x^2/2 + x^3/3 + o(x^3)$
$e^x = 1 + x + x^2/2! + x^3/3! + o(x^3)$
$(1 + x)^alpha = 1 + alpha x + (alpha (alpha - 1) x^2)/2! + o(x^2)$
$ln(x + sqrt(1 + x^(2))) = x + (1)/(6) x^(3) + o(x^(3))$
]至于分块模式我不是很了解，不知道怎样分比较好。 |
Beta Was this translation helpful? Give feedback.
-
|
java和python有现有的分块方案吗,请求加上 |
Beta Was this translation helpful? Give feedback.
-
|
最简单的就是当成一整个文件来切分，这样啥也不用做就直接能支持。但这样的问题是如果是超长代码块，可能会有太长的问题 |
Beta Was this translation helpful? Give feedback.
-
|
请问Unstructed.io如何配置，项目里有集成 吗？ |
Beta Was this translation helpful? Give feedback.
-
|
Please add ePub support as well to the list of files supported? Most of the books are in ePub or PDF format, also seems like LLM's are better at reading ePub (due to less clutter than PDF's formatting). You also could add Mobi support, but Mobi support is dying since AMZN (it was only AMZN who were supporting it) stopped supporting it and even they have moved onto ePub now. Thanks |
Beta Was this translation helpful? Give feedback.
-
|
I will check it. Thanks for your advice~ |
Beta Was this translation helpful? Give feedback.
-
|
Unstructed.io 怎么使用？ |
Beta Was this translation helpful? Give feedback.
-
|
楼主好，我想到一个办法，可以把excel文件转换成YAML格式的文件，然后按一定的规则分块。 |
Beta Was this translation helpful? Give feedback.
-
|
贴一个站，可以把Excel转成YAML格式：https://tableconvert.com/zh-tw/excel-to-yaml ※分块想法，每两个“-xxx-”之间分一个块。商品ID: 605081056762
|
Beta Was this translation helpful? Give feedback.
-
|
我在另外一个项目上是把excel表格转换成Markdown格式……就能向量了 |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/nanbingxyz/5ire 发现一个支持向量EXCEL的 |
Beta Was this translation helpful? Give feedback.
-
|
AnythingLLM也可以向量化xls，不知道为什么LobeChat弄不好，本来一直用Lobe的，因为没有Excel向量功能，我打算换平台了。 |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/microsoft/markitdown 现在微软官方出了，office转markdown，py工具 |
Beta Was this translation helpful? Give feedback.
-
背景
在 RAG 中，只有将文件合理分块后，才能做好检索与查询，但是市面上文件类型是非常多的，目前一期只做了一部分的分块支持。
目前支持的分块类型：
纯文本类：
代码类：
富文本类：
表格类：
音频类：
视频类：
如果有对文件类型的分块诉求，请在下面留言，并说明对此类文件的分块设想
Beta Was this translation helpful? Give feedback.
All reactions