Add option for cache management #319
Conversation
Extract an explicit function expression to a variable to fix broken syntax highlight. For example add an `-spec` line or `-define` after function `index_of` and those will not be syntax highlighted.
Erlang/OTP `supervisor` documentation says: [1] > It is also allowed to set it to `infinity`, if the child process is a > worker. > > Warning > > Be careful when setting the shutdown time to `infinity` when the child > process is a worker. Because, in this situation, the termination of > the supervision tree depends on the child process, it must be > implemented in a safe way and its cleanup procedure must always > return. Therefore, the shutdown time of `gen_lsp_server` is changed from `infinity` to default value, that is `5000` milliseconds for worker processes. Additionally, a sketch of the application supervision tree is added to module header of `vscode_lsp_app` for easier understanding. The `-behaviour` attributes are added to the corresponding modules, also for easier understanding. [1] https://www.erlang.org/doc/man/supervisor.html
Add a configuration setting to change the storage mode for large cache tables (mainly used for code navigation). Available modes: - `memory` (default, original): use ETS tables - `compressed memory`: use ETS tables with `compressed` flag turned on. - `file`: use DETS tables saved into temporary files. Why is this configuration option needed? In shared environment, where many instances of `vscode_erlang` run in the same time on large code bases, noticeable amount of memory can be consumed by this extension that eventually can exhaust all the memory. Notes for `compressed memory`: Some simple tests, done in the above explained scenario, showed memory consumption dropped by 50% when `compressed memory` is configured. Notes for `file`: As multiple extension instances can run in the same time on the same host, even in the same workspace, we cannot use static filenames, nor workspace specific filenames to create DETS tables. The only option is to use unique filenames, like temporary files. Of course, it has a drawback, temporary files must be deleted on exit otherwise those will consume a lot of disk space after a while. If the extension is shut down normally, it's not a problem, the extension deletes it's own cache. However, if some extension is terminated brutally, the cache files left on the disk. Hence, an automatic mechanism is hooked to the points when configuration is received from Visual Studio Code, and to normal shutdown, to look for cache directories created by yet not running extension, and delete them. Reusing old cache files is theoretically possible if only one extension instance run in a workspace, VSCode is closed and reopened, then the new extension instance could continue where the previous one stopped. But, as multiple extension instances can run in the same workspace, it can easily go wrong. Therefore it is simpler to use instance specific DETS files, just like instance specific ETS tables.
|
This is a continuation of PR #313 that is canceled and evolved into this PR. |
|
There is a yet unsolved issue in this PR, namely the deletion of an old cache directory created by an old extension instance with old PID, and there is a non-Erlang process during the check using the same PID. It could be filtered out, but it doesn't really cause a noticeable problem, as next or next-next time it will be deleted. Yet, it's not nice. |
|
I list some measurements here for reference. The test project contains 2209 Erlang modules, approximately 110 MB. memory This is the baseline as this is the legacy and default method. Total memory: 2.34 GB, ETS memory: 2.26 GB ETS tables:
compressed memory Total memory: 944.86 MB, ETS memory: 872.12 MB ETS tables:
file Total memory: 535.23 MB, ETS memory: 453.16 MB, DETS disk usage: 716.1 MB ETS tables:
DETS tables:
|
|
Hi @KornelH , Sorry for the delay. You choose to store file in temporary file, but what do think about using storage locations provided by VsCode ? On Windows: C:\Users<user>\AppData\Roaming\Code\User\globalStorage<extension-id> documentation : https://code.visualstudio.com/api/extension-capabilities/common-capabilities#data-storage
|
|
Good idea. I haven't checked it yet but this is what I'm thinking about:
I shall test 2) and if it's not unique I think simply using My other big concern is 4). Storing config on NFS is good, but large cache files is another story. This is why I wanted to use the temp directory as that is always resides on local disk and usually has no or high quota limit and fast. |
|
I meant to discuss the alternatives ... but I'm fine with the current version too, hence I did it. Let's see if there will be any feedback for it. |
|
For me 2 things
I agree, let's see. |
Add a configuration setting to change the storage mode for large cache tables (mainly used for code navigation). Available modes:
memory(default, original): use ETS tablescompressed memory: use ETS tables withcompressedflag turned on.file: use DETS tables saved into temporary files.Why is this configuration option needed?
In shared environment, where many instances of
vscode_erlangrun in the same time on large code bases, noticeable amount of memory can be consumed by this extension that eventually can exhaust all the memory.Notes for
compressed memory:Some simple tests, done in the above explained scenario, showed memory consumption dropped by 50% when
compressed memoryis configured.Notes for
file:As multiple extension instances can run in the same time on the same host, even in the same workspace, we cannot use static filenames, nor workspace specific filenames to create DETS tables. The only option is to use unique filenames, like temporary files. Of course, it has a drawback, temporary files must be deleted on exit otherwise those will consume a lot of disk space after a while. If the extension is shut down normally, it's not a problem, the extension deletes it's own cache.
However, if some extension is terminated brutally, the cache files left on the disk. Hence, an automatic mechanism is hooked to the points when configuration is received from Visual Studio Code, and to normal shutdown, to look for cache directories created by yet not running extension, and delete them.
Reusing old cache files is theoretically possible if only one extension instance run in a workspace, VSCode is closed and reopened, then the new extension instance could continue where the previous one stopped. But, as multiple extension instances can run in the same workspace, it can easily go wrong. Therefore it is simpler to use instance specific DETS files, just like instance specific ETS tables.