BUG: minKNOW non-default data output location broken in LINUX

[cement-head]

Using Ubuntu 22.04 & 18.04 - latest versions of minKNOW do not let one choose and output folder other than the default.

I get a "Path not writable" error, meaning that more than likely the folder permissions are not be changed properly. This is a new bug.

[0x55555555]

Hi @cement-head,

Can you go into a little more detail about how youre using the API, arguments you're passing and what the exact response/errors you receive are?

Additionally, what versions of minknow + the API youre using?

Thanks,

• George

[cement-head]

@jorj1988 Will do, currently in the middle of a sequencing run, so it'll be a day or two - don't want to bugger the current run.

[cement-head]

@jorj1988 Okay, here's the situation. I have an Intel NUC (small computer) that has TWO (2) physically separate SSDs; one for the operating system (Ubuntu 22.04 LTS; about 500 GB) and one for the minKNOW data (1 TB).

The ONLY location that I can save data to during a run is the default </var/lib/minknow/data>. Choosing ANY other location gives an error when attempting to start run that says "ERROR: PATH IS NOT WRITABLE".

The permissions for </var/lib/minknow/data> are:

The permissions for the second SSD (1 TB) where I used to be able to select my storage location for the data are:

The issue (BUG) is that one cannot choose a location other than the default. This is a CRITICAL bug, as many people, including myself use a second SSD for the data only collection. This is a regression/new bug, as older versions of minKNOW did allow choosing a different path/locations to save the run data.

I am using Ubuntu 22.04 LTS; the minKNOW GUI is 5.7.14; the minKNOW is 23.07.12; minKNOW CORE 5.75; GUPPY 7.1.4; BREAM 7.7.6; SCRIPT 5.7.11

[cement-head]

Nanopore Community:

(1) https://community.nanoporetech.com/posts/path-not-writebable

(2) https://community.nanoporetech.com/posts/an-error-occurred-while-at

[0x55555555]

@cement-head - the second image in your post seems to be the same as the first - I am interested to know the permissions on your external HDD, if possible.

We're trying to work back to what the issue is here - none of the changes (running as minknow user, needing write permissions for minknow user to dest) are new - so it's not clear what's broken the workflow.

[juliano60]

@cement-head In addition to @jorj1988 's question above, could you please also tell us how you mounted your secondary SSD? Did you manually mount it from the command line? Many thanks.

Kindest Regards,
Jules

[cement-head]

@jorj1988 Whoops! Thanks for the catch - updated the screenshot to the proper one. The cbfg-minion account is the user account (the only one on the machine).

@juliano60 Via the GUI UDISK2 -> Disks Utility

[Edward-Knight]

Hi @cement-head, could you show us the permissions of the DATA directory on your mounted drive instead of the contents? You can do this with ls -ld DATA instead of ls -l DATA.

[cement-head]

[juliano60]

@cement-head ,

Thanks for that.

We can see (from your screenshot) that the DATA folder is lacking some permissions for the minknow user.
Could you via the same GUI Disk Utility edit your mount options and append the following options (see screenshots below):
rw,umask=0000

We believe that should resolve the permissions issue. To confirm this, could you also re-run the "ls -ld DATA" on your mounted drive and paste the new permissions here? Many thanks.

[cement-head]

Okay, will try and post back.

NOPE. It a mess now, and I can't even access the drive.

---

EDIT: is that umask=ZEROZEROZEROZERO, or umask=OOOO ?

[juliano60]

@cement-head , apologies that was ZEROZEROZEROZERO.
Would you also please be able to provide a screenshot of your mount options page (as you did in a previous post) whilst you're at it? Many thanks.

Kindest Regards,
Jules

[cement-head]

@juliano60 Okay, so reversing (removing & rebooting) the addition of rw,umask=0000 allows me (USER=cbfg-minion) to access the drive again, but again the minKNOW program (USER=minknow) cannot/doesn't have write permissions.

[juliano60]

@cement-head ,
Okay, let's try it again but this time from the command-line.

I am assuming in the following discussion that the drive you are trying to mount is /dev/sdb (as per one of your screenshots above). I am also assuming that the mount point will be located at: /media/cbfg-minion/app-data .

Could you please run the following commands from a terminal:
sudo mkdir /media/cbfg-minion/app-data
sudo mount -t vfat /dev/sdb /media/cbfg-minion/app-data -o rw,umask=0000

The first command creates the directory whilst the second command mounts the device on that directory. If your device is no longer called /dev/sdb you will need to update the command above as appropriate. The Disk Utility shows what the device is called.

Then once done, could you please run a ls -d /media/cbfg-minion/app-data and post a screenshot on here. It is important that the permissions be correct in order for it to work. Many thanks.

Kindest Regards,
Jules

[juliano60]

Note that -o is lowercase O as in option
I should also point out that the device must NOT already be mounted before you run the commands above (you can unmount it using the Disk Utility).

[juliano60]

@cement-head ,
It turns out that your external drive has a different filesystem (namely ext4 as per one of your screenshots above) to what I had in mind, which might explain why you experienced some issues with the "umask" option earlier. I would have to try it here and get back to you. Many thanks.

Kindest Regards,
Jules

[juliano60]

@cement-head ,
My previous post now becomes as follows.

I am assuming in the following discussion that the drive you are trying to mount is /dev/sdb (as per one of your screenshots above). I am also assuming that the mount point will be located at: /media/cbfg-minion/app-data .

Could you please run the following commands from a terminal:

sudo mkdir /media/cbfg-minion/app-data
sudo mount -t ext4 /dev/sdb /media/cbfg-minion/app-data -o rw
sudo chown -R minknow:minknow /media/cbfg-minion/app-data
sudo chmod 777 /media/cbfg-minion/app-data

The first command creates the directory.
The second command mounts the device on that directory.
The third command reconfigures the permissions for the mount point.
The last command achieves a similar purpose but is used in case the third command fails for some reason.

If your device is no longer called /dev/sdb you will need to update the command above as appropriate. The Disk Utility shows what the device is called.

Then once done, could you please run a ls -d /media/cbfg-minion/app-data and post a screenshot on here. It is important that the permissions be correct in order for it to work. Many thanks.

Once again, please ensure that your device is NOT already mounted before running the command above (you can use the Disk Utility to unmount).

Kindest Regards,

[cement-head]

We will try this tomorrow AM, as we're finishing up a large run right now.

[juliano60]

Sounds good @cement-head , do let us know how it goes.

Kindest Regards,
Jules

[cement-head]

$ ls -ld /media/cbfg-minion/app-data/
drwxrwxrwx 4 minknow minknow 4096 Oct 30 15:53 /media/cbfg-minion/app-data/

Still the same error "/path/ not writable"

[juliano60]

Thanks @cement-head ,

The permissions look good now. Just to confirm, did you also select /media/cbfg-minion/app-data as the output directory at the start of the experiment?

Kindest Regards,
Juls

[cement-head]

@juliano60 Yes, I did - and the same error popped up. I'm going to try a fresh install next week and see if that makes a difference - let's put a hold on trouble shooting until then. We have another run that needs to get started, but once that's done (Mon/Tues). I'll reinstall Ubuntu 22.04 from scratch and see if I can get it to work. There maybe something buggered with the current install, and a clean system might be the fastest way to track down this problem.

[juliano60]

@cement-head ,
Sounds good. Do let us know how it goes next week.

Thanks and Regards,
Jules

[ChristopherAdelmann]

Hello @cement-head,
as I am facing a similar issue, I wonder if this could be resolved?
Do you have any updates on this?

Best, Christopher

[jackwadden]

Any updates on this? We've been unable to write to alternate directories for about..... five years now.

[juliano60]

@cement-head Any luck with the re-install?

Kindest Regards,
Jules

[0x55555555]

Hi @jackwadden + @ChristopherAdelmann,

Can you provide info on your specific systems (OS, minknow version) and where you are trying to write, the permissions of that location etc.

Thanks,

• George

[jackwadden]

I'm on Ubuntu 20.04. I'm trying to write to a folder in my home directory. The permissions are 777 and the owner and group are 'minknow:minknow'. Installed Minknow version is 24.02.10 core 5.9.7.

[juliano60]

Thanks @jackwadden ,
Just for absolute clarity, would you mind pasting a screenshot of the permissions for that folder on here. That is, the output of running an ls -ald on that folder.

Kindest Regards,
Jules

[jackwadden]

drwxrwxrwx 6 minknow minknow 4096 Apr 23 17:59 sequencing_runs/

[juliano60]

Thanks @jackwadden

Those look good.
It is worth noting that for the user minknow to be able to access a folder, say folder dir-c, example /dir-a/dir-b/dir-c
where dir-a and dir-b are intermediate directories, the user "minknow" must also have at least both the read and execute permissions set on folders dir-a and dir-b.

For your use-case, we want
At least Read and Execute on / for the minknow user
At least Read and Execute on home for the minknow user
At least Read and Execute on wadden for the minknow user
At least Read and Execute on sequencing_runs for the minknow user (that's already the case, so no change required)

Could you please double-check each of them (by running ls -ald on each folder)? Or paste a screenshot showing the permissions for each folder on here so we can double-check those also.

Kindest Regards,
Jules

[jackwadden]

OK, thanks for the detailed response, and understood. I'll check/change those permissions and reply back later today when I'm able to retest. Is there any (easy) way to test without a live flowcell?

[juliano60]

@jackwadden

One way to test (and to make sure) that the permissions are correct would be to login as that user (that is, as the user "minknow") and to attempt to create a dummy file in that folder of interest. Otherwise checking each of the intermediate folders' permissions should cover it.

Kindest Regards,
Jules

[jackwadden]

Yeah, perfect. I'll do that now and reply later when I've tested with a live run.

Thanks again for the quick and helpful responses.

[juliano60]

Cheers @jackwadden ,
Note that loging in as "minknow" might be tricky depending on whether that user has a login shell configured. If you are able to login, great, otherwise we are back to checking all the permissions individually.

Kindest Regards,
Jules

[0x55555555]

If that doesnt resolve your issue @jackwadden I'd like to take a look at your logs, are you able to contact support (via https://nanoporetech.com/support) and pass over a log dump from your machine?

Thanks,

• George

[jackwadden]

Just updating here because I was able to make some progress but am still am having issues. Examining the log files was extremely helpful.

• My particular issue stemmed from the fact that I deleted the /var/lib/minknow/data folder and moved those folders to my home directory thinking that they should reside in the same directory as the output folder.
• Minknow really didn't like this and after a few re-installs, configuration files and permissions were really messed up. Log files indicated that minknow couldn't write to the /var/lib/minknow/data and log folders because it didn't have permissions.
• I deleted all /var/lib/minknow /opt/minknow related folders manually
• I re-installed minknow
• I manually created the /var/lib/minknow folder and then modified all permissions and ownership to minknow:minknow
• Once I did this, Minknow was able to write to those folders properly, and also seamlessly wrote to my chosen output directory after modifying permissions for those folders.
• To avoid chmod -R 777 on my entire home directory, I instead used ''setfacl -R -m u:minknow:rwx minknow"
• This led to Minknow letting me start a sequencing run. It properly identified the output directory set to my home directory and emitted a message saying 'Disk /home had 3023GB space remaining'. Awesome.
• Then, the run failed after sequencing started with 'Error: No disk space'.
• Presumably, this is because /var needs to be on a partition with a particular size. I'm not sure if this is outlined in the IT requirements, but my university managed machine only has 12GB available. This is why I'm anxious to move the output dir to a different partition on the same drive.
• I cleaned the apt cache to reclaim a few GB and ended up with a total of 5.8GB free
• I restarted the run and got the same "all clear" message that told me I had 3023GB available.
• Then the run failed again with the error message: "Disk too low to continue. Stopping data acquisition. Please clear some space and re-start the experiment" and "Error: No disk space"
• I will email support now to pass along the log files, but any information advice in the meantime to help resolve this in this forum would be appreciated.

[jackwadden]

OK, I think all is good now

• I re-read the instructions here: https://help.nanoporetech.com/en/articles/6627927-how-do-i-change-the-directory-for-the-reads-folder
• modified the user_conf file so that the 'base' directory points to a folder with correct permissions on the correct, >1TB partition.
• rebooted the machine
• restarted the run
• And things started working no problem

So to summarize:

• users don't follow directions
  But seriously, it would be nice for the IT requirements and the modification of the output folder to be a bit more streamlined in the GUI (including more helpful error messages) to avoid these issues (like having the temp files default to the same output directory when changed).

To successfully adjust your output directory for all output files in the case /var is too small, you need to follow these steps in order:

• Delete all ont related folders/files to clean up any leftover configuration files from previous installs
• Perform a clean install
• Make an output directory on the output drive with wrx permissions and ownership for minknow:minknow. These permissions must be recursive all the way to root (if I'm understanding above correct?). To avoid relaxing all permissions for all users all the way to the root directory, you can run "setfacl -R -m u:minknow:rwx minknow" per level. I did this to a particular folder in my home directory.
• This only modifies the final output directory for fastq/pod5 data and DOES NOT modify the output directory for intermediate files. If /var is particularly small, your runs will still fail with "no space" errors.
• To move the output for intermediate files you need to modify /opt/ont/minknow/conf/user/conf according to point to the new intermediate output directory, and then restart the machine. This directory must also have the same permissions as the final output directory, and should be modified like the output directory to have relaxed permissions above. I simply made the output folder path of the intermediate files the same as the final files and that worked great.

Hopefully this helps anyone running into the same issues. And thanks to George and Jules for the quick responses.

-Jack

[0x55555555]

Thanks @jackwadden,

I'm glad you got it working - I will take your advice here and get the docs updated to reflect the advice!

• George