rsync: mkdir ".packages" file exists

[ibobo]

Hi, I'm using firelink to deploy from a GitLab CI script, and most times everything runs smoothly. Sometimes instead I get this error:

$ yarn deploy:only --token $FIREBASE_TOKEN
rsync: mkdir "/builds/XXXXXX/./.packages" failed: File exists (17)
rsync error: error in file IO (code 11) at main.c(682) [Receiver=3.1.3]
(node:455) UnhandledPromiseRejectionWarning: false
(Use `node --trace-warnings ...` to show where the warning was created)
(node:455) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:455) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I couldn't find any way to replicate it, and I tried to fix it by adding a rm -rf .packages before running the deploy command to no avail.

[murphman300]

Running into this atm - any clue what is up?

[Stradivario]

Interesting. I am assuming that if it is a clean deploy this would not happen but probably there is some cache which leaves .packages folder available like artefacts from the build itself. Maybe i am wrong but it sounds to me that we need to check in firelink if this folder exists and to remove it if it does not exists. Or if it exists to not throw error but it may polute the artefacts so probably it is better to remove the folder before.

I will try to replicate this one and please if it is possible to write down nodejs version that you are using so i can debug it better.

Regards,
Kristiyan Tachev

[ibobo]

I'm running this on the node:14 docker image, so it happens on the latest node 14 release.

[Stradivario]

I'm running this on the node:14 docker image, so it happens on the latest node 14 release.

Thanks for sharing since today is my birthday i will try to find some time to reproduce it but probably tomorrow i will have more clue and some PR onto how to fix it :)

It should be realatively simple fix.

[Stradivario]

Unfortunate i cannot reproduce it using docker or any other method.

I have executed multiple times rsync command with already existing folders and everything works smooth like a butter...

This is one Dockerfile that i build and i am running it from the example folder on this repo.

FROM node:14.19-slim

WORKDIR /usr/src/app

RUN apt-get update

RUN apt-get upgrade -y

RUN apt-get install rsync -y

RUN npm i -g @rxdi/firelink 

COPY . .

RUN firelink install

Any more clues onto how to replicate this one ?

I have started to think about that it may be related with different versions of rsync accross the docker images or environments.

There is also a native copy method that can be used which will use nodejs fs module to copy appropriate files but it will be far more slower than rsync this is why in the first place i decided to put rsync as one of the main parts of this library because of the performance.

1. Anyway if you want you guys to switch to native copy method you could try --use-native-copy argument

firelink --use-native-copy

2. Other possible solution like you already mentioned is deleting .packages folder before firelink deploy.... command.

3. Third option is to find a way to replicate this and probably to have a better alternative than native rsync

4. Fourth option is to delete folder programatically everytime before rsync method is executed which i will introduce as a PR.

Is there anything that may lead me how to debug it ?

Do you have some example Dockerfile which i may test ?

It is pretty interesting to happen since rsync is pretty idempotent function which has the same output everytime executed and even if folder exists it shouldn't throw error that file exists.

Thank you guys for the report this is the way we can make the world a better place :)

@ibobo
@murphman300

[Stradivario]

It is interesting that there is ongoing ticket inside rsync package reported back in 2011 (2011-07-07) :D

https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/806880

rsync: make_bak_dir mkdir "/media/2TBBackup/backup/.versions/media/hp320/Skannat#2011-07-07#11:21:33" failed: File exists (17)

Also i have found out this tread https://www.experts-exchange.com/questions/28318040/rsync-giving-'file-exists'-errors.html

Where in the end the guy says Problem seemed to go away on it's own. 💯

hahaha i don't understand what is happening here actually :D

[ibobo]

Any more clues onto how to replicate this one ?

I'm not executing this directly through Docker, there is the GitLab CI machinery in the middle, that is maybe reusing runners through execution, but I really have no idea where's the culprit.

I have started to think about that it may be related with different versions of rsync accross the docker images or environments.

If I look at two subsequent runs, one of which failed, I see the same rsync version (3.1.3-6).

There is also a native copy method that can be used which will use nodejs fs module to copy appropriate files but it will be far more slower than rsync this is why in the first place i decided to put rsync as one of the main parts of this library because of the performance.

1. Anyway if you want you guys to switch to native copy method you could try --use-native-copy argument

firelink --use-native-copy

I will give this a try, it could solve the problem despite the lower performance... but that's not a serious concern. I rather prefer having a consistent result.

2. Other possible solution like you already mentioned is deleting .packages folder before firelink deploy.... command.

I already tried that as a workaround, but doesn't help at all... this is a snippet from the GitLab CI log:

$ rm -rf .packages
$ ls -la
total 80
drwxrwxrwx  7 root root  4096 Feb 20 10:40 .
drwxrwxrwx 12 root root  4096 Feb 20 10:40 ..
-rw-rw-rw-  1 root root    58 Feb 20 10:39 .firebaserc
-rw-rw-rw-  1 root root   144 Feb 20 10:39 .gitignore
drwxrwxrwx  3 root root  4096 Feb 20 10:39 assets
drwxrwxrwx  2 root root  4096 Feb 20 10:39 bqschema
-rw-rw-rw-  1 root root   435 Feb 20 10:39 firebase.json
-rw-rw-rw-  1 root root 25 Feb 20 10:39 firestore.rules
drwxr-xr-x  3 root root  4096 Feb 20 10:36 lib
drwxr-xr-x  3 root root  4096 Feb 15 16:25 node_modules
-rw-rw-rw-  1 root root  2059 Feb 20 10:39 package.json
drwxrwxrwx  8 root root  4096 Feb 20 10:39 src
-rw-rw-rw-  1 root root  1379 Feb 20 10:39 storage.rules
-rw-rw-rw-  1 root root   585 Feb 20 10:39 tsconfig.json
$ yarn deploy:only --token $FIREBASE_TOKEN
rsync: mkdir "/builds/XXXXX/functions/./.packages" failed: File exists (17)

3. Third option is to find a way to replicate this and probably to have a better alternative than native rsync

You should really try using GitLab CI. Don't know if @murphman300 also uses that or has any other info.

4. Fourth option is to delete folder programatically everytime before rsync method is executed which i will introduce as a PR.

That could be a safety countermeasure that doesn't harm anyone and could help.

Is there anything that may lead me how to debug it ?

Do you have some example Dockerfile which i may test ?

Really don't have any more info than what I already said.

It is pretty interesting to happen since rsync is pretty idempotent function which has the same output everytime executed and even if folder exists it shouldn't throw error that file exists.

Your remark on the idempotent nature of rsync got me thinking... could there be some sort of -f flag to force it to discard that kind of errors? And yes, there's a --force flag that force deletion of dirs even if not empty. Can you try adding that?

Thank you guys for the report this is the way we can make the world a better place :)

You're welcome! Thanks for helping us solve this hairy issue.

[Stradivario]

@ibobo hello there again!

I have released 0.9.2 version of firelink with added additional --force argument to rsync function. #66

I really hope that this would not break anything else on some other people builds.

Can you give it a try by installing it npm i @rxdi/firelink@0.9.2 ?

Will leave this issue open since i am not sure that this argument will fix the main cause of it...

Cheers!

[MattSkala]

We still experience the same issue (the deployment fails in ~50% of cases) even with the latest version of firelink 0.9.2