Emulator stops without specific error

[exetico]

[REQUIRED] Environment info

GitHub Action execution on ubuntu-latest with Firebase CLI installed with npm install -g firebase-tools

firebase-tools: 13.20.2

Platform: Ubuntu

[REQUIRED] Test case

• I've not been able to trim down the current case, to reproduce this. But I'm working on that, too.

[REQUIRED] Steps to reproduce

• Started the emulator with firebase emulators:exec --only auth,functions,firestore,hosting,pubsub 'npm run __jest' --debug and it's stopping without throwing a reason why it's terminating the emulator.
• The __jest command looks like this: node --experimental-vm-modules node_modules/jest/bin/jest.js --detectOpenHandles --verbose --ci --reporters=default --reporters=jest-junit

[REQUIRED] Expected behavior

I expect the emulator to continue starting the Emulators, or throw a error, so I know what's causing the issue.

[REQUIRED] Actual behavior

The emulator stops, without throwing a specific error. I've notived a "VSCode notifiation" debug-log, but that shouldn't trigger a shutdown. It's printed with logger.debug.

[2024-10-02T10:44:09.289Z] openjdk version "11.0.24" 2024-07-16

[2024-10-02T10:44:09.291Z] OpenJDK Runtime Environment Temurin-11.0.24+8 (build 11.0.24+8)
OpenJDK 64-Bit Server VM Temurin-11.0.24+8 (build 11.0.24+8, mixed mode)

[2024-10-02T10:44:09.304Z] Parsed Java major version: 11
i  emulators: Starting emulators: auth, functions, firestore, hosting, pubsub {"metadata":{"emulator":{"name":"hub"},"message":"Starting emulators: auth, functions, firestore, hosting, pubsub"}}
[2024-10-02T10:44:09.309Z] [logging] Logging Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2024-10-02T10:44:09.309Z] [auth] Authentication Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2024-10-02T10:44:09.309Z] [firestore] Firestore Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2024-10-02T10:44:09.309Z] [firestore.websocket] websocket server for firestore only supports listening on one address (127.0.0.1). Not listening on ::1
[2024-10-02T10:44:09.310Z] [hosting] Hosting Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2024-10-02T10:44:09.310Z] [pubsub] Pub/Sub Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2024-10-02T10:44:09.310Z] assigned listening specs for emulators {"user":{"logging":[{"address":"127.0.0.1","family":"IPv4","port":4500}],"auth":[{"address":"127.0.0.1","family":"IPv4","port":9099}],"firestore":[{"address":"127.0.0.1","family":"IPv4","port":5002}],"firestore.websocket":[{"address":"127.0.0.1","family":"IPv4","port":9150}],"hosting":[{"address":"127.0.0.1","family":"IPv4","port":5004}],"pubsub":[{"address":"127.0.0.1","family":"IPv4","port":5003}]},"metadata":{"message":"assigned listening specs for emulators"}}
[2024-10-02T10:44:09.312Z] [functions] Functions Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2024-10-02T10:44:09.312Z] [eventarc] Eventarc Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2024-10-02T10:44:09.312Z] [tasks] Cloud Tasks Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2024-10-02T10:44:09.312Z] late-assigned ports for functions and eventarc emulators {"user":{"logging":[{"address":"127.0.0.1","family":"IPv4","port":4500}],"auth":[{"address":"127.0.0.1","family":"IPv4","port":9099}],"firestore":[{"address":"127.0.0.1","family":"IPv4","port":5002}],"firestore.websocket":[{"address":"127.0.0.1","family":"IPv4","port":9150}],"hosting":[{"address":"127.0.0.1","family":"IPv4","port":5004}],"pubsub":[{"address":"127.0.0.1","family":"IPv4","port":5003}],"functions":[{"address":"127.0.0.1","family":"IPv4","port":5001}],"eventarc":[{"address":"127.0.0.1","family":"IPv4","port":9299}],"tasks":[{"address":"127.0.0.1","family":"IPv4","port":9499}]},"metadata":{"message":"late-assigned ports for functions and eventarc emulators"}}
[2024-10-02T10:44:09.314Z] Could not find VSCode notification endpoint: FetchError: request to http://localhost:40001/vscode/notify failed, reason: connect ECONNREFUSED ::1:40001
i  emulators: Shutting down emulators. {"metadata":{"emulator":{"name":"hub"},"message":"Shutting down emulators."}}
[2024-10-02T10:44:09.317Z] Could not find VSCode notification endpoint: FetchError: request to http://localhost:40001/vscode/notify failed, reason: connect ECONNREFUSED ::1:40001

Before this, I also tried running the emulator without the --only param. I expected the "extensions" support in firebase-tools to be the problem, even though, we're not using any extensions. But, it's the same story. It still stops without any specific error.

Is it out of scope, to expect a thrown error, to be related, to why it's simply stopping the emulator right away?

firebase-tools/src/dataconnect/webhook.ts

Line 38 in a717346

logger.debug(`Could not find VSCode notification endpoint: ${e}`);

Locally things works as expected. I'm meet by the You are not signed in to the Firebase CLI ➕ ⚠ firestore: Did not find a Cloud Firestore rules file specified in a firebase.json config file. ➕ ⚠ firestore: The emulator will default to allowing all reads and writes. ➕ Ignoring unsupported arg: auto_download ➕ Ignoring unsupported arg: single_project_mode_error. HereafterStarting Firestore Emulator with command are printed out, and the Emulator starts up, and the jest test spins up as expected.

Using "host": "0.0.0.0" / "127.0.0.1" / "localhost" does not change things.

I've previously submitted #7499 (solved) and #7624 (workaround applied), but after a few weeks, the CI/CD solution broke. I've tried to search through the issues, but I've not found anything like this.

How am I going to spot what's causing fire Emulator to stop in the Github Actions env.?

[joehan]

Hey @exetico, sorry for the inconvenience here - we've recently made some changes to this part of the codebase to support a new VSCode extension, and it seems like that is causing your issue here. I'm looking into the error handling around this now to ensure that we are not innapropriately swallowing errors or triggering a shutdown somewhere.

[exetico]

@joehan Thanks for the reply. Let me know, how it goes.

[exetico]

Hi @joehan , how is it going with the patch?

[aalej]

Hey @exetico, thanks for following up. I'm currently working on reproducing this, and it looks like the issue might have been fixed on v13.21.0 of firebase-tools. Here's the current mcve I have. Could you try updating to firebase-tools v13.21.0 to see if you'd still encounter this issue?

[exetico]

Hi @aalej. Thank you for the information. I've bumped the version and executed the same workflow. I can confirm that the issue is not present anymore.

Side-note: What a impressive amount of mcve's! I'll try and remember that, if I need to reproduce a given bug 🐛 in the future.

[exetico]

@aalej I just reopened this, as I'm experincing the same thing again. Now we're using 13.22.0 (latest version right now)

I get the following errors at the end, just before the emulators: Shutting down emulators. {"metadata":{"emulator":{"name":"hub"},"message":"Shutting down emulators."}}.

[2024-10-10T13:44:32.908Z] Could not find VSCode notification endpoint: FetchError: request to http://localhost:40001/vscode/notify failed, reason: connect ECONNREFUSED ::1:40001. If you are not running the Firebase Data Connect VSCode extension, this is expected and not an issue.
i  emulators: Shutting down emulators. {"metadata":{"emulator":{"name":"hub"},"message":"Shutting down emulators."}}
[2024-10-10T13:44:32.909Z] Could not find VSCode notification endpoint: FetchError: request to http://localhost:40001/vscode/notify failed, reason: connect ECONNREFUSED ::1:40001. If you are not running the Firebase Data Connect VSCode extension, this is expected and not an issue.

I also get this:

[2024-10-10T13:44:32.905Z] HTTP Error: 401, Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

But again, if I remove the --debug flag, the Emulator does not state EXACTLY why it's terminated. That makes it very tricky to debug, and find the root cause.

Update:
I've cloned your mcve, and everything works. But that's a bit confusion, cause why does things works as expected on your end?

[exetico]

Update:
Using firebase emulators:exec --only auth,functions,firestore,hosting,pubsub 'npm run __jest' and therefore not using extensions fixes it.

Are we still in a situation where extensions are problematic, in some flows?

We don't use extensions.

Again; The emulator should not just stop without actually inform the user about why it's actually stops; even before functions are started

[exetico]

@aalej Take a look at the mcve to see the issues with the extensions.

https://github.com/exetico/issues-7779/actions

With only (=extensions excluded): https://github.com/exetico/issues-7779/actions/runs/11292281349/job/31407950327
Without only: https://github.com/exetico/issues-7779/actions/runs/11292281353/job/31407950329

The mcve looks a bit too complex, but I ended up trying to replicate something which came close to what I'm doing in one of the application where we're facing the mentioned error.

We're not using extensions, so we don't expect the emulator to fail. On top of that, it fails without being specific about anything, and breaks the deploy flow for users who simply bumps the version of firebase-tools.

[aalej]

Hey @exetico, sorry to hear you're still encountering this issue, and thanks for all the additional details. I appreciate your effort in gathering this information!

I’m also now able to reproduce the issue on firebase-tools v13.22.0. As you mentioned, running with the --only flag, firebase emulators:exec --only auth,functions,firestore,hosting,pubsub 'npm run test is successful. But running firebase emulators:exec 'npm run test' outputs:

i  emulators: Starting emulators: auth, functions, firestore, hosting, pubsub, extensions
i  emulators: Shutting down emulators.

At the moment, I’m not sure if extensions is causing this error. Let me raise this to our engineering team so they can take a look into this and verify if this is indeed the cause of the issue.

[joehan]

Hey @exetico - thanks for the detailed MCVE. I noticed that in the MCVE, you're using what is maybe a fake project ('projectname-default') but are not using the demo- prefix (seehttps://firebase.google.com/docs/emulator-suite/use_extensions#choose_a_firebase_project). When I run firebase emulators:exec 'npm run test_jest' -P demo-test, the tests pass as expected even with the extensions emulator running.

I suspect this is the root cause of your issue here. The reason that excluding extensions made the test pass even with the fake project is because the extensions emulator tries to look up project number for real projects (as it is needed for correct secret param resolution).

Having said that - this clearly is also a failure of our error handling and shouldn't fail silently like this. PR incoming to handle this more cleanly.

[exetico]

@aalej Thank you.

I still think it's important to know, why extensions are the solution which does kill the emulator. I mean, as the other services are able to work as expected, I also expect the "default pack" to work. The extensions have raised a few issues in the last couple of months, so personally I still think it's related to some newer changes and/or implementations.

@joehan The project name are from @aalej 's MCVE. Normally I use a valid project ID, but the test session are not authorized to Google (like the GitHub Action workflow provided).

The fact there's a demo prefix was not part of my current knowledge, so I'll revisit that part on the upcoming week. However, I'm not sure I agree with your general thoughts about the root cause.

We're talking about a tool, which does fail if extensions is included. Extensions handled in firebase-tools are fairly new (if I remember correctly), at least in the current way. Before that, it have been working just fine, with the minimal example provided (where --only are not in use). Therefore, as seen from my view, there's still a bug to be resolved, implemented in one of the last couple of releases.

To me it sounds like there's three things:

1. Something I personally need to look at, regarding the test prefix. But normally I use a real project ID. I don't get why I should start using a project with a test prefix, as it have been working just fine for a long period of time, at this point 😄
2. Firebase Tools should not shutdown the emulator as it does, without providing any user information. Even the --debug flag does not help much
3. There's still some problems introduced with extensions. As pointed out, things have been working just fine, but broke after the latest changes to the extensions part. However, the important part here is, that the extensions part of things break the current experience, even though I'm not using extensions at all. Here the --only flag shouldn't be required. The empty extensions:{} doesn't solve this. You're stating that the extensions needs a correct project ID for secret param resolution, however, all other things does work with a none-authorized user with the current project ID, and the current project does not even use extensions at the first place?

It sounds like we're able to explain what's going on, but I don't see that point 2 + 3 are fixed. I still see those two things as bugs in the code-base. The update and support for extensions even broke current CI/CD solutions, due to those changes.

Do you guys agree on this?