[Bug] I can't upload symbols zip file to firebase crashlytics

[Awais6]

Description

I can't upload debug symbols to firebase crashylytics.
I have updated firebase SDK version in my unity project to 12.2.1, and build my game for android (aab) with debug symbols (zip file).

i Generating symbols for D:\Projects\Multi-Projects\Ludo Paisa\Unity\Ludo\Builds\9.6.166-9.6.166-v166-IL2CPP.symbols.zip
! [CRASHLYTICS LOG DEBUG] Crashlytics Buildtools initialized.
[CRASHLYTICS LOG DEBUG] Skipping dumpsyms extraction, file exists: C:\Users\Awais.crashlytics\dump_syms.exe
[CRASHLYTICS LOG DEBUG] Breakpad symbol generator initialized: C:\Users\Awais.crashlytics\dump_syms.exe
[CRASHLYTICS LOG DEBUG] Zipped input file detected: D:\Projects\Multi-Projects\Ludo Paisa\Unity\Ludo\Builds\9.6.166-9.6.166-v166-IL2CPP.symbols.zip; unzipping to temp location: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache
[CRASHLYTICS LOG DEBUG] Generating native symbol files for C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache; writing output to: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad
[CRASHLYTICS LOG DEBUG] 53 native libraries found at C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache
[CRASHLYTICS LOG DEBUG] Crashlytics generating Breakpad Symbol file for: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache\arm64-v8a\libcrashlytics-trampoline.so
[CRASHLYTICS LOG DEBUG] Extracting Breakpad symbols to temp file: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\libcrashlytics-trampoline.so4702762137075756530.tmp
[CRASHLYTICS LOG DEBUG] GNU Build Id for C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache\arm64-v8a\libcrashlytics-trampoline.so: acf6365f1a57d0513d7e78424b14f07c579aa5fc
[CRASHLYTICS LOG DEBUG] Renaming Breakpad symbol file to: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\crashlytics-trampoline-arm64-acf6365f1a57d0513d7e78424b14f07c579aa5fc.sym
[CRASHLYTICS LOG DEBUG] Generated symbol file: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\crashlytics-trampoline-arm64-acf6365f1a57d0513d7e78424b14f07c579aa5fc.sym (839 bytes)
[CRASHLYTICS LOG DEBUG] Crashlytics generating Breakpad Symbol file for: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache\arm64-v8a\libFirebaseCppAnalytics.so
[CRASHLYTICS LOG DEBUG] Extracting Breakpad symbols to temp file: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\libFirebaseCppAnalytics.so15280945419020714535.tmp
[CRASHLYTICS LOG DEBUG] Cleaning up unzip target dir: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache
java.io.IOException: Breakpad symbol generation failed (exit=1), see STDERR
at com.google.firebase.crashlytics.buildtools.ndk.internal.breakpad.BreakpadSymbolGenerator.generateSymbols(BreakpadSymbolGenerator.java:129)
at com.google.firebase.crashlytics.buildtools.Buildtools.generateNativeSymbolFiles(Buildtools.java:306)
at com.google.firebase.crashlytics.buildtools.Buildtools.generateNativeSymbolFiles(Buildtools.java:280)
at com.google.firebase.crashlytics.buildtools.CommandLineHelper.executeGenerateSymbols(CommandLineHelper.java:193)
at com.google.firebase.crashlytics.buildtools.CommandLineHelper.executeCommand(CommandLineHelper.java:125)
at com.google.firebase.crashlytics.buildtools.CommandLineHelper.main(CommandLineHelper.java:72)
at com.google.firebase.crashlytics.buildtools.Buildtools.main(Buildtools.java:114)

Error: java command failed with args: -jar,C:\Users\Awais.cache\firebase\crashlytics\buildtools\crashlytics-buildtools-3.0.2.jar,-symbolGenerator,breakpad,-symbolFileCacheDir,C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad,-verbose,-generateNativeSymbols,-unstrippedLibrary,D:\Projects\Multi-Projects\Ludo Paisa\Unity\Ludo\Builds\9.6.166-9.6.166-v166-IL2CPP.symbols.zip,-clientName,firebase-cli;crashlytics-buildtools

Reproducing the issue

No response

Firebase Unity SDK Version

12.2.1

Unity editor version

6000.0.25f1 LTS

Installation Method

.unitypackage

Problematic Firebase Component(s)

Authentication, Crashlytics, Database, Firestore, Functions, Messaging, Remote Config

Other Firebase Component(s) in use

No response

Additional SDKs you are using

No response

Targeted Platform(s)

Android

Unity editor platform

Windows

Scripting Runtime

IL2CPP

Release Distribution Type

Pre-built SDK from https://firebase.google.com/download/unity

Relevant Log Output

i  Generating symbols for D:\Projects\Multi-Projects\Ludo Paisa\Unity\Ludo\Builds\9.6.166-9.6.166-v166-IL2CPP.symbols.zip
!  [CRASHLYTICS LOG DEBUG] Crashlytics Buildtools initialized.
[CRASHLYTICS LOG DEBUG] Skipping dumpsyms extraction, file exists: C:\Users\Awais\.crashlytics\dump_syms.exe
[CRASHLYTICS LOG DEBUG] Breakpad symbol generator initialized: C:\Users\Awais\.crashlytics\dump_syms.exe
[CRASHLYTICS LOG DEBUG] Zipped input file detected: D:\Projects\Multi-Projects\Ludo Paisa\Unity\Ludo\Builds\9.6.166-9.6.166-v166-IL2CPP.symbols.zip; unzipping to temp location: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache
[CRASHLYTICS LOG DEBUG] Generating native symbol files for C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache; writing output to: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad
[CRASHLYTICS LOG DEBUG] 53 native libraries found at C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache
[CRASHLYTICS LOG DEBUG] Crashlytics generating Breakpad Symbol file for: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache\arm64-v8a\libcrashlytics-trampoline.so
[CRASHLYTICS LOG DEBUG] Extracting Breakpad symbols to temp file: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\libcrashlytics-trampoline.so4702762137075756530.tmp
[CRASHLYTICS LOG DEBUG] GNU Build Id for C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache\arm64-v8a\libcrashlytics-trampoline.so: acf6365f1a57d0513d7e78424b14f07c579aa5fc
[CRASHLYTICS LOG DEBUG] Renaming Breakpad symbol file to: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\crashlytics-trampoline-arm64-acf6365f1a57d0513d7e78424b14f07c579aa5fc.sym
[CRASHLYTICS LOG DEBUG] Generated symbol file: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\crashlytics-trampoline-arm64-acf6365f1a57d0513d7e78424b14f07c579aa5fc.sym (839 bytes)
[CRASHLYTICS LOG DEBUG] Crashlytics generating Breakpad Symbol file for: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache\arm64-v8a\libFirebaseCppAnalytics.so
[CRASHLYTICS LOG DEBUG] Extracting Breakpad symbols to temp file: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\libFirebaseCppAnalytics.so15280945419020714535.tmp
[CRASHLYTICS LOG DEBUG] Cleaning up unzip target dir: C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad\unzippedLibsCache
java.io.IOException: Breakpad symbol generation failed (exit=1), see STDERR
        at com.google.firebase.crashlytics.buildtools.ndk.internal.breakpad.BreakpadSymbolGenerator.generateSymbols(BreakpadSymbolGenerator.java:129)
        at com.google.firebase.crashlytics.buildtools.Buildtools.generateNativeSymbolFiles(Buildtools.java:306)
        at com.google.firebase.crashlytics.buildtools.Buildtools.generateNativeSymbolFiles(Buildtools.java:280)
        at com.google.firebase.crashlytics.buildtools.CommandLineHelper.executeGenerateSymbols(CommandLineHelper.java:193)
        at com.google.firebase.crashlytics.buildtools.CommandLineHelper.executeCommand(CommandLineHelper.java:125)
        at com.google.firebase.crashlytics.buildtools.CommandLineHelper.main(CommandLineHelper.java:72)
        at com.google.firebase.crashlytics.buildtools.Buildtools.main(Buildtools.java:114)


Error: java command failed with args: -jar,C:\Users\Awais\.cache\firebase\crashlytics\buildtools\crashlytics-buildtools-3.0.2.jar,-symbolGenerator,breakpad,-symbolFileCacheDir,C:\Users\Awais\AppData\Local\Temp\crashlytics-5c117779-203c-4a35-a431-39f037256e65\nativeSymbols\1-79023085-android-dd73ceb161b56d31113724\breakpad,-verbose,-generateNativeSymbols,-unstrippedLibrary,D:\Projects\Multi-Projects\Ludo Paisa\Unity\Ludo\Builds\9.6.166-9.6.166-v166-IL2CPP.symbols.zip,-clientName,firebase-cli;crashlytics-buildtools

If using CocoaPods for Apple platforms, the project's Podfile.lock

Expand Podfile.lock snippet

👀 Replace this line with the contents of your Podfile.lock!

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[UnwindGamesDeveloper]

+1 getting same error on Mac Seq 15.1 & Firebase Unity SDK - 12.4.1

[argzdev]

Hey folks, thanks for reporting this. I'm experiencing very inconsistent behavior with this issue (version 12.2.0). Initially I was able to reproduce this behavior yesterday, however today when I started re-testing with higher versions (version 12.4.1). It seems to upload the symbols correctly. Could you try it again and see if it works? I'm wondering if it's an issue with the backend which caused it to momentarily not work.

[UnwindGamesDeveloper]

@argzdev I had been trying for last 2 days didn't work. Tested it now , got uploaded fine. Have been on 12.4.1 thoughout. Can we assume then it was a backend issue ?

[argzdev]

@UnwindGamesDeveloper, yes I'm guessing this is the case. Ive finished testing versions 12.1.0, 12.2.0, 12.2.1, 12.3.0, 12.4.0 so far all these versions are working. I'll go ahead and close this thread for now. Let me know if there's any other concerns regarding the SDK, we can always reopen this thread.

In case this issue pops up again without any changes. Feel free to contact our Firebase support team. They'll be able to redirect this issue to our backend engineers.

[Awais6]

I think issue is related to java version mismatch or maybe firebase-tools npm package, Breakpad symbol generation is failed, it happened on client side, not backend (server) side. (as shown in above figure)

If crashylytics SDK is involved in producing symbols zip file (created with build), maybe the bug is related to SDK.

Thanks for your testing @argzdev

[argzdev]

That's also possible that there is a version compatibility of firebase-tools with the Crashlytics SDK being used. That said, are you still encountering the issue, @Awais6? Could you share with me which firebase-tools version are you currently using? I'm currently using firebase-tools version 13.7.3.

[UnwindGamesDeveloper]

@argzdev So I found some more details. If I remove, libFirebaseCppAnalytics.so, libFirebaseCppApp-12_4_1.so, libFirebaseCppCrashlytics.so from zip it works fine (All three as I am only using analytics and crashlytics).

That means its breaking on trying to generate breakpad symbols for these 3 libs and maybe other firebase specific libs

[Awais6]

Yes @argzdev, The issue is still there on my side. I am using firebase-tools 13.26.0 Latest and firebase SDK 12.2.1.

[Awais6]

@argzdev So I found some more details. If I remove, libFirebaseCppAnalytics.so, libFirebaseCppApp-12_4_1.so, libFirebaseCppCrashlytics.so from zip it works fine (All three as I am only using analytics and crashlytics).

That means its breaking on trying to generate breakpad symbols for these 3 libs and maybe other firebase specific libs

I tested your workaround, which was to remove all Firebase libraries from the zip file, and it successfully generated and uploaded breakpad symbols.
I am using firebase-tools 13.26.0