Size of Zapdos? (9.0 GB!)

[smpeyres]

I recently made a fresh clone of Zapdos and noticed it took >20 minutes to get the whole thing onto my machine. The most significant steps were the compression & receiving stages, and it seems this is likely due to the sheer size. The total contents of the repository on my machine clocks in at 9.0 GB, whereas my clone of the moose repo only takes up 2.1 GB.

Zapdos is considerably smaller than the entire MOOSE framework. Where is all of this memory requirement coming from?

[cticenhour]

Without looking too deeply, there are likely large data files either in the repository currently or in its git history that is pulled down with the clone as well. I can make a note to look into ways we might shrink this down. Thanks for the note!

[cticenhour]

I was finally able to take a look at this today. Below is a list of the top 20 largest files in zapdos git history (neglecting the gh-pages branch):

9de863be80a2  8.7MiB tests/1d_dc/gold/mean_en_out.e
7d3ea8659718  9.3MiB doc/Lindsay_thesis_coupling_of_plasmas_and_liquids.pdf
b668c969d84a   10MiB problems/Schottky_emission/transient/no_ballast/cycle_0.3_ns_21_ns/V_199_V_d_2_um/V_199.o743290
393f30bd5bd3   10MiB tests/Schottky_emission/Example6/.__afs7BBF
ac75728e2d3f   12MiB test/tests/DriftDiffusionAction/gold/RF_out.e
a5109ef572d0   12MiB tests/DriftDiffusionAction/gold/RF_out.e
332a6a5191a9   13MiB tests/accelerations/gold/Acceleration_By_Averaging_main_out.e
aba2ce5f2f22   13MiB test/tests/accelerations/gold/Acceleration_By_Averaging_main_out.e
27a52670cb51   13MiB test/tests/Conference_Syntax_Tests/GEC_mesh.msh
324b8e2223a6   14MiB tests/steady_state_field_emission/5um-field-emission_out.e
2abb3a093a99   15MiB logEnergyFormFDPSerialIonHighTransport.txt
4fc7b2e6e3f7   19MiB tests/Lymberopoulos_rf_discharge/gold/Lymberopoulos_with_argon_metastables_2D_At100mTorr_CourseMesh_out.e
b1b67294b3bb   19MiB tests/Lymberopoulos_rf_discharge/gold/Lymberopoulos_with_argon_metastables_2D_At100mTorr_CourseMesh_out.e
d1f45babb00c   27MiB test/tests/PeriodicRelativeNodalDifference/gold/Sample_Periodic_Diffusion_out.e
039ed5ad2253   28MiB tests/steady_state_field_emission/50um-no_field_emission_out.e
3a103e2b7eb2   32MiB test/tests/PeriodicRelativeNodalDifference/gold/Sample_Periodic_Diffusion_Log_out.e
db25441fba5a   33MiB tests/steady_state_field_emission/50um-field_emission_out.e
29a3162d31d7   59MiB test/tests/accelerations/Acceleration_By_Shooting_Method_Initial_Conditions.e
ebfa070cbdf3   66MiB log_DCPlasma_argon_energy_interp_resid_and_jacobian.txt
e92383324639   85MiB problems/Schottky_emission/transient/no_ballast/cycle_0.3_ns_21_ns/V_199.5_d_2_um/V_199.5.o743616

MOOSE generally doesn't allow files larger than 5MB (1MB ~ 0.9537 MiB), so we're easily exceeding that. I think I might add CIVET warnings to future PR prechecks based on this result, so avoid the checking in of large files in the future.

If we include gh-pages, however, this drastically increases the size of the repository. Each doxygen HTML file is ~3.5MB, which is normally fine, but there are hundreds of these in the history for that branch. I believe this is the primary reason for the bloat in the Zapdos repo as it stands right now. The fix for this aspect of the repo size is to overwrite history for this branch every time a new website is generated (so only one commit is ever present in gh-pages at any given time, and only one copy of any doxygen HTML file is present in the git file reference list).

The "nuclear" option is to remove some of the worse offenders in the above list from history altogether, which is tricky and error-prone. So, I don't think we'll be exploring this option, but I think the gh-pages change will help alot.

[smpeyres]

I like the idea of modifying the gh-pages branch/setup to help get rid of the extra copies of files, if I understand correctly.

Nonetheless, some of these files are still huge!

1. It looks like the 'problems' directory is no longer a thing for the master nor devel branch. Some of the bloat from the Schottky_emission problems then are hanging out in branches that may no longer be used.
2. Can't seem to find the giant .txt files on the master nor devel branch, either.
3. Most of these are tests. This can definitely be improved by coarsing the mesh, increasing time steps, reducing complexity, reducing number of time steps, etc for some or most of these.

[cticenhour]

The current gh-pages is now a single commit, which I set up manually for now. I have also deleted all stale branches except for one, which involves code for an out-standing issue. So this should also help. (1) and (2) is related to the fact that these files exist in Zapdos git history. Some of these no longer exist in the current branches as they are, but still exist as historical references so that they could be returned to. Removing them would involve changes to history, which is not usually a good thing to do.

Regarding (3), another good point when performing general testing (and one we adhere to more strongly than we used to), but still undercut by the fact that these files are still present in history.

With the changes I've performed today, I cloned a fresh copy of Zapdos and received ~750MB. Still large, (I think the gh-pages branch with only one commit could also be ~400MB of this), but much more manageable than before! If you would, please clone a fresh copy of Zapdos in a temp directory to see if this is consistent with your machine.

[smpeyres]

Yes, ~750 MB is what my clone is reported. Much faster!