Practical questions for monitoring applications

[vsoch]

Hi! I tested out parca today for the default tutorial on a small Kubernetes cluster, and it's awesome! I was able to run a few pods with an HPC (MPI) application, lammps, and see an expected pattern:

I have some practical questions. First, I'm looking to run many different HPC applications and collect metrics. Is the easiest way to do this to run one HPC application per CRD (with some associated pod with labels parca filters to) and then assess performance at the level of the pod? Or is there a way to get a more fine grained view of different applications running in the same container over time?

And then the only metric I see in this default is related to CPU. How do I add others?

Second, what is the best way to save and then retrieve data? it looks like I can export data likely from the pod that the agent is sending to - I'm guessing there is a database backing that, and that is exportable to keep. Is that the best way to do it?

Many thanks!