Feature/gaugefield unity

[maddyscientist]

This PR's primary purpose is to unify the cpuGaugeField and cudaGaugeField classes, along with an assortment of other cleanups.

• Removed cpuGaugeField and cudaGaugeField classes, we have now only GaugeField
  • Similar to ColorSpinorField the communiciation routines for host and device are still separate within the GaugeField class, unification of these is deferred until later
  • Added move and copy constructors and assignment operators for GaugeField to facilitate significant cleanup
  • Raw field pointer is now obtained using the data() method. This is to align with STL design. Though here we also have an optional template parameter that can be used to perform a cast to the desired type. (Also applied to ColorSpinorField and CloverField)
• Introduced a smart pointer class quda_ptr, which provides heterogenous memory allocation
  • Deployed in GaugeField, ColorSpinorField and CloverField
• Introduced qudaMemcpy and qudaMemset functions that accept a quda_ptr
  • This allows for unified calls to memcpy and memset without needing to worry about the location of the field
• Introduced the concept of a profile stack, with pushProfile and popProfile functions, together with getProfile for getting a reference to the current top of stack profile.
  • This facilitates a significant cleanup of the profiling, we can now profile in arbitrary functions without needing to pass the profile to it explicitly
• Profiles now no longer complain about being double started: we now have a ref counter to allow things to just work (as long as the number of starts and stops match)
• Significant cleanup of the interface for the non-solver functions to utilize the technology introduced as part of this PR
• Jitify
  • Updated the version of Jitify included with QUDA
  • Some fixes for issues that creeped in (e.g., moved some function definitions out of headers to cpp files to avoid the run-time compiler from parsing them)

Outstanding items

• Add quda_ptr for ROCm
• Apply clang-format

[hummingtree]

Have verified (again) that CPS works good with this branch.

[weinbe2]

Tentative approve, I noticed a few cosmetic things, and this needs a fresh merge of develop + clang-format. After that it's good to go!

[maddyscientist]

@SaltyChiang I have merged in your changes into this branch, that will shortly be merged into develop.

One line I have not included is this one:

gauge[d] = (param.gauge) ? ((void **)param.gauge)[d] : nullptr;

as this line doesn't make sense to me. If it's a reference to a pre-existing field, why should the value be a nullptr? Can you explain why this change is needed?

Thanks

[SaltyChiang]

Hi @maddyscientist, there are some if statements about the input gauge link, like

quda/lib/interface_quda.cpp

Line 3873 in 6e90b49

if (longlink) {

and

quda/lib/interface_quda.cpp

Line 3903 in 6e90b49

if (ulink) {

. I believe the logic here is originally used for QUDA_MILC_GAUGE_ORDER, but this will fail if QUDA_QDP_GAUGE_ORDER is used here. For example, if the ulink is nullptr here, the unitarized fat7 link will not be calculated, which is right; but cpuUnitarizedLink defined in

quda/lib/interface_quda.cpp

Line 3847 in 6e90b49

cpuGaugeField cpuUnitarizedLink(gParam);

fails now as you cannot get ((void **)nullptr)[d]. A null QUDA_QDP_GAUGE_ORDER field should be something like void *ulink[4] = {nullptr, nullptr, nullptr, nullptr} but now the if (ulink) will return true which we do not expect.

So I made all four reference fields nullptr when the input value itself is nullptr and I think this will not affect the existing logic.

[maddyscientist]

@SaltyChiang thanks for the explanation. I have fixed my branch accordingly (4c308f6), so it should work for your usecase. If you could test this branch, that would be appreciated, thanks.

[weinbe2]

Looks great, thanks @maddyscientist !

[SaltyChiang]

Hi @maddyscientist, very sorry for the late feedback. I think it should be !unitarizedLink.StaggeredPhaseApplied() && param->staggered_phase_applied here.

quda/lib/interface_quda.cpp

Lines 3748 to 3749 in 0c016af

	if (unitarizedLink.StaggeredPhaseApplied() && param->staggered_phase_applied)
	unitarizedLink.applyStaggeredPhase();

Edit: I believe gauge_param.location should be changed to QUDA_CUDA_FIELD_LOCATION here.

quda/lib/interface_quda.cpp

Lines 765 to 774 in 0c016af

	case QUDA_SMEARED_LINKS:
	gauge_param.create = QUDA_NULL_FIELD_CREATE;
	gauge_param.reconstruct = param->reconstruct;
	gauge_param.setPrecision(param->cuda_prec, true);
	gauge_param.ghostExchange = QUDA_GHOST_EXCHANGE_PAD;
	gauge_param.pad = param->ga_pad;
	cudaGauge = new GaugeField(gauge_param);
	copyExtendedGauge(*cudaGauge, *gaugeSmeared, QUDA_CUDA_FIELD_LOCATION);
	break;
	default: errorQuda("Invalid gauge type");

[maddyscientist]

@SaltyChiang thanks for pointing put my bug. I've pushed a fix to my next PR that will be merged (#1393). This is a fairly simple PR, so will likely be merged soon.

[SaltyChiang]

@maddyscientist thanks for the fix. You might miss another issue that will happen while saving the smeared gauge through saveGaugeQuda(). I updated the comment above and I repeat it here.

Edit: I believe gauge_param.location should be changed to QUDA_CUDA_FIELD_LOCATION here.

quda/lib/interface_quda.cpp

Lines 765 to 774 in 0c016af

	case QUDA_SMEARED_LINKS:
	gauge_param.create = QUDA_NULL_FIELD_CREATE;
	gauge_param.reconstruct = param->reconstruct;
	gauge_param.setPrecision(param->cuda_prec, true);
	gauge_param.ghostExchange = QUDA_GHOST_EXCHANGE_PAD;
	gauge_param.pad = param->ga_pad;
	cudaGauge = new GaugeField(gauge_param);
	copyExtendedGauge(*cudaGauge, *gaugeSmeared, QUDA_CUDA_FIELD_LOCATION);
	break;
	default: errorQuda("Invalid gauge type");

[maddyscientist]

Thanks again @SaltyChiang. Fix is here 44c4100 (in #1393).