Suggestion for new Makefile

[ggs67]

I have altered the Makefile and would like to know if this could be an acceptable contribution and, if so how to contribute, via fork (overkill?) or just post here by attachment ?

The changed/new features;

• result outputs are generated in the Prime directory tree under results/run_ (results directory is currently expected to exist) being automatically incremented for each run
• the above change from temporary directories allows for the following changes:
  • "make report" can be used to regenerate the report of the last run
  • future extension to generate report of any previous run "export RUN= ; make report"
  • future extension to generate average report of all runs (interesting on systems that have other workloads as well)
  • "make purge-results" removes all results
  • "make purge-last-result" remove last run results
  • future "export RUN= ; make purge-result" to remove specific run results
• resolved some duplicate code for reporting
• "export LANGUAGE= ; make language" (example CPP) allows to run all solutions of one given language. Interesting if you try to develop a faster solution

Future features:

• "make distclean" to purge all results and remove the docker container
• in a second phase add "make clean" to purge results and solution compilation outputs (this will then be added to distclean)

NOTE: usually one would implement clean first and then distclean, but the solution cleaning needs some thinking in this case before implementation so maybe distclean gest first while not being real distclean without solution-cleaner

solution-cleaner

I have several possible paths for the solution cleaner. The traditional path would be that each solution get invoked by the cleaner to clean its own files with its knowledge about itself.

The second path would be to generate a file cache at first run of make and to clean every file not in the cache including itself

And last but not least, use git ls-files to identify the native files.

Each of these paths has its own advantages and drawbacks, and I am currently in favor of solution 2.

The reason against the first path is that solutions are added from many sources and many already there which may make solution local cleaning a hard work as many contributors may not update their solution, and the cleaner will only work when all solutions have been updated

The reason against path 3 is that it assumes that the environment is a checkout of git not installed by any other means (like tarball). This could be solved by adding the git URI but this then requires git and access to github, both potential reasons why git was not used in the first place (for example system without internet access).

But as I said, this is still under investigation. I did only detail it here for discussion)

[marghidanu]

I have altered the Makefile and would like to know if this could be an acceptable contribution and, if so how to contribute, via fork (overkill?) or just post here by attachment ?

Why do you need to alter the Makefile? In the future, we will provide a way for these reports to be uploaded and made available as a source of comparison for everyone.

You can already do a make report OUTPUT_DIR=... to specify the output directory same for the make benchmark OUTPUT_DIR. We opted to use a temporary directory to make the cleaning process easier.

[ggs67]

Well, I guess you replied yourself ;-) From all the features I named you came up with one that is somehow possible right now. So the first reason are all the other features. But even the one you name, with temp directory I do not understand how cleaning would be easier, its a mess when not on a personal system. The mktemp is not templated and there may be many others form other processes around and I need to cleanup manually if I can't wait for the reboot. On some hosts mktemp to /tmp is prohibited for normal users, ... I need to note down each single directory to find it back (even mktemp templating is not 100% sure and could collide on the base/suffix). and, and, and... In general /tmp temporaries are bad for general use, I prefer something that can be run on a campus system as well as on a personnel system

I also don't see (I admit I did not test before I changed) how make report OUTPUT_DIR=... can work as in your Makefile report depends on benchmark and will run the benchmarks again. This was actually one of the first things I change, separating reporting from benchmarking to allow separate reporting without running the benchmarks. In addition it allows to cleanup the Makefile by eliminating redundant code for reporting in "all" and "one"

I don't see how manually seek and rm is easier cleanup than "make purge-results"

Regarding the upload this surprises me as the goal was more a language than a system benchmarking. But even with uploading I believe that my apporoach (as opposed to cryptic temps) is the better one allowing easy separate upload from runnign the benchmark and allowing averaging n runs.

[marghidanu]

From all the features I named you came up with one that is somehow possible right now.

Of course, I did, these are options that are available now.

I'm working on several features that will drop the build and run from the Makefile directly, instead, this will move to code and the temporary files will not be used anymore; all execution results will be aggregated in memory and dumped in the preferred when the report is generated.