Chatting with Rob

[BadgerOnABike]

Just trying to document our chats on things that may hold up opening source in a more cohesive manner

Multithreading systems moving to arrays. New stuff works on linux and we don't need the old stuff. (99% sure I misinterpreted this)

Getting rid of the wtime library in favor of a standardized approach to represent time. Started using a newer library that updates a lot due to geopolitics. Moving away from wtime is a big change and breaks a lot. [Manage this after opening source]

We have no test tool currently.

Bad error messages.

Validation of the code base.

Neal wanted to spend more time on managing fault tolerance. How do we deal with systems dying during a distributed workload.

[RobBryce]

For clarification:

1.) multithreading - it may be in Github already, or maybe in Redmine. I think we already had that. History: 18ish years ago, I wrote my own library to deal with thread groups - multiple threads that work on the same task. And they are managed by the thread that starts the work. It's worked well, and when we started to integrate openmp, that version of openmp did not support traversing linked lists (it requires either locking of structures, or an atomic instruction call to flush the cache, dereference and return a pointer, update memory with the new pointer value - which doesn't exist). Anyway, experiments show that you need to queue up a certain amount of work per thread before synchronization along a list, or you spend more time dealing with synchronization of threads, than doing the work. That's where my library worked well. openmp really works best with arrays and work of equal size so it can pre-divide up the set of work among available threads that it manages explicitly. Fast forward to today, and openmp and C++ have matured, and may have options for us to walk along a linked list, and certainly have ways to deal with arrays of work where the work per array element doesn't have to be the same. So let's drop custom solutions where we can.

[RobBryce]

2.) WTime - I believe this work item is also in Github - it works, and is extremely efficient. But there are modern alternatives that deal with timezones in a manner that we don't have to maintain. Pulling out this object-oriented construct for systime which isn't is going to be painful, even if we wrap it with similar constructs. I'm not sure if this is a trivial task for a newbie into the source code, and I'd like to rearchitect some stuff at the same time (mentioned a year ago or so) to clean up some stuff that had to be done the way it was, to keep COM objects separate from each other.

[RobBryce]

3.) GDAL / PROJ - discussed, shouldn't be an issue if GDAL follows its own rules for major and minor version numbers - which they have broken in the past. Also discussed how environment variables already exist to deal with updating the code to new versions of GDAL, or compile to many targets (after a clean between the builds). Discussed priority of matching to versions of R (not consistent with itself) or maybe Python, and various pain points along the way. Some issues go away after dropping COM because we only have to work with the directory structure and the system path, rather than the old way where COM may take us to a completely different directory where we have trouble finding to get to the correct DLL's for other dependencies.

[RobBryce]

Note that this stuff has already been discussed with Franco and this is for situational awareness and to clean up comments (e.g.) around updating GDAL.

[RobBryce]

4.) 2c in one Github item - seems like a reasonable approach to use C or Python approaches for string formatting (%d, %s were cited as acceptable) for subscenarios. So I won't research a common expression builder that (eg.) BOOST or another library implements. E.g. no approach unique to PowerShell or shell in Linux will be used. For now, we'll just use %d for index into a subscenario, and if we need to revisit it in the future, we will then.

[BadgerOnABike]

3.) GDAL / PROJ - discussed, shouldn't be an issue if GDAL follows its own rules for major and minor version numbers - which they have broken in the past. Also discussed how environment variables already exist to deal with updating the code to new versions of GDAL, or compile to many targets (after a clean between the builds). Discussed priority of matching to versions of R (not consistent with itself) or maybe Python, and various pain points along the way. Some issues go away after dropping COM because we only have to work with the directory structure and the system path, rather than the old way where COM may take us to a completely different directory where we have trouble finding to get to the correct DLL's for other dependencies.

The more I think about this, the more I wonder how other systems do it. but moving from COM does seem like it'll help. I guess I'd hope we can get to a point of GDAL >= 3.x or >= 3.2. As you mentioned there were breaking changes and I think those are an inevitability, but if every update can be allowed to function that would likely be a positive. I know when they moved into 3.4 there were a number of performance enhancements in the changelogs specifically with raster writing and it would be lovely to leverage those when/if they come!

Found a version agnostic example:
https://github.com/qgis/QGIS/blob/master/cmake/FindGDAL.cmake

[spydmobile]

Found a version agnostic example:
https://github.com/qgis/QGIS/blob/master/cmake/FindGDAL.cmake

I ❤️ Open Source 😉

[RobBryce]

Moving from COM saves a lot because the registry was used to bypass the path for looking for DLL's. The registry said where to find the DLL (and assumed it was unique on the system) that implemented the COM object.

[RobBryce]

There were also changes from 2.x to 3.x, in addition to the change(s) on the API to 3.2.

[spydmobile]

Hi Boys, just chiming in again, I think the idea is that we want to start coding to DYNAMIC values (Correct me if im wrong @BadgerOnABike ) and so @RobBryce - How do we do it? Is the example I provided of any use? If not what are the roadblocks? @RobBryce your last 2 comments, dont seem to shed light, only add more confusing details. Again, I apologize if I misunderstand here. Currently the hard version approach which once held more benefit than problems, is now reversed, more often than not, this approach is hamstringing us, and of course, this is NOT how we do things if we want automated builds, with automated builds EVERYTHING must be abstracted and dynamic.

[RobBryce]

C and C++ are statically linked programming languages. To use a DLL, we need to link to a LIB file which provides entry points into the DLL. As @BadgerOnABike mentioned, GDAL now has rules they've imposed (but haven't consistently followed) in revision control of their API. So in theory, the LIB file shouldn't change even if the DLL changes, so long as the major version stays the same. That's about as dynamic as we can get without creating something like COM - that was one of the roles of COM to allow purely dynamic linking for statically compiled and linked languages.