Using decorators to link pybind geometry classes to pure Python classes with Numpy arrays

[evbernardes]

This was mostly just a little personal experiment to see what was possible to do with decorators, so bear with me. This is also using legacy geometry elements.

I decided to create the abomination that I will be describing in this post. This is definitely quite ugly, but I wonder if something like this could be actually implemented in a better way or if someone could point out to me where the bugs would be!

"Problems" I want to solve

Problem 1

One of the "problems" I wanted to solve is that all properties of the geometry classes are of type std::vector<Eigen::Something>. Example:

>>> from open3d.geometry import PointCloud
>>> PointCloud().points
std::vector<Eigen::Vector3d> with 0 elements.
Use numpy.asarray() to access data.

And also, when giving points, I have to first transform them to Eigen classes. This fails:

>>> import numpy as np
>>> points = np.random.random([2, 3])
>>> PointCloud(points)
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported

But this works:

>>> from open3d.utility import Vector3dVector
>>> PointCloud(Vector3dVector(points))
PointCloud with 2 points.

(As far as I know, this is not a problem with Tensor classes).

Problem 2

There are also problems with inheritance, which was the main problem I had, as I wanted to implement some other stuff with both pointclouds and meshes.

>>> class Test(PointCloud):
>>>    pass
>>>
>>> Test()
TypeError: pybind11::init(): unable to convert returned instance to required alias class: no `Alias<Class>(Class &&)` constructor available

As far as I know, this is also a problem with Tensor classes:

>>> from open3d.t.geometry import PointCloud
>>> class Test(PointCloud):
>>>    pass
>>>
>>> Test()
RuntimeError: Tried to call pure virtual function "GeometryBase::"

Solution

My idea to solve Problem 2 was to create a class that has an internal property that points to the Open3D equivalent class. But having to call it all the time, then convert everything to Eigen classes, and take the return values and convert them to numpy arrays was a bit clunky, so I decided to create some ugly decorators to do this for me.

Decorator 1: converting returns to Numpy arrays

This decorator takes the results of methods and, one of the results is a member of open3d.utility, it converts it to a Numpy array recursively:

def _in_module(value, module):
    return type(value).__module__ == module.__name__

def _convert_args_to_numpy(args):
    if isinstance(args, (list, tuple)):
        for i, value in enumerate(args):
            args[i] = _convert_args_to_numpy(value)
    
    elif _in_module(args, utility):
        return np.asarray(args)
    return args

# this is the decorator
def result_as_numpy(func):
    def wrapper(*args, **kwargs):
        return _convert_args_to_numpy(func(*args, **kwargs))
    return wrapper

Decorator 2: converting args to Eigen arrays

The second, arguably the ugliest, would be to do the inverse: check all elements on the args and see if there is an equivalent array type on open3d.utility. This string-based solution is obviously ridiculous, but this could be replaced by some actual implementation inside utility that is a bit more general.

NOTE: this would also convert any self argument to the internal as_open3d element, which is important.

def _convert_args_to_open3d(*args, **kwargs):
    args = list(args)
    
    for i, arg in enumerate(args):
        if hasattr(arg, '_open3d'):
            args[i] = arg._open3d
        
        if isinstance(arg, (list, tuple)):
            arg = np.array(arg)
        
        if isinstance(arg, np.ndarray):
            if arg.ndim > 2:
                continue
            
            if arg.ndim == 1:
                if arg.dtype == int:
                    args[i] = utility.IntVector(arg.tolist())
                else:
                    args[i] = utility.DoubleVector(arg.tolist())
                    
            if arg.ndim == 2:
                d = 'i' if arg.dtype == int else 'd'
                class_name = f'Vector{arg.shape[1]}{d}Vector'
                
                converter = getattr(utility, class_name, lambda x: x)
                args[i] = converter(arg)
                
    return tuple(args), kwargs

# this is the decorator:
def args_to_open3d(func):
    if func is None:
        return None
    
    def wrapper(*args, **kwargs):
        args, kwargs = _convert_args_to_open3d(*args, **kwargs)
        return func(*args, **kwargs)
    return wrapper

Decorator 3: a fusion of 1 and 2:

def to_open3d_and_back(func):
    if func is None:
        return None
    
    def wrapper(*args, **kwargs):
        args, kwargs = _convert_args_to_open3d(*args, **kwargs)
        return _convert_args_to_numpy(func(*args, **kwargs))
    return wrapper

Decorator 4: the class decorator

Ok, I lied. THIS is the ugliest one yet. The idea is to copy the definitions of all attributes of the original class, but using the decorators (including for the properties), creating a new __init__ that actually initializes the original class.

def link_to_open3d_geometry(original_class):
    def _copy_from_open3d(cls):
        
        setattr(cls, '__open3d_class__', original_class)

        # create new __init__ that generates an internal instance
        @args_to_open3d
        def new_init(self, *args, **kwargs):
            self._open3d = original_class(*args, **kwargs)
        setattr(cls, '__init__', new_init)

        # removing two last subclasses, 'object' and 'pybind11_object'
        for subclass in original_class.__mro__[:-2]:
            for attr_name, attr_value in subclass.__dict__.items():
                if attr_name == '__init__':
                    continue

                if attr_name not in cls.__mro__[0].__dict__:
                    if callable(attr_value):
                        setattr(cls, attr_name, to_open3d_and_back(attr_value))
                    
                    elif isinstance(attr_value, property):
                        
                        # Apply decorators to the original getter and setter functions                        
                        getter = to_open3d_and_back(attr_value.fget)
                        setter = args_to_open3d(attr_value.fset)
                        
                        # Create a new property with the decorated getter and setter functions
                        new_property = property(getter, setter, attr_value.fdel, attr_value.__doc__)
                        
                        setattr(cls, attr_name, new_property)
                    else:
                        setattr(cls, attr_name, attr_value)
        return cls
    return _copy_from_open3d

Base linker class

The idea is then to finally just create a base class that gives the internal instance as a property:

class Open3D_Geometry():
    
    @property
    def as_open3d(self):
        return self._open3d

Testing

Here I demonstrate the use of this experiment. Suppose I want to use pointclouds, then:

import open3d

@link_to_open3d_geometry(open3d.geometry.PointCloud)
class PointCloud(Open3D_Geometry):
    pass

Now I can init the PointCloud normally (note that it automatically copies the __repr__ element too):

>>> points = np.random.random([5, 3])
>>> pcd = PointCloud(points)
>>> pcd
PointCloud with 5 points.
>>> pcd.as_open3d
PointCloud with 5 points.

I can set the points normally from the property setter, and it sets the points inside of the internal object:

>>> pcd = PointCloud()
>>> pcd
>>> PointCloud with 0 points.
>>> pcd.points = np.random.random([5, 3])
>>> pcd
PointCloud with 5 points.
>>> pcd.as_open3d
PointCloud with 5 points.

I can use methods like translate for example with no problem:

# note that I have to specify zeroes as float here:
>>> pcd = PointCloud([[0., 0., 0.]])
>>> pcd.translate([10, 0, -10])
>>> pcd.points
array([[ 10.,   0., -10.]])
>>> np.asarray(pcd.as_open3d.points)
array([[ 10.,   0., -10.]])

Indexing also words:

>>> pcd.points = np.random.random([2, 3])
>>> pcd.points
array([[0.60446516, 0.86566047, 0.9840189 ],
       [0.3483691 , 0.85513877, 0.33398857]])
       
>>> pcd.points[0] = [-1]*3
>>> pcd.points
array([[-1.        , -1.        , -1.        ],
       [ 0.3483691 ,  0.85513877,  0.33398857]])

>>> np.asarray(pcd.as_open3d.points)
array([[-1.        , -1.        , -1.        ],
       [ 0.3483691 ,  0.85513877,  0.33398857]])

Conclusion

And this is the end of the experiment, in which I achieved what I wanted. There are of course some issues there, specially coming from the bad conversion method from args to Eigen types, but that could potentially be solved by some kind of proper compatibility utility converter.

What are your thoughts? Where would it fail? Could something actually usable and safe be implemented in a similar manner, hipothetically, in order to improve the compatibility with Python?