Skip to content

Work in progress to improve Python 3 support

Michael Kirschner edited this page Apr 6, 2022 · 25 revisions

Introduction

This document lists issues we detected related to compatibility between Iron Python 2.7 and CPython 3.8 (via Python.NET) engines. Next to each issue we provide a status of it:

  • Fixed: The issue has been fixed. Either a release version is specified or it can be tested with our latest daily build.
  • Planned: The issue is either being worked on or is planned for a future release.
  • Not planned: The issue has been noted but has not been considered to be worked on.

We ask the community to provide feedback about these or other issues we might have overlooked.

Migrations from version 2 to 3 issues are purposely left out from this document, as they will be handled by a Migration Assistant which is currently in development. Documentation about it will be added later.

Iron Python and Python.NET

(If you are already aware of what these libraries are, you may skip this section, but if not, read on.)

Dynamo has supported Python execution through the Iron Python runtime for a long time, even though the Engine selector surfaced that only recently. As you may have heard, Iron Python currently supports Python 2, but it doesn't support Python 3. Something else you should know is that Iron Python does not use CPython, which is the standard Python runtime, but instead implements its own runtime over the .NET Framework. If you are not familiar with .NET, let's just say it is a framework to develop applications. It's pretty popular, even Dynamo and other tools you love are built on top of it 😄

The CPython engine, on the other hand, uses the standard Python runtime, also called CPython. Since Dynamo and its libraries are based on .NET, you need somethings that acts like a "glue" to make execution and data flow correctly between the two. This is achieved by using a library called Python.NET.

So if you can write Python and it will run in any case, you may ask yourself why we bring .NET to the discussion. Basically there are three reasons:

  1. Some of the issues mentioned here deal with incompatibilities around .NET integration when comparing the two engines
  2. It helps to understand what's going on behind the scenes: in one case everything runs in .NET, while in the other execution and data flows between CPython and .NET.
  3. Most of your favourite applications here at Autodesk use the .NET framework, so if you want your Python node to be able to do things in these environments (such as talk to the Revit API) then your Python code needs a way to talk to them!

Issues

Python primitives are not .NET objects (Not planned)

Iron Python supports calling .NET functions on Python objects. This can be done for a big variety of classes and functions, what follows are only a few examples:

import clr

# Calling 'ToString' on a Python int
python_int = 1
my_str = python_int.ToString()

# Calling 'Add' on a Python list
python_list = ['hello']
python_list.Add('world')

OUT = my_str, python_list

Those examples work because for Iron Python every Python object defined is backed up by a .NET object representation. This means a Python int also has the behaviour of a .NET int, which happens to implement ToString as expected. The same thing happens for the other examples.

The previous code does not work for native Python, because the Python native types don't implement these functions. However, in Python you could achieve the same result by doing things differently:

# Convert a Python int to a string using plain Python
python_int = 1
my_str = str(python_int)

# Adding an element to a list using plain Python
python_list = ['hello']
python_list.append('world')

OUT = my_str, python_list

Certain types cannot be returned from a Python node (Fixed in 2.8)

An easy way to reproduce this issue would be to write the following code in a Python node, with the CPython3 engine:

s = { 'hello', 'world' }
OUT = s

The previous code tried to return a Python set from a Python node. This type was not supported, and in previous versions also led to a crash due to a bug. Support for Python types has been improved so set is now supported along with other collection types.

TypeError : No method matches given arguments (Fixed in 2.8)

A TypeError is an error that might be seen from a .NET function call that receives Python types as arguments. One example that is reproducible in Dynamo 2.7 with the addition of CPython3 is:

import clr
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *

point = Point.ByCoordinates(0,0,0)
OUT = point

In this case, the TypeError issue was caused by an inability of Python.NET to determine that an overload of Point.ByCoordinates should have been called. The overload Point.ByCoordinates(double x, double y, double z) could have been called by converting the int inputs with values of 0 to float inputs with values of 0.0. However, Python.NET did not attempt this and failed with the TypeError: No method matches given arguments message.

This specific problem for int to float conversions was fixed. Unfortunately, the message is quite generic and might describe other failed conversions or scenarios where the method was not found due to being misspelled, for instance. If you run into this type of error, please reach out to us with detailed information about the issue.

No line number on Python errors (Fixed in 2.8)

Most often than not, the CPython3 engine would omit the line number where an error occurred, making it harder to debug long Python scripts. Errors of type TypeError would never provide a line number, while others of type SyntaxError would.

This was a rather inconvenient omission in Python.NET that was fixed.

Python with blocks don't call Dispose for wrapped IDisposable .NET objects (Fixed in 2.11)

The following example shows what this issue is about:

with ClassThatImplementsIDisposable() as something:
  pass

Iron Python would call Dispose() automatically for something when exiting the with block. This is not the case for Python.NET, so developers should be aware and call Dispose() explicitly if required.

Very big integers are not supported (Fixed in 2.8)

Python supports through its int type integer values of any size. However, taking that value from Python to .NET requires special considerations, because the target type needs to be able to support very large values as well. Iron Python converts to BigInteger automatically, so the following code example would work:

# That is bigger than what would fit in an Int64
x = 11111111111111111111
OUT = x

Python.NET did not support converting large integers to BigInteger out of the box, so additional work was required to make it work.

Custom Python objects cannot transcend a Python script node (Fixed in 2.8)

Iron Python allowed to return objects of custom classes defined in the same Python script node. An example of this would be the following:

class Dog:
  kind = 'canine'
  
  def __init__(self, name):
    self.name = name

OUT = Dog('Rex')

Even though the output from that node would not be of much use in Dynamo, it could be reused in another Python node, allowing certain degree of code reuse between nodes.

Conversion differences between engines for custom objects that are iterable (Fixed in 2.8)

Related to the previous issue, we have found examples of objects that are subject to conversion in the CPython3 engine, while they would be passed unconverted using the IronPython2 engine. To better illustrate this, here is an example of such a class:

class MyIterable:
    def __str__(self):
        return 'I want to participate in conversion'
    def __iter__(self):
        return iter([0,1,2,3])
    def __getitem__(self,key):
        return key

OUT = MyIterable()

The IronPython2 engine would return this object as an IronPython.Runtime type, which can be passed around between Python nodes. On the other hand, for the CPython3 engine, this would be recognized as a collection and converted to a List. This is because the class implement __iter__ and __getitem__, which is something only the CPython3 engine checks for. This check is required to support a wide range of built-in collections, so simply removing it was not an option we cared for.

But what if the author of this class wants the old behaviour offered by the IronPython2 engine and intends to move this object between Python nodes? This is still possible using a Dynamo specific attribute we have introduced: __dynamoskipconversion__

class MyIterable:
    def __dynamoskipconversion__(self):
        pass
    def __str__(self):
        return "I don't want to participate in conversion"
    def __iter__(self):
        return iter([0,1,2,3])
    def __getitem__(self,key):
        return key

OUT = MyIterable()

By including this attribute, even with an empty implementation, Dynamo will return this object as-is, instead of attempting any conversion.

dir does not list classes or child namespaces of a .NET namespace (Fixed in 2.8)

The dir function allows to get a list of the attributes defined for a given Python object. Here is an example of how it can be used to get the list of attributes for a .NET namespace:

import clr
clr.AddReference('DSCoreNodes')

import DSCore

OUT = dir(DSCore)

When run with the IronPython2 engine, the previous code would give as a result the list of classes and namespaces contained in the DSCore namespace from Dynamo. However, with the CPython3 engine, the result would be mostly empty, only including some internal attributes. This was fixed by enabling the preload of modules in Python .NET.

getattr returns AttributeError for some attributes listed with dir (Fixed in 2.8)

When requesting certain internal attributes using getattr, the CPython3 engine would fail with an AttributeError stating that the requested attribute did not exist. This would happen even for attributes obtained from dir, which should be guaranteed to exist. Here is an example of a case that failed:

import clr
clr.AddReference('DSCoreNodes')

import DSCore
# This failed with AttributeError!
getattr(DSCore, '__delattr__')

This was a problem originating in the Python .NET library which should now be fixed.

Lack of support for extension methods (Fixed in 2.8)

One nice feature Iron Python provides is the ability to call extension methods similarly to how it is done in .NET. For a quick overview of extension methods, please take a look at this brief article. What follows is an example of how it could be used in a Python script:

import clr
# Calculator is a .NET class implemented in an external library
# It defines a 'Sum' function to sum two numbers
clr.AddReference('ExternalLib')
from ExternalLib import Calculator

# MyExtensions is a library that defines extension methods
# For Calculator, it defines the 'Sub' extension to subtract two numbers
clr.AddReference('MyExtensions')
import MyExtensions
clr.ImportExtensions(MyExtensions)

calc = Calculator()
result = calc.Sum(1,2)
# Note that 'Sub' can be called on the instance, even if it was
# defined as an extension
result = calc.Sub(result,3)
OUT = result

The previous code example would work without issues using the IronPython2 engine, given the mentioned libraries were loaded in Dynamo. For CPython3, on the other hand, several things were missing to make this work.

In order to fix this, the ImportExtensions function had to be added to Python .NET, along with other internal changes that were required to allow calling extension methods.

Extension methods are not listed when using dir on the instance (Not planned)

Related to the previous issue, one thing Iron Python also allows is listing extension methods when using dir. Imagine the previous example but changing the final lines like this:

...
# The result of 'dir' includes 'Sub' only in Iron Python!
result = dir(calc)
OUT = result

Making the previous code example work correctly in Python .NET would require more involved changes in the library. Currently, we haven't found this to be a blocker for known workflows so we have not planned to work on this yet. If you do happen to be blocked by this, please let us know.

Accessing a .NET property or enum named None fails with SyntaxError (Not planned)

This is an error that might surface in scripts that use System.Windows.Forms, like the following example:

import clr
clr.AddReference('System.Windows.Forms')
from System.Windows.Forms import DockStyle

dockstyle = DockStyle.None
OUT = dockstyle

The fact that DockStyle.None crates a SyntaxError when using the CPython3 engine is due to changes in Python 3 with regards to how the special None keyword can be used. This fact makes the problem impossible to fix, although there are workarounds in case you happen to need this, like the follow snippet shows:

import clr
clr.AddReference('System.Windows.Forms')
from System.Windows.Forms import DockStyle

dockstyle = getattr(DockStyle, 'None')
OUT = dockstyle

Python classes cannot implement .NET interfaces (Not planned)

This is probably an advanced usage scenario of Python and interoperability with .NET. Here a Python class is defined in such a way that it implements a .NET interface. The idea behind this is that instances of the class can be created and passed as arguments to methods accepting them. Here is an example of how this could look like:

import clr
clr.AddReference('SomeLibrary')
from SomeLibrary import SomeInterface, SomeClass

class MyClass(SomeInterface):
  def __init__(self):
    pass

inst = MyClass()
result = SomeClass.SomeMethodTakingSomeInterface(inst)
OUT = result

Given a valid library was provided, the previous code sample would work without issues in the IronPython2 engine. When using CPython3, however, you can get TypeError : interface takes exactly one argument or TypeError : object does not implement SomeInterface, depending on the interface and method definitions. The required changes in Python .NET to make this work seem big when compared to the relevance we have detected for this use case, so we have decided to not plan this work yet. Please let us know if this is blocking you.

.NET enums are returned as integers from Python (Not planned)

While Iron Python preserves type information for enums, Python .NET loses it and deals only with their corresponding integer value. As a consequence of this, returning an enum from a Python script will result in a number when using the CPython3 engine. While it's possible to convert a number back into the corresponding enum, this difference in behaviour between engines may lead to subtle compatibility issues, so please be aware.

Parameters using ref and out work differently (Not planned)

Calling .NET methods that use parameters with the ref and out modifiers in Iron Python relies on a special function to create a reference. Here is a code sample showing how this works in Iron Python when calling the Dictionary.TryGetValue function:

import clr
from System.Collections.Generic import Dictionary

d = {"a": 1, "b": 2}
dict = Dictionary[str,int](d)
value = clr.Reference[int]()
key = "a"
found = dict.TryGetValue(key,value)
if found:
  OUT = value.Value
else:
  OUT = -1

Python .NET, on the other hand, does not provide support for clr.Reference. However, ref and out parameters are supported in a different way. They need to be provided when calling the method but they are considered as inputs only. The output is actually obtained from the result of the function call, which is actually a Python tuple. Here is an equivalent code sample using the CPython3 engine:

import clr
from System.Collections.Generic import Dictionary

d = {"a": 1, "b": 2}
dict = Dictionary[str,int](d)
dummy = 0
key = "a"
found,value = dict.TryGetValue(key,dummy)
if found:
  OUT = value
else:
  OUT = -1

In the previous example, dummy is used just to comply with the method signature, while value is filled with the actual value obtained from the dictionary.

For completeness, here is a code sample showing how to call a method using ref parameters, although it works in a very similar fashion:

import clr
from System.Threading import Interlocked

original = 1
replacement = 2
old,new = Interlocked.Exchange(original, replacement)
OUT=old,new

Import module works differently in IronPython and CPython (WIP)

While IronPython will happily reload certain modules (usually those imported from .py files, though the exact behavior is quite complex) - CPython / Python .Net will not. This is standard CPython behavior. See here for more info: https://docs.python.org/3/reference/import.html

When CPython encounters an import statement it will check sys.modules to see if it has cached that module. If it has, it will simply return the cached module.

As a first improvement we have added an option to manually reload certain modules. Accessible from the Python preferences dialog, pressing the Reset CPython button will currently perform a few process wide reset steps. Currently it will:

  • reload modules that have __file__ attributes that point to .py files and where the module name is not __main__
  • attempt to cleanup Python objects created using the now out-of-date version of the reloaded module.
  • mark the graph as needing to be re-executed.
  • the reloaded modules will be logged to the Dynamo console for debugging purposes.

In general this is a hard problem to solve reliably for all cases, so it's good to know about some tools to take control of module loading/reloading while developing a Dynamo graph that uses your own Python modules.

The standard Python function to use to replicate this behavior in CPython3 is importlib.reload(module) https://docs.python.org/3/library/importlib.html#importlib.reload. You can use this function to reload a module from Python code after previously importing it.

import mymodule
import importlib
importlib.reload(mymodule)

This will reload the module into sys.modules and other CPython nodes that execute after the reload will have access to it. It's important to understand that importlib.reload will not modify existing instances of Python objects created using code from the outdated module. For that functionality you'll want to look at something like autoreload or reloadr.

⚠️ Besides sometimes confusing behavior where outdated and up to date Python types could co-exist, there is also a performance cost to always reloading modules - so while these solutions make development much easier, they should be removed in the final code you deploy if possible. For example, if you publish a package to the Dynamo Package Manager where you used importlib.reload to iterate on your module, you can try removing it before publishing and revert back to regular import statements to get the best performance possible for your released package.

This entry will be updated if improvements are merged to Dynamo core for this issue.

Releases

Roadmap

How To

Dynamo Internals

Contributing

Python3 Upgrade Work

Libraries

FAQs

API and Dynamo Nodes

Clone this wiki locally