-
Notifications
You must be signed in to change notification settings - Fork 636
Work in progress to improve Python 3 support
This document lists issues we detected related to compatibility between Iron Python 2.7 and CPython 3.8 (via Python.NET) engines. Next to each issue we provide a status of it:
- Fixed: The issue has been fixed. Either a release version is specified or it can be tested with our latest daily build.
- Planned: The issue is either being worked on or is planned for a future release.
- Not planned: The issue has been noted but has not been considered to be worked on.
We ask the community to provide feedback about these or other issues we might have overlooked.
Migrations from version 2 to 3 issues are purposely left out from this document, as they will be handled by a Migration Assistant which is currently in development. Documentation about it will be added later.
(If you are already aware of what these libraries are, you may skip this section, but if not, read on.)
Dynamo has supported Python execution through the Iron Python runtime for a long time, even though the Engine
selector surfaced that only recently. As you may have heard, Iron Python currently supports Python 2, but it doesn't support Python 3. Something else you should know is that Iron Python does not use CPython, which is the standard Python runtime, but instead implements its own runtime over the .NET Framework. If you are not familiar with .NET, let's just say it is a framework to develop applications. It's pretty popular, even Dynamo and other tools you love are built on top of it 😄
The CPython engine, on the other hand, uses the standard Python runtime, also called CPython. Since Dynamo and its libraries are based on .NET, you need somethings that acts like a "glue" to make execution and data flow correctly between the two. This is achieved by using a library called Python.NET.
So if you can write Python and it will run in any case, you may ask yourself why we bring .NET to the discussion. Basically there are three reasons:
- Some of the issues mentioned here deal with incompatibilities around .NET integration when comparing the two engines
- It helps to understand what's going on behind the scenes: in one case everything runs in .NET, while in the other execution and data flows between CPython and .NET.
- Most of your favourite applications here at Autodesk use the .NET framework, so if you want your Python node to be able to do things in these environments (such as talk to the Revit API) then your Python code needs a way to talk to them!
Iron Python supports calling .NET functions on Python objects. This can be done for a big variety of classes and functions, what follows are only a few examples:
import clr
# Calling 'ToString' on a Python int
python_int = 1
my_str = python_int.ToString()
# Calling 'Add' on a Python list
python_list = ['hello']
python_list.Add('world')
OUT = my_str, python_list
Those examples work because for Iron Python every Python object defined is backed up by a .NET object representation. This means a Python int
also has the behaviour of a .NET int
, which happens to implement ToString
as expected. The same thing happens for the other examples.
The previous code does not work for native Python, because the Python native types don't implement these functions. However, in Python you could achieve the same result by doing things differently:
# Convert a Python int to a string using plain Python
python_int = 1
my_str = str(python_int)
# Adding an element to a list using plain Python
python_list = ['hello']
python_list.append('world')
OUT = my_str, python_list
An easy way to reproduce this issue would be to write the following code in a Python node, with the CPython3
engine:
s = { 'hello', 'world' }
OUT = s
The previous code tried to return a Python set
from a Python node. This type was not supported, and in previous versions also led to a crash due to a bug. Support for Python types has been improved so set
is now supported along with other collection types.
A TypeError
is an error that might be seen from a .NET function call that receives Python types as arguments. One example that is reproducible in Dynamo 2.7 with the addition of CPython3 is:
import clr
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *
point = Point.ByCoordinates(0,0,0)
OUT = point
In this case, the TypeError
issue was caused by an inability of Python.NET to determine that an overload of Point.ByCoordinates
should have been called. The overload Point.ByCoordinates(double x, double y, double z)
could have been called by converting the int
inputs with values of 0
to float
inputs with values of 0.0
. However, Python.NET did not attempt this and failed with the TypeError: No method matches given arguments
message.
This specific problem for int
to float
conversions was fixed. Unfortunately, the message is quite generic and might describe other failed conversions or scenarios where the method was not found due to being misspelled, for instance. If you run into this type of error, please reach out to us with detailed information about the issue.
Most often than not, the CPython3
engine would omit the line number where an error occurred, making it harder to debug long Python scripts. Errors of type TypeError
would never provide a line number, while others of type SyntaxError
would.
This was a rather inconvenient omission in Python.NET that was fixed.
The following example shows what this issue is about:
with ClassThatImplementsIDisposable() as something:
pass
Iron Python would call Dispose()
automatically for something
when exiting the with
block. This is not the case for Python.NET, so developers should be aware and call Dispose()
explicitly if required.
Python supports through its int
type integer values of any size. However, taking that value from Python to .NET requires special considerations, because the target type needs to be able to support very large values as well. Iron Python converts to BigInteger
automatically, so the following code example would work:
# That is bigger than what would fit in an Int64
x = 11111111111111111111
OUT = x
Python.NET did not support converting large integers to BigInteger
out of the box, so additional work was required to make it work.
Iron Python allowed to return objects of custom classes defined in the same Python script node. An example of this would be the following:
class Dog:
kind = 'canine'
def __init__(self, name):
self.name = name
OUT = Dog('Rex')
Even though the output from that node would not be of much use in Dynamo, it could be reused in another Python node, allowing certain degree of code reuse between nodes.
Related to the previous issue, we have found examples of objects that are subject to conversion in the CPython3
engine, while they would be passed unconverted using the IronPython2
engine. To better illustrate this, here is an example of such a class:
class MyIterable:
def __str__(self):
return 'I want to participate in conversion'
def __iter__(self):
return iter([0,1,2,3])
def __getitem__(self,key):
return key
OUT = MyIterable()
The IronPython2
engine would return this object as an IronPython.Runtime
type, which can be passed around between Python nodes. On the other hand, for the CPython3
engine, this would be recognized as a collection and converted to a List
. This is because the class implement __iter__
and __getitem__
, which is something only the CPython3
engine checks for. This check is required to support a wide range of built-in collections, so simply removing it was not an option we cared for.
But what if the author of this class wants the old behaviour offered by the IronPython2
engine and intends to move this object between Python nodes? This is still possible using a Dynamo specific attribute we have introduced: __dynamoskipconversion__
class MyIterable:
def __dynamoskipconversion__(self):
pass
def __str__(self):
return "I don't want to participate in conversion"
def __iter__(self):
return iter([0,1,2,3])
def __getitem__(self,key):
return key
OUT = MyIterable()
By including this attribute, even with an empty implementation, Dynamo will return this object as-is, instead of attempting any conversion.
The dir
function allows to get a list of the attributes defined for a given Python object. Here is an example of how it can be used to get the list of attributes for a .NET namespace:
import clr
clr.AddReference('DSCoreNodes')
import DSCore
OUT = dir(DSCore)
When run with the IronPython2
engine, the previous code would give as a result the list of classes and namespaces contained in the DSCore
namespace from Dynamo. However, with the CPython3
engine, the result would be mostly empty, only including some internal attributes. This was fixed by enabling the preload of modules in Python .NET.
When requesting certain internal attributes using getattr
, the CPython3
engine would fail with an AttributeError
stating that the requested attribute did not exist. This would happen even for attributes obtained from dir
, which should be guaranteed to exist. Here is an example of a case that failed:
import clr
clr.AddReference('DSCoreNodes')
import DSCore
# This failed with AttributeError!
getattr(DSCore, '__delattr__')
This was a problem originating in the Python .NET library which should now be fixed.
One nice feature Iron Python provides is the ability to call extension methods similarly to how it is done in .NET. For a quick overview of extension methods, please take a look at this brief article. What follows is an example of how it could be used in a Python script:
import clr
# Calculator is a .NET class implemented in an external library
# It defines a 'Sum' function to sum two numbers
clr.AddReference('ExternalLib')
from ExternalLib import Calculator
# MyExtensions is a library that defines extension methods
# For Calculator, it defines the 'Sub' extension to subtract two numbers
clr.AddReference('MyExtensions')
import MyExtensions
clr.ImportExtensions(MyExtensions)
calc = Calculator()
result = calc.Sum(1,2)
# Note that 'Sub' can be called on the instance, even if it was
# defined as an extension
result = calc.Sub(result,3)
OUT = result
The previous code example would work without issues using the IronPython2
engine, given the mentioned libraries were loaded in Dynamo. For CPython3
, on the other hand, several things were missing to make this work.
In order to fix this, the ImportExtensions
function had to be added to Python .NET, along with other internal changes that were required to allow calling extension methods.
Related to the previous issue, one thing Iron Python also allows is listing extension methods when using dir
. Imagine the previous example but changing the final lines like this:
...
# The result of 'dir' includes 'Sub' only in Iron Python!
result = dir(calc)
OUT = result
Making the previous code example work correctly in Python .NET would require more involved changes in the library. Currently, we haven't found this to be a blocker for known workflows so we have not planned to work on this yet. If you do happen to be blocked by this, please let us know.
This is an error that might surface in scripts that use System.Windows.Forms
, like the following example:
import clr
clr.AddReference('System.Windows.Forms')
from System.Windows.Forms import DockStyle
dockstyle = DockStyle.None
OUT = dockstyle
The fact that DockStyle.None
crates a SyntaxError
when using the CPython3
engine is due to changes in Python 3 with regards to how the special None
keyword can be used. This fact makes the problem impossible to fix, although there are workarounds in case you happen to need this, like the follow snippet shows:
import clr
clr.AddReference('System.Windows.Forms')
from System.Windows.Forms import DockStyle
dockstyle = getattr(DockStyle, 'None')
OUT = dockstyle
This is probably an advanced usage scenario of Python and interoperability with .NET. Here a Python class is defined in such a way that it implements a .NET interface. The idea behind this is that instances of the class can be created and passed as arguments to methods accepting them. Here is an example of how this could look like:
import clr
clr.AddReference('SomeLibrary')
from SomeLibrary import SomeInterface, SomeClass
class MyClass(SomeInterface):
def __init__(self):
pass
inst = MyClass()
result = SomeClass.SomeMethodTakingSomeInterface(inst)
OUT = result
Given a valid library was provided, the previous code sample would work without issues in the IronPython2
engine. When using CPython3
, however, you can get TypeError : interface takes exactly one argument
or TypeError : object does not implement SomeInterface
, depending on the interface and method definitions. The required changes in Python .NET to make this work seem big when compared to the relevance we have detected for this use case, so we have decided to not plan this work yet. Please let us know if this is blocking you.
While Iron Python preserves type information for enums, Python .NET loses it and deals only with their corresponding integer value. As a consequence of this, returning an enum from a Python script will result in a number when using the CPython3
engine. While it's possible to convert a number back into the corresponding enum, this difference in behaviour between engines may lead to subtle compatibility issues, so please be aware.
Calling .NET methods that use parameters with the ref
and out
modifiers in Iron Python relies on a special function to create a reference. Here is a code sample showing how this works in Iron Python when calling the Dictionary.TryGetValue
function:
import clr
from System.Collections.Generic import Dictionary
d = {"a": 1, "b": 2}
dict = Dictionary[str,int](d)
value = clr.Reference[int]()
key = "a"
found = dict.TryGetValue(key,value)
if found:
OUT = value.Value
else:
OUT = -1
Python .NET, on the other hand, does not provide support for clr.Reference
. However, ref
and out
parameters are supported in a different way. They need to be provided when calling the method but they are considered as inputs only. The output is actually obtained from the result of the function call, which is actually a Python tuple. Here is an equivalent code sample using the CPython3
engine:
import clr
from System.Collections.Generic import Dictionary
d = {"a": 1, "b": 2}
dict = Dictionary[str,int](d)
dummy = 0
key = "a"
found,value = dict.TryGetValue(key,dummy)
if found:
OUT = value
else:
OUT = -1
In the previous example, dummy
is used just to comply with the method signature, while value
is filled with the actual value obtained from the dictionary.
For completeness, here is a code sample showing how to call a method using ref
parameters, although it works in a very similar fashion:
import clr
from System.Threading import Interlocked
original = 1
replacement = 2
old,new = Interlocked.Exchange(original, replacement)
OUT=old,new
While IronPython
will happily reload certain modules (usually those imported from .py files, though the exact behavior is quite complex) - CPython / Python .Net
will not. This is standard CPython behavior. See here for more info:
https://docs.python.org/3/reference/import.html
When CPython encounters an import
statement it will check sys.modules
to see if it has cached that module. If it has, it will simply return the cached module.
As a first improvement we have added an option to manually reload certain modules. Accessible from the Python preferences dialog, pressing the Reset CPython
button will currently perform a few process wide reset steps.
Currently it will:
- reload modules that have
__file__
attributes that point to.py
files and where the module name is not__main__
- attempt to cleanup Python objects created using the now out-of-date version of the reloaded module.
- mark the graph as needing to be re-executed.
- the reloaded modules will be logged to the Dynamo console for debugging purposes.
In general this is a hard problem to solve reliably for all cases, so it's good to know about some tools to take control of module loading/reloading while developing a Dynamo graph that uses your own Python modules.
The standard Python function to use to replicate this behavior in CPython3 is importlib.reload(module)
https://docs.python.org/3/library/importlib.html#importlib.reload. You can use this function to reload a module from Python code after previously importing it.
import mymodule
import importlib
importlib.reload(mymodule)
This will reload the module into sys.modules
and other CPython nodes that execute after the reload will have access to it. It's important to understand that importlib.reload
will not modify existing instances of Python objects created using code from the outdated module.
For that functionality you'll want to look at something like autoreload or reloadr.
importlib.reload
to iterate on your module, you can try removing it before publishing and revert back to regular import
statements to get the best performance possible for your released package.
This entry will be updated if improvements are merged to Dynamo core for this issue.
Looking for help with using the Dynamo application? Try dynamobim.org.
- Dynamo 2.0 Language Changes Explained
- How Replication and Replication Guide work: Part 1
- How Replication and Replication Guide work: Part 2
- How Replication and Replication Guide work: Part 3