-
Notifications
You must be signed in to change notification settings - Fork 431
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ValueError: tensorflow op ExtractImagePatches is not supported #436
Comments
currently, for TF ExtractImagePatches op, there is no equivalent op in onnx. |
@nbcsm Thank you for reply. What would be the process for that? |
go to https://github.com/onnx/onnx, and open an issue for ExtractImagePatches |
Great, thanks |
@nbcsm can it be composed in the converter? |
Along the same lines. Translated in to something else in special cases: |
for that case (ExtractImagePatch with ksizes = [1, blk, blk, 1], stride = [1, blk, blk, 1], rates = [1, 1, 1, 1]), yes, SpaceToDepth seems to work. it feels like we may use MaxPool + SpaceToDepth to convert ExtractImagePatch, but need to double check. |
Hi, I'm having the same issue here when trying to convert YOLOv2 full network (416x416) into ONNX and I really need ExtractImagePatch op, I was trying to use tensorflow model instead but it doesnt worked, as ML.NET failed to load it. Please add support for this or at least some workaround code? How do I define own special op for that? |
@nbcsm Hi can someone just paste here the ExtractImagePatch handler that is using SpaceToDepth ? I'm new in this area and writing own operations seems quite hard to me, I need some assistence, thanks. |
Are you using DarkFlow? |
Yes, I have used DarkFlow and converted darknet (.weights) model into tensorflow (.pb), because only tensorflow and ONNX are supported formats in ML.NET which I'm using. However tensorflow model throwed error while loading it using ML.NET (I have tested that model using darkflow and detections were working there, will report that issue to MS). So logical step was to try to convert my model into ONNX that is also supported. Gonna to try your commit, thanks a lot. |
Hi, any update on this ? |
#1543 features another model using this op. ksizes = [1, 3, 3, 1], rates = [1, 1, 1, 1], strides = [1, 1, 1, 1], padding = 'SAME'/'VALID' |
I have tried to make a numpy implementation of ExtractImagePatches: def numpy_eip(arr, ksizes, strides, rates, padding):
sizes = [1, ksizes[1]*rates[1] - (rates[1]-1), ksizes[2]*rates[2] - (rates[2]-1), 1]
if padding == 'SAME':
extra_i = max(0, (arr.shape[1]-1) // strides[1] * strides[1] + sizes[1] - arr.shape[1])
extra_j = max(0, (arr.shape[2]-1) // strides[2] * strides[2] + sizes[2] - arr.shape[2])
arr = np.pad(arr, [(0,0), (extra_i//2, extra_i//2 + extra_i%2), (extra_j//2, extra_j//2 + extra_j%2), (0,0)])
elif padding != 'VALID':
raise Exception('Padding type "%s" is not supported' % padding)
def make_range(in_size, k_size, rate, stride):
return range(0, in_size - (k_size*rate - rate), stride)
indexes_i = make_range(arr.shape[1], ksizes[1], rates[1], strides[1])
indexes_j = make_range(arr.shape[2], ksizes[2], rates[2], strides[2])
batch_size = arr.shape[0]
channel_size = ksizes[1]*ksizes[2]*arr.shape[3]
return np.concatenate([np.concatenate([
arr[:, i : sizes[1]+i : rates[1], j : sizes[2]+j : rates[2], :].reshape([batch_size, 1, 1, channel_size])
for j in indexes_j], axis=2)
for i in indexes_i], axis=1) Note that the TF documentation says that |
If we want to do an onnx op, could we do a conv with an identity kernel? |
I have done a quick test with my |
I think you should be able to move the M and C values of 2304 and 256 to the batch dimension, do the conv with a simple [1, 1, 3, 3] kernel and put the batch dims back. That should make the kernel much smaller. |
Minimal effort test without batch handling, seems to work, and down to 1 KiB with a [9, 1, 3, 3] kernel: eip_valid.zip |
Perfect. Actually if reshape sees a -1 in the shape it will dump all the remaining dimensions into there, and a 0 indicates to keep that dim the same. Since the input is [9, 256, 30, 30], try [1, -1, 0, 0] instead of [1, 2304, 30, 30]. It is also common to use Shape, Slice, Concat, and Reshape together to make the shapes dynamic. Should be cheap to execute since all those ops are small. |
That is really useful, I could fix the rest of the Reshapes in my model using that as well.
Assuming we have a way to convert ExtractImagePatches to either Conv or SpaceToDepth in either all cases or in specific cases, what can/needs to be done to move this forward and what can I help with? Could this be done by the converter (the Conv approach technically is a change of semantics even though the behavior is the same), or could we make a guide on what options you have to deal with these models instead (i.e. custom OP implementation, replacing it with common OPs, etc.). Is there still a need for pushing ExtractImagePatches to get standalized assuming we have a workaround with standard operators? |
@spillerrec it would be fantastic if you could add support for this op in tf2onnx. Add a handler for ExtractImagePatches in nn.py https://github.com/onnx/tensorflow-onnx/blob/master/tf2onnx/onnx_opset/nn.py You can copy CTCGreedyDecoder as a template. Use the lowest onnx opset you can get away with https://github.com/onnx/onnx/blob/master/docs/Operators.md. Then add some unit tests to test_backend.py https://github.com/onnx/tensorflow-onnx/blob/master/tests/test_backend.py It is perfectly fine to only be able to convert certain cases. Use It's fine that this conversion uses a composition of ops and might have different performance characteristics from the original TF op. That's pretty common in the converter. Since this is a pre-processing op, hopefully the inference time won't be dominated by it. For large sizes hopefully we can use SpaceToDepth instead, since I think those cases have strides matching the size (otherwise the output would be huge anyway). As you are working, open a PR and I'll be happy to comment if you run into any issues. |
@TomWildenhain-Microsoft Thanks for the pointers, I will give it a go. |
@spillerrec Just checking if you've had a chance to work on this. Happy to help if you are stuck. |
@TomWildenhain-Microsoft I'm sorry about the lack of updates, no, I have not looked at it yet. However I have next week off, so I will not have an excuse now for not getting some progress done on it. ; ) |
@spillerrec Are there any updates on this issue? |
@savvaki No, not anything else than I looked into the To sum up, the current status is that we have a decent workaround using standard operators and most people would probably be happy with it. It is currently stalled on me to actually add it. I honestly forgot about this issue, so thanks for the ping. I found out that mingw can't be used with onnxruntime on Windows (and I have no intention to use Visual Studio), so this stalled my own ONNX related project (and will probably stay so until I switch back to Linux). I'm still willing to look at it, just don't assume it will be anytime soon with my track record and certainly not before January. If anyone else is interested in getting this done, don't hold back because of me. |
@spillerrec Do you perhaps have the tf code lying around that you used to generate the onnx models that you posted above? |
I created an "equivalent" operation to import tensorflow as tf
import numpy as np
# Define some parameters
input_shape = [5, 10, 10, 6]
patch_size = 3 # height/width of the square image patch size
padding = 'SAME'
# Generate example input
x_original = tf.cast(tf.reshape(tf.range(np.prod(input_shape)), input_shape), dtype=tf.float32)
x = tf.transpose(x_original, perm=[0, 3, 1, 2]) # Move feature map dimension next to the batch dimension
x = tf.expand_dims(x, -1) # Add extra channel at the end
# Create an identity kernel
kernel = tf.reshape(tf.eye(patch_size**2), [patch_size, patch_size, 1, patch_size**2]) # [filter_height, filter_width, in_channels, out_channels]
# Convolve with identity kernel
patches_simulation = tf.nn.conv2d(x, kernel, strides=[1, 1, 1, 1], padding='VALID')
patches_simulation = tf.transpose(patches_simulation, perm=[0 ,2 ,3, 4, 1]) # Move filter dim to last
patches_simulation_shape = tf.shape(patches_simulation)
patches_simulation = tf.reshape(patches_simulation, [patches_simulation_shape[0], patches_simulation_shape[1], patches_simulation_shape[2], -1]) # Merge last two dims into one
# Intended output to compare against
patches = tf.image.extract_patches(x_original, sizes=[1, patch_size, patch_size, 1], strides=[1, 1, 1, 1], rates=[1, 1, 1, 1], padding='VALID')
# Print out check to see if the alternative method is identical to tf.image.extract_patches
print(f'tf.image.extract_patches shape {patches.shape} simulation shape {patches_simulation.shape} same shape: {patches.shape == patches_simulation.shape}')
print(f'Simulation is correct? {tf.reduce_all(tf.math.equal(patches, patches_simulation)).numpy()}') |
I am trying to convert a custom developed model with a custom convolution layer and I implemented it with |
Hi! Any updates? I'm also waiting for this update :( [UPDATE] |
I am trying to convert yolov2-voc generated by DarkFlow in to ONNX, and running in to this error
"ValueError: tensorflow op ExtractImagePatches is not supported"
I am using tensorflow=1.9.0, onnx=1.4.1, opset=7, tfonnx=1.5.0/9fcb06.
Is this tensorflow issue, or the converter? How would I got about fixing this?
Thank You
The text was updated successfully, but these errors were encountered: