low framerate when drawing graphical tiles

[jonbro]

when attempting to update a full console worth of graphical tiles (80x40 of 8x8 tiles) using colorized mode, I am getting really bad framerates, somewhere on the order of 700ms per frame. You can see a simple example here:
https://grateful-kilometer.glitch.me/

I built a version that uses more bit twiddling to get the images onto the canvas, and for my simple case (a single layer of tiles with background and foreground colors:
https://grateful-kilometer.glitch.me/imgdata_drawTiles.html

This is running at 60fps, and checking the core loop, it seems to take around 3-5ms to do all the operations. There would be a bit of work to extend this test to support all the behaviors that the current colorized graphical tiles do, but I just wanted to open up this issue.

My use case for this is doing a full screen ansi art editor with swappable text sets, but things like brogue with graphical tiles would also suffer from this low frame rate.

[ondras]

Hi @jonbro,

thanks for the report. It would be very beneficial to find out what part of the whole task is being the framerate bottleneck: is it the actual canvas drawing (i.e. drawImage), the colorizing phase, some internal rot.js housekeeping, or something different? Were you able to profile? Your fast implementation does not happen to use any parts of rot.js, so it is difficult to see what shall we focus on when working on perf here.

[jonbro]

yep, of course! I did a few tests on my way to this conclusion, though unfortunately I was having difficulty using performance.now() to isolate just the drawing functions. It seemed like the best test that I could get was via measuring the time from the beginning of the loop to the beginning of the next loop. My suspicion is that the browser is deferring the canvas draw calls until after the javascript thread is done, so you can't actually measure how much they cost :(

At first I tried hacking up rot js, and seeing if I could get the core loop fast there. It seems like rotjs isn't adding much overhead to the drawcalls. So I pulled out the core loop as best I could, and started hacking it up.

all these examples are dumping out frametimes, avg'd over the the last 20 frames to the console.

https://grateful-kilometer.glitch.me/rotNoColor.html
uses rotjs, with colorized: false - approximately 120ms per frame. Better, but still only ~10fps.

https://grateful-kilometer.glitch.me/onlyDrawImageCall.html
this copies the core call (with colorized:false) from rotjs, but removes any potential rotjs overhead. ~70ms per frame.

https://grateful-kilometer.glitch.me/rotjsColorizeApproach.html
this copies the core call (with colorized:true) from rotjs, but removes any potential rotjs overhead. ~700ms per frame. Leads me to believe that rotjs is adding some overhead, but not much

https://grateful-kilometer.glitch.me/fast_draw.html
here is a test that is using globalCompositeOperation = "multiply" to do the coloring. ~100ms per frame here. (note, this approach will only work for a single layer)

One thing to look at is that once you do enough ROT.display.draw calls to get an actual number out of the loop (i.e. something higher than a reasonable vsync division), then the time scales linearly with the number of draw calls. on my computer, it seems like each call is taking about 0.2ms

I don't have it any more, but I did another test to see if batching each type of call would help, but unfortunately it didn't... I did:

batchedDrawRequests = [...];
batchedDrawRequests.forEach(dr.DrawImgTocolorizeCanvas);
batchedDrawRequests.forEach(dr.drawFgOverlay);
batchedDrawRequests.forEach(dr.drawBg);
batchedDrawRequests.forEach(dr.DrawColorizedToMainCanvas);

My suspicion is that canvas operations are just slow. it isn't just drawImage, but also fillRect seems slow as well - minimizing the number of calls to it seems critical for getting a fast drawing loop.

[jonbro]

going back to the early tickets on this topic #22 and #50 it seems like the pixel based approach that I am suggesting was proposed but rejected in favor of the current approach. I wonder if under certain circumstances, the globalCompositeOperation is actually faster. Possibly with fewer, but more high resolution tiles.

[ondras]

Wow, thanks for the detailed post! I will try to look into this, even though it looks like a more complex issue and my time is pretty limited these days.

going back to the early tickets on this topic #22 and #50 it seems like the pixel based approach that I am suggesting was proposed but rejected in favor of the current approach. I wonder if under certain circumstances, the globalCompositeOperation is actually faster. Possibly with fewer, but more high resolution tiles.

I am pretty certain that globalCompositeOperation is faster than individual per-pixel processing, but only when you compare comparable: your code is way faster because you are only drawing the canvas once per frame (the ctx.putImageData(imageData, 0, 0) call). If you were to hand-craft every tile and then render the tile with drawImage, I would guess that globalCompositeOperation is faster.

I see two ways of handling the issue now:

1. re-implementing the tiling ROT.Display in a way similar to your demo, so that all draw calls operate on a hidden Uint8ClampedArray that is flushed to the real canvas only once per frame;

2. re-implementing the tiling ROT.Display using WebGL, so that the per-pixel operations are implemented in a fragment shader.

[jonbro]

Yeah, I recognize that this is a big issue :) and its a real pain to get it right too - I ended up reimplementing the parts of Rot.Display that I needed for my particular project - it is doing a number of hacks to get speed even higher than my demos that I posted above. This is passing every tile via a normal Display.draw(x,y,char) call.

https://glitch.com/edit/#!/asc-paint?path=src/Display.js:7:20

The feature that I cut here was the multilayer tiles, and there is also some cleverness happening so I don't need to support the tiles alpha channels. Both of those things feel like they could be addable with a switch that regains performance if they aren't necessary.

The WebGL is something I have considered for sure, and I think that is probably the most reasonable option for someone like me that wants the most speed.

[ondras]

The WebGL is something I have considered for sure, and I think that is probably the most reasonable option for someone like me that wants the most speed.

This would make sense the most to me as well. I would, on the other hand, prefer to not have as many performance-related switches (multilayer, alpha) as possible, to keep the stuff readable and minimize complexity. I believe WebGL provides enough performance to comfortably fit within the 60 FPS cap of the requestAnimationFrame loop.

[ondras]

Hi @jonbro,

please feel free to check out my work-in-progress demo implementation of the WebGL-based tile rendering mechanism: https://beta.observablehq.com/@ondras/new-rot-js-display-backend-for-image-tiles

I would say that it performs just fine. The demo is slowed down by CSS-based canvas resizing, because I am using huge (64x64) tiles on a large (80x25) display. It uses colorization (tinting) and currently only draws one image per tile, but I think it might be sufficient as a proof of concept.

A more performant version would probably batch individual draw calls, passing multiple tile definitions (coordinates, colors, ...) as WebGL attributes. I will try to stick with the current implementation, though.

[jonbro]

Impressive! I was seeing a teeny bit of slowdown on my phone, but like you say, batching will make that issue go away completely. This is great.

[Gerhard-Wonner]

Hello Ondřej,
I am also very interested in a higher frame rate when using colorized tiles since my game is intended to be played fast paced. When I set the "tileColorize" to "false" the performance is much better. But I really need the colorization since I want to draw a navigation-system-like path over my tiles.
Are you still working on this issue?
Kind regards
Gerhard

[ondras]

Hi @Gerhard-Wonner,

I got distracted and forgot about this feature. Sorry. Let me put it back to my priority list to see what can be done.

[Gerhard-Wonner]

Good morning @ondras, you are awesome! 👍
Thank you so much for providing this fantastic game-engine! I used it in this year's 7DRL challenge and I am quite satisfied with the result. My game is a turn-based racing game with roguelike elements. It is all about overtaking, speed and speed-boosts so your "Speed scheduler" matched my needs just perfectly. It was received quite well and has been played by 5 youtubers so far. If you are interested in what I have been done so far with your engine you are invited to give it a quick look here.
Now I do some post-challenge-development. One of my goals is to replace the ASCII-graphics by tiles. Since I implemented a navigation system in my ASCII-game I need some way to still display it in the tiled version. I already implemented it (not published so far) with your "Colorizing tiles" feature and it looks quite nice. The only problem is the slow update rate.

[ondras]

I pushed some initial preparations to the tile-gl branch.

The implementation is taking much longer than I thought, though, because I constantly run into corner issues regarding blending, opacity and redrawing. WebGL is not something I am 100% versed in :-(

[ondras]

I just merged the tile-gl branch to master. This means that you are invited to test the webgl functionality!

Please use the master branch or the 2.1.0 version from NPM; swap your layout:"tile" to layout:"tile-gl" and report how does that work for you. I will leave this issue open, should someone encounter a bug related to this very fresh webgl implementation.

This new WebGL renderer is also mentioned in the interactive manual, along with a code sample.

/cc @Gerhard-Wonner @jonbro

[Gerhard-Wonner]

Hello @ondras, thanks a lot for the quick help and the enhancment of the performance. Please excuse my late response. I had troubles to get the tiles displayed in my local version of the game and got the error message "Cross origin requests are only supported for HTTP". But after I uploaded everything to the server it worked and I can see that the performance is better now. :-)
Do you have any idea how I could get it to load the tiles-sheet correctly in my local version of the game? It would help me a lot with debugging things.

[ondras]

Hi @Gerhard-Wonner,

I very briefly linked the image loading issue at the very bottom of the manual page.

TL;DR: use a (local) web server to serve images; develop either using http://localhost or configure your web server to send proper CORS headers.

Longer read: https://hacks.mozilla.org/2011/11/using-cors-to-load-webgl-textures-from-cross-domain-images/

[Gerhard-Wonner]

@ondras Thank you for the explanation. I will try it soon. Sorry if I ask silly questions, but I just used the 7DRL-challenge as an opportunity to learn JavaScript. So this whole web developing thing is quite new to me.

[Gerhard-Wonner]

@ondras I found a solution for me that works well: I just open my local version using the "Live Server"-extension of Visual Studio Code and all the tiles are displayed correctly. :-)

[ondras]

Hi @Gerhard-Wonner,

Sorry if I ask silly questions, but I just used the 7DRL-challenge as an opportunity to learn JavaScript. So this whole web developing thing is quite new to me.

no problem, feel free to ask more! You will soon find that although the whole frontend tech stack looks simple, it contains many convoluted difficulties. But this particular complexity (cross-domain image requests) is actually very reasonable.

@ondras I found a solution for me that works well: I just open my local version using the "Live Server"-extension of Visual Studio Code and all the tiles are displayed correctly. :-)

Right, that is exactly what I meant by localhost development. Good to hear that it works for you.

[Gerhard-Wonner]

Hello @ondras, there seems to be a problem with some browsers when opening http://ondras.github.io/rot.js/manual/#tiles.
On Microsoft Edge I get the error-message "Unable to get property 'createShader' of undefined or null reference".
On Apple-Safari (IPad) I get "null is not an object (evaluating 'gl.createShader')".
On Chrome, Firefox and Opera everything seems to work fine.

[ondras]

Hi @Gerhard-Wonner,

that would correspond to the WebGL2 support status across browsers: https://caniuse.com/#feat=webgl2

Someone more skilled with WebGL might be able to re-factor the code using an older WebGL implementation (version 1), though.

[Gerhard-Wonner]

Hello @ondras,
thank you for the information.
Since the most common browsers already support WebGL 2.0 I decided to publish the graphical version of my game anyway.
For the players lacking a browser capable of WebGL 2.0 I also kept the ASCII-version.
Does anyone know or could anyone check, if the page http://ondras.github.io/rot.js/manual/#tiles works on a mac using chrome?
Kind regards
Gerhard

[ldd]

I don't think you ever got a confirmation, but yes @Gerhard-Wonner ,
everything works smoothly on a Macbook Pro on Mojave running chrome 80