I went looking for examples of isometric maps using the new Unity Tilemap feature, but found none. Some people even believe it cannot be done, but I remember the old games.
This was part exploration, part confirmation of what I knew: old school techniques for doing isometric maps did not rely on special 3d transforms, per-tile sorting or wasteful single-tile atlases where most of the image is spent on transparent edges. They were packed tight, transparency was avoided for speed and grids where drawn in one go.
I've spent a lot of time studying techniques so I decided to try this one out and see if my understanding was correct.
I managed to adapt existing art into the needed atlas and use it to draw a flat ground with diagonal features. After that, I managed to draw raised features that a sprite can go behind, without scripting or special ordering (just changing the draw order per object/tilemap in the inspector).
ISO tiles must be placed side to side and they must be continuous, so the four edges of the iso tile continue into their neighbors. This means you might need to repeat tiles for different transitions. This will sometimes be wasteful, but will allow you to draw all the ground floor in one pass. This part is very important, as good arrangement of the tiles will lead to less confusion later and will avoid tiling artifacts at the edges, letting you even keep bilinear filtering on.
Each ISO tile will be split in at least 4 rectangular tiles (there could be benefits to splitting into 8 squares). My source art was 64x32, so I split it into 32x16 tiles, where each Unity tile fully contains one edge of an iso tile.
At this point you'll realize that you'll have to be careful with the art. If a feature (ie: a rock in the above example) crosses over the center of the iso tile, it will cross over a Unity tile boundary and now you will be unable to reuse that tile by itself, as it will look cut off. There's many tricks that you can do with this arrangement, giving you more possibilities for variation.
When you want a raised platform as seen above, the tiles belonging to the top edges will have transparent top edges so sprites can get behind them. These are not used in the ground floor, but above, as this will be another tilemap anyway. The tiles at the bottom edge of the raised platform, however, will have one side of the iso edge as flat ground and one side as the wall, so it shouldn't be transparent. The base of the raised platform is a ground tile, as seen in the video, so it's also fully opaque. Tall props are modeled as raised platforms, keeping the base as ground tiles and the rest can be split into multiple floors, so they can be put one in front of the other without sorting issues (and you could, say, have a tree going through 3 floors and walk around it at any floor with no glitches, even with proper collision at each level).
I edited the colliders tile by tile, removing the colllider from the tiles that can be walked on, while removing one corner of the tiles that are half-walkable.
I avoided complex sorting for this project, so everything has been split in "floors". Sprites are expected to lie between floors, so floors (one tilemap per floor) are given even numbers while sprites are given odd numbers. A character walking between floors is not demonstrated, but could be achieved with Scriptable Tiles, adding a slope property per tile and using it to change the movement of the characters walking on them. By then, we'll need to keep track of the implicit height of each sprite and this height can be used to switch floors when appropriate. If you're changing floors, you might want to use several layers instead of one as in the tutorial, so you can get the right collision layer for free.
I did not try to solve tile edge issues for this tutorial, as the solutions are known and it would take time away from the issues I really wanted to address. Sampling was set to nearest neighbor, but the camera was left with default values.
Sprites taller than 1.5 Unity tiles may be covered by tiles in the next floor, specially when going around sharp corners. This can be addressed by avoiding thin obstacles (no 90 degree corners!), so you can shift more of the wall to the ground tile while ensuring the player won't be able to walk behind those tiles. Another way would be spliting characters into multiple-floored sprites, but this sounds error-prone. A third way is to have larger tiles so the player can be larger and still be less than 2 tiles high (here splitting in each iso tile in 8 may help keep the world in a reasonable size). Some advanced sorting techniques might be a better choice (such as custom axis sorting), but I did not investigate them for this experiment.
In the video you can see weird tile half-positioning in the grid. I tried setting the pivot at the bottom of the tile, which makes it very confusing as the tiles are also shifted in the palette so picking and placement becomes slow and error-prone. This might be required for advanced sorting, but I ended up not trying that, so I don't know if it's necessary. I feel like picking from the palette should not be shifted, as you should be able to easily identify what you are grabbing. Placement in the tilemap seems correct, but I'm not so convinced either.
Editing colliders is extremely awkward. A lot of my colliders (and for most users, I'd imagine) ended up being the same, but I cannot see them until I click on them and I cannot copy/paste them, so I did a lot of careful clicking and triple checking. Made a couple of mistakes, too. Removing the collider from a tile is extra-awkward, since you can't do it in the physics shape editor (found in the sprite editor). You have to go to the tile asset after adding the tiles to a palette (extra cumbersome as ours are tiny and hard to recognize, I would select them in the palette, remember the name and then find it in the assets) and in properties select no collider. This means, of course, if you dump the palette (I did several times when rearranging the art) you also lose that setting. I really wish unity would streamline this bit: I'd like to see all collision shapes in the atlas with a solid overlay in a color of my choice and be able to copy/paste replacing the original collider for that tile, and keep that setting in the atlas (if you delete enough edges in the physics shape, it just resets).
Arranging the art was a ton of work, but I could immediately see gains: the source had repeated angles everywhere, which with more planning could have been put only once in the atlas, making it easier to combine and reuse and also reducing texture size and setup work.
Mind the size: an iso tile usually has a one pixel corner at each side (left, right), which cannot be split into two. To properly decide tiling, I drew a grid (shown in the examples above) but I ended up misaligning anyway. I found that the original art had extra pixels sometimes, as the original format is much less strict. I made a new grid, but this time the grid is supposed to be the INSIDE EDGE of each iso tile. I also added a half-transparent inside, so I could see when I was getting close to the edge. This helped a lot, but it would have been much easier if the art was planned for this style from the beginning.
Success! I really like how Tilemap is coming together, though there's still plenty of room for improvement. I also look forward to geting deeper into this style of maps, with art made ad-hoc for it.
Further experimentation: a ramp between floors! There's much more to try before the next tutorial.
Original tiles by Yar at https://opengameart.org/content/isometric-64x64-outside-tileset
I used 2D Character controller by MIMU STUDIO at https://assetstore.unity.com/packages/tools/2d-character-controller-78532
I used Pro Camera 2D by Luis Pedro Fonseca at https://assetstore.unity.com/packages/tools/camera/pro-camera-2d-the-definitive-2d-2-5d-unity-camera-plugin-42095