Working hexagonal tiles into a rectangular tilemap
The following is based on my own research. Feel free to use any of the resources and techniques presented in this page. If these are useful to you, please mention me in your work (and let me know about it), so I can point to it when people ask me if my stuff is any good. This also motivates me to keep researching.
IMPORTANT NOTE (about Unity 2018.2 hex grids)
This article was written just before Unity 2018.2 released with support for hexagonal tiles. This article describes building hexagonal grids using SQUARE TILES. Unity's hexagonal grid requires using hexagons with alpha, which might leave holes or overlap at diagonals. While this could result in issues, it might also not. Smooth transitions as described in this article would not work with the new hex grid, as placement is staggered and overdraw is unavoidable. You cannot use the tilesets shown here with the new hex grid. I'll do some further testing.
This article is a part of a series. Here is the main article.
This was a difficult problem. We want to give the illusion of hexagonal tiles, while retaining the advantages of rectangular tiles.
There's two common proportions used in hexagonal grids: 2/4 and 2/3 (seen here with rectangular 2x1 tiles). Diagonals are always 1 rectangular tile, but at the middle the hexagon might be two diagonals high, for a top-down look, or just one diagonal high, for a squashed perspective look.
We'll use 2/3 since the math fits when tiling, as seen here.
We'll move to square tiles now, so each hexagon is split by half in the middle and there's a top and a bottom half. We'll use 16x16 square tiles, but you can use any power of 2: 8x8, 32x32, 64x64, 256x256, etc.
Inspired by Corner Wang Tiles (ARES LAGAE and PHILIP DUTR), we want the corners of the hexagon (all 6) to land inside a tile (not the edge), so we can draw a smooth transition: This means 6 tiles, so our hexagon will be split among 6 square tiles, and we scroll our hexagon within the tiles such that corners are in the middle. The upper corner won't be at the same height as the lower one, but as long as they fit inside a tile and are at a distance from the edge, we'll be fine.
How many tiles do we need to represent every possible combination of color? The number of permutations of 2 colors, picking 3 at a time is 2x2x2 = 8, but we have "up" and "down" tiles, for a total of 8x2=16 tiles, which is great because 16=4^2.
How do we arrange the tiles so that every combination matches a neighbor and also wraps around? After fiddling with layouts for a long time, I came up with this one:
And that is all you need to know! Now let's try it out.
Let's build a tileset!
I use Pro Motion NG. Here is a brush for you to use as a seamless grid.
Start by drawing each of your 2 hexagon tiles. Be strict about the edges, we'll soften them later.
Now copy each completed hexagon and paste it in the pattern shown here.
Now you may fix up the transitions between hexagon, but keep a band of one pixel (at least) around each square tile intact. This edge will connect to different tiles, which must match exactly. Mipmapping will require a bigger edge, but this will be explored in another article.
Now let's try it out in Unity! Sprite must be set to Multiple, compression to none (due to small size) and wrap mode to Repeat (for correct filtering on tiles in the edge of the tileset). In the sprite editor, you'll need to slice by tile count (4x4).
Now you may have large maps made out of solid-looking (even in motion! ) hexagons, with good framerate and nice transitions.
Placing the tiles is a bit difficult. I've used a two-color plain hexagon tileset as a giude and I've replaced the image with the final tileset afterwards, to make it easier. Still, I'll work on a method to make placement easier.