How to use unity for multi-projection displays?
Published 19 days ago
Introduction (Multi-Displays in Unity)
Blackbox Realities is a child company of PK Pictures, which is a media production company. This allows the Blackbox team to work on entertainment installations, including displaying content on massive surfaces, like a 50ft wall. These large projections are often made up of 2 or more projectors casting onto a wall and overlapping where the two projectors come together. This allows for a blend between the two projections without an increase or decrease in the screen’s emission. I’ve attached a diagram to further describe what the team was trying to accomplish.

The Problem

While unity supports multiple displays, an overlap requires for each screen to render repeated content. Additionally, to render on two displays, unity requires that your create two cameras and target which display they will render on. This is causing performance issues in addition to time consuming setup, especially if you wish to display UI stretched across both displays.
*Tools like Nvidia Mosaic require quadro cards, the installation that I was developing for used a 2080 TI so that software didn’t work. Other tools didn’t offer enough overlap for the projection to work. We needed ~20% overlap total.

Solution #1 | Using two cameras to render 1 scaled screen buffer

Before I talk about external tools, I want to share a solution that works natively in unity. I suppose that this would be easier in SRP since you could extend the render pipeline to display on two displays without having to catch the Pre and Post render callbacks on specific cameras.
I have linked a script with comments below. I’ve decided to condense the solution into one large script for the purpose of this blog post. That way, anyone could copy and paste into into their project to see how it works without having to worry about setup or downloaded additional packages.
using UnityEngine; using UnityEngine.Rendering; //Render 1 image to two displays with overlap that calculates the total resolution. public class MultiScreenManager : MonoBehaviour { [Tooltip("Overlap Percentage")] [Range(0, 100)] public float TotalOverlap = 59.4f; [Tooltip("The resolution of a single display")] public int SingleDisplayResolutionX = 1920; public int SingleDisplayResolutionY = 1080; //Values used to display the in-game menu so that overlap can be adjusted private bool showmenu; private string overlapString; private string resolutionString; //Aspect ratio of the screen when the resolution of with overlap has been calculated private float aspectRatio; private Camera leftCamera; private float leftCamDepth; private Camera rightCamera; private float rightCamDepth; //Render texture that contains the screen contents. private CustomRenderTexture targetRenderTexture; //Buffers to render from render texture to the screen private CommandBuffer leftCameraBuffer; private CommandBuffer rightCameraBuffer; private void Awake() { //Create a camera for each display leftCamera = CreateCamera(0); leftCamDepth = leftCamera.depth; rightCamera = CreateCamera(1); rightCamDepth = rightCamera.depth; leftCameraBuffer = new CommandBuffer(); leftCamera.AddCommandBuffer(CameraEvent.AfterEverything, leftCameraBuffer); rightCameraBuffer = new CommandBuffer(); rightCamera.AddCommandBuffer(CameraEvent.AfterEverything, rightCameraBuffer); } void Start() { //#if !UNITY_EDITOR if (Display.displays.Length == 1) { Debug.LogWarning("Only One Display Detected."); enabled = false; return; } //#endif UpdateOverlap(TotalOverlap); //This could be avoided if each camera had a script to capture the post render event Camera.onPostRender += OnPostRenderCam; //This is used to make sure that each camera is rendering at the correct aspect ratio. This is important for Camera overlay UI canvases Camera.onPreRender += OnPreRenderCam; //Toggle FullScreen UseFullScreen(); } private void UpdateOverlap(float overlap) { Vector2 totalResolution = GetResolution(overlap); resolutionString = totalResolution.x + "x" + totalResolution.y; overlapString = overlap.ToString(); SetResolution(totalResolution); SetDisplayCameras(totalResolution); if (verbos) { Debug.Log("Setting total resolution to " + resolutionString); } } private Vector2 GetResolution(float overlap) { Vector2 resolution = new Vector2(); //Horizontal resolution float overlapPercentage = overlap * .01f; float horizontalScreenMultiplier = 2 - (1 * overlapPercentage); resolution.x = Mathf.RoundToInt(SingleDisplayResolutionX * horizontalScreenMultiplier); //Vertical Resolution resolution.y = SingleDisplayResolutionY; return resolution; } private void SetResolution(Vector2 totalResolution) { //Set Resolution Screen.SetResolution((int)totalResolution.x, (int)totalResolution.y, FullScreenMode.FullScreenWindow); //Aspect Ratios aspectRatio = totalResolution.x / totalResolution.y; if (targetRenderTexture != null) { targetRenderTexture.Release(); Destroy(targetRenderTexture); } //Create a render texture and set it as the target render texture targetRenderTexture = new CustomRenderTexture((int)totalResolution.x, (int)totalResolution.y, RenderTextureFormat.ARGB32); } /// <summary> /// Set the buffers on the left and right display cameras that blit to the target screens. /// </summary> /// <param name="totalResolution"></param> private void SetDisplayCameras(Vector2 totalResolution) { //Clear the buffers if they had calls associated rightCameraBuffer.Clear(); leftCameraBuffer.Clear(); //How much percent does each camera see float horizontalScale = SingleDisplayResolutionX / (float)totalResolution.x; Vector2 singleTextureScale = new Vector2(horizontalScale, 1); Vector2 camera2Offset = new Vector2(1 - horizontalScale, 0); //Set the left camera renderer leftCameraBuffer.Blit(targetRenderTexture, BuiltinRenderTextureType.CurrentActive, singleTextureScale,; //Set the right camera renderer rightCameraBuffer.Blit(targetRenderTexture, BuiltinRenderTextureType.CurrentActive, singleTextureScale,camera2Offset); } /// <summary> /// Set the aspect ratio of all the other cameras to match the resolution. /// </summary> /// <param name="cam">the camera that is going to render</param> private void OnPreRenderCam(Camera cam) { if (!IsDisplayCamera(cam)) { cam.aspect = aspectRatio; } } /// <summary> /// Once we are rendering left display camera, blit the screen buffer to the target texture /// </summary> /// <param name="cam"> the camera that will render</param> private void OnPostRenderCam(Camera cam) { if (cam.Equals(leftCamera)) { Graphics.Blit(null, targetRenderTexture); } } /// <summary> /// Create a camera that renders to a display and renders after all of the other camras in the scene /// </summary> /// <param name="targetDisplay">The display that we want to target</param> /// <returns>Returns the newly created camera</returns> private Camera CreateCamera(int targetDisplay) { GameObject newGameObject = new GameObject(targetDisplay + "_Display Camera"); Camera newCamera = newGameObject.AddComponent<Camera>(); newCamera.clearFlags = CameraClearFlags.Nothing; newCamera.cullingMask = 0; newCamera.targetDisplay = targetDisplay; newCamera.depth = 99.9f + (targetDisplay / (float)100); return newCamera; } /// <summary> /// Compare a camera to check if it is a camera that is rendering to Display 0 or 1 /// </summary> /// <param name="cam">The camera to compare</param> /// <returns>Returns true if the camera is a display camera</returns> private bool IsDisplayCamera(Camera cam) { float currentDepth = cam.depth; if (currentDepth != leftCamDepth && currentDepth != rightCamDepth) { return false; } return cam == leftCamera || cam == rightCamera; } /// <summary> /// Activate FullScreen and if there are two displays activate the second display. /// </summary> private void UseFullScreen() { // Activate displays in builds or set fullscreen if not using multiple displays if (Application.platform == RuntimePlatform.WindowsPlayer || Application.platform == RuntimePlatform.OSXPlayer) { if (Display.displays.Length > 1) { Display.displays[1].Activate(); } else { Screen.fullScreen = true; } } } private void Update() { //Toggle IGUI menu by pressing R if (Input.GetKeyDown(KeyCode.R)) { showmenu = !showmenu; } } //Using IGUI to adjust the overlap of the screen in game private void OnGUI() { if (showmenu) { if (GUI.Button(new Rect(150, 5, 30, 30), "<")) { showmenu = !showmenu; } if (showmenu) { GUI.Label(new Rect(25, 185, 100, 30), "Overlap"); overlapString = GUI.TextField(new Rect(25, 200, 100, 30), overlapString); GUI.Label(new Rect(25, 235, 100, 30),resolutionString); if (GUI.Button(new Rect(25, 280, 100, 30), "Apply")) { UpdateOverlap(float.Parse(overlapString)); //Toggle FullScreen UseFullScreen(); } } } } }

Solution #2 | Using Spout and a render texture

If you have a complex projection installation consider using external tools like MadMapper to make sure the projection is aligned correctly. To revive the video frames in real time without incurring significant performance overhead, I recommend using Spout. (Link below)
To make it work with our mutli-camera setup, we used the custom Render Texture Sender and blit the Screen Buffer to a Render Texture. I’ve written a simple script to demonstrate this.
using UnityEngine; //Render the screen to a render Texture public class SpoutScreenManager : MonoBehaviour { [Tooltip("The resolution of the display")] public int DisplayResolutionX = 2700; public int DisplayResolutionY = 1080; [Tooltip("The render texture that Spout will reference")] public CustomRenderTexture targetRenderTexture; private Camera leftCamera; private void Awake() { //Create a camera with the depth of 100 to make sure it is the last to render //This script could have been added to the camera directly and you could avoid //checking which camera is being rendered. leftCamera = CreateCamera(0); Screen.SetResolution(DisplayResolutionX, DisplayResolutionY, FullScreenMode.MaximizedWindow); Camera.onPostRender += OnPostRenderCam; } private void OnPostRenderCam(Camera cam) { if (cam.Equals(leftCamera)) { Graphics.Blit(null, targetRenderTexture); } } //Create a camera with the depth of 100 private Camera CreateCamera(int targetDisplay) { GameObject newGameObject = new GameObject(targetDisplay + "_Display Camera"); Camera newCamera = newGameObject.AddComponent<Camera>(); newCamera.clearFlags = CameraClearFlags.Nothing; newCamera.cullingMask = 0; newCamera.targetDisplay = targetDisplay; newCamera.depth = 100; return newCamera; } private void OnDestroy() { targetRenderTexture.Release(); } } </Code>
Krystian Babilinski
Unity Developer - Programmer