Unity has given me a better understanding on how to use the stencil buffer to do portal rendering and I'd like to share how to do impossible space in Unity.
How it works is the portal is a separate 2D game object like a quad or a plane and a sector is a separate game object that is part of the map.
The map is separated into sectors to deal with the impossible space.
The portal has a stencil shader that writes to the stencil buffer and is set to always comparison.
The sector game object has a stencil shader on it that is read by the portal stencil shader and the comparison is set to equal.
The portal and sector stencil shader reference number is the same for both because the read stencil shader comparison is equal.
An impossible space hallway would have two portals with write stencil shaders for the openings and the hallway sector has a read stencil shader.
There are two overlapping sectors with different reference numbers and two portals for each sector then that would be impossible space hallways.
The overlapping sectors are different layers and the layers are changed in the project settings to not interact with each other.
The portal has a box collider trigger around it and a script attached that activates on trigger stay.
The portal has a mathematical plane in the direction of the portal normals.
Front side of the portal is positive and the back is negative.
When the player is at a distance of less than zero in front of the portal plane, the player layer changes to layer 8 or 9 and the comparison on the sector stencil shader changes from equal to always.
When the player is at a distance in front of the plane greater than zero then the sector stencil comparison changes back to equal and the player layer changes back to layer 0.
The player layer changes to match the sectors layer and layer 0 interacts with both layers 8 and 9, but layers 8 and 9 don't interact with each other.
The sector is rendered by the portals when compare is equal and is rendered normally when compare is always.
The sector stencil shader comparison changes to always before the player camera near clip reaches the portal.
The portals can be made in blender or quads can be used.
If you make portals in blender, move the portals origin from x 0, y 0, z 0 to directly on the portals center then the script will work correctly.
If the map is imported then you have to enable read/write on the model so the portals work.
Render queue is important.
Different reference number portals and sectors will need a different render queue.
Portals render before sectors.
lower number is rendered before higher numbered.
Portal script - attach this script to the portal game objects.
using UnityEngine;
using UnityEngine.Rendering;
public class PortalSwitch : MonoBehaviour
{
public GameObject Sector;
public int NextSector;
public int LastSector;
public float PortalDistance = 0f;
public GameObject[] AddPortals;
public GameObject[] RemovePortals;
private Material[] SectorMaterials;
private Plane portalPlane;
private GameObject Player;
void Start()
{
Player = GameObject.FindWithTag("Player");
Mesh portalMesh = GetComponent<MeshFilter>().sharedMesh;
Vector3[] normals = portalMesh.normals;
for (int i = 0; i < normals.Length; ++i)
{
portalPlane.SetNormalAndPosition(transform.TransformDirection(normals[i]), transform.position);
}
SectorMaterials = Sector.GetComponent<Renderer>().sharedMaterials;
for (int i = 0; i < SectorMaterials.Length; ++i)
{
SectorMaterials[i].SetInt("_SectorComp", (int)CompareFunction.Equal);
}
}
void OnTriggerStay(Collider other)
{
for (int i = 0; i < SectorMaterials.Length; ++i)
{
SectorMaterials[i].SetInt("_SectorComp", (int)CompareFunction.Always);
}
for (int i = 0; i < AddPortals.Length; ++i)
{
AddPortals[i].GetComponent<Renderer>().enabled = true;
}
if (portalPlane.GetDistanceToPoint(Camera.main.transform.position) < PortalDistance)
{
Player.gameObject.layer = NextSector;
}
else
{
Player.gameObject.layer = LastSector;
}
}
void OnTriggerExit(Collider other)
{
if (portalPlane.GetDistanceToPoint(Camera.main.transform.position) < PortalDistance)
{
for (int i = 0; i < SectorMaterials.Length; ++i)
{
SectorMaterials[i].SetInt("_SectorComp", (int)CompareFunction.Always);
}
for (int i = 0; i < RemovePortals.Length; ++i)
{
RemovePortals[i].GetComponent<Renderer>().enabled = false;
}
}
else
{
for (int i = 0; i < SectorMaterials.Length; ++i)
{
SectorMaterials[i].SetInt("_SectorComp", (int)CompareFunction.Equal);
}
for (int i = 0; i < AddPortals.Length; ++i)
{
AddPortals[i].GetComponent<Renderer>().enabled = false;
}
for (int i = 0; i < RemovePortals.Length; ++i)
{
RemovePortals[i].GetComponent<Renderer>().enabled = true;
}
}
}
void OnDestroy()
{
for (int i = 0; i < SectorMaterials.Length; ++i)
{
SectorMaterials[i].SetInt("_SectorComp", (int)CompareFunction.Equal);
}
}
}
How to use the PortalSwitch script.
The player game object has the tag player added to it.
The main camera is part of the player game object.
The sector you want to change rendering on goes in the sector area.
The next sector is used to change the player layer when the player enters the portal to the next sector.
The last sector is used to change the player layer when the player exits the portal to the last sector the player was in.
Sectors can be different layers so the player layer changes to that sectors layer
Portals can be different layers so the player layer interacts only with layers specified in project settings.
Add portals is for turning on the mesh renderer of portals on when you enter a portal and turning it off when you exit the portal.
Remove portals is for turning the mesh renderer of portals off when you enter a portal and turning it on when you exit the portal.
You can put more than one portal in a spot so you can render two or more sectors while the portal game objects are turned off.
Portal shader - This is the portal material shader.
Shader "Custom/Portal"
{
Properties
{
[IntRange] _PortalRef("Portal Ref Number", Range(0,255)) = 0
}
SubShader
{
Tags { "RenderType" = "Opaque" "Queue" = "Geometry-1"}
Stencil
{
Ref[_PortalRef]
Comp Always
Pass Replace
}
Pass
{
Blend Zero One
ZWrite Off
}
}
}
Sector shader - This is the overlapping rooms sector material shader.
Shader "Custom/Sector"
{
Properties
{
_Color("Color", Color) = (1,1,1,1)
_MainTex("Albedo (RGB)", 2D) = "white" {}
_Glossiness("Smoothness", Range(0,1)) = 0.5
_Metallic("Metallic", Range(0,1)) = 0.0
[HDR] _Emission("Emission", color) = (0,0,0)
[Enum(Equal,3,Always,8)] _SectorComp("Sector Comp", int) = 3
[IntRange] _SectorRef("Sector Ref Number", Range(0,255)) = 0
}
SubShader
{
Tags { "RenderType" = "Opaque" }
LOD 300
Stencil
{
Ref[_SectorRef]
Comp[_SectorComp]
}
CGPROGRAM
// Physically based Standard lighting model, and enable shadows on all light types
#pragma surface surf Standard fullforwardshadows
// Use shader model 3.0 target, to get nicer looking lighting
#pragma target 3.0
sampler2D _MainTex;
struct Input
{
float2 uv_MainTex;
};
half _Glossiness;
half _Metallic;
fixed4 _Color;
half3 _Emission;
// Add instancing support for this shader. You need to check 'Enable Instancing' on materials that use the shader.
// See https://docs.unity3d.com/Manual/GPUInstancing.html for more information about instancing.
// #pragma instancing_options assumeuniformscaling
UNITY_INSTANCING_BUFFER_START(Props)
// put more per-instance properties here
UNITY_INSTANCING_BUFFER_END(Props)
void surf(Input IN, inout SurfaceOutputStandard o)
{
// Albedo comes from a texture tinted by color
fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
o.Albedo = c.rgb;
// Metallic and smoothness come from slider variables
o.Metallic = _Metallic;
o.Smoothness = _Glossiness;
o.Alpha = c.a;
o.Emission = _Emission * tex2D(_MainTex, IN.uv_MainTex);
}
ENDCG
}
FallBack "Diffuse"
}
Impossible space is when two or more rooms geometry overlap.
The player has a height of 3.2 and a radius of 0.8 with the camera at 0.8 y on player.
The map scale is 4 = 1 world unit in Marathon.
