Games, like all complex programs, absolutely must consider object lifetimes. Objects need to be created at well-specified times, live from one designated time to another designated time, and then be destroyed and cleaned up at the end of their lives.
Whatever plan you follow, you need to follow it consistently. That seems to be the problem here.
Historically there have been many restrictions on memory, because game consoles had very little memory available, and a failed allocation due to many reasons (like memory fragmentation) generally results in program-terminating behavior. This in turn led to various practices. Some only allow allocations/released during level transitions or loading screens. Some have required no allocations/releases at all, only loading the entire world as a single block, then unloading the entire world as a single block. Some games have limited allocations/releases to only be done outside of certain times, like outside of the main loop during inter-frame delays.
Many object lifetimes must be guaranteed by policies. When a function receives an object pointer as a parameter, a general guarantee is that the object will remain in memory for the duration of the function. However, unless it is specified in the broad design saving the pointer to the object is bad because the pointer may be a temporary object, or may be unloaded through other systems. You may decide that these pointers are no longer guaranteed at the end of a graphics frame, or that they are no longer guaranteed after a level load, or that the code must always follow a pair of registering an object exactly once, then unregistering the object a single time before it is destroyed.
Smart pointers can solve a few problems related to object management when there are multiple systems that must share certain responsibilities for object lifetimes, but they do not replace a solid design. Smart pointers still fail terribly under many conditions, and they still suffer from resource leaks under conditions like circular references. They do not replace the fundamenal issue that object lifetime must be well understood.
Getting back to the issue with this code...
Under one set of circumstances, the class is using an external object. Other systems own the lifetime of the object, other systems create the object, and other systems destroy the object. This is the most common pattern in software.
Under another set of circumstances, the class acquires ownership of an external object. Other systems create the object, but under various circumstances this system will destroy the object. This is NOT the typical pattern in software. The typical object lifetime rules are being violated.
In general you should write code that follows a source/sink model. If a system is the source of creating objects, it should also sink or destroy them when they are done. That is, you should always have a pair of functions for create/destroy, allocate/free, new/delete, load/unload, add/remove, and keep those responsibilities paired together. Either this object controls ALL aspects of sub-object lifetime, or it controls NONE of the sub-object lifetime.
With an 'all or none' guarantee, code that uses the system can follow more broad policy rules about when it is safe to allocate or release resources.
This also allows for alternate forms of memory management. You can use different systems for pools of memory, allocating some stuff from one pool or global memory, allocating other stuff from different pools or different memory managers. If you do ALL the work you can use your specified pool or memory manager. If you do NONE of the work those who use the system can use whatever memory management they wish. But if you take partial control, outside systems are limited to only use the forms you use in your half of the system.