Reference counting as memory management has a drawback in that it is difficult to detect circular references when determining dead objects. AngelScript allows the application to register types with special behaviours to support the garbage collection for detecting circular references. These behaviours make the class a bit more complex, but you should only have to register them for a few types, e.g. generic container classes.
The difference between the garbage collected and non-garbage collected types is in the addref and release behaviours, the class constructor, and the extra support behaviours.
The GC determines when objects should be destroyed by counting the references it can follow for each object. If the GC can see all references that points to an object, it knows that the object is part of a circular reference. If all the objects involved in that circular reference have no outside references it means that they should be destroyed.
The process of determining the dead objects uses the first four of the behaviours below, while the destruction of the objects is done by forcing the release of the object's references.
Whenever a garbage collected class is created, the garbage collector must be notified of it's existence. The easiest way of doing that is to have the factory behaviour, or the class constructor call the NotifyGarbageCollectorOfNewObject()
method on the engine when initializing the class.
You may want to consider caching the typeId, so that it doesn't have to be looked up through the relatively expensive call to GetTypeIdByDecl every time an object of this type is created.
Note, if you create objects of this type from the application side, you must also notify the garbage collector of its existence, so it's a good idea to make sure all code use the same way of creating objects of this type.
For garbage collected objects it is important to make sure the AddRef and Release behaviours clear the GC flag. Otherwise it is possible that the GC incorrectly determine that the object should be destroyed.
Value types are normally not thought of as being part of circular references as they themselves cannot be referenced, however if a value type can hold a reference to a type, and then that type can have the value type as a member then a circular reference can be established preventing the reference type from being released.
To solve these situations the value types can also be registered with some of the garbage collector behaviours.
Only the asBEHAVE_ENUMREFS and asBEHAVE_RELEASEREFS should be registered for value types. These work the same way as for reference types, i.e. the asBEHAVE_ENUMREFS should call the engine's GCEnumCallback for all references held, and asBEHAVE_RELEASEREFS should clear all references held.
Reference types that contain value types that have GC behaviours need to have the asBEHAVE_ENUMREFS and asBEHAVE_RELEASEREFS behaviours adapted for this by forwarding the enum and release call to the value type. This forward is done by calling the engine's ForwardGCEnumReferences or ForwardGCReleaseReferences respectively.
If you plan on executing scripts from multiple threads with automatic garbage collection turned on, or if you plan on running the garbage collector manually from a background thread, then you must make sure that the object type behaviours that support the garbage collector are thread-safe. Especially the ADDREF, RELEASE, and ENUMREFS behaviours have a high probability of being invoked from multiple threads simultaneously. The RELEASEREFS behaviour will only be invoked when the Garbage Collector has determined that the object is already dead so it is guaranteed not to be invoked by multiple threads. The others, GETREFCOUNT, SETGCFLAG, and GETGCFLAG, are not sensitive as the garbage collector just use the information as a hint.
Making the ADDREF and RELEASE behaviours thread-safe is easy with the use of asAtomicInc and asAtomicDec. If the object is static container, i.e. the memory layout of the contents cannot change, then ENUMREFS is already thread-safe, but if memory layout can change, e.g. dynamic arrays or hash maps, then the iteration over the content in ENUMREFS must be protected so that it doesn't break in case the memory happen to change in the middle of the iteration.