[FEATURE] Buffers for entity creation, destruction, structural changes and value insertation.

[genaray]

Well its me again... yeah i know, but i recently used Unitys ECS and came across [CommandBufferSystems] (https://docs.unity3d.com/Packages/com.unity.entities@0.4/manual/entity_command_buffer.html).

Those are used to "buffer" the creation, destruction and composition of entities, great if you wanna create a bunch of entities at the start of the frame for example. Or if you have multiple threads working on entities.

As i did for the Job-System Feature Request, i came along my own implementation aswell. It would be great if artemis ODB would provide such "simply" but "essential" tools itself :) Its honestly a pain to take care of that, but i hope that my "idea" of the buffer system is a good point to start.

Currently im using it in my own game, there probably still some little bugs.
I havent found a nice solution for buffering changes to component attributes... but the current implementation should do the trick for the first.

Dont get confused by all the maps, buffered maps contains all buffered elements for entites not existant yet. And the other maps ( without buffered in their name ) contain buffered elements for already existing entities. The system either uses the artemis entity id for applying buffered changes to existing entities or generates a internal id for the buffered creation of entities, that one is also used to modify the buffered entities.

package com.parallelorigin.extension.code.ecs.systems.lifecycle.buffer;

import com.artemis.Archetype;
import com.artemis.BaseSystem;
import com.artemis.Component;
import com.artemis.Entity;
import com.parallelorigin.extension.base.classes.pattern.typeobject.IDManager;
import com.parallelorigin.extension.base.classes.structure.Tuple;
import com.parallelorigin.extension.base.interfaces.structure.ITuple;

import java.util.Map;
import java.util.Queue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.function.Consumer;

/**
 * A system that is used to buffer {@link Entity} creation, destruction and modification calls.
 */
public abstract class BufferSystem extends BaseSystem {

    /** A simply class that holds values for a buffered change of a component **/
    public class BufferedChange {

        public Class<? extends Component> target;
        public Consumer<Component> change;

        /**
         * Creates the buffered change.
         * @param target The target component class
         * @param change A callback that modifies the passed object
         */
        public BufferedChange(Class<? extends Component> target, Consumer<Component> change) {
            this.target = target;
            this.change = change;
        }
    }

    protected Queue<Runnable> onProcess = new ConcurrentLinkedQueue<>();

    // Creation & Destruction
    protected Queue<Integer> createRawQueue = new ConcurrentLinkedQueue<>();
    protected Queue<ITuple<Integer, Archetype>> createArcheTypeQueue = new ConcurrentLinkedQueue<>();
    protected Queue<Integer> destroyQueue = new ConcurrentLinkedQueue<>();

    // Structural changes
    protected Map<Integer, Queue<Class<? extends Component>>> createToBufferedQueue = new ConcurrentHashMap<>();
    protected Map<Integer, Queue<Component>> addToBufferedQueue = new ConcurrentHashMap<>();
    protected Map<Integer, Queue<Class<? extends Component>>> removeFromBufferedQueue = new ConcurrentHashMap<>();

    protected Map<Integer, Queue<Class<? extends Component>>> createQueue = new ConcurrentHashMap<>();
    protected Map<Integer, Queue<Component>> addQueue = new ConcurrentHashMap<>();
    protected Map<Integer, Queue<Class<? extends Component>>> removeQueue = new ConcurrentHashMap<>();

    // Value changes

    protected Map<Integer, Queue<BufferedChange>> bufferedChangesQueue = new ConcurrentHashMap<>();
    protected Map<Integer, Queue<BufferedChange>> changesQueue = new ConcurrentHashMap<>();

    // Internal ID to real ID
    protected Map<Integer, Integer> bufferIdToEntity = new ConcurrentHashMap<>();

    @Override
    protected void processSystem() {

        while(!onProcess.isEmpty()) onProcess.poll().run();

        // Create entities and put them into the list
        while(!createRawQueue.isEmpty()){

            int bufferID = createRawQueue.poll();
            var entity = world.create();

            bufferIdToEntity.put(bufferID, entity);
        }

        // Create entities by their archetype and put them into the list
        while(!createArcheTypeQueue.isEmpty()){

            var tuple = createArcheTypeQueue.poll();

            int bufferID = tuple.getKey();
            var entity = world.create(tuple.getValue());

            bufferIdToEntity.put(bufferID, entity);
        }

        // Create buffered components to the newly created buffered entities
        for(var entry : createToBufferedQueue.entrySet()){

            int bufferID = entry.getKey();
            int entity = bufferIdToEntity.get(bufferID);

            var createComponentsQueue = entry.getValue();
            while(!createComponentsQueue.isEmpty()){ world.edit(entity).create(createComponentsQueue.poll()); }
        }

        // Add buffered components to the newly created buffered entities
        for(var entry : addToBufferedQueue.entrySet()){

            int bufferID = entry.getKey();
            int entity = bufferIdToEntity.get(bufferID);

            var addComponentsQueue = entry.getValue();
            while(!addComponentsQueue.isEmpty()){ world.edit(entity).add(addComponentsQueue.poll()); }
        }

        // Remove buffered components from the newly created buffered entities
        for(var entry : removeFromBufferedQueue.entrySet()){

            int bufferID = entry.getKey();
            int entity = bufferIdToEntity.get(bufferID);

            var addComponentsQueue = entry.getValue();
            while(!addComponentsQueue.isEmpty()){ world.edit(entity).remove(addComponentsQueue.poll()); }
        }


        // Remove buffered components from the newly created buffered entities
        for(var entry : bufferedChangesQueue.entrySet()){

            int bufferID = entry.getKey();
            int entity = bufferIdToEntity.get(bufferID);

            var changesQueue = entry.getValue();
            while(!changesQueue.isEmpty()){

                var bufferedReplace = changesQueue.poll();
                var cmp = world.getEntity(entity).getComponent(bufferedReplace.target);

                bufferedReplace.change.accept(cmp);
            }
        }

        // Create buffered components to the existing entities
        for(var entry : createQueue.entrySet()){

            int entity = entry.getKey();

            var createComponentsQueue = entry.getValue();
            while(!createComponentsQueue.isEmpty()){ world.edit(entity).create(createComponentsQueue.poll()); }
        }

        // Add buffered components to the existing entities
        for(var entry : addQueue.entrySet()){

            int entity = entry.getKey();

            var addComponentsQueue = entry.getValue();
            while(!addComponentsQueue.isEmpty()){ world.edit(entity).add(addComponentsQueue.poll()); }
        }

        // Remove buffered components from the existing entities
        for(var entry : removeQueue.entrySet()){

            int entity = entry.getKey();

            var addComponentsQueue = entry.getValue();
            while(!addComponentsQueue.isEmpty()){ world.edit(entity).remove(addComponentsQueue.poll()); }
        }

        // Change component value from the existing entities
        for(var entry : changesQueue.entrySet()){

            int entity = entry.getKey();

            var changesQueue = entry.getValue();
            while(!changesQueue.isEmpty()){

                var bufferedChange = changesQueue.poll();
                var cmp = world.getEntity(entity).getComponent(bufferedChange.target);

                bufferedChange.change.accept(cmp);
            }
        }


        // Clearing, prevents that we write the same buffered components multiple times into the same entity
        addToBufferedQueue.clear();
        removeFromBufferedQueue.clear();
        bufferedChangesQueue.clear();
        addQueue.clear();
        removeQueue.clear();
        changesQueue.clear();
        bufferIdToEntity.clear();

        // Destroy entities
        while(!destroyQueue.isEmpty()) world.delete(destroyQueue.poll());
    }

    /**
     * Enlists a {@link Runnable} Callback getting executed while this system processes.
     * @param execute The runnable callback we wanna execute during this frame.
     */
    public void buffer(Runnable execute){ onProcess.add(execute); }

    /**
     * Creates a buffer {@link Entity} and returns its internal ID which can be used to modify that buffered {@link Entity}
     * @return The internal id of the buffered entity.
     */
    public int create(){

        var id = (int)IDManager.getID(BufferSystem.class);;
        createRawQueue.add(id);
        bufferIdToEntity.put(id, 0);

        return id;
    }

    /**
     * Creates a buffer {@link Entity} by its {@link Archetype} and returns its internal ID which can be used to
     * modify that buffered {@link Entity}
     * @return The internal id of the buffered entity.
     */
    public int create(Archetype archetype){

        var id = (int)IDManager.getID(BufferSystem.class);;
        createArcheTypeQueue.add(new Tuple<>(id, archetype));
        bufferIdToEntity.put(id, 0);

        return id;
    }

    /**
     * Buffers a new component for being created, to a already buffered {@link Entity} or existing {@link Entity}which
     * gets added once this system was processed.
     * @param id The unique internal id of the buffered {@link Entity} or the global id of a {@link Entity}
     * @param component The component to add to the {@link Entity}
     */
    public void create(int id, Class<? extends Component> component){

        // Either put the modification into the queue that works on non existent entities or into the queue that
        // works with existing entities.
        if(bufferIdToEntity.containsKey(id)){

            if(createToBufferedQueue.containsKey(id)) createToBufferedQueue.get(id).add(component);
            else createToBufferedQueue.put(id, new ConcurrentLinkedQueue<>(){{add(component);}});
        }
        else {

            if(createQueue.containsKey(id)) createQueue.get(id).add(component);
            else createQueue.put(id, new ConcurrentLinkedQueue<>(){{add(component);}});
        }
    }

    /**
     * Buffers a component to add or to set,  to a already buffered {@link Entity} or existing {@link Entity} which
     * gets added once this system was processed.
     * @param id The unique internal id of the buffered {@link Entity} or the global id of a {@link Entity}
     * @param component The component to add to the {@link Entity}
     */
    public void add(int id, Component component){

        // Either put the modification into the queue that works on non existent entities or into the queue that
        // works with existing entities.
        if(bufferIdToEntity.containsKey(id)){

            if(addToBufferedQueue.containsKey(id)) addToBufferedQueue.get(id).add(component);
            else addToBufferedQueue.put(id, new ConcurrentLinkedQueue<Component>(){{add(component);}});
        }
        else {

            if(addQueue.containsKey(id)) addQueue.get(id).add(component);
            else addQueue.put(id, new ConcurrentLinkedQueue<Component>(){{add(component);}});
        }
    }


    /**
     * Enlist a {@link BufferedChange} to modify a {@link Component} in the buffer
     * @param id The unique internal id of the buffered {@link Entity} or the global id of a {@link Entity}
     * @param component The component we wanna modify a attribute in
     * @param changeCallback The callback that is used to change the component and its attributes
     */
    public <T extends Component> void change(int id, Class<T> component, Consumer<T> changeCallback){

        if(bufferIdToEntity.containsKey(id)){
            if(bufferedChangesQueue.containsKey(id)) bufferedChangesQueue.get(id).add(new BufferedChange(component, (Consumer<Component>) changeCallback));
            else bufferedChangesQueue.put(id, new ConcurrentLinkedQueue<>(){{add(new BufferedChange(component, (Consumer<Component>) changeCallback));}});
        }
        else {

            if(changesQueue.containsKey(id)) changesQueue.get(id).add(new BufferedChange(component, (Consumer<Component>) changeCallback));
            else changesQueue.put(id, new ConcurrentLinkedQueue<>(){{add(new BufferedChange(component, (Consumer<Component>) changeCallback));}});
        }
    }

    /**
     * Buffers a component removal to a already buffered {@link Entity} or existing one which gets removed as soon as
     * this system was processed.
     * @param id The unique internal id of the buffered {@link Entity} or the id of the global entity.
     * @param component The component to remove to the buffered {@link Entity}
     */
    public void remove(int id, Class<? extends Component> component){

        if(bufferIdToEntity.containsKey(id)){
            if(removeFromBufferedQueue.containsKey(id)) removeFromBufferedQueue.get(id).add(component);
            else removeFromBufferedQueue.put(id,new ConcurrentLinkedQueue<Class<? extends Component>>(){{add(component);}});
        }
        else {

            if(removeQueue.containsKey(id)) removeQueue.get(id).add(component);
            else removeQueue.put(id,new ConcurrentLinkedQueue<Class<? extends Component>>(){{ add(component);}});
        }
    }

    /**
     * Enlists a entity for being destroyed at the end of the frame.
     * @param entityID The entity-id
     */
    public void delete(int entityID){ destroyQueue.add(entityID); }
}

/**
 * This class is used to generate unique id's for different class types. <p>
 * It reuses old ID's which arent used anymore.
 */
public class IDManager {

    private static ConcurrentHashMap<Class<?>, Queue<Long>> freed = new ConcurrentHashMap<>();

    /**
     * This method returns a unique long id for each class type.
     * @param object
     * @return
     */
    public static synchronized long getID(Class<?> object){

        // If theres a free ID... reuse it !
        if(freed.containsKey(object) && !freed.get(object).isEmpty())
            return freed.get(object).poll();

        // If no free ID's... create a new one !
        return UUID.randomUUID().getMostSignificantBits() & Long.MAX_VALUE;
    }

    /**
     * Removes a generated ID for a specifc class type. <p>
     * This id gets reused later on.
     * @param id
     * @param object
     */
    public static synchronized void removeID(long id, Class<?> object){

        if(freed.containsKey(object))
            freed.get(object).add(id);
        else freed.put(object, new ArrayDeque<Long>(){{add(id);}});
    }
}

/**
 * Represents a tuple... containing two values.
 */
public class Tuple<T,E> implements ITuple<T, E> {

    private T key;
    private E value;

    public Tuple(T key, E value){

        this.key = key;
        this.value = value;
    }

    @Override
    public T getKey() { return key; }

    @Override
    public E getValue() { return value; }

    @Override
    public String toString() { return "("+key+","+value+")"; }
}

/**
 * A interface for a class which is able to store a key and a value as a single tuple.
 * @param <K> The key
 * @param <V> The Value
 */
public interface ITuple<K, V> {

    /**
     * Returns the key.
     * @return The key
     */
    K getKey();

    /**
     * Returns the value.
     * @return The value
     */
    V getValue();
 }

Here a little example of how to use this buffer system ;)

        // Creating the entity once the buffer system is processing
        var playerEntityID = myBufferSystem.create(myPlayerArcheType);

        // Adding stuff to the not existent yet entity
        myBufferSystem.add(playerEntityID , new Invincible(10.0f));

        // Change some stuff
        myBufferSystem.change(playerEntityID , Player.class, (player) -> player.name = "God");
        myBufferSystem.change(playerEntityID , Velocity.class, (velo) -> velo.speed = 10000.0f);

         // Remove some components
        myBufferSystem.remove(playerEntityID, Collider.class);

        // Congratulations, you just buffered a entity which will get created and modified once the used buffer system is processing 
        // :) 

[DaanVanYperen]

Man you are an idea machine. Appreciate the posts. I'm reading it in backwards order.

Do you have any usecases of ReactiveSystem? I'm looking for a reason why everyone should be using this (and it should be integrated in core instead of a plugin). It feels fairly specialized.

A command pattern might improve the design to preserve order of operations, otherwise you get strange behavior when somebody adds, deletes and adds again. Something like what the operations plugin does (functionally very close to this but currently no callbacks).
https://github.com/DaanVanYperen/artemis-odb-contrib/wiki/Operations-Plugin