Francis Tao

Francis Tao

3D UI of Box3 Game Editor

Combine Webgl and React to create 3d UI

In modeling software and game editors, 3dUI is an indispensable part, through which user can intuitively manipulate the objects they edit. Some common 3dUIs are looks like:

common 3dUIs

In the Box3 game editor, we also need such manipulators. If we think about it a little bit, we will find that these 3d ui and ordinary React components actually have a lot in common. For example, they can be easily created and destroyed, they can accept props to calculate the initial state, and they have to actively respond to the user’s actions (translation, rotation, scale).

That why we want to use the lifecycle of traditional React components to control the creation and destruction of 3d UI. But here’s another problem, the data used in webgl is outside of React, but with the React Context API, we can inject that data into React.

Here is the diagram show how this system was designed.

system design diagram

So, what exactly is a 3dUI?

If you look at it from a React’s perspective, the combination of a UI3DContext.Consumer and a EliminateWrapper in the diagram is a 3dUI component. If you look at it from the perspective of the WebGL rendering engine, then the 3dUI data generated by the UI3DContext.Consumer is actually a 3dUI Element.

At the specific code implementation level, we use Mudb to build the schema of UI3dState.(Mudb is the core opensource library that we use in ‘Box3’, it was build by our team). You may use any method you like, but the overall design pattern has shown in the diagram.

The simplest example would be a 3d Point(or you can call it Sphere), I will use it for a detailed demonstration. First, we define the schema for UI3DState and UI3DSphere.

// define mouse event schema
export const UI3DMouseEventSchema = new MuStruct({
    eye: Vec3Schema,
    mouseButton: new MuUint8(),
    mouseState: new MuUint8(),
    worldPosition: Vec3Schema,
    screenPosition: Vec2Schema,
});
export type UI3DMouseEvent = typeof UI3DMouseEventSchema.identity;

export type UI3DMouseHandler = (event:UI3DMouseEvent) => boolean;

export class UI3DEventHandlerNode {
    public next:UI3DEventHandlerNode|null = null;
    public prev:UI3DEventHandlerNode|null = null;
}

// Schema for UI3DSphere
export const UI3DSphereSchema = new MuStruct({
    id: new MuUint32(),
    destroyed: new MuBoolean(false),
    position: Vec3Schema,
    radius: new MuFloat64(1),
    color: Vec3Schema,
    onMouseChange: new MuRefSchema<UI3DMouseHandler>(),
});

// you can define more different kind of UI3DElement
export const UI3DElementSchema = new MuUnion({
    point: UI3DSphereSchema,
});
export type UI3DElement = typeof UI3DElementSchema.identity;

// an event record
export const UI3DEventSchema = new MuStruct({
    // hit distance
    hitDistance: new MuFloat64(),
    // event type
    event: new MuUnion({
        mouse: UI3DMouseEventSchema,
    }),
    // event handler
    handler: new MuRefSchema<(event:any) => boolean>(),
});
export const UI3DEventQueueSchema = new MuArray(UI3DEventSchema, Infinity);

export type UI3DEventQueue = typeof UI3DEventQueueSchema.identity;

// Schema for UI3DState
export const UI3DStateSchema = new MuStruct({
    idCounter: new MuUint32(),

    // elements list
    elements: new MuArray(UI3DElementSchema, Infinity),

    // event handler list
    eventHandlers: new MuRefSchema<UI3DEventHandlerNode>(),

    // previous event handler list
    prevEventStack: UI3DEventQueueSchema,
});
export type UI3DState = typeof UI3DStateSchema.identity;

Now let’s come back to React, we need to create the UI3DContext and defined a component to provide the context provider.

export const UI3DContext = React.createContext(UI3DStateSchema.identity);

export type UI3DContainerProps = {
    state:UI3DContext;
};
// define the UI3DContainer component to provide UI3DContext.Provider
// the reason why we don't call UI3DContext.Provider directly is because 
// we want to manage our UI3d state with the help of the React component's lifecycle method.
export class UI3DContainer extends React.Component<UI3DContainerProps, {}> {
    private _node:UI3DEventHandlerNode|null = null;

    public componentWillUnmount() {
        const ui3d = this.props.state;
        if (this._node) {
            if (this._node === ui3d.eventHandlers) {
                ui3d.eventHandlers = this._node.next;
            }
            if (this._node.next) {
                this._node.next.prev = this._node.prev;
            }
            if (this._node.prev) {
                this._node.prev.next = this._node.next;
            }
            this._node = null;
        }
    }

    public render() {
        const ui3d = this.props.state;
        if (!this._node) {
            this._node = new UI3DEventHandlerNode();
            this._node.next = ui3d.eventHandlers;
            if (this._node.next) {
                this._node.next.prev = this._node;
            }
            this._node.prev = null;
            ui3d.eventHandlers = this._node;
        }

        return (
            <UI3DContext.Provider value={this.props.state}>
                {this.props.children}
            </UI3DContext.Provider>
        );
    }
}

Now, we can create the actual 3dUI component of 3d Point, remember, from React’s perspective, it is a UI3DContext.Comsumer and a EliminateWrapper.

interface EliminateWrapperProps {
    elements:UI3DElement[];
    element:UI3DElement;
    clearMethod:(fn:() => void) => void;
}

class EliminateWrapper extends React.Component<EliminateWrapperProps, {}> {
    constructor(props) {
        super(props);
        this.componentWillUnMount = this.componentWillUnMount.bind(this);
    }

    // this is very strange, that componentWillUnMount will not be called by react when the component is in the Consumer
    // I have search a lot but could not find much resources about it, that why it end up call by an hook from parent ....
    public componentWillUnMount() {
        for (let i=0; i < this.props.elements.length;  i++) {
            if (this.props.elements[i].data.id === this.props.element.data.id) {
                this.props.elements.splice(i, 1);
                break;
            }
        }
    }

    public componentDidMount() {
        this.props.klarMethod(this.componentWillUnMount);
    }

    public render() {
        return '';
    }
}

export function UI3DContextWrapper<Props, ElementType extends keyof typeof UI3DElementSchema.muData, Schema extends MuSchema<any>>(
    schema:Schema,
    elementType:ElementType,
    factory:(element:typeof UI3DElementSchema.muData[ElementType]['identity'], props:Props, state:typeof UI3DStateSchema['identity']) => void) {

    class UI3DContextInjector extends React.Component<Props, {}> {

        private _element?:null|UI3DElement;
        private _clearData:() => void = () => {};

        private clearMethod(childUmont:() => void) {
            this._clearData = childUmont;
        }

        constructor (props) {
            super(props);
            this.klarMethod = this.klarMethod.bind(this);
        }

        public componentWillUnmount() {
            if (this._element) {
                this._element.data.destroyed = true;
                if (this._clearData) {
                    this._clearData();
                }
                this._element = null;
            }
        }

        public render () {
            return (
                <UI3DContext.Consumer>
                    {(context) => {
                        if (this._element) {
                            factory(this._element.data, this.props, context);
                        } else {
                            const element = UI3DElementSchema.alloc();
                            element.type = elementType;
                            const data = element.data = schema.clone(schema.identity);
                            data.id = context.idCounter++;
                            data.destroyed = false;
                            this._element = element;
                            context.elements.push(element);
                            factory(data, this.props, context);
                        }
                        return <EliminateWrapper elements={context.elements} element={this._element} clearMethod={this.clearMethod}/>;
                    }}
                </UI3DContext.Consumer>
            );
        }
    }
    return UI3DContextInjector;
}

Look closely at UI3DContextInjector’s render function, the ui3d element is created in the UI3DContext.Consumer, and the element will be add into UI3D State, it will also pass into EliminateWrapper as a prop. Finally, we can defined UI3DPoint component

export const UI3DPoint = UI3DContextWrapper(UI3DSphereSchema, 'point', (
    element,
    props:{
        position:vec3,
        radius?:number,
        color?:vec3,
        onMouse?:UI3DMouseHandler,
    },
    context) => {
    vec3.copy(element.position, props.position);
    if ('color' in props) {
        vec3.copy(element.color, (props.color as vec3));
    } else {
        vec3.set(element.color, 0, 0, 0);
    }
    if ('radius' in props) {
        element.radius = (props.radius as number);
    } else {
        element.radius = 1;
    }
    if ('onMouse' in props) {
        element.onMouseChange = (props.onMouse as UI3DMouseHandler);
    } else {
        element.onMouseChange = null;
    }
});

And this is how we can use it:

<UI3DContainer state={ui3dState}>
    ...
    <UI3DPoint
        position={[5, 5.5, 5]}
        radius={2}
        color={[1,0,0]}
        onMouse={(ev) => {
            ....
        }}
    ></UI3DPoint>
    ...
</UI3DContainer>

With this design, you can easily combine multiple 3d components into one more advanced component. This can be done with React, without modifying any WebGL-related code. Here are examples of our 3d component.

← Back to home
Your browser doesn't appear to support the HTML5 <canvas> element.