Multi-thread, Pthread APIs And Thread Management

Note: This document describes the old pthread implementation. See Pthread implementations.

WAMR provides a built-in library to support pthread APIs. You can call pthread APIs in your application source code.

Build and run

Suppose you have written a C program calling pthread_create() to create a thread, and the file name is main.c

#include <stdio.h>
#include <pthread.h>

void *thread_routine(void *arg)
{
    printf("Enter thread\n");
    pthread_exit(NULL);
    return NULL;
}

int main(int argc, char** argv)
{
    pthread_t tid;

    if (0 != pthread_create(&tid, NULL, thread_routine, NULL)) {
        printf("Failed to create thread\n");
    }

    if (0 != pthread_join(tid, NULL)) {
        printf("Failed to join thread %d.\n", tid);
    }

    printf("Exit\n");

    return 0;
}

Build with libc-builtin

To build this C program into WebAssembly app with libc-builtin, you can use this command:

/opt/wasi-sdk/bin/clang  --target=wasm32        \
    --sysroot=${WAMR_ROOT}/wamr-sdk/app/libc-builtin-sysroot   \
    -O3 -pthread -nostdlib -z stack-size=32768      \
    -Wl,--shared-memory             \
    -Wl,--initial-memory=131072,--max-memory=131072 \
    -Wl,--allow-undefined-file=${WAMR_ROOT}/wamr-sdk/app/libc-builtin-sysroot/share/defined-symbols.txt \
    -Wl,--no-entry -Wl,--export=main                \
    -Wl,--export=__heap_base,--export=__data_end    \
    -Wl,--export=__wasm_call_ctors  \
    main.c -o test.wasm
# -pthread: it will enable some dependent WebAssembly features for thread
# -nostdlib: disable the WASI standard library as we are using WAMR builtin-libc
# -z stack-size=: specify the total aux stack size
# -Wl,--export=__heap_base,--export=__data_end: export these globals so the runtime can resolve the total aux stack size and the start offset of the stack top
# -Wl,--export=__wasm_call_ctors: export the init function to initialize the passive data segments

Build with libc-WASI

You can also build this program with WASI, but we need to make some changes to wasi-sysroot:

1. disable malloc/free of wasi, as they are not atomic operations:

   /opt/wasi-sdk/bin/llvm-ar -d /opt/wasi-sdk/share/wasi-sysroot/lib/wasm32-wasi/libc.a dlmalloc.o

2. copy the pthread.h to wasi-sysroot so the compiler can find it:

   cp ${WAMR_ROOT}/wamr-sdk/app/libc-builtin-sysroot/include/pthread.h /opt/wasi-sdk/share/wasi-sysroot/include

Note:

1. Remember to back up the original sysroot files

Then build the program with this command:

/opt/wasi-sdk/bin/clang -pthread -O3                \
    -Wl,--shared-memory,--max-memory=196608         \
    -Wl,--allow-undefined,--no-check-features       \
    -Wl,--export=__heap_base,--export=__data_end    \
    main.c -o test.wasm
# -Wl,--no-check-features: the errno.o in wasi-sysroot is not compatible with pthread feature, pass this option to avoid errors

Build with EMCC

Note: This document is based on emcc 2.0.26, other version may not work with these commands

EMCC's -pthread option is not compatible with standalone mode, we need to pass -mbulk-memory -matomics to the compiler and --shared-memory,--no-check-features to linker manually

EMCC provides some empty implementation for pthread related APIs, we need to remove them from emcc's libc.

cd ${emsdk_dir}/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten
emar d libc.a library_pthread_stub.o
emranlib libc.a

emcc -O3 -mbulk-memory -matomics -s MALLOC="none"   \
     -Wl,--export=__data_end,--export=__heap_base   \
     -Wl,--shared-memory,--no-check-features        \
     -s ERROR_ON_UNDEFINED_SYMBOLS=0                \
     main.c -o test.wasm

Build AOT module

You can build the wasm module into AOT module with pthread support, please pass option --enable-multi-thread to wamrc:

wamrc --enable-multi-thread -o test.aot test.wasm

Currently WAMR disables pthread library by default. To run the module with pthread support, please build the runtime with -DWAMR_BUILD_LIB_PTHREAD=1

cd ${WAMR_ROOT}/product-mini/platforms/linux
mkdir build && cd build
cmake .. -DWAMR_BUILD_LIB_PTHREAD=1
make
# Then you can run the wasm module above:
./iwasm test.wasm
# Or the AOT module:
# ./iwasm test.aot

Here is also a sample to show how wasm-apps use pthread APIs to create threads, and how to build it with cmake. You can build this sample and have a try:

cd ${WAMR_ROOT}/samples/multi-thread
mkdir build && cd build
cmake ..
make
# Run wasm application
./iwasm wasm-apps/test.wasm

Aux stack separation

The compiler may use some spaces in the linear memory as an auxiliary stack. When pthread is enabled, every thread should have its own aux stack space, so the total aux stack space reserved by the compiler will be divided into N + 1 parts, where N is the maximum number of threads that can be created by the user code.

The default value of N is 4, which means you can create 4 threads at most. This value can be changed by an option if you are using product-mini:

./iwasm --max-threads=n test.wasm

If you are going to develop your own runtime product, you can use the API wasm_runtime_set_max_thread_num or init arg init_args.max_thread_num to set the value, or you can change the macro CLUSTER_MAX_THREAD_NUM in config.h.

Note: the total size of aux stack reserved by compiler can be set with -z stack-size option during compilation. If you need to create more threads, please set a larger value, otherwise it is easy to cause aux stack overflow.

Supported APIs

/* Thread APIs */
int pthread_create(pthread_t *thread, const void *attr,
                   void *(*start_routine) (void *), void *arg);

int pthread_join(pthread_t thread, void **retval);

int pthread_detach(pthread_t thread);

int pthread_cancel(pthread_t thread);

pthread_t pthread_self(void);

void pthread_exit(void *retval);

/* Mutex APIs */
int pthread_mutex_init(pthread_mutex_t *mutex, const void *attr);

int pthread_mutex_lock(pthread_mutex_t *mutex);

int pthread_mutex_unlock(pthread_mutex_t *mutex);

int pthread_mutex_destroy(pthread_mutex_t *mutex);

/* Cond APIs */
int pthread_cond_init(pthread_cond_t *cond, const void *attr);

int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex);

int pthread_cond_timedwait(pthread_cond_t *cond, pthread_mutex_t *mutex,
                           unsigned int useconds);

int pthread_cond_signal(pthread_cond_t *cond);

int pthread_cond_broadcast(pthread_cond_t *cond);

int pthread_cond_destroy(pthread_cond_t *cond);

/* Pthread key APIs */
int pthread_key_create(pthread_key_t *key, void (*destructor)(void *));

int pthread_setspecific(pthread_key_t key, const void *value);

void *pthread_getspecific(pthread_key_t key);

int pthread_key_delete(pthread_key_t key);

Known limits

• pthread_attr_t, pthread_mutexattr_t and pthread_condattr_t are not supported yet, so please pass NULL as the second argument of pthread_create, pthread_mutex_init and pthread_cond_init.

• The errno.o in wasi-sysroot is not compatible with this feature, so using errno in multi-thread may cause unexpected behavior.

• Currently struct timespec is not supported, so the prototype of pthread_cond_timedwait is different from the native one, it takes an unsigned int argument useconds to indicate the waiting time.