# Building the GNU Go WASM engine Compiles GNU Go 3.8 (2008-era autoconf C) to a WebAssembly module with Emscripten, wrapped by a small C API (`wasm/wrapper/wq_wrapper.c`) tailored to this app's board/move representation. Produces `wasm/gnugo.js` (an ES module, `MODULARIZE`d, factory `createGnuGo`) and `wasm/gnugo.wasm`. Both are checked into git -- they are the actual build output the app ships, not intermediate artifacts. ## Prerequisites ``` brew install emscripten emcc --version # verify; this doc was written against 6.0.6 ``` `node` is also required (used both as part of Emscripten's own toolchain and to run the smoke test). ## Source ``` mkdir -p wasm/build curl -Lo wasm/build/gnugo-3.8.tar.gz https://ftp.gnu.org/gnu/gnugo/gnugo-3.8.tar.gz ``` `wasm/build/` is git-ignored -- the GNU Go source tree (and a second, native copy of it, see Deviation 3 below) live there but are never committed. Only the compiled `gnugo.js`/`gnugo.wasm` and this wrapper source are. ## Build ``` wasm/wrapper/build.sh ``` Idempotent (skips steps whose output already exists), and does the whole thing end to end: downloads the source if missing, patches it (Deviation 1), builds the wasm libraries, does a **second, native** build of just enough of the tree to generate GNU Go's pattern-database C sources (Deviation 3), and links the final module. Read the script; each phase is commented with which deviation below it corresponds to. ## Deviations from a stock GNU Go 3.8 build This is the actual list of 2008-vs-modern-toolchain friction hit while building, in the order encountered. Two failed strategies were tried before the ones below worked; see "Rejected approaches" at the end. ### Deviation 0: config.guess/config.sub, `--host` -- NOT hit The task brief anticipated `config.guess`/`config.sub` not recognizing the host triple. On this machine (Apple Silicon macOS, Homebrew emscripten 6.0.6), `emconfigure ./configure` ran clean on the first try with no `--host` override and no config.guess/config.sub errors. Not needed here; noted in case a different emscripten/host combination hits it. ### Deviation 1: tentative global definitions in a header (real bug, was silently "working" by accident) `engine/liberty.h` declares two globals with no `extern`: ```c /* Unconditionally meaningless moves. */ int meaningless_black_moves[BOARDMAX]; int meaningless_white_moves[BOARDMAX]; ``` Every `.c` file that `#include`s `liberty.h` (nearly the whole engine, plus the pattern-compiler tools) then also "defines" these two arrays -- a **tentative definition** in old C terms. Traditional Unix linkers (BSD `ld` on macOS, GNU `ld`) silently merge multiple tentative definitions of the same symbol via COMMON-symbol semantics, so this happened to link fine on every platform GNU Go 3.8 was ever built on natively. `wasm-ld` (LLVM's linker, what Emscripten uses) does not implement COMMON symbols for the wasm target at all -- not "off by default", genuinely unimplemented: ``` error: common symbols are not yet implemented for Wasm: gg_optarg ``` (that specific message came from testing `-fcommon` as documented in "Rejected approaches" below; without `-fcommon` the failure instead surfaces at the final link, once two or more `.o` files defining the same array both end up needed): ``` wasm-ld: error: duplicate symbol: meaningless_black_moves >>> defined in engine/libengine.a(aftermath.o) >>> defined in engine/libengine.a(cache.o) ``` **Fix** (applied by `build.sh`, idempotent, via `sed`): add `extern` to both declarations in `engine/liberty.h`, and add the one real definition to `engine/globals.c` -- which is already, by its own comment, "Define all global variables used within the engine." This is the only source patch this build needed. ### Deviation 2: generated Makefiles default `AR=ar`, not `emar` `emconfigure ./configure` produces Makefiles with `AR = ar` (the system/BSD archiver), not Emscripten's `emar` wrapper. BSD `ar`/`ranlib` accept `.o` files that are actually wasm binary modules without complaint or error -- but silently produce a near-empty `.a` (96 bytes: just a symbol-table header, no members): ``` $ ar cru libengine.a aftermath.o board.o ... # (36 files) ranlib: warning: archive member 'aftermath.o' not a mach-o file $ ls -la libengine.a -rw-r--r-- 1 percy staff 96 ... ``` Silently broken -- no error, only a warning that's easy to miss in a long log. **Fix**: pass `AR=emar` explicitly on every `make`/`emmake make` invocation that builds a `.a` (`utils`, `sgf`, `engine`, `patterns`). `emranlib` is already selected correctly by `emconfigure` (it's `RANLIB` in the generated Makefile), only `AR` needed the override. ### Deviation 3: pattern databases are generated at build time by host tools that cannot themselves be wasm `patterns/libpatterns.a`'s sources are not all committed to the GNU Go tarball -- most (`conn.c`, `patterns.c`, `eyes.c`, `fuseki19.c`, `mcpat.c`, 19 files in all, one of them (`fuseki19.c`) over 1MB of generated tables) are produced at build time by four small host tools (`mkpat`, `joseki`, `mkeyes`, `mkmcpat`) that read GNU Go's `*.db` pattern-database files and print C source to stdout, e.g.: ```make conn.c : mkpat$(EXEEXT) conn.db ./mkpat -c$(DFAFLAGS) -o conn.c $(srcdir)/conn.db "connection patterns" ``` Those tools need to *run* on the machine doing the build, producing text output consumed by the next compile step. A `noinst_PROGRAMS` built by `emcc` is a WebAssembly module (a `.js` + `.wasm` pair, or -- since `EXEEXT` is empty for this target -- a Node-shebang'd `.js` renamed without extension); attempting to have `make` invoke it directly like a normal host binary at build time is unreliable within the constraints of this build (and in initial testing, the *compile* of `mkpat.c`/`transform.c`/`dfa.c` independently hit Deviation 1's duplicate-symbol problem, since those three files also include `liberty.h`). **Fix**: `build.sh` does a **second, entirely separate, native build** of just `utils/`, `sgf/`, `engine/`, and `patterns/` (using the system `cc`, no `emconfigure`) in `wasm/build/native/gnugo-3.8/`. That native build's `make -C patterns libpatterns.a` runs `mkpat`/`joseki`/etc. as ordinary native executables and produces the 19 generated `.c` files plus 7 generated `.db` files (`gogo.db`, `hoshi_keima.db`, etc. -- joseki databases compiled from `.sgf` game records) exactly as a stock build would. `build.sh` copies just those generated files into the emscripten tree's `patterns/` directory. To stop `emmake make -C patterns libpatterns.a` from then trying to rebuild `mkpat`/`joseki`/`mkeyes`/`mkmcpat`/`uncompress_fuseki` itself (which would hit the wasm-can't-be-a-build-tool problem above), the script places inert placeholder scripts at those paths (`exit 1` if ever actually invoked -- a loud, obvious failure rather than silent wrong output) and runs: ``` emmake make AR=emar -o mkpat -o joseki -o mkeyes -o mkmcpat -o uncompress_fuseki libpatterns.a ``` `-o` / `--assume-old` tells GNU Make to treat those five files as already up to date and skip their entire prerequisite chain -- it never even looks at `mkpat.o`/`transform.o`/`dfa.o`, so the placeholder scripts are never executed; make just compiles the already-present generated `.c` files straight into `libpatterns.a`. Verified: with the placeholders in place and `-o` used, `emmake make` output shows only the 19 generated files (plus `connections.c`, `helpers.c`, `transform.c`) being compiled and archived -- no attempt to link `mkpat`. The two build trees are independent; only the *generated pattern C/db files* cross from native to emscripten, never object code or binaries. ### Deviation 4: a real, pre-existing GNU Go 3.8 bug, confirmed on a pristine native build The smoke test's first scenario (9x9, komi 7.5, level 5, Black plays tengen, White to `genmove`) reliably crashed with: ``` ***assertion failure: board.c:2540 - ON_BOARD1(str) near PASS*** ``` This is `findstones()` (`engine/board.c`) getting called with `str == PASS_MOVE` (0) during White's reading at level 5 (worm/dragon analysis that levels 1-4 don't reach). Before concluding this was a wasm-specific issue, it was reproduced identically on the **fully native** build -- same source, same generated pattern data, same everything, compiled and run entirely with the system `cc`, no Emscripten involved: ``` $ printf 'boardsize 9\nkomi 7.5\nlevel 5\nplay black E5\ngenmove white\nquit\n' \ | ./gnugo --mode gtp --seed 0 ... ***assertion failure: board.c:2540 - ON_BOARD1(str) near PASS*** ``` using the actual, unmodified `gnugo` binary built via the project's own standard `interface/Makefile.am` (curses, GTP loop, everything) -- not this wrapper's minimal init sequence. So this is a genuine defect in GNU Go 3.8 itself, unrelated to this port, that a real game reaching a similar reading situation at level >= 5 could hit. Ruled out along the way: it isn't seed-dependent (tried 0, 1, 2, 42, 12345), isn't sensitive to the `gnugo_clear_board()`/`init_gnugo()` call order (both orders crash identically, contradicting the task brief's assumption that order might matter -- see next paragraph), and isn't about the reading cache being enabled vs the real default-disabled state (`DEFAULT_MEMORY == -1`, which `main.c` passes straight through, giving `reading_cache_init()` a *negative* byte count that disables caching -- reproduced that exact condition too, no change). One side note on init order: the task brief says to call `init_gnugo()` before `gnugo_clear_board()`; `interface/main.c` actually calls `gnugo_clear_board()` **first**, then `init_gnugo()`. Both orders were tested against this bug (no difference) and both produce a working engine for the smoke test's purposes, so `wq_wrapper.c` keeps the brief's stated order (`init_gnugo()` then `gnugo_clear_board()`) rather than silently diverging from what was asked for. **Fix**: GNU Go 3.8 ships its own documented escape hatch for exactly this class of problem -- `GG_TURN_OFF_ASSERTS` (`engine/board.h`) compiles all of these internal sanity checks (`ASSERT1`/`ASSERT2`/`gg_assert`) out to no-ops. This is not a targeted patch for this one bug; it's a blanket, upstream-provided switch. It's justified here less as a build-time workaround and more as a **production requirement**: without it, any real game that happens to reach a similar internal edge case at moderate-to-high levels would abort the entire WASM module for a real user mid-game (an unrecoverable `RuntimeError`, not a catchable game-logic error). With assertions off, `findstones()` instead reads `string_number[0]` -- an in-bounds but semantically-meaningless slot (`POS(-1,-1) == 0` by construction, i.e. `pos 0` is a valid array index representing "off the board") -- and the one internal computation using it degrades quietly rather than aborting. Applied via `CFLAGS=... -DGG_TURN_OFF_ASSERTS` on every library build and on the final wrapper compile. ### Deviation 5: Emscripten's default 64KiB wasm stack is far too small Before finding Deviation 4, the *first* symptom investigated (at level 1, not 5) was a different-looking crash in the same neighborhood -- `matchpat.c:439 - ON_BOARD1(pos) near [1224]` under `-O2`, or a raw `wasm-ld` ... `RuntimeError: memory access out of bounds` inside `find_string_connection_moves`/`recursive_connect2`/`string_connect` under `-O0` (ruling out an `-O2` miscompilation). This one *did* turn out to be wasm-specific: it did not reproduce on the native build at all, at any level, with the same generated pattern data. GNU Go's reading/owl/connection search is deeply recursive with substantial per-frame state, and Emscripten's default `-sSTACK_SIZE` is 64KiB (visible in the `wasm-ld` invocation as `-z stack-size=65536`) -- easily exhausted, and a wasm stack overflow does not fail cleanly the way a native one usually does (it can silently corrupt whatever's adjacent in linear memory instead of trapping immediately, matching the "garbage position value" symptom observed). **Fix**: `-sSTACK_SIZE=16777216` (16MiB) on the final link. Verified sufficient for level 1 through level 10 on a 9x9 board (see smoke test and the level 1/10 strength-plumbing check in `task-6-report.md`); no attempt was made to find the exact minimum, 16MiB was simply generous enough headroom after 64KiB visibly wasn't and 16MiB visibly was, at low cost. `ALLOW_MEMORY_GROWTH` does *not* cover this -- wasm's stack is sized once at instantiation and does not participate in the growable-heap mechanism. Two other explanations were ruled out along the way, both faster to check than they sound and worth recording so a future session doesn't retread them: `-fcommon` as a fix for Deviation 1 made things worse, not better (see "Rejected approaches" below); and whether `DFA_ENABLED` (patterns' alternate, packed pattern-matching format) differed between the native generation host and the wasm consumption target -- it didn't (verified via `config.status`), and disabling DFA entirely on both sides reproduced the exact same crash, ruling out the pattern-data-format angle entirely before the real cause (stack size) was found. ## Final link ``` emcc -O2 -DGG_TURN_OFF_ASSERTS -I . -I engine -I sgf -I utils \ wasm/wrapper/wq_wrapper.c \ engine/libengine.a patterns/libpatterns.a sgf/libsgf.a utils/libutils.a \ -sMODULARIZE=1 -sEXPORT_ES6=1 -sEXPORT_NAME=createGnuGo \ -sEXPORTED_FUNCTIONS=_wq_init,_wq_fixed_handicap,_wq_play,_wq_pass,_wq_genmove,_wq_final_status_dead,_wq_estimate_score,_malloc,_free \ -sEXPORTED_RUNTIME_METHODS=cwrap \ -sALLOW_MEMORY_GROWTH=1 -sSTACK_SIZE=16777216 \ -o wasm/gnugo.js ``` Library link order matches `interface/Makefile.am`'s own `LDADD` for the real `gnugo` binary (`libengine.a libpatterns.a libsgf.a libutils.a`) -- proven by upstream to resolve in a single left-to-right archive scan, no `--start-group`/duplicate-archive tricks needed. `libboard.a` is not linked separately: its object files (`board.o`, `boardlib.o`, `hash.o`, `printutils.o`) are already a subset of `libengine.a`'s members. `-I .` (the emscripten build tree's top directory) is required because `engine/board.h` does `#include "config.h"` unconditionally (not guarded by `HAVE_CONFIG_H`), and `config.h` is generated by `./configure` at the tree root. ## Wrapper API mapping (`wasm/wrapper/wq_wrapper.c`) Every `wq_*` function mirrors a specific call sequence from GNU Go's own GTP interface (`interface/play_gtp.c`, `interface/main.c`, `interface/gtp.c`): | Wrapper function | Modeled after | |---|---| | `wq_init` | `init_gnugo()` once (interface/main.c) + `gnugo_clear_board()` + `komi = ...` (global, `engine/board.h`) + `set_level()` | | `wq_fixed_handicap` | `gtp_fixed_handicap` -> `place_fixed_handicap()` | | `wq_play` | `gtp_play` -> `is_allowed_move()` guard, then `gnugo_play_move()` | | `wq_pass` | same as `wq_play` but with `PASS_MOVE` (0) | | `wq_genmove` | `gtp_genmove` -> `genmove(color, NULL, &resign)`, play the result unless resigning | | `wq_final_status_dead` / `wq_estimate_score` | `finish_and_score_game()` (interface/play_gtp.c): play the position out to two passes with `genmove_conservative()` under `doing_scoring=1`, score with `aftermath_compute_score()`, get per-point status with `aftermath_final_status()`, then `restore_board()` to undo the playout | ### Color encoding: wrapper (1=B, 2=W) is NOT the engine's own enum GNU Go's internal `enum colors` (`engine/board.h`) is `EMPTY=0, WHITE=1, BLACK=2` -- the *opposite* numbering from the wrapper's public API (`1=B, 2=W`, per the task brief). `wq_wrapper.c`'s `wq_engine_color()` converts on every call; nothing else in the wrapper assumes the two numberings match. ### Coordinate encoding: `(x, y)`, y=0 = TOP -- verified empirically, not assumed GNU Go addresses the board as `POS(i, j)` where `i`/`j` come from the `I(pos)`/`J(pos)` macros (`engine/board.h`). Which edge `i=0` represents isn't stated as such anywhere in the headers -- it's implicit in how `gtp_print_vertex()` (`interface/gtp.c`) formats it for GTP output: ```c gtp_printf("%c%d", 'A' + rj + (rj >= 8), gtp_boardsize - ri); ``` GTP's own convention is row 1 = bottom, row `boardsize` = top. Since `ri=0` maps to `gtp_boardsize - 0 = boardsize` (the top row in GTP numbering), internal `i=0` **is** the top row. That means the wrapper's `y` (0 = top, per the task brief) equals GNU Go's internal `i` directly, and `x` equals internal `j`, with **no flip needed**: `wq_play`/`wq_genmove` use `POS(y, x)` and `I(move)`/`J(move)` as-is. This was verified empirically, not just trusted from reading the source: a throwaway native (non-wasm) program was compiled against the native `libengine.a`/`libpatterns.a`, calling `gnugo_play_move(POS(0,0), BLACK)` and `gnugo_play_move(POS(8,8), WHITE)` on a 9x9 board, then `showboard(0)`: ``` A B C D E F G H J 9 X . . . . . . . . 9 ... 1 . . . . . . . . O 1 A B C D E F G H J ``` `POS(0, 0)` (Black, `X`) prints at the top-left (row 9 = the top in GTP's bottom-up row numbering, column A = leftmost); `POS(8, 8)` (White, `O`) prints at the bottom-right. So internal `i` (`I(pos)`) is confirmed to be row-from-top and `j` (`J(pos)`) column-from-left -- exactly the wrapper's own `(x, y)`/`y=0=top` convention, no flip needed. The smoke test's tengen assertion (`play(1, 4, 4)` on a 9x9 board must be legal) is a cheap regression check for this on every run, though by itself it wouldn't catch an axis flip (tengen is the middle of both axes on an odd board) -- the `showboard()` check above is what actually rules that out. ### `wq_genmove` return encoding `>= 0`: `y*size+x` (i.e. `I(move)*board_size + J(move)`). `-1`: pass (`move == PASS_MOVE`). `-2`: resign (`genmove`'s `resign` out-param was set; per `gtp_genmove`, the move is *not* played in this case). `-3`: invalid `color` argument (added in the fix round below) -- deliberately not `-1`, so a caller bug can't be misread as a legitimate pass. ## Input validation (added in code review fix round 1) The initial implementation trusted its arguments: `wq_play` passed `POS(y, x)` straight to the engine with no check that `0 <= x, y < board_size`, and `wq_init` never validated `boardsize` before it's used as an array bound throughout the engine. In a native build GNU Go's own internal `ON_BOARD1()`/`gg_assert()` checks would have caught bad values; in this build they're compiled to no-ops by `GG_TURN_OFF_ASSERTS` (Deviation 4), which exists for a real, confirmed-upstream reason (see above) but as a side effect also removes that safety net for ordinary caller mistakes -- an off-by-one or a stale `boardsize` from the JS side could have silently corrupted engine state or hit a raw wasm out-of-bounds trap (aborting the whole module) instead of returning a clean error. Fixed by having every `wq_*` function validate its own arguments and return an error code *before* calling into the engine at all: - `wq_init`: rejects `boardsize` outside `[2, MAX_BOARD]` (19, `board.h`). This is a **signature change** -- `wq_init` now returns `int` (`0` ok, `-1` invalid), not `void`. Chosen over silently clamping because a silent clamp would let the JS side and the engine disagree about the board size without either one noticing. `cwrap`'s `null` return type (used by the two pre-existing smoke tests) still works unchanged after this -- Emscripten's `cwrap` only special-cases `"string"`/`"boolean"` return types, so a numeric return under a `null` declared type is just passed through, not dropped -- but new code should declare it `'number'` for clarity (the new third smoke test does). - `wq_play`: rejects an invalid `color` (not 1 or 2) or `x`/`y` outside `[0, board_size)` for the *current* board size (not the `MAX_BOARD` ceiling -- a 9x9 game must reject `x=15` even though 15 < 19). Folded into the existing `-1` ("illegal") return; no new sentinel needed, an out-of-range coordinate is definitionally an illegal move. - `wq_pass`: rejects an invalid `color`, returns `-1` (the brief didn't document a return-value meaning for this function beyond "int", so this doesn't conflict with anything). - `wq_genmove`: rejects an invalid `color`, returns `-3` (see above -- `-1`/`-2` were already taken by legitimate outcomes). - `wq_fixed_handicap`: rejects `n < 0` or `n > MAX_HANDICAP` (9, `board.h`). `n < 2` (0 or 1) is *not* rejected -- that's GNU Go's own convention for "no fixed handicap requested", and `place_fixed_handicap()` already returns `0` for it correctly. Values above `MAX_HANDICAP` would actually be handled safely by `place_fixed_handicap()` itself even without this check (it clamps internally, and its loop is bounded by the clamped value regardless of how large the input was -- verified by reading `engine/handicap.c`), so this validation isn't fixing a live memory-safety bug, but `place_fixed_handicap()`'s own comment says checking the input against what's placeable is explicitly "up to the caller", so the wrapper now does that rather than silently returning fewer stones than requested with no signal. Also fixed, a related but separate bug (Minor in the same review): in `wq_final_status_dead`, the accumulation loop added `snprintf`'s return value into `used` without clamping it. `snprintf` returns how many bytes it *would* have written even when the buffer was too small to hold them, so if that value had ever exceeded the remaining space, `used` could run past `sizeof(buf)`, and the next iteration's `sizeof(buf) - used` -- `used` being `size_t`, i.e. unsigned -- would underflow into a huge bogus size passed as the buffer size to the following `snprintf` call. Not reachable in practice with the current buffer size (4096 bytes comfortably fits all 19*19 possible vertices at ~4 bytes each), but fixed anyway: `wq_final_status_dead` now formats each vertex into a small fixed-size stack buffer first, computes the exact space it needs in the accumulation buffer, and only appends if it fits -- `used` can no longer run past `sizeof(buf)` regardless of input. ## Level / strength `wq_init`'s `level` argument is passed straight to `set_level()` (`engine/clock.h`) -- same knob GNU Go's own `--level`/`gtp level` command uses. This build does not attempt to verify strength differences between levels beyond "both produce a legal move" (see smoke test) -- meaningfully testing playing strength would need many games and is out of scope for a build-verification smoke test. Level 1 and level 10 were both spot-checked on an empty 9x9 board (`task-6-report.md`); both return a legal move. **Residual risk for whoever builds Task 7 (the worker)**: `GG_TURN_OFF_ASSERTS` (Deviation 4) turns a caught internal inconsistency into a quiet degraded-but-continuing computation in the *specific, verified* case found here (`findstones()` reading an off-board sentinel position). It does not prove every other internal assertion in the codebase is equally safe to skip -- it's plausible, though not observed in testing, that some other edge case now reads further out of bounds and trips wasm's own linear-memory bounds trap instead, which *would* still abort the module (an uncatchable `RuntimeError`, same as before this fix, just rarer). The worker should treat any exception out of a `wq_*` call as fatal to that engine instance and be prepared to `createGnuGo()` a fresh one rather than assuming the existing instance's C-level state is still trustworthy after an exception. ## Rejected approaches (for the record) 1. **`-fcommon`**: tried first, since the duplicate-symbol failure looks exactly like the textbook case `-fcommon` exists to fix. Made things *worse*: Emscripten's wasm backend has no representation for COMMON symbols at all, so `-fcommon` turns even a single-translation-unit tentative definition into a hard compile error (`error: common symbols are not yet implemented for Wasm: gg_optarg`, from `utils/getopt.c`, which compiled fine under the default `-fno-common`). Reverted; the real fix is Deviation 1's source patch. 2. **Letting `emmake make -C patterns` build `mkpat` etc. as wasm/Node executables and have `make` invoke them directly** (`./mkpat ... > conn.c`): in principle `EXEEXT` is empty and Emscripten can emit a Node-shebang'd script that's directly executable, which would have avoided the whole native-build detour. In practice this path first hit Deviation 1 (duplicate symbols compiling `mkpat.o`/`transform.o`/`dfa.o` together) and, after that was patched, was abandoned in favor of the native-build approach once it was already working, rather than also verified end-to-end -- noted here as an untried alternative for a future session that wants to drop the native-build phase entirely. ## Rebuilding after a change `wasm/build/` is git-ignored and persists between runs; `build.sh` is idempotent and will skip phases whose output already exists (source already extracted, `Makefile` already configured, etc.). To force a full rebuild, delete `wasm/build/` and re-run `wasm/wrapper/build.sh`.