Contributing
Code of Conduct
Contributors to Tree-sitter should abide by the Contributor Covenant.
Developing Tree-sitter
Prerequisites
To make changes to Tree-sitter, you should have:
- A C compiler, for compiling the core library and the generated parsers.
- A Rust toolchain, for compiling the Rust bindings, the highlighting library, and the CLI.
- Node.js and NPM, for generating parsers from
grammar.js
files. - Either Emscripten, Docker, or podman for compiling the library to WASM.
Building
Clone the repository:
git clone https://github.com/tree-sitter/tree-sitter
cd tree-sitter
Optionally, build the WASM library. If you skip this step, then the tree-sitter playground
command will require an internet
connection. If you have Emscripten installed, this will use your emcc
compiler. Otherwise, it will use Docker or Podman:
cd lib/binding_web
npm install # or your JS package manager of choice
npm run build
Build the Rust libraries and the CLI:
cargo build --release
This will create the tree-sitter
CLI executable in the target/release
folder.
If you want to automatically install the tree-sitter
CLI in your system, you can run:
cargo install --path cli
If you're going to be in a fast iteration cycle and would like the CLI to build faster, you can use the release-dev
profile:
cargo build --release --profile release-dev
# or
cargo install --path cli --profile release-dev
Testing
Before you can run the tests, you need to fetch some upstream grammars that are used for testing:
cargo xtask fetch-fixtures
To test any changes you've made to the CLI, you can regenerate these parsers using your current CLI code:
cargo xtask generate-fixtures
Then you can run the tests:
cargo xtask test
Similarly, to test the WASM binding, you need to compile these parsers to WASM:
cargo xtask generate-fixtures --wasm
cargo xtask test-wasm
Debugging
The test script has a number of useful flags. You can list them all by running cargo xtask test -h
.
Here are some of the main flags:
If you want to run a specific unit test, pass its name (or part of its name) as an argument:
cargo xtask test test_does_something
You can run the tests under the debugger (either lldb
or gdb
) using the -g
flag:
cargo xtask test -g test_does_something
Part of the Tree-sitter test suite involves parsing the corpus tests for several languages and performing randomized edits
to each example in the corpus. If you just want to run the tests for a particular language, you can pass the -l
flag.
Additionally, if you want to run a particular example from the corpus, you can pass the -e
flag:
cargo xtask test -l javascript -e Arrays
Published Packages
The main tree-sitter/tree-sitter
repository contains the source code for
several packages that are published to package registries for different languages:
-
Rust crates on crates.io:
tree-sitter
— A Rust binding to the core librarytree-sitter-highlight
— The syntax-highlighting librarytree-sitter-cli
— The command-line tool
-
JavaScript modules on npmjs.com:
web-tree-sitter
— A WASM-based JavaScript binding to the core librarytree-sitter-cli
— The command-line tool
There are also several other dependent repositories that contain other published packages:
tree-sitter/node-tree-sitter
— Node.js bindings to the core library, published astree-sitter
on npmjs.comtree-sitter/py-tree-sitter
— Python bindings to the core library, published astree-sitter
on PyPI.org.tree-sitter/go-tree-sitter
— Go bindings to the core library, published astree_sitter
on pkg.go.dev.
Developing Documentation
Our current static site generator for documentation is mdBook
, with a little bit of custom JavaScript to handle
the playground page. Most of the documentation is written in Markdown, including this file! You can find these files
at docs/src
. If you'd like to submit a PR to improve the documentation, navigate to the page you'd like to
edit and hit the edit icon at the top right of the page.
Prerequisites for Local Development
To run and iterate on the docs locally, the
mdbook
CLI tool is required, which can be installed with
cargo install mdbook
You might have noticed we have some fancy admonitions sprinkled throughout the documentation, like the note above.
These are created using mdbook-admonish
, a preprocessor for mdBook
. As such, this is also
a requirement for developing the documentation locally. To install it, run:
cargo install mdbook-admonish
Once you've installed it, you can begin using admonitions in your markdown files. See the reference for more information.
Spinning it up
Now that you've installed the prerequisites, you can run the following command to start a local server:
cd docs
mdbook serve --open
mdbook
has a live-reload feature, so any changes you make to the markdown files will be reflected in the browser after
a short delay. Once you've made a change that you're happy with, you can submit a PR with your changes.
Improving the Playground
The playground page is a little more complicated, but if you know some basic JavaScript and CSS you should be able to make
changes. The playground code can be found in docs/src/assets/js/playground.js
, and its corresponding css
at docs/src/assets/css/playground.css
. The editor of choice we use for the playground is CodeMirror,
and the tree-sitter module is fetched from here. This, along with the wasm module and wasm parsers, live in the
.github.io repo.