A nicer, faster and more featureful frontend to TUCaN.
TUCaN't consists of three components: a fontend, a backend and a database. The frontend only communicates with the backend, which in turn communicates with the database.
The frontend is written using React and TypeScript. It should be a much faster, nicer looking and more featureful frontend to TUCaN.
The backend is written in Rust and is supposed to crawl TUCaN when first logging in. This data is then stored in a database to allow arbitrary analysis with it. There are also some web API endpoints for common things like navigating modules and full text search.
The database is a PostgreSQL database. It is used to store the crawled data from TUCaN.
When resetting the database remember to also remove sessions.key
.
The following bookmarklet opens the tucan page in tucant (and optionally logs you in):
javascript:window.location.href = `http://localhost:8080/login-hack?${document.querySelector("#logoutButton") ? new URL(document.querySelector("#logoutButton").href).searchParams.get("ARGUMENTS").split(",")[0].replace("-N", "session_nr=") + "&" : ""}${document.cookie.split(";").find((item) => item.trim().startsWith("cnsc=")) ? "session_id=" + document.cookie.split(";").find((item) => item.trim().startsWith("cnsc=")).split("=")[1] + "&" : ""}redirect=${encodeURIComponent(window.location.href)}`
- Docker
- Node.js
- Yarn Berry
- Rust
- libpq-dev[el] (might be called differently on other distributions)
cd backend-rust
podman build . --pull -f Dockerfile-postgres --tag postgres-hunspell
podman run --name tucant-postgres -d --restart unless-stopped -e POSTGRES_INITDB_ARGS="--data-checksums" -e POSTGRES_PASSWORD=password -p 5432:5432 -it postgres-hunspell
https://sqlite.org/wasm/doc/trunk/index.md
https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/emscripten-packages.nix
https://users.rust-lang.org/t/wasm32-unknown-unknown-vs-wasm32-wasi/78325
almost works: export EM_CACHE=$(pwd)/.emscriptencache cargo build --target wasm32-unknown-emscripten
https://github.com/strawlab/iana-time-zone/blob/main/Cargo.toml seems like some dependencies use wasm-bindgen so we can't use emscripten?
cargo install diesel_cli --no-default-features --features sqlite
cp env.sample .env
cd backend-rust
$HOME/.cargo/bin/diesel setup
# run this each time you want to run the backend
RUST_BACKTRACE=1 RUST_LOG=tucan_scraper=info,info cargo run --bin server
cd frontend-react
# install dependencies each time the package.json changed
yarn install --immutable
# run this each time you want to run the frontend
yarn run dev
Rome on NixOS (waiting for rome/tools#4516):
cp $(nix build --print-out-paths nixpkgs#rome)/bin/rome frontend-react/.yarn/unplugged/@rometools-cli-linux-x64-npm-*/node_modules/@rometools/cli-linux-x64/rome
cp $(nix build --print-out-paths nixpkgs#rome)/bin/rome /home/moritz/.vscode-oss/extensions/rome.rome-0.24.3-linux-x64/server/rome
If you want automatic formatting and linting on commit
ln -srf pre-commit.sh .git/hooks/pre-commit
If you want the backend to automatically restart on file change
cargo install cargo-watch
cargo watch -x check -s 'touch .trigger'
RUST_BACKTRACE=1 cargo watch --no-gitignore -w ./.trigger -s 'cargo run --bin server'
To test the backend
cd backend-rust
RUST_BACKTRACE=1 cargo test -- -Z unstable-options --nocapture --report-time
To get a nice GUI of the database on Linux
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
flatpak install flathub io.dbeaver.DBeaverCommunity
flatpak run io.dbeaver.DBeaverCommunity
To access the database from using a CLI on Linux
posgresql
needs to be installed on the host system
psql postgres://postgres:password@localhost:5432/tucant
Tracing:
# https://www.jaegertracing.io/docs/1.39/getting-started/
sudo docker run -d --name jaeger \
-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-e COLLECTOR_OTLP_ENABLED=true \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
-p 14250:14250 \
-p 14268:14268 \
-p 14269:14269 \
-p 9411:9411 \
jaegertracing/all-in-one:latest
cargo run --bin server
echo http://localhost:16686/
Add license headers
reuse
needs to be installed on the host system
reuse addheader --copyright "The tucant Contributors" --license AGPL-3.0-or-later --exclude-year --recursive --skip-unrecognised .
Optimize dependencies:
cargo tree -d --format "{p} {f}"
cargo install cargo-hack
cargo hack build --workspace --all-targets
cargo install cargo-udeps --locked
cargo udeps --workspace --all-targets
cargo install cargo-machete
cargo machete --workspace --all-targets
cargo install --locked cargo-deny
cargo deny check --workspace --all-targets
cargo install --locked cargo-outdated
cargo outdated --workspace
cargo tree --no-dedupe --prefix none | sort -k 1 | uniq -c | sort -k 1 -n -r
RUSTFLAGS="-Z time-passes" time cargo +nightly build &> time-passes.log
RUSTFLAGS="-Z time-passes" ../../rustc_codegen_cranelift/dist/cargo-clif run &> time-passes.log
Clippy
cargo clippy --all-targets --all-features
https://opensearch.org/docs/latest/opensearch/search-template/
https://opensearch.org/docs/latest/opensearch/ux/ autocomplete
https://opensearch.org/docs/latest/api-reference/document-apis/index-document/
https://opensearch.org/docs/latest/api-reference/document-apis/bulk/
https://opensearch.org/docs/latest/api-reference/explain/
https://opensearch.org/docs/latest/api-reference/search/
cargo install cargo-fuzz cargo install afl
echo core | sudo tee /proc/sys/kernel/core_pattern cd tucant-language-server/fuzz_afl mkdir in echo 1 > in/trivial cargo afl build cargo afl fuzz -i in -o out ../../target/debug/fuzz_target_1
cd tucant-language-server/fuzz_libfuzzer/ cargo fuzz run --jobs 8 --fuzz-dir . fuzz_target_1