pg_tokenizer

pg_tokenizer : Tokenizers for full-text search

Overview

ID	Extension	Package	Version	Category	License	Language
2160	pg_tokenizer	pg_tokenizer	`0.1.1`	FTS	Apache-2.0	Rust

Attribute	Has Binary	Has Library	Need Load	Has DDL	Relocatable	Trusted
--s-dt-	No	Yes	No	Yes	no	yes

Relationships
See Also	pg_search pgroonga pg_bigm zhparser pgroonga_database pg_bestmatch vchord_bm25 pg_trgm

PG18 fix by Vonng

Packages

Type	Repo	Version	PG Major Compatibility	Package Pattern	Dependencies
EXT	PIGSTY	`0.1.1`	18 17 16 15 14 13	`pg_tokenizer`	-
RPM	PIGSTY	`0.1.1`	18 17 16 15 14 13	`pg_tokenizer_$v`	-
DEB	PIGSTY	`0.1.1`	18 17 16 15 14 13	`postgresql-$v-pg-tokenizer`	-

Linux / PG	PG18	PG17	PG16	PG15	PG14	PG13
el8.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el8.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el9.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el9.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el10.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el10.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
d12.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
d12.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
d13.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
d13.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
u22.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
u22.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
u24.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
u24.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_18`	`0.1.1`	el8.x86_64	pigsty	11.7 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_18`	`0.1.1`	el8.aarch64	pigsty	11.5 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_18`	`0.1.1`	el9.x86_64	pigsty	11.0 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_18`	`0.1.1`	el9.aarch64	pigsty	10.9 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_18`	`0.1.1`	el10.x86_64	pigsty	10.9 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_18`	`0.1.1`	el10.aarch64	pigsty	11.0 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-18-pg-tokenizer`	`0.1.1`	d12.x86_64	pigsty	5.0 KiB	postgresql-18-pg-tokenizer_0.1.1-1PIGSTY~bookworm_amd64.deb
`postgresql-18-pg-tokenizer`	`0.1.1`	d12.aarch64	pigsty	5.0 KiB	postgresql-18-pg-tokenizer_0.1.1-1PIGSTY~bookworm_arm64.deb
`postgresql-18-pg-tokenizer`	`0.1.1`	d13.x86_64	pigsty	5.0 KiB	postgresql-18-pg-tokenizer_0.1.1-1PIGSTY~trixie_amd64.deb
`postgresql-18-pg-tokenizer`	`0.1.1`	d13.aarch64	pigsty	5.0 KiB	postgresql-18-pg-tokenizer_0.1.1-1PIGSTY~trixie_arm64.deb
`postgresql-18-pg-tokenizer`	`0.1.1`	u22.x86_64	pigsty	4.9 KiB	postgresql-18-pg-tokenizer_0.1.1-1PIGSTY~jammy_amd64.deb
`postgresql-18-pg-tokenizer`	`0.1.1`	u22.aarch64	pigsty	4.9 KiB	postgresql-18-pg-tokenizer_0.1.1-1PIGSTY~jammy_arm64.deb
`postgresql-18-pg-tokenizer`	`0.1.1`	u24.x86_64	pigsty	4.9 KiB	postgresql-18-pg-tokenizer_0.1.1-1PIGSTY~noble_amd64.deb
`postgresql-18-pg-tokenizer`	`0.1.1`	u24.aarch64	pigsty	4.9 KiB	postgresql-18-pg-tokenizer_0.1.1-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_17`	`0.1.1`	el8.x86_64	pigsty	11.7 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_17`	`0.1.1`	el8.aarch64	pigsty	11.5 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_17`	`0.1.1`	el9.x86_64	pigsty	11.0 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_17`	`0.1.1`	el9.aarch64	pigsty	10.9 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_17`	`0.1.1`	el10.x86_64	pigsty	10.9 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_17`	`0.1.1`	el10.aarch64	pigsty	11.0 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-17-pg-tokenizer`	`0.1.1`	d12.x86_64	pigsty	9.9 MiB	postgresql-17-pg-tokenizer_0.1.1-1PIGSTY~bookworm_amd64.deb
`postgresql-17-pg-tokenizer`	`0.1.1`	d12.aarch64	pigsty	9.7 MiB	postgresql-17-pg-tokenizer_0.1.1-1PIGSTY~bookworm_arm64.deb
`postgresql-17-pg-tokenizer`	`0.1.1`	d13.x86_64	pigsty	9.9 MiB	postgresql-17-pg-tokenizer_0.1.1-1PIGSTY~trixie_amd64.deb
`postgresql-17-pg-tokenizer`	`0.1.1`	d13.aarch64	pigsty	9.7 MiB	postgresql-17-pg-tokenizer_0.1.1-1PIGSTY~trixie_arm64.deb
`postgresql-17-pg-tokenizer`	`0.1.1`	u22.x86_64	pigsty	10.8 MiB	postgresql-17-pg-tokenizer_0.1.1-1PIGSTY~jammy_amd64.deb
`postgresql-17-pg-tokenizer`	`0.1.1`	u22.aarch64	pigsty	10.7 MiB	postgresql-17-pg-tokenizer_0.1.1-1PIGSTY~jammy_arm64.deb
`postgresql-17-pg-tokenizer`	`0.1.1`	u24.x86_64	pigsty	10.8 MiB	postgresql-17-pg-tokenizer_0.1.1-1PIGSTY~noble_amd64.deb
`postgresql-17-pg-tokenizer`	`0.1.1`	u24.aarch64	pigsty	10.7 MiB	postgresql-17-pg-tokenizer_0.1.1-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_16`	`0.1.1`	el8.x86_64	pigsty	11.7 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_16`	`0.1.1`	el8.aarch64	pigsty	11.5 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_16`	`0.1.1`	el9.x86_64	pigsty	11.0 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_16`	`0.1.1`	el9.aarch64	pigsty	10.9 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_16`	`0.1.1`	el10.x86_64	pigsty	10.9 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_16`	`0.1.1`	el10.aarch64	pigsty	11.0 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-16-pg-tokenizer`	`0.1.1`	d12.x86_64	pigsty	10.0 MiB	postgresql-16-pg-tokenizer_0.1.1-1PIGSTY~bookworm_amd64.deb
`postgresql-16-pg-tokenizer`	`0.1.1`	d12.aarch64	pigsty	9.8 MiB	postgresql-16-pg-tokenizer_0.1.1-1PIGSTY~bookworm_arm64.deb
`postgresql-16-pg-tokenizer`	`0.1.1`	d13.x86_64	pigsty	9.9 MiB	postgresql-16-pg-tokenizer_0.1.1-1PIGSTY~trixie_amd64.deb
`postgresql-16-pg-tokenizer`	`0.1.1`	d13.aarch64	pigsty	9.8 MiB	postgresql-16-pg-tokenizer_0.1.1-1PIGSTY~trixie_arm64.deb
`postgresql-16-pg-tokenizer`	`0.1.1`	u22.x86_64	pigsty	10.8 MiB	postgresql-16-pg-tokenizer_0.1.1-1PIGSTY~jammy_amd64.deb
`postgresql-16-pg-tokenizer`	`0.1.1`	u22.aarch64	pigsty	10.7 MiB	postgresql-16-pg-tokenizer_0.1.1-1PIGSTY~jammy_arm64.deb
`postgresql-16-pg-tokenizer`	`0.1.1`	u24.x86_64	pigsty	10.8 MiB	postgresql-16-pg-tokenizer_0.1.1-1PIGSTY~noble_amd64.deb
`postgresql-16-pg-tokenizer`	`0.1.1`	u24.aarch64	pigsty	10.7 MiB	postgresql-16-pg-tokenizer_0.1.1-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_15`	`0.1.1`	el8.x86_64	pigsty	11.7 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_15`	`0.1.1`	el8.aarch64	pigsty	11.5 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_15`	`0.1.1`	el9.x86_64	pigsty	11.0 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_15`	`0.1.1`	el9.aarch64	pigsty	10.9 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_15`	`0.1.1`	el10.x86_64	pigsty	10.9 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_15`	`0.1.1`	el10.aarch64	pigsty	11.0 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-15-pg-tokenizer`	`0.1.1`	d12.x86_64	pigsty	9.9 MiB	postgresql-15-pg-tokenizer_0.1.1-1PIGSTY~bookworm_amd64.deb
`postgresql-15-pg-tokenizer`	`0.1.1`	d12.aarch64	pigsty	9.7 MiB	postgresql-15-pg-tokenizer_0.1.1-1PIGSTY~bookworm_arm64.deb
`postgresql-15-pg-tokenizer`	`0.1.1`	d13.x86_64	pigsty	9.9 MiB	postgresql-15-pg-tokenizer_0.1.1-1PIGSTY~trixie_amd64.deb
`postgresql-15-pg-tokenizer`	`0.1.1`	d13.aarch64	pigsty	9.8 MiB	postgresql-15-pg-tokenizer_0.1.1-1PIGSTY~trixie_arm64.deb
`postgresql-15-pg-tokenizer`	`0.1.1`	u22.x86_64	pigsty	10.8 MiB	postgresql-15-pg-tokenizer_0.1.1-1PIGSTY~jammy_amd64.deb
`postgresql-15-pg-tokenizer`	`0.1.1`	u22.aarch64	pigsty	10.7 MiB	postgresql-15-pg-tokenizer_0.1.1-1PIGSTY~jammy_arm64.deb
`postgresql-15-pg-tokenizer`	`0.1.1`	u24.x86_64	pigsty	10.8 MiB	postgresql-15-pg-tokenizer_0.1.1-1PIGSTY~noble_amd64.deb
`postgresql-15-pg-tokenizer`	`0.1.1`	u24.aarch64	pigsty	10.7 MiB	postgresql-15-pg-tokenizer_0.1.1-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_14`	`0.1.1`	el8.x86_64	pigsty	11.7 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_14`	`0.1.1`	el8.aarch64	pigsty	11.5 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_14`	`0.1.1`	el9.x86_64	pigsty	11.0 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_14`	`0.1.1`	el9.aarch64	pigsty	10.9 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_14`	`0.1.1`	el10.x86_64	pigsty	10.9 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_14`	`0.1.1`	el10.aarch64	pigsty	11.0 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-14-pg-tokenizer`	`0.1.1`	d12.x86_64	pigsty	9.9 MiB	postgresql-14-pg-tokenizer_0.1.1-1PIGSTY~bookworm_amd64.deb
`postgresql-14-pg-tokenizer`	`0.1.1`	d12.aarch64	pigsty	9.7 MiB	postgresql-14-pg-tokenizer_0.1.1-1PIGSTY~bookworm_arm64.deb
`postgresql-14-pg-tokenizer`	`0.1.1`	d13.x86_64	pigsty	9.9 MiB	postgresql-14-pg-tokenizer_0.1.1-1PIGSTY~trixie_amd64.deb
`postgresql-14-pg-tokenizer`	`0.1.1`	d13.aarch64	pigsty	9.7 MiB	postgresql-14-pg-tokenizer_0.1.1-1PIGSTY~trixie_arm64.deb
`postgresql-14-pg-tokenizer`	`0.1.1`	u22.x86_64	pigsty	10.8 MiB	postgresql-14-pg-tokenizer_0.1.1-1PIGSTY~jammy_amd64.deb
`postgresql-14-pg-tokenizer`	`0.1.1`	u22.aarch64	pigsty	10.7 MiB	postgresql-14-pg-tokenizer_0.1.1-1PIGSTY~jammy_arm64.deb
`postgresql-14-pg-tokenizer`	`0.1.1`	u24.x86_64	pigsty	10.8 MiB	postgresql-14-pg-tokenizer_0.1.1-1PIGSTY~noble_amd64.deb
`postgresql-14-pg-tokenizer`	`0.1.1`	u24.aarch64	pigsty	10.7 MiB	postgresql-14-pg-tokenizer_0.1.1-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_13`	`0.1.1`	el8.x86_64	pigsty	11.7 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_13`	`0.1.1`	el8.aarch64	pigsty	11.5 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_13`	`0.1.1`	el9.x86_64	pigsty	11.0 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_13`	`0.1.1`	el9.aarch64	pigsty	10.9 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_13`	`0.1.1`	el10.x86_64	pigsty	10.9 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_13`	`0.1.1`	el10.aarch64	pigsty	11.0 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-13-pg-tokenizer`	`0.1.1`	d12.x86_64	pigsty	5.0 KiB	postgresql-13-pg-tokenizer_0.1.1-1PIGSTY~bookworm_amd64.deb
`postgresql-13-pg-tokenizer`	`0.1.1`	d12.aarch64	pigsty	5.0 KiB	postgresql-13-pg-tokenizer_0.1.1-1PIGSTY~bookworm_arm64.deb
`postgresql-13-pg-tokenizer`	`0.1.1`	d13.x86_64	pigsty	5.0 KiB	postgresql-13-pg-tokenizer_0.1.1-1PIGSTY~trixie_amd64.deb
`postgresql-13-pg-tokenizer`	`0.1.1`	d13.aarch64	pigsty	5.0 KiB	postgresql-13-pg-tokenizer_0.1.1-1PIGSTY~trixie_arm64.deb
`postgresql-13-pg-tokenizer`	`0.1.1`	u22.x86_64	pigsty	4.9 KiB	postgresql-13-pg-tokenizer_0.1.1-1PIGSTY~jammy_amd64.deb
`postgresql-13-pg-tokenizer`	`0.1.1`	u22.aarch64	pigsty	4.9 KiB	postgresql-13-pg-tokenizer_0.1.1-1PIGSTY~jammy_arm64.deb
`postgresql-13-pg-tokenizer`	`0.1.1`	u24.x86_64	pigsty	4.9 KiB	postgresql-13-pg-tokenizer_0.1.1-1PIGSTY~noble_amd64.deb
`postgresql-13-pg-tokenizer`	`0.1.1`	u24.aarch64	pigsty	4.9 KiB	postgresql-13-pg-tokenizer_0.1.1-1PIGSTY~noble_arm64.deb

Source

Repository

github.com/tensorchord/pg_tokenizer.rs

Source Tarball

pg_tokenizer.rs-0.1.1.tar.gz

pig build pkg pg_tokenizer;		# build rpm / deb with pig

Install

Make sure PGDG and PIGSTY repo available:

pig repo add pgdg pigsty -u   # add both repo and update cache

Install this extension with pig:

pig install pg_tokenizer;		# install via package name, for the active PG version

pig install pg_tokenizer -v 18;   # install for PG 18
pig install pg_tokenizer -v 17;   # install for PG 17
pig install pg_tokenizer -v 16;   # install for PG 16
pig install pg_tokenizer -v 15;   # install for PG 15
pig install pg_tokenizer -v 14;   # install for PG 14
pig install pg_tokenizer -v 13;   # install for PG 13

Create this extension with:

CREATE EXTENSION pg_tokenizer;

Last updated on 2025-11-10

vchord_bm25 hunspell_cs_cz