pg_tokenizer

Tokenizers for full-text search

Overview

ID	Extension	Package	Version	Category	License	Language
2160	pg_tokenizer	pg_tokenizer	`0.1.1`	FTS	Apache-2.0	Rust

Attribute	Has Binary	Has Library	Need Load	Has DDL	Relocatable	Trusted
--s-dt-	No	Yes	No	Yes	no	yes

Relationships
See Also	pg_search pgroonga pg_bigm zhparser pgroonga_database pg_bestmatch vchord_bm25 pg_trgm

PG18 fix by Vonng

Packages

Type	Repo	Version	PG Major Availability	Package Pattern	Dependencies
EL	PIGSTY	`0.1.1`	18 17 16 15 14 13	`pg_tokenizer_$v`	-
Debian	PIGSTY	`0.1.1`	18 17 16 15 14 13	`postgresql-$v-pg-tokenizer`	-

Linux / PG	PG18	PG17	PG16	PG15	PG14	PG13
el8.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el8.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el9.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el9.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el10.x86_64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
el10.aarch64	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1	PIGSTY 0.1.1
d12.x86_64	MISS	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	MISS
d12.aarch64	MISS	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	MISS
d13.x86_64	MISS	MISS	MISS	MISS	MISS	MISS
d13.aarch64	MISS	MISS	MISS	MISS	MISS	MISS
u22.x86_64	MISS	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	MISS
u22.aarch64	MISS	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	MISS
u24.x86_64	MISS	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	MISS
u24.aarch64	MISS	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	PIGSTY 0.1.0	MISS

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_18`	0.1.1	`el8.x86_64`	pigsty	11.7 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_18`	0.1.1	`el8.aarch64`	pigsty	11.5 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_18`	0.1.1	`el9.x86_64`	pigsty	11.0 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_18`	0.1.1	`el9.aarch64`	pigsty	10.9 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_18`	0.1.1	`el10.x86_64`	pigsty	10.9 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_18`	0.1.1	`el10.aarch64`	pigsty	11.0 MiB	pg_tokenizer_18-0.1.1-1PIGSTY.el10.aarch64.rpm

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_17`	0.1.1	`el8.x86_64`	pigsty	11.7 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_17`	0.1.1	`el8.aarch64`	pigsty	11.5 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_17`	0.1.1	`el9.x86_64`	pigsty	11.0 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_17`	0.1.1	`el9.aarch64`	pigsty	10.9 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_17`	0.1.1	`el10.x86_64`	pigsty	10.9 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_17`	0.1.1	`el10.aarch64`	pigsty	11.0 MiB	pg_tokenizer_17-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-17-pg-tokenizer`	0.1.0	`d12.x86_64`	pigsty	9.8 MiB	postgresql-17-pg-tokenizer_0.1.0-1PIGSTY~bookworm_amd64.deb
`postgresql-17-pg-tokenizer`	0.1.0	`d12.aarch64`	pigsty	9.6 MiB	postgresql-17-pg-tokenizer_0.1.0-1PIGSTY~bookworm_arm64.deb
`postgresql-17-pg-tokenizer`	0.1.0	`u22.x86_64`	pigsty	10.7 MiB	postgresql-17-pg-tokenizer_0.1.0-1PIGSTY~jammy_amd64.deb
`postgresql-17-pg-tokenizer`	0.1.0	`u22.aarch64`	pigsty	10.6 MiB	postgresql-17-pg-tokenizer_0.1.0-1PIGSTY~jammy_arm64.deb
`postgresql-17-pg-tokenizer`	0.1.0	`u24.x86_64`	pigsty	10.6 MiB	postgresql-17-pg-tokenizer_0.1.0-1PIGSTY~noble_amd64.deb
`postgresql-17-pg-tokenizer`	0.1.0	`u24.aarch64`	pigsty	10.5 MiB	postgresql-17-pg-tokenizer_0.1.0-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_16`	0.1.1	`el8.x86_64`	pigsty	11.7 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_16`	0.1.1	`el8.aarch64`	pigsty	11.5 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_16`	0.1.1	`el9.x86_64`	pigsty	11.0 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_16`	0.1.1	`el9.aarch64`	pigsty	10.9 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_16`	0.1.1	`el10.x86_64`	pigsty	10.9 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_16`	0.1.1	`el10.aarch64`	pigsty	11.0 MiB	pg_tokenizer_16-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-16-pg-tokenizer`	0.1.0	`d12.x86_64`	pigsty	9.8 MiB	postgresql-16-pg-tokenizer_0.1.0-1PIGSTY~bookworm_amd64.deb
`postgresql-16-pg-tokenizer`	0.1.0	`d12.aarch64`	pigsty	9.6 MiB	postgresql-16-pg-tokenizer_0.1.0-1PIGSTY~bookworm_arm64.deb
`postgresql-16-pg-tokenizer`	0.1.0	`u22.x86_64`	pigsty	10.7 MiB	postgresql-16-pg-tokenizer_0.1.0-1PIGSTY~jammy_amd64.deb
`postgresql-16-pg-tokenizer`	0.1.0	`u22.aarch64`	pigsty	10.6 MiB	postgresql-16-pg-tokenizer_0.1.0-1PIGSTY~jammy_arm64.deb
`postgresql-16-pg-tokenizer`	0.1.0	`u24.x86_64`	pigsty	10.6 MiB	postgresql-16-pg-tokenizer_0.1.0-1PIGSTY~noble_amd64.deb
`postgresql-16-pg-tokenizer`	0.1.0	`u24.aarch64`	pigsty	10.5 MiB	postgresql-16-pg-tokenizer_0.1.0-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_15`	0.1.1	`el8.x86_64`	pigsty	11.7 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_15`	0.1.1	`el8.aarch64`	pigsty	11.5 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_15`	0.1.1	`el9.x86_64`	pigsty	11.0 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_15`	0.1.1	`el9.aarch64`	pigsty	10.9 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_15`	0.1.1	`el10.x86_64`	pigsty	10.9 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_15`	0.1.1	`el10.aarch64`	pigsty	11.0 MiB	pg_tokenizer_15-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-15-pg-tokenizer`	0.1.0	`d12.x86_64`	pigsty	9.8 MiB	postgresql-15-pg-tokenizer_0.1.0-1PIGSTY~bookworm_amd64.deb
`postgresql-15-pg-tokenizer`	0.1.0	`d12.aarch64`	pigsty	9.6 MiB	postgresql-15-pg-tokenizer_0.1.0-1PIGSTY~bookworm_arm64.deb
`postgresql-15-pg-tokenizer`	0.1.0	`u22.x86_64`	pigsty	10.7 MiB	postgresql-15-pg-tokenizer_0.1.0-1PIGSTY~jammy_amd64.deb
`postgresql-15-pg-tokenizer`	0.1.0	`u22.aarch64`	pigsty	10.5 MiB	postgresql-15-pg-tokenizer_0.1.0-1PIGSTY~jammy_arm64.deb
`postgresql-15-pg-tokenizer`	0.1.0	`u24.x86_64`	pigsty	10.6 MiB	postgresql-15-pg-tokenizer_0.1.0-1PIGSTY~noble_amd64.deb
`postgresql-15-pg-tokenizer`	0.1.0	`u24.aarch64`	pigsty	10.5 MiB	postgresql-15-pg-tokenizer_0.1.0-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_14`	0.1.1	`el8.x86_64`	pigsty	11.7 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_14`	0.1.1	`el8.aarch64`	pigsty	11.5 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_14`	0.1.1	`el9.x86_64`	pigsty	11.0 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_14`	0.1.1	`el9.aarch64`	pigsty	10.9 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_14`	0.1.1	`el10.x86_64`	pigsty	10.9 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_14`	0.1.1	`el10.aarch64`	pigsty	11.0 MiB	pg_tokenizer_14-0.1.1-1PIGSTY.el10.aarch64.rpm
`postgresql-14-pg-tokenizer`	0.1.0	`d12.x86_64`	pigsty	9.8 MiB	postgresql-14-pg-tokenizer_0.1.0-1PIGSTY~bookworm_amd64.deb
`postgresql-14-pg-tokenizer`	0.1.0	`d12.aarch64`	pigsty	9.6 MiB	postgresql-14-pg-tokenizer_0.1.0-1PIGSTY~bookworm_arm64.deb
`postgresql-14-pg-tokenizer`	0.1.0	`u22.x86_64`	pigsty	10.7 MiB	postgresql-14-pg-tokenizer_0.1.0-1PIGSTY~jammy_amd64.deb
`postgresql-14-pg-tokenizer`	0.1.0	`u22.aarch64`	pigsty	10.5 MiB	postgresql-14-pg-tokenizer_0.1.0-1PIGSTY~jammy_arm64.deb
`postgresql-14-pg-tokenizer`	0.1.0	`u24.x86_64`	pigsty	10.6 MiB	postgresql-14-pg-tokenizer_0.1.0-1PIGSTY~noble_amd64.deb
`postgresql-14-pg-tokenizer`	0.1.0	`u24.aarch64`	pigsty	10.5 MiB	postgresql-14-pg-tokenizer_0.1.0-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tokenizer_13`	0.1.1	`el8.x86_64`	pigsty	11.7 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el8.x86_64.rpm
`pg_tokenizer_13`	0.1.1	`el8.aarch64`	pigsty	11.5 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el8.aarch64.rpm
`pg_tokenizer_13`	0.1.1	`el9.x86_64`	pigsty	11.0 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el9.x86_64.rpm
`pg_tokenizer_13`	0.1.1	`el9.aarch64`	pigsty	10.9 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el9.aarch64.rpm
`pg_tokenizer_13`	0.1.1	`el10.x86_64`	pigsty	10.9 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el10.x86_64.rpm
`pg_tokenizer_13`	0.1.1	`el10.aarch64`	pigsty	11.0 MiB	pg_tokenizer_13-0.1.1-1PIGSTY.el10.aarch64.rpm

Source

Repository

github.com/tensorchord/pg_tokenizer.rs

Source Tarball

pg_tokenizer.rs-0.1.1.tar.gz

pig build get pg_tokenizer; # get pg_tokenizer source code
pig build dep pg_tokenizer; # install build dependencies
pig build pkg pg_tokenizer; # build extension rpm or deb
pig build ext pg_tokenizer; # build extension rpms

Install

To add the required PGDG / PIGSTY upstream repository, use:

pig repo add pgsql -u   # add PGDG + Pigsty repo and update cache (leave existing repos)

Install this extension with:

pig ext install pg_tokenizer; # install by extension name, for the current active PG version
pig ext install pg_tokenizer; # install via package alias, for the active PG version
pig ext install pg_tokenizer -v 18;   # install for PG 18
pig ext install pg_tokenizer -v 17;   # install for PG 17
pig ext install pg_tokenizer -v 16;   # install for PG 16
pig ext install pg_tokenizer -v 15;   # install for PG 15
pig ext install pg_tokenizer -v 14;   # install for PG 14
pig ext install pg_tokenizer -v 13;   # install for PG 13

Create this extension with:

CREATE EXTENSION pg_tokenizer CASCADE SCHEMA tokenizer_catalog;

Last updated on 2025-11-02

vchord_bm25 hunspell_cs_cz