pg_tiktoken

tiktoken tokenizer for use with OpenAI models in postgres

Overview

ID	Extension	Package	Version	Category	License	Language
1870	pg_tiktoken	pg_tiktoken	`0.0.1`	RAG	Apache-2.0	Rust

Attribute	Has Binary	Has Library	Need Load	Has DDL	Relocatable	Trusted
--s-d--	No	Yes	No	Yes	no	no

Relationships
See Also	vectorize pg_summarize pg4ml pgml vector vchord vectorscale pg_graphql

Packages

Type	Repo	Version	PG Major Availability	Package Pattern	Dependencies
EL	PIGSTY	`0.0.1`	18 17 16 15 14 13	`pg_tiktoken_$v`	-
Debian	PIGSTY	`0.0.1`	18 17 16 15 14 13	`postgresql-$v-pg-tiktoken`	-

Linux / PG	PG18	PG17	PG16	PG15	PG14	PG13
el8.x86_64	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
el8.aarch64	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
el9.x86_64	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
el9.aarch64	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
el10.x86_64	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
el10.aarch64	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
d12.x86_64	MISS	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
d12.aarch64	MISS	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
d13.x86_64	MISS	MISS	MISS	MISS	MISS	MISS
d13.aarch64	MISS	MISS	MISS	MISS	MISS	MISS
u22.x86_64	MISS	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
u22.aarch64	MISS	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
u24.x86_64	MISS	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1
u24.aarch64	MISS	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1	PIGSTY 0.0.1

Package	Version	OS	ORG	SIZE	File URL
`pg_tiktoken_18`	0.0.1	`el8.x86_64`	pigsty	1.7 MiB	pg_tiktoken_18-0.0.1-2PIGSTY.el8.x86_64.rpm
`pg_tiktoken_18`	0.0.1	`el8.aarch64`	pigsty	1.6 MiB	pg_tiktoken_18-0.0.1-2PIGSTY.el8.aarch64.rpm
`pg_tiktoken_18`	0.0.1	`el9.x86_64`	pigsty	1.6 MiB	pg_tiktoken_18-0.0.1-2PIGSTY.el9.x86_64.rpm
`pg_tiktoken_18`	0.0.1	`el9.aarch64`	pigsty	1.5 MiB	pg_tiktoken_18-0.0.1-2PIGSTY.el9.aarch64.rpm
`pg_tiktoken_18`	0.0.1	`el10.x86_64`	pigsty	1.7 MiB	pg_tiktoken_18-0.0.1-2PIGSTY.el10.x86_64.rpm
`pg_tiktoken_18`	0.0.1	`el10.aarch64`	pigsty	1.5 MiB	pg_tiktoken_18-0.0.1-2PIGSTY.el10.aarch64.rpm

Package	Version	OS	ORG	SIZE	File URL
`pg_tiktoken_17`	0.0.1	`el8.x86_64`	pigsty	1.7 MiB	pg_tiktoken_17-0.0.1-2PIGSTY.el8.x86_64.rpm
`pg_tiktoken_17`	0.0.1	`el8.aarch64`	pigsty	1.6 MiB	pg_tiktoken_17-0.0.1-2PIGSTY.el8.aarch64.rpm
`pg_tiktoken_17`	0.0.1	`el9.x86_64`	pigsty	1.6 MiB	pg_tiktoken_17-0.0.1-2PIGSTY.el9.x86_64.rpm
`pg_tiktoken_17`	0.0.1	`el9.aarch64`	pigsty	1.5 MiB	pg_tiktoken_17-0.0.1-2PIGSTY.el9.aarch64.rpm
`pg_tiktoken_17`	0.0.1	`el10.x86_64`	pigsty	1.7 MiB	pg_tiktoken_17-0.0.1-2PIGSTY.el10.x86_64.rpm
`pg_tiktoken_17`	0.0.1	`el10.aarch64`	pigsty	1.5 MiB	pg_tiktoken_17-0.0.1-2PIGSTY.el10.aarch64.rpm
`postgresql-17-pg-tiktoken`	0.0.1	`d12.x86_64`	pigsty	4.9 KiB	postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
`postgresql-17-pg-tiktoken`	0.0.1	`d12.aarch64`	pigsty	4.9 KiB	postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
`postgresql-17-pg-tiktoken`	0.0.1	`u22.x86_64`	pigsty	4.9 KiB	postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
`postgresql-17-pg-tiktoken`	0.0.1	`u22.aarch64`	pigsty	4.9 KiB	postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
`postgresql-17-pg-tiktoken`	0.0.1	`u24.x86_64`	pigsty	4.9 KiB	postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
`postgresql-17-pg-tiktoken`	0.0.1	`u24.aarch64`	pigsty	4.9 KiB	postgresql-17-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tiktoken_16`	0.0.1	`el8.x86_64`	pigsty	1.7 MiB	pg_tiktoken_16-0.0.1-2PIGSTY.el8.x86_64.rpm
`pg_tiktoken_16`	0.0.1	`el8.aarch64`	pigsty	1.6 MiB	pg_tiktoken_16-0.0.1-2PIGSTY.el8.aarch64.rpm
`pg_tiktoken_16`	0.0.1	`el9.x86_64`	pigsty	1.6 MiB	pg_tiktoken_16-0.0.1-2PIGSTY.el9.x86_64.rpm
`pg_tiktoken_16`	0.0.1	`el9.aarch64`	pigsty	1.5 MiB	pg_tiktoken_16-0.0.1-2PIGSTY.el9.aarch64.rpm
`pg_tiktoken_16`	0.0.1	`el10.x86_64`	pigsty	1.7 MiB	pg_tiktoken_16-0.0.1-2PIGSTY.el10.x86_64.rpm
`pg_tiktoken_16`	0.0.1	`el10.aarch64`	pigsty	1.5 MiB	pg_tiktoken_16-0.0.1-2PIGSTY.el10.aarch64.rpm
`postgresql-16-pg-tiktoken`	0.0.1	`d12.x86_64`	pigsty	1.3 MiB	postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
`postgresql-16-pg-tiktoken`	0.0.1	`d12.aarch64`	pigsty	1.2 MiB	postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
`postgresql-16-pg-tiktoken`	0.0.1	`u22.x86_64`	pigsty	1.5 MiB	postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
`postgresql-16-pg-tiktoken`	0.0.1	`u22.aarch64`	pigsty	1.5 MiB	postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
`postgresql-16-pg-tiktoken`	0.0.1	`u24.x86_64`	pigsty	1.5 MiB	postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
`postgresql-16-pg-tiktoken`	0.0.1	`u24.aarch64`	pigsty	1.4 MiB	postgresql-16-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tiktoken_15`	0.0.1	`el8.x86_64`	pigsty	1.7 MiB	pg_tiktoken_15-0.0.1-2PIGSTY.el8.x86_64.rpm
`pg_tiktoken_15`	0.0.1	`el8.aarch64`	pigsty	1.6 MiB	pg_tiktoken_15-0.0.1-2PIGSTY.el8.aarch64.rpm
`pg_tiktoken_15`	0.0.1	`el9.x86_64`	pigsty	1.6 MiB	pg_tiktoken_15-0.0.1-2PIGSTY.el9.x86_64.rpm
`pg_tiktoken_15`	0.0.1	`el9.aarch64`	pigsty	1.5 MiB	pg_tiktoken_15-0.0.1-2PIGSTY.el9.aarch64.rpm
`pg_tiktoken_15`	0.0.1	`el10.x86_64`	pigsty	1.7 MiB	pg_tiktoken_15-0.0.1-2PIGSTY.el10.x86_64.rpm
`pg_tiktoken_15`	0.0.1	`el10.aarch64`	pigsty	1.5 MiB	pg_tiktoken_15-0.0.1-2PIGSTY.el10.aarch64.rpm
`postgresql-15-pg-tiktoken`	0.0.1	`d12.x86_64`	pigsty	1.3 MiB	postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
`postgresql-15-pg-tiktoken`	0.0.1	`d12.aarch64`	pigsty	1.2 MiB	postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
`postgresql-15-pg-tiktoken`	0.0.1	`u22.x86_64`	pigsty	1.5 MiB	postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
`postgresql-15-pg-tiktoken`	0.0.1	`u22.aarch64`	pigsty	1.5 MiB	postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
`postgresql-15-pg-tiktoken`	0.0.1	`u24.x86_64`	pigsty	1.5 MiB	postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
`postgresql-15-pg-tiktoken`	0.0.1	`u24.aarch64`	pigsty	1.4 MiB	postgresql-15-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tiktoken_14`	0.0.1	`el8.x86_64`	pigsty	1.7 MiB	pg_tiktoken_14-0.0.1-2PIGSTY.el8.x86_64.rpm
`pg_tiktoken_14`	0.0.1	`el8.aarch64`	pigsty	1.6 MiB	pg_tiktoken_14-0.0.1-2PIGSTY.el8.aarch64.rpm
`pg_tiktoken_14`	0.0.1	`el9.x86_64`	pigsty	1.6 MiB	pg_tiktoken_14-0.0.1-2PIGSTY.el9.x86_64.rpm
`pg_tiktoken_14`	0.0.1	`el9.aarch64`	pigsty	1.5 MiB	pg_tiktoken_14-0.0.1-2PIGSTY.el9.aarch64.rpm
`pg_tiktoken_14`	0.0.1	`el10.x86_64`	pigsty	1.7 MiB	pg_tiktoken_14-0.0.1-2PIGSTY.el10.x86_64.rpm
`pg_tiktoken_14`	0.0.1	`el10.aarch64`	pigsty	1.5 MiB	pg_tiktoken_14-0.0.1-2PIGSTY.el10.aarch64.rpm
`postgresql-14-pg-tiktoken`	0.0.1	`d12.x86_64`	pigsty	1.3 MiB	postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
`postgresql-14-pg-tiktoken`	0.0.1	`d12.aarch64`	pigsty	1.2 MiB	postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
`postgresql-14-pg-tiktoken`	0.0.1	`u22.x86_64`	pigsty	1.5 MiB	postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
`postgresql-14-pg-tiktoken`	0.0.1	`u22.aarch64`	pigsty	1.5 MiB	postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
`postgresql-14-pg-tiktoken`	0.0.1	`u24.x86_64`	pigsty	1.5 MiB	postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
`postgresql-14-pg-tiktoken`	0.0.1	`u24.aarch64`	pigsty	1.4 MiB	postgresql-14-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb

Package	Version	OS	ORG	SIZE	File URL
`pg_tiktoken_13`	0.0.1	`el8.x86_64`	pigsty	1.7 MiB	pg_tiktoken_13-0.0.1-2PIGSTY.el8.x86_64.rpm
`pg_tiktoken_13`	0.0.1	`el8.aarch64`	pigsty	1.6 MiB	pg_tiktoken_13-0.0.1-2PIGSTY.el8.aarch64.rpm
`pg_tiktoken_13`	0.0.1	`el9.x86_64`	pigsty	1.6 MiB	pg_tiktoken_13-0.0.1-2PIGSTY.el9.x86_64.rpm
`pg_tiktoken_13`	0.0.1	`el9.aarch64`	pigsty	1.5 MiB	pg_tiktoken_13-0.0.1-2PIGSTY.el9.aarch64.rpm
`pg_tiktoken_13`	0.0.1	`el10.x86_64`	pigsty	1.7 MiB	pg_tiktoken_13-0.0.1-2PIGSTY.el10.x86_64.rpm
`pg_tiktoken_13`	0.0.1	`el10.aarch64`	pigsty	1.5 MiB	pg_tiktoken_13-0.0.1-2PIGSTY.el10.aarch64.rpm
`postgresql-13-pg-tiktoken`	0.0.1	`d12.x86_64`	pigsty	1.3 MiB	postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~bookworm_amd64.deb
`postgresql-13-pg-tiktoken`	0.0.1	`d12.aarch64`	pigsty	1.2 MiB	postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~bookworm_arm64.deb
`postgresql-13-pg-tiktoken`	0.0.1	`u22.x86_64`	pigsty	1.5 MiB	postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~jammy_amd64.deb
`postgresql-13-pg-tiktoken`	0.0.1	`u22.aarch64`	pigsty	1.5 MiB	postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~jammy_arm64.deb
`postgresql-13-pg-tiktoken`	0.0.1	`u24.x86_64`	pigsty	1.5 MiB	postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~noble_amd64.deb
`postgresql-13-pg-tiktoken`	0.0.1	`u24.aarch64`	pigsty	1.4 MiB	postgresql-13-pg-tiktoken_0.0.1-1PIGSTY~noble_arm64.deb

Source

Repository

github.com/kelvich/pg_tiktoken

Source Tarball

pg_tiktoken-0.0.1.tar.gz

pig build get pg_tiktoken; # get pg_tiktoken source code
pig build dep pg_tiktoken; # install build dependencies
pig build pkg pg_tiktoken; # build extension rpm or deb
pig build ext pg_tiktoken; # build extension rpms

Install

To add the required PGDG / PIGSTY upstream repository, use:

pig repo add pgsql -u   # add PGDG + Pigsty repo and update cache (leave existing repos)

Install this extension with:

pig ext install pg_tiktoken; # install by extension name, for the current active PG version
pig ext install pg_tiktoken; # install via package alias, for the active PG version
pig ext install pg_tiktoken -v 18;   # install for PG 18
pig ext install pg_tiktoken -v 17;   # install for PG 17
pig ext install pg_tiktoken -v 16;   # install for PG 16
pig ext install pg_tiktoken -v 15;   # install for PG 15
pig ext install pg_tiktoken -v 14;   # install for PG 14
pig ext install pg_tiktoken -v 13;   # install for PG 13

Create this extension with:

CREATE EXTENSION pg_tiktoken;

Last updated on 2025-11-02

pg_summarize pg4ml