ᚴᛁᛁᛚᛐᛁᛒᚢᛌᛐ · SRC-377

ᛐᛁᚴᛐᚢᚴᛁᚿ ᚱᛁᚠᛁᚱᛆᚿᛌᛁᛁᛘᛒᛚᛁᛘᛁᚿᛐᛆᛌᛁᚢᚿ

SRC-377 · ᛐᛁᚴᛐᚢᚴᛁᚿ ᚱᛁᚠᛁᚱᛆᚿᛌᛁᛁᛘᛒᛚᛁᛘᛁᚿᛐᛆᛌᛁᚢᚿ · ᚴᛁᛁᛚᛐᛁᛒᚢᛌᛐ ᚠᚱᚭ ᚢᛒᛁᚿAI ᛘᛁᛐ ᛒᚢᛒᛚᛁᛌᛁᚱᛁᚿᚴᛌᛐᛆᛐᚢ 2022-12-01, ᚴᛁᛁᛚᛐᛁᚴᛚᛆᛌᛌᛁ ᚢᚠᚠᛁᛌᛁᛁᛚᛚ ᚢᛒᛁᚿ-ᛌᚢᚢᚱᚴᛁ ᛐᚢᚴᛁᚿᛁᛌᛁᚱᛁᛘᛒᛚᛁᛘᛁᚿᛐᛆᛌᛁᚢᚿ, ᚱᛁᚴᛁᛌᛐᚱᛁᚱᛐ ᛆᚴᛐᚢᛆᛚᛁᛐᛁᛐ ᛆᚴᛐᚢᛁᛚᛚ ᚢᚴ ᛘᛁᛐᛆᛐᛆᛐᛆᚱᛁᚠᛁᛌᛁᚢᚿ ᛁᚴᚴᛁᛁ ᚴᚱᛆᛐᛁᚱᛐ.

  • ᛆᚴᛐᚢᛁᛚᛚ CURRENT
  • ᛐᛁᛚᚴᛁᛁᚿᚴᛁᛚᛁᚴ AVAILABLE

ᛐᛁᚱᛁᚴᛐᛁ ᛌᚠᛆᚱ

SRC-377 · ᛐᛁᚴᛐᚢᚴᛁᚿ ᚱᛁᚠᛁᚱᛆᚿᛌᛁᛁᛘᛒᛚᛁᛘᛁᚿᛐᛆᛌᛁᚢᚿ · ᚴᛁᛁᛚᛐᛁᛒᚢᛌᛐ ᚠᚱᚭ ᚢᛒᛁᚿAI ᛘᛁᛐ ᛒᚢᛒᛚᛁᛌᛁᚱᛁᚿᚴᛌᛐᛆᛐᚢ 2022-12-01, ᚴᛁᛁᛚᛐᛁᚴᛚᛆᛌᛌᛁ ᚢᚠᚠᛁᛌᛁᛁᛚᛚ ᚢᛒᛁᚿ-ᛌᚢᚢᚱᚴᛁ ᛐᚢᚴᛁᚿᛁᛌᛁᚱᛁᛘᛒᛚᛁᛘᛁᚿᛐᛆᛌᛁᚢᚿ, ᚱᛁᚴᛁᛌᛐᚱᛁᚱᛐ ᛆᚴᛐᚢᛆᛚᛁᛐᛁᛐ ᛆᚴᛐᚢᛁᛚᛚ ᚢᚴ ᛘᛁᛐᛆᛐᛆᛐᛆᚱᛁᚠᛁᛌᛁᚢᚿ ᛁᚴᚴᛁᛁ ᚴᚱᛆᛐᛁᚱᛐ.

ᛌᛁᛘᛆᚿᛐᛁᛌᚴ ᚴᛁᛁᛚᛐᛁᛐᛁᛐᛐᛁᛚ

ᛐᛁᚴᛐᚢᚴᛁᚿ ᚱᛁᚠᛁᚱᛆᚿᛌᛁᛁᛘᛒᛚᛁᛘᛁᚿᛐᛆᛌᛁᚢᚿ

ᛐᛁᛐᛐᛁᛚᛁᚿ ᛁᚱ ᛁᛁ ᚱᚯᚴ-ᚿᚢᚱᛐᛁᛌᚴ ᛌᚴᛁᛚᛐᚱᛁᚿᚴ ᚠᚢᚱ ᛐᛁᚿᚿᛁ ᚴᛆᛐᛆᛚᚢᚴᛁᚿ ᚢᚴ ᚠᛁᚱᛐ ᛁᚴᚴᛁᛁ ᚠᚱᛆᛘᛌᛐᛁᛚᛐ ᛌᚢᛘ ᚢᛐᚴᛁᛁᚠᛆᚱᛁᚿ ᛌᛁᚿ ᚢᚠᚠᛁᛌᛁᛁᛚᛚᛁ ᛐᛁᛐᛐᛁᛚ.

ᚴᛁᛁᛚᛐᛁᚴᛚᛆᛌᛌᛁ

ᚢᚠᚠᛁᛌᛁᛁᛚᛚ ᚢᛒᛁᚿ-ᛌᚢᚢᚱᚴᛁ ᛐᚢᚴᛁᚿᛁᛌᛁᚱᛁᛘᛒᛚᛁᛘᛁᚿᛐᛆᛌᛁᚢᚿ · ᛒᚱᛁᛘᛅᚱ

ᛒᚭᛚᛁᛐᛁᛚᛁᚴᚽᛁᛁᛐᛌᚽᛁᛌᛐᚢᚱᛁᛁ

ᚢᚠᚠᛁᛌᛁᛁᛚᛚ ᛌᛐᛆᚿᛐᛆᚱᛐ-, ᚱᛁᚴᚢᛚᛆᛐᚢᚱ-, ᛚᛆᛒᚢᚱᛆᛐᚢᚱᛁᚢᛘ- ᛁᛚᛚᛁᚱ ᚢᛒᛒᚽᛆᚠᛌᛁᛘᛒᛚᛁᛘᛁᚿᛐᛆᛌᛁᚢᚿᛌᚴᛁᛁᛚᛐᛁ ᛘᛁᛐ ᛆᚠᚴᚱᛁᚿᛌᛆ ᛒᚭᛌᛐᛆᚿᛐᛌᚢᛘᚠᛆᚿᚴ. ᚴᚠᛆᚱ ᛒᚭᛌᛐᛆᚿᛐ ᛘᚭ ᛚᛁᛌᛆᛌᛐ ᛘᚢᛐ ᛐᚢᚴᚢᛘᛁᚿᛐᚠᛁᚱᛌᛁᚢᚿ, ᛐᛆᛐᚢ ᚢᚴ ᚴᛁᛁᛚᛐᛆᚿᛐᛁ ᛒᚱᚢᚠᛁᛚ.

ᛘᛁᛐᛆᛐᛆᛐᛆᚱᛁᚠᛁᛌᛁᚢᚿ

ᚢᚠᚠᛁᛌᛁᛁᛚᛚ ᛐᚢᚴᚢᛘᛁᚿᛐ- ᛁᛚᛚᛁᚱ ᛁᛘᛒᛚᛁᛘᛁᚿᛐᛆᛌᛁᚢᚿᛌᚱᛁᚠᛁᚱᛆᚿᛌᛁ ᛁᚱ ᚱᛁᚴᛁᛌᛐᚱᛁᚱᛐ ᚠᚢᚱ ᛌᛁᚯᛚᚠᛌᛐᛁᚿᛐᛁᚴ ᚴᚢᚿᛐᚱᚢᛚᛚ. ᛌ12 ᚢᛐᚠᚯᚱᛁᚱ ᛁᚿᚴᛆ ᚿᛁᛐᛐᛁᚿᚿᚽᛁᚿᛐᛁᚿᚴ ᚠᛁᛐ ᛌᛁᛐᛁᚠᛁᛌᛁᚿᚴ ᚢᚴ ᛌᛁᚱᛐᛁᚠᛁᛌᛁᚱᛁᚱ ᛁᚴᚴᛁᛁ ᚠᚢᛚᛚᛌᛐᛁᚿᛐᛁ ᛁᛚᛚᛁᚱ ᛘᛆᛐᛁᚱᛁᛁᛚᛚ ᛌᛆᚿᚿᛁᚿᚴ.

ᚢᛒᚿᛆ ᛌᛆᛘᛚᛆ ᛆᚴᛐᚢᛆᛚᛁᛐᛁᛐᛌᚱᛁᚴᛁᛌᛐᛁᚱ

ᚴᛁᛁᛚᛐᛁᛌᛒᛁᛌᛁᚠᛁᚴᚴᛁ ᛆᚠᚴᚱᛁᚿᛌᛁᚿᚴᛆᚱ

ᛒᚱᚢᚴᚱᛆᛘᚠᛆᚱᛁ ᚢᚴ ᛁᚿᚴᚢᛐᛁᚿᚴᛆᚱᛐᛁᚠᛆᚴᛐ ᛘᚭ ᛒᛁᚿᛐᛆᛌᛐ ᚴᚠᛆᚱ ᚠᚢᚱ ᛌᛁᚴ ᛘᛁᛐ ᚠᛁᚱᛌᛁᚢᚿ ᚢᚴ ᚽᛆᛌᚽ. ᚱᛁᛒᚢᛁᛐ ᛁᛐᛆᛒᛚᛁᚱᛁᚱ ᛁᚴᚴᛁᛁ ᛁᛁᛐᛐ-ᛐᚢᚴᛁᚿ-ᛒᛁᚱ-ᚱᚢᚿᛁ ᛁᛚᛚᛁᚱ ᚢᚿᛁᚠᛁᚱᛌᛁᛚᛚ ᚴᚢᛘᛒᚱᛁᛌᛁᚢᚿ.

ᛆᚴᛐᚢᛆᛚᛁᛐᛁᛐᛌᚴᚱᛁᚿᛌᛁ

ᚴᛁᛁᛚᛐᛆ ᛘᚭ ᚠᚢᚱᛐᛁᚱᛆᛌᛐ ᛘᚢᛐ ᛐᛁᚿ ᚴᚢᚿᚴᚱᛁᛐᛁ ᛒᚭᛌᛐᛆᚿᛐᛁᚿ, ᚠᛁᚱᛌᛁᚢᚿᛁᚿ ᚢᚴ ᚢᛒᛌᛁᚱᚠᛆᛌᛁᚢᚿᛌᛐᛁᛐᛆ. ᛁᛚᛐᚱᛁ ᚴᛁᛁᛚᛐᛁᚱ ᚴᛆᚿ ᚠᛁᚱᛆ ᛆᚢᛐᚢᚱᛁᛐᛆᛐᛁᚠᛁ ᚠᚢᚱ ᚴᚱᚢᚿᚿᚱᛁᚴᛚᛆᚱ ᚢᚴ ᛚᛁᚴᛁᚠᛁᛚ ᚢᛐᛁᛚᛌᛐᚱᛁᚴᚴᛁᛚᛁᚴᛁ ᚠᚢᚱ ᚿᚢᚠᛁᚱᛆᚿᛐᛁ ᛐᛁᛚᛌᛐᛆᚿᛐ.

ᛁᚴᛌᛐᛁᚱᚿ ᛐᛁᛚᚴᛆᚿᚴ

ᚢᛒᚿᛆ ᛐᛁᚿ ᛁᚴᛌᛐᛁᚱᚿᛁ ᚴᛁᛁᛚᛐᛆ

https://github.com/openai/tiktoken

ᚴᚠᛆ ᚴᛁᛁᛚᛐᛁᛒᚢᛌᛐᛁᚿ ᛁᚴᚴᛁᛁ ᛒᚱᚢᚠᛆᚱ

ᛆᚢᛐᛁᚿᛐᛁᛌᚴ ᚢᛒᛒᚽᛆᚠ ᛒᚱᚢᚠᛆᚱ ᛁᚴᚴᛁᛁ ᛆᛐ ᛁᚿᚿᚽᛆᛚᛐᛁᛐ ᛁᚱ ᚠᚢᛚᛚᛌᛐᛁᚿᛐᛁᚴ, ᛆᚴᛐᚢᛁᛚᛐ, ᚱᛁᛚᛁᚠᛆᚿᛐ ᚠᚢᚱ ᛆᛚᛚᛁ ᛌᛒᚯᚱᛌᛘᚭᛚ ᛁᛚᛚᛁᚱ ᚱᛁᛐᛐ ᛐᚢᛚᚴᛆ.

ᛌᛆᛘᛒᚢᚿᛐᚿᛁ ᛌᛁᚴᚿᛆᛚ