Tokenization

What is tokenization in simple words

Tokenization is the process of converting data or assets into digital equivalents — tokens. Depending on the context, this word means different things, but the essence is one: replace sensitive information with a safe substitute that has no value for attackers.

In simple words, imagine that you have a key to an apartment. Instead of showing it to everyone, you make a special “duplicate” with a modified shape that fits only your lock. Even if someone sees this duplicate, they will not be able to make a real key from it. Tokenization works the same way — you replace real data (card number, passport data) with a random set of characters (a token) that is useless to attackers without a decryption system.

Tokenization is used in three main areas: information security (protection of payment data), blockchain (digital assets) and artificial intelligence (text processing). According to Juniper Research, the global tokenization market will reach $5.2 billion by 2027, and more than 70% of all transactions in the world will go through tokenized payment systems. Read about data protection in the article Information security.

Tokenization in information security

This is a method of protecting confidential data in which real data (bank card number, passport data, SNILS, INN) is replaced with a random set of characters — a token. The token does not carry the original information, so if the seller's system is hacked, hackers will only get useless tokens.

How it works in practice: When you pay for a purchase through a smartphone (Apple Pay, Google Pay, Mir Pay), your real bank card number is not transmitted to the seller. Instead, a unique token is generated that is valid only for this transaction or for this seller. Even if the seller's database is compromised, the tokens cannot be used for other payments.

Main advantages of tokenization for information security:

• Data protection: Even if the database is hacked, tokens are useless without a decryption system that is stored separately in a secure HSM (Hardware Security Module).
• Compliance with standards: Helps comply with PCI DSS (Payment Card Industry Data Security Standard) requirements for payment systems, as well as 152-FZ requirements for personal data protection.
• Transaction security: Each token is unique for a specific operation or seller, which prevents fraud.
• Reduced liability: Companies do not store real customer data, which reduces the risks of leaks and regulator fines.

Read about security standards in the article PCI DSS. In Russia, tokenization is actively used in the systems of the Fast Payment System (SBP) and the Mir national payment system to protect transactions.

Tokenization in blockchain and cryptocurrencies

In the crypto industry, tokenization is the representation of real assets (real estate, stocks, gold, works of art) in the form of digital tokens in the blockchain. This allows:

• Dividing indivisible assets: For example, selling 1% of an apartment or 0.1% of a painting. This makes investments accessible to a wide range of people.
• Accelerating transactions: Digital tokens are transferred instantly around the world without intermediaries and paperwork.
• Increasing liquidity: Assets that were previously difficult to sell (real estate, works of art) become available on the global market.
• Ensuring transparency: All transactions with tokens are recorded in the blockchain and available for verification.

Examples of tokenized assets: tokenized real estate (RealT), tokenized gold (PAX Gold, Tether Gold), tokenized company stocks (Securitize). Read about the blockchain in the article Blockchain.

Tokenization in artificial intelligence and NLP

In neural networks and natural language processing (NLP), tokenization is the division of text into small fragments (words, subwords, syllables or characters) — tokens. Models (for example, language models in ChatGPT, Yandex GPT, GigaChat) cannot read words as a whole, so the text is translated into numerical identifiers that the computer can process.

Example: The phrase “I am studying tokenization” can be divided by the algorithm into tokens: [“I”, “am”, “study”, “ing”, “token”, “ization”] (subword tokenization) or into separate words: [“I”, “am”, “studying”, “tokenization”] (word tokenization).

There are different approaches to tokenization in NLP: word (division into words), subword (BPE — Byte-Pair Encoding, WordPiece), character (division into separate characters). The choice of method affects the quality of the model's work and its ability to process unknown words. Read about artificial intelligence in the article Artificial intelligence.

Comparison of tokenization and encryption

Tokenization is often confused with encryption, but these are different methods of data protection. Understanding their differences is important for choosing the right approach:

• Encryption: This is a reversible transformation of data using a key. If you have the key, you can decrypt the data and get the original information. Encryption can be broken mathematically with sufficient computing power.
• Tokenization: This is replacing data with a random substitute (a token) that does not contain the original information. The token cannot be “decrypted” mathematically, since it simply does not store data. The connection between the token and the real data is stored separately in a secure system (Token Vault).

The choice between tokenization and encryption depends on the task: for storing data in databases, encryption is often used, for payment transactions — tokenization, since it is more secure and complies with PCI DSS requirements. Read about encryption in the article Data encryption.