6 commits

Author	SHA1	Message	Date
Eduard Urbach	f31ea5e825	Implemented token splitting as a generator	2025-03-20 00:26:42 +01:00
Eduard Urbach	f7bc903aa4	Added more tests	2025-02-05 11:11:15 +01:00
Eduard Urbach	a466281307	Simplified file structure	2024-08-07 19:39:10 +02:00
Eduard Urbach	722d07c321	Reorganized file structure	2024-06-10 15:51:39 +02:00
Eduard Urbach	18993e0855	Added token tests	2024-06-06 21:35:14 +02:00
Eduard Urbach	21cf4b5c6d	Added a tokenizer	2023-10-31 11:57:37 +01:00