From: "Luke A. Guest" <laguest@archeia.com>
Subject: Re: How to read in a (long) UTF-8 file, incrementally?
Date: Wed, 3 Nov 2021 08:48:58 +0000 [thread overview]
Message-ID: <sltigk$43o$1@gioia.aioe.org> (raw)
In-Reply-To: d1c5ba75-bc0a-4e7b-a2df-394bc710cbcen@googlegroups.com
On 02/11/2021 17:42, Marius Amado-Alves wrote:
> As I understand it, to work with Unicode text inside the program it is better to use the Wide_Wide (UTF-32) variants of everything.
You can take a look at my simple lib: https://github.com/Lucretia/uca
> Now, Unicode files usually are in UTF-8.
>
> One solution is to read the entire file in one gulp to a String, then convert to Wide_Wide. This solution is not memory efficient, and it may not be possible in some tasks e.g. real time processing of lines of text.
It can read into a large string buffer.
> If the files has lines, I guess we can also work line by line (Text_IO). But the text may not have lines. Can be a long XML object, for example.
And can break it up into lines. There's no Unicode consistency checks.
The lib is a bit hacky, but seems to work for now. There's nothing more
than what I've mentioned so far.
next prev parent reply other threads:[~2021-11-03 8:48 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-02 17:42 How to read in a (long) UTF-8 file, incrementally? Marius Amado-Alves
2021-11-02 18:17 ` Dmitry A. Kazakov
2021-11-03 7:43 ` Vadim Godunko
2021-11-03 8:48 ` Luke A. Guest [this message]
2021-11-04 11:43 ` Marius Amado-Alves
2021-11-04 12:13 ` Dmitry A. Kazakov
2021-11-04 14:30 ` Luke A. Guest
2021-11-05 10:56 ` Marius Amado-Alves
2021-11-05 19:55 ` Simon Wright
2021-11-16 11:55 ` Marius Amado-Alves
2021-11-16 12:36 ` Dmitry A. Kazakov
2021-11-16 13:52 ` Marius Amado-Alves
2021-11-16 20:23 ` Randy Brukardt
2021-11-16 15:25 ` Luke A. Guest
2021-11-16 17:38 ` Vadim Godunko
replies disabled
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox