Re: Ada and Unicode

[Vadim Godunko <vgodunko@gmail.com>]

On Sunday, April 18, 2021 at 1:03:14 AM UTC+3, DrPi wrote:
> 
> I have a good knowledge of Unicode : code points, encoding... 
> What I don't understand is how to manage Unicode strings with Ada. I've 
> read part of ARM and did some tests without success. 
> 
> I managed to be partly successful with source code encoded in Latin-1. 
> Any other encoding failed. 
> Any way to use source code encoded in UTF-8 ? 
> In some languages, it is possible to set a tag at the beginning of the 
> source file to direct the compiler which encoding to use. 
> I wasn't successful using -gnatW8 switch. But maybe I made to many tests 
> and my brain was scrambled. 
> 
> Even with source code encoded in Latin-1, I've not been able to manage 
> Unicode strings correctly. 
> 
> What's the way to manage Unicode correctly ? 
> 

Ada doesn't have good Unicode support. :( So, you need to find suitable set of "workarounds".

There are few different aspects of Unicode support need to be considered:

1. Representation of string literals. If you want to use non-ASCII characters in source code, you need to use -gnatW8 switch and it will require use of Wide_Wide_String everywhere.
2. Internal representation during application execution. You are forced to use Wide_Wide_String at previous step, so it will be UCS4/UTF32.
3. Text encoding/decoding on input/output operations. GNAT allows to use UTF-8 by providing some magic string for Form parameter of Text_IO.

It is hard to say that it is reasonable set of features for modern world. To fix some of drawbacks of current situation we are developing new text processing library, know as VSS. 

https://github.com/AdaCore/VSS

At current stage it provides encoding independent API for text manipulation, encoders and decoders API for I/O, and JSON reader/writer; regexp support should come soon.

Encoding independent API means that application always use Unicode characters to process text, independently from the real encoding used to store information in memory (UTF-8 is used for now, UTF-16 will be added later for interoperability with Windows API and WASM). Coders and encoders allow translation from/to different encodings when application exchange information with the world.