Re: Ada and Unicode

[Simon Wright <simon@pushface.org>]

"Randy Brukardt" <randy@rrsoftware.com> writes:

> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> wrote in message 
> news:t2q3cb$bbt$1@gioia.aioe.org...
>> On 2022-04-08 21:19, Simon Wright wrote:
>>> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
>>>
>>>> On 2022-04-08 10:56, Simon Wright wrote:
>>>>> "Randy Brukardt" <randy@rrsoftware.com> writes:
>>>>>
>>>>>> If you had an Ada-like language that used a universal UTF-8 string
>>>>>> internally, you then would have a lot of old and mostly useless
>>>>>> operations supported for array types (since things like slices are
>>>>>> mainly useful for string operations).
>>>>>
>>>>> Just off the top of my head, wouldn't it be better to use
>>>>> UTF32-encoded Wide_Wide_Character internally?
>>>>
>>>> Yep, that is the exactly the problem, a confusion between interface
>>>> and implementation.
>>>
>>> Don't understand. My point was that *when you are implementing this* it
>>> mught be easier to deal with 32-bit charactrs/code points/whatever the
>>> proper jargon is than with UTF8.
>>
>> I think it would be more difficult, because you will have to convert from 
>> and to UTF-8 under the hood or explicitly. UTF-8 is de-facto interface 
>> standard and I/O standard. That would be 60-70% of all cases you need a 
>> string. Most string operations like search, comparison, slicing are 
>> isomorphic between code points and octets. So you would win nothing from 
>> keeping strings internally as arrays of code points.
>
> I basically agree with Dmitry here. The internal representation is an 
> implementation detail, but it seems likely that you would want to store 
> UTF-8 strings directly; they're almost always going to be half the size 
> (even for languages using their own characters like Greek) and for most of 
> us, they'll be just a bit more than a quarter the size. The amount of bytes 
> you copy around matters; the number of operations where code points are 
> needed is fairly small.

Well, I don't have any skin in this game, so I'll shut up at this point.