Sockets, Streams, and Element_Arrays: Much confusion

Sockets, Streams, and Element_Arrays: Much confusion

[Mark Gardner]

Hello, I've been having a bit of difficulty doing some UDP socket 
programming in Ada. As outlined in my stackoverflow question here 
(https://stackoverflow.com/q/74953052/7105391), I'm trying to reply to 
messages I am getting over UDP.

GNAT.Sockets gives me a Stream_Element_Array, which I can't find any 
documentation on how to make use of other than "You should also be able 
to get a Stream, which you should use instead" (About ten years ago, on 
this very newsgroup, somebody said not to use streams with UDP, or at 
least not GNAT.Sockets.Stream).

Adasockets gives me a String, which I can work with, except it throws 
away the from data recvfrom gives it, apparently making it impossible to 
reply to the querying address.

At this point, I'm half-tempted to make my own binding, but as I've 
never done that sort of thing before, I thought I'd ask the wisdom of 
the Usenet if there is a way to convert a Stream_Element_Array into the 
exotic types of Unsigned_16 and String.

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Dmitry A. Kazakov]

On 2022-12-31 13:11, Mark Gardner wrote:

> GNAT.Sockets gives me a Stream_Element_Array, which I can't find any 
> documentation on how to make use of other than "You should also be able 
> to get a Stream, which you should use instead" (About ten years ago, on 
> this very newsgroup, somebody said not to use streams with UDP, or at 
> least not GNAT.Sockets.Stream).

Stream_Element_Array is declared in Ada.Streams as

    type Stream_Element_Array is
       array(Stream_Element_Offset range <>) of
          aliased Stream_Element;

For communication purpose it is an array of octets. Your datagram is 
represented as a Stream_Element_Array or a slice of.

As for streams, yes, it does not make sense to use them for networking, 
unless you override all stream primitives. The reasons for that are

- non-portability of predefined primitives
- low efficiency for complex data types
- encoding inefficiency as well

You will need to handle some application protocol artifacts, checksums, 
counters, strange encodings, sequence number etc. It is easier to this 
directly on the Stream_Element_Array elements.

And, well, do not use UDP, expect for broadcasting. There is no reason 
to use it. For multicast consider delivery-safe protocols like PGM. For 
single cast use TCP/IP. (If you need low latency see the socket NO_DELAY 
option)

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Mark Gardner]

On 31/12/2022 15:11, Dmitry A. Kazakov wrote:
> On 2022-12-31 13:11, Mark Gardner wrote:
> 
>> ...
> 
> Stream_Element_Array is declared in Ada.Streams as
> 
>     type Stream_Element_Array is
>        array(Stream_Element_Offset range <>) of
>           aliased Stream_Element;
> 
> For communication purpose it is an array of octets. Your datagram is 
> represented as a Stream_Element_Array or a slice of.
> 

According to RM 13.13.1, "Stream_Element is mod implementation-defined" 
which to me says there is no guarantee that they will be octets, unless 
this is specified elsewhere?

> As for streams, yes, it does not make sense to use them for networking, 
> unless you override all stream primitives. The reasons for that are
> 
> - non-portability of predefined primitives
> - low efficiency for complex data types
> - encoding inefficiency as well
> 
> You will need to handle some application protocol artifacts, checksums, 
> counters, strange encodings, sequence number etc. It is easier to this 
> directly on the Stream_Element_Array elements.

So, how would I do this directly on the elements? I mean, if it is an 
octet-array to a string, I expect an element-to-element copy, or type 
conversion to work, but what about integers? Do I need to do something like
My_Int:=Unsigned_8(octet(1))+2**8*Unsigned_8(octet(2));
or whatever endianness demands? Or is this the time to learn how to use 
Unchecked_Conversion?

> And, well, do not use UDP, expect for broadcasting. There is no reason 
> to use it. For multicast consider delivery-safe protocols like PGM. For 
> single cast use TCP/IP. (If you need low latency see the socket NO_DELAY 
> option)
> 

  Well, my use case just so happens to be broadcasting, and 
re-broadcasting data across a binary-tree-like p2p network.

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Dmitry A. Kazakov]

On 2022-12-31 14:50, Mark Gardner wrote:
> On 31/12/2022 15:11, Dmitry A. Kazakov wrote:
>> On 2022-12-31 13:11, Mark Gardner wrote:
>>
>>> ...
>>
>> Stream_Element_Array is declared in Ada.Streams as
>>
>>     type Stream_Element_Array is
>>        array(Stream_Element_Offset range <>) of
>>           aliased Stream_Element;
>>
>> For communication purpose it is an array of octets. Your datagram is 
>> represented as a Stream_Element_Array or a slice of.
> 
> According to RM 13.13.1, "Stream_Element is mod implementation-defined" 
> which to me says there is no guarantee that they will be octets, unless 
> this is specified elsewhere?

GNAT.Sockets is GNAT-specific. All GNAT compilers have Stream_Element 8 
bits. I can imagine some DSP implementation with Stream_Element of 32 
bits. But realistically add

    pragma Assert (Stream_Element'Size >= 8);

and be done with that.

> So, how would I do this directly on the elements? I mean, if it is an 
> octet-array to a string, I expect an element-to-element copy, or type 
> conversion to work, but what about integers?

Hmm, it cannot be string. It is a string encoded in some specific way 
(usually most peculiar (:-)). Then you will have to decode it into your 
machine type e.g. Wide_String or String.

An UTF-8 string you could put into String ignoring Ada's Latin-1 stuff, 
as most people would do:

    function To_String (S : Stream_Element_Array) return String is
    begin
       return Result : String (1..S'Length) do
          for I in S'Range loop
             Result (Positive (I - S'First + 1) := Character'Val (S (I));
          end loop;
       end return;
    end To_String;

> Do I need to do something like
> My_Int:=Unsigned_8(octet(1))+2**8*Unsigned_8(octet(2));
> or whatever endianness demands? Or is this the time to learn how to use 
> Unchecked_Conversion?

There are many ways to convert Stream_Element_Array "in situ" to string. 
However, in network protocols you rarely have any strings at all. 
Usually it is some binary data you need to decode into some machine 
type. (So is String or Wide_String actually)

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Jeffrey R.Carter]

On 2022-12-31 14:50, Mark Gardner wrote:
> 
> According to RM 13.13.1, "Stream_Element is mod implementation-defined" which to 
> me says there is no guarantee that they will be octets, unless this is specified 
> elsewhere?

The ARM has always tried to ensure that the language could be implemented on any 
kind of processor. Thus you have implementation-defined separate definitions of 
Storage_Element and Stream_Element, which need not be the same, and no guarantee 
that Interfaces contains declarations of Integer_8 or Unsigned_8.

But these days almost everything is byte oriented, so unless you need what 
you're writing to work on some unusual H/W, you can presume that both of these 
are bytes, and that Interfaces contains those declarations.

-- 
Jeff Carter
"My legs are gray, my ears are gnarled, my eyes are old and bent."
Monty Python's Life of Brian
81

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Jeffrey R.Carter]

On 2022-12-31 23:32, philip...@gmail.com wrote:
> 
> Getting between Stream_Element_Array and a byte array is a pain and I wound up just looping over arrays, copying one byte at a time.  If somebody has a better idea, let me know.

You should be able to use Unchecked_Conversion for that.

-- 
Jeff Carter
"My legs are gray, my ears are gnarled, my eyes are old and bent."
Monty Python's Life of Brian
81

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Dmitry A. Kazakov]

On 2022-12-31 23:32, philip...@gmail.com wrote:

> I have to disagree here.  UDP is perfectly fine for RPC-like (Remote Procedure Call) transactions on a local area network.

RPC and other synchronous exchange policies should be avoided as much as 
possible.

Saying said that, implementation of RPC on top of streams is 
incomparable more easier than on top of UDP.

> And it is orders of magnitude easier to implement on microcontrollers than TCP.

Not at all. You need:

- Safe transmission and error correction on top UDP;
- Buffering and sorting out incoming datagrams;
- Maintaining sequence numbers;
- Splitting messages that do not fit into a single datagram and 
reassembling them on the receiver side;
- Buffering on the sender side to service resend requests.

This is extremely difficult and huge load for a microcontroller.

> Getting between Stream_Element_Array and a byte array is a pain and I wound up just looping over arrays, copying one byte at a time.  If somebody has a better idea, let me know.

Use "in situ" conversion if you are concerned about copying. E.g.

    pragma Import (Ada, Y);
    for Y'Address use X'Address;

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Simon Wright]

Mark Gardner <magardner2017@gmail.com> writes:

> GNAT.Sockets gives me a Stream_Element_Array, which I can't find any
> documentation on how to make use of other than "You should also be
> able to get a Stream, which you should use instead" (About ten years
> ago, on this very newsgroup, somebody said not to use streams with
> UDP, or at least not GNAT.Sockets.Stream).

The reasoning behind the recommendation not to use streams with UDP was
as follows (there's a faint possibility that it no longer applies!)

If the data type you want to send is e.g.

   type Message is record
      Id  : Integer;
      Val : Boolean;
   end record;

and you create a datagram socket and from that a stream, then use
Message'Write to the stream, GNAT will transmit each component of
Message separately in canonical order (the order they're written in the
type declaration). This results in two datagrams being sent, one of 4
bytes and one of 1 byte.

If you take the same approach at the destination, Message'Read reads one
datagram of 4 bytes, and one of 1 byte, and it all looks perfect from the
outside. If the destination is expecting a 5 byte record, of course,
things won't work so well.

The approach we adopted was to create a 'memory stream', which is a
chunk of memory that you can treat as a stream (see for example
ColdFrame.Memory_Streams at [1]). With Ada2022, you should be able to
use Ada.Streams.Storage.Bounded[2].

Message'Write the record into the memory stream;
transmit the written contents as one datagram.

To read, create a memory stream large enough for the message you expect;
read a datagram into the memory stream;
Message'Read (Stream => the_memory_stream, Item => a_message);

You can use gnatbind's switch -xdr to "Use the target-independent XDR
protocol for stream oriented attributes instead of the default
implementation which is based on direct binary representations and is
therefore target-and endianness-dependent".

[1]
https://github.com/simonjwright/coldframe/blob/master/lib/coldframe-memory_streams.ads
[2] http://www.ada-auth.org/standards/22rm/html/RM-13-13-1.html#p25

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Mark Gardner]

On 31/12/2022 19:39, Simon Wright wrote:
> [...]
> 
> The approach we adopted was to create a 'memory stream', which is a
> chunk of memory that you can treat as a stream (see for example
> ColdFrame.Memory_Streams at [1]). With Ada2022, you should be able to
> use Ada.Streams.Storage.Bounded[2].
> 

Wait, so if I know what shape my data is, and use a memory_stream (like 
the one in the Big Online Book of Linux Ada Programming chapter 11 [1]), 
I'm fine using Stream, in conjunction with Get_Address? That's 
wonderful. Not at all frustrated that I just wasted approximately three 
working days looking for a solution to a problem that didn't exist.

> Message'Write the record into the memory stream;
> transmit the written contents as one datagram.

I'm guessing with Memory_Stream'Write(Socket_Stream, Buffer);?

> 
> To read, create a memory stream large enough for the message you expect;
> read a datagram into the memory stream;
> Message'Read (Stream => the_memory_stream, Item => a_message);

Does this second buffer need to be added? If the datagram arrives (UDP), 
shouldn't GNAT.Sockets.Stream() be able to handle it?
> 
> You can use gnatbind's switch -xdr to "Use the target-independent XDR
> protocol for stream oriented attributes instead of the default
> implementation which is based on direct binary representations and is
> therefore target-and endianness-dependent".

Oh fun, I didn't think of that aspect. Thanks! Would I have to pass it 
as a command line flag, or would there be some kind of pragma I could use?

Thanks for the help so far, and happy new year!

[1] http://www.pegasoft.ca/resources/boblap/11.html#11.12

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Dmitry A. Kazakov]

On 2022-12-31 20:36, Mark Gardner wrote:
> On 31/12/2022 19:39, Simon Wright wrote:

>> Message'Write the record into the memory stream;
>> transmit the written contents as one datagram.
> 
> I'm guessing with Memory_Stream'Write(Socket_Stream, Buffer);?

No, you create a memory stream object. Then you write your packet into it:

    My_Message'Write (My_Memory_Stream'Access);

Once written you use the accumulated stream contents to write it into 
the socket. An implementation of a memory-resident stream is very 
simple. E.g. see:

http://www.dmitry-kazakov.de/ada/strings_edit.htm#Strings_Edit.Streams

My advise would be not to do this. It is wasting resources and 
complicated being indirect when 'Write and 'Read are compiler-generated. 
If you implement 'Write and 'Read yourself, then why not calling these 
implementations directly. It just does not make sense to me. I always 
wonder why people always overdesign communication stuff.

Build messages directly in a Stream_Element_Array. Use 
system-independent ways to encode packet data. E.g. chained codes for 
integers. Mantissa + exponent for real numbers. If you have Booleans and 
enumerations it is a good idea to pack them into one or two octets to 
shorten the packets. All this is very straightforward and easy to implement.

You can also consider using some standard data representation format, 
e.g. ASN.1. An Ada ASN.1 implementation is here:

    http://www.dmitry-kazakov.de/ada/components.htm#ASN.1

You describe your message in ASN.1 as an Ada tagged type derived from 
building blocks. Then you can encode and decode it directly from 
Stream_Element_Array. I would not recommend that either. ASN.1 is quite 
overblown.

Happy New Year!

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Simon Wright]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> My advise would be not to do this. It is wasting resources and
> complicated being indirect when 'Write and 'Read are
> compiler-generated. If you implement 'Write and 'Read yourself, then
> why not calling these implementations directly. It just does not make
> sense to me. I always wonder why people always overdesign
> communication stuff.
>
> Build messages directly in a Stream_Element_Array. Use
> system-independent ways to encode packet data. E.g. chained codes for
> integers. Mantissa + exponent for real numbers. If you have Booleans
> and enumerations it is a good idea to pack them into one or two octets
> to shorten the packets. All this is very straightforward and easy to
> implement.

It has to depend on the design criteria.

If you need something now, and it's not performance critical, and you
have control over both ends of the channel, why not go for a
low-brain-power solution?

On the other hand, when faced with e.g. SNTP, why not use Ada's
facilities (e.g. [1]) to describe the network packet and use unchecked
conversion to convert to/from the corresponding stream element array to
be sent/received?

I'd have thought that building messages directly in a stream element
array would be the least desirable way to do it.

[1] https://sourceforge.net/p/coldframe/adasntp/code/ci/default/tree/SNTP.impl/sntp_support.ads

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Dmitry A. Kazakov]

On 2023-01-01 00:41, Simon Wright wrote:
> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
> 
>> My advise would be not to do this. It is wasting resources and
>> complicated being indirect when 'Write and 'Read are
>> compiler-generated. If you implement 'Write and 'Read yourself, then
>> why not calling these implementations directly. It just does not make
>> sense to me. I always wonder why people always overdesign
>> communication stuff.
>>
>> Build messages directly in a Stream_Element_Array. Use
>> system-independent ways to encode packet data. E.g. chained codes for
>> integers. Mantissa + exponent for real numbers. If you have Booleans
>> and enumerations it is a good idea to pack them into one or two octets
>> to shorten the packets. All this is very straightforward and easy to
>> implement.
> 
> It has to depend on the design criteria.
> 
> If you need something now, and it's not performance critical, and you
> have control over both ends of the channel, why not go for a
> low-brain-power solution?

Because it is still much simpler (and safer) than record layouts.

> On the other hand, when faced with e.g. SNTP, why not use Ada's
> facilities (e.g. [1]) to describe the network packet and use unchecked
> conversion to convert to/from the corresponding stream element array to
> be sent/received?

Well decoding NTP query from Stream_Element_Array takes 2 statements 
(extracting two big-endian 32-bit numbers). You can check the 
implementation here:

http://www.dmitry-kazakov.de/ada/components.htm#17.17

which is under-zero brained (:-)) compared to dealing with definition of 
record layout and bit orders you suggest.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Simon Wright]

"Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:

> Well decoding NTP query from Stream_Element_Array takes 2 statements
> (extracting two big-endian 32-bit numbers). You can check the
> implementation here:
>
> http://www.dmitry-kazakov.de/ada/components.htm#17.17
>
> which is under-zero brained (:-)) compared to dealing with definition
> of record layout and bit orders you suggest.

It's obviously OK to use just the fields of interest in the received
packet.

But this strikes me as something I would have questioned at review:

      Data     : Stream_Element_Array (1..NTP_Packet_Size) :=
                    (  1  => 2#1110_0011#, -- LI, Version, Mode
                       2  => 0,            -- Stratum, or type of clock
                       3  => 0,            -- Polling Interval
                       4  => 16#EC#,       -- Peer Clock Precision
                       13 => 49,
                       14 => 16#4E#,
                       15 => 49,
                       16 => 52,
                       others => 0
                    );

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Niklas Holsti]

On 2023-01-01 1:41, Simon Wright wrote:
> "Dmitry A. Kazakov" <mailbox@dmitry-kazakov.de> writes:
> 
>> My advise would be not to do this. It is wasting resources and
>> complicated being indirect when 'Write and 'Read are
>> compiler-generated. If you implement 'Write and 'Read yourself, then
>> why not calling these implementations directly. It just does not make
>> sense to me. I always wonder why people always overdesign
>> communication stuff.
>>
>> Build messages directly in a Stream_Element_Array. Use
>> system-independent ways to encode packet data. E.g. chained codes for
>> integers. Mantissa + exponent for real numbers. If you have Booleans
>> and enumerations it is a good idea to pack them into one or two octets
>> to shorten the packets. All this is very straightforward and easy to
>> implement.
> 
> It has to depend on the design criteria.
> 
> If you need something now, and it's not performance critical, and you
> have control over both ends of the channel, why not go for a
> low-brain-power solution?
> 
> On the other hand, when faced with e.g. SNTP, why not use Ada's 
> facilities (e.g. [1]) to describe the network packet and use
> unchecked conversion to convert to/from the corresponding stream
> element array to be sent/received?
> 
> [1]
https://sourceforge.net/p/coldframe/adasntp/code/ci/default/tree/SNTP.impl/sntp_support.ads


One reason is that the Scalar_Storage_Order representation aspect is 
GNAT-specific, so that code is not really "Ada code", but "GNAT code". 
With most processors now being little-endian, while network traffic is 
still big-endian, byte-endianness conversion is usually necessary.

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Daniel Norte de Moraes]

Em Sat, 31 Dec 2022 14:11:55 +0200, Mark Gardner escreveu:
> At this point, I'm half-tempted to make my own binding, but as I've
> never done that sort of thing before, I thought I'd ask the wisdom of
> the Usenet if there is a way to convert a Stream_Element_Array into the
> exotic types of Unsigned_16 and String.

 You are obliged to use gnat-sockets ?
 No? then use https://github.com/danieagle/adare-net

 And Be Happy! (Enjoy!!)

 p.s.: see the client and server in udp in example diretory.
 p.s.: nowdays Adare_Net has a manual in pdf, too

 Thanks All,
 Dani.

Re: Sockets, Streams, and Element_Arrays: Much confusion

[Dmitry A. Kazakov]

On 2023-01-05 08:55, Daniel Norte de Moraes wrote:
> Em Sat, 31 Dec 2022 14:11:55 +0200, Mark Gardner escreveu:
>> At this point, I'm half-tempted to make my own binding, but as I've
>> never done that sort of thing before, I thought I'd ask the wisdom of
>> the Usenet if there is a way to convert a Stream_Element_Array into the
>> exotic types of Unsigned_16 and String.
> 
>   You are obliged to use gnat-sockets ?
>   No? then use https://github.com/danieagle/adare-net

The OP's question was about Stream_Element_Array being used for payload. 
In that regard there is no difference between GNAT.Sockets and the 
library you suggest.

P.S. I am not sure if the library supports socket select and setting all 
necessary socket flags (e.g. NODELAY).

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de