ANN: Ada Resource Embedder for C, Ada and Go

ANN: Ada Resource Embedder for C, Ada and Go

[Stephane Carrez]

Hi all,

I created a new tool to allow embedding any file in an Ada, C or Go binary.
While embedding files, you can apply some transformations such as
running a Javascript minifier (closure), compressing the file, encrypting it, ...
The tool generates Ada, C or Go files that you compile with your program.
In its simplest form, you can access the embedded content as a:

type Content_Access is access constant Ada.Streams.Stream_Element_Array;

So the generated code only depends on Ada.Streams.

There are many modes that are explained in the documentation.
For an overview, have a look at:

https://blog.vacs.fr/vacs/blogs/post.html?post=2021/06/11/Advanced-Resource-Embedder

And don't hesitate to fork, hack, and submit pull requests to:

https://github.com/stcarrez/resource-embedder

Well, for me it was a fun project :-)

Stephane

Ps: Go has a `go:embed` but It was fun to write the Go generator :-)

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Stéphane Rivière]

Hi Stephane,

> I created a new tool to allow embedding any file in an Ada, C or Go binary.

This is typically what I needed to improve my current AIDE v2 project
(Ada Instant Development Environment - source GNAT CE 2019 2020 2021 -
target Debian / Ubuntu with subtarget station (with GNATStudio, HAC,
libs, debug aware RTS, and goodies) or server (bare minimal).

Many thanks !!!

-- 
Be Seeing You
Number Six

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Doctor Who]

On Fri, 11 Jun 2021 15:51:52 +0200, Stéphane Rivière
<stef@genesix.org> wrote:

>Hi Stephane,
>
>> I created a new tool to allow embedding any file in an Ada, C or Go binary.
>
>This is typically what I needed to improve my current AIDE v2 project
>(Ada Instant Development Environment - source GNAT CE 2019 2020 2021 -
>target Debian / Ubuntu with subtarget station (with GNATStudio, HAC,
>libs, debug aware RTS, and goodies) or server (bare minimal).
>
>Many thanks !!!


Your projects are very interesting, can I have the link for AIDE ?

Thank you.

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Dmitry A. Kazakov]

On 2021-06-11 15:51, Stéphane Rivière wrote:
> Hi Stephane,
> 
>> I created a new tool to allow embedding any file in an Ada, C or Go binary.
> 
> This is typically what I needed to improve my current AIDE v2 project
> (Ada Instant Development Environment - source GNAT CE 2019 2020 2021 -
> target Debian / Ubuntu with subtarget station (with GNATStudio, HAC,
> libs, debug aware RTS, and goodies) or server (bare minimal).

I considered embedding into relocatable libraries similar to Windows' 
resource, e.g. for versioning plugins etc, but then decided to use an 
easier and more universal method.

I simply put an Ada constructing function into the library. The function 
is exported. The address returned by GetProcAddress or dlsym goes to 
Unchecked_Conversion to an access to subprogram, and here you are.

The obvious advantage of this method over embedding is that the object 
can be tagged of any derived type, which no embedding can do. And you 
can add whatever further initialization or checks you might need.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Stephane Carrez]

Hi Dimitry,

Le vendredi 11 juin 2021 à 18:19:18 UTC+2, Dmitry A. Kazakov a écrit :
> On 2021-06-11 15:51, Stéphane Rivière wrote: 
> > Hi Stephane, 
> > 
> >> I created a new tool to allow embedding any file in an Ada, C or Go binary. 
> > 
> > This is typically what I needed to improve my current AIDE v2 project 
> > (Ada Instant Development Environment - source GNAT CE 2019 2020 2021 - 
> > target Debian / Ubuntu with subtarget station (with GNATStudio, HAC, 
> > libs, debug aware RTS, and goodies) or server (bare minimal).
> I considered embedding into relocatable libraries similar to Windows' 
> resource, e.g. for versioning plugins etc, but then decided to use an 
> easier and more universal method. 
> 
> I simply put an Ada constructing function into the library. The function 
> is exported. The address returned by GetProcAddress or dlsym goes to 
> Unchecked_Conversion to an access to subprogram, and here you are. 
> 
> The obvious advantage of this method over embedding is that the object 
> can be tagged of any derived type, which no embedding can do. And you 
> can add whatever further initialization or checks you might need. 
> 

Can you elaborate a little?
I don't see what you put in your Ada constructing function.
I do see how you use it but not how and where you put the original content or file.

Let's suppose you have some documentation file 'config/example.conf'.
How would you integrate it in a binary with your solution?

Best regards,

Stephane

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Dmitry A. Kazakov]

On 2021-06-11 19:32, Stephane Carrez wrote:

> Le vendredi 11 juin 2021 à 18:19:18 UTC+2, Dmitry A. Kazakov a écrit :
>> On 2021-06-11 15:51, Stéphane Rivière wrote:
>>> Hi Stephane,
>>>
>>>> I created a new tool to allow embedding any file in an Ada, C or Go binary.
>>>
>>> This is typically what I needed to improve my current AIDE v2 project
>>> (Ada Instant Development Environment - source GNAT CE 2019 2020 2021 -
>>> target Debian / Ubuntu with subtarget station (with GNATStudio, HAC,
>>> libs, debug aware RTS, and goodies) or server (bare minimal).
>> I considered embedding into relocatable libraries similar to Windows'
>> resource, e.g. for versioning plugins etc, but then decided to use an
>> easier and more universal method.
>>
>> I simply put an Ada constructing function into the library. The function
>> is exported. The address returned by GetProcAddress or dlsym goes to
>> Unchecked_Conversion to an access to subprogram, and here you are.
>>
>> The obvious advantage of this method over embedding is that the object
>> can be tagged of any derived type, which no embedding can do. And you
>> can add whatever further initialization or checks you might need.
> 
> Can you elaborate a little?
> I don't see what you put in your Ada constructing function.
> I do see how you use it but not how and where you put the original content or file.

In my case I do not deal with files, it is always objects. In the 
simplest case it could be a record type like:

    type Library_Data is record
       Do_This  : not null access procedure;
       Get_That : not null access function return That'Class;
    end record;

That the library provides.

> Let's suppose you have some documentation file 'config/example.conf'.

The documentation would be an object ready for rendering. Say the 
documentation renderer is GTK text view widget. Then that would require 
a text buffer:

    type Library_Data is record
       Do_This       : not null access procedure;
       Get_That      : not null access function return That'Class;
       Documentation : not null Gtk_Text_Buffer;
    end record;

The caller will drop the text buffer Documentation into a Gtk_Text_View 
widget to show the documentation.

I usually use programmatically generated content. E.g. HTML 
documentation would be a set of subprograms with parameters that put a 
portion of HTML into a stream/string. I then decorate their output as 
necessary, e.g. <tr>...</tr> if that must be a table cell etc.

The point that documentation is almost never a static text, but has all 
sorts of parameters the provider of does not know in advance.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Stephane Carrez]

Thanks Dimitry for the clarification.

Different requirements leads to different solutions.

With ARE, I want to embed a Javascript file (such as jQuery), minify it with closure, compress it with gzip
and make it available as raw content so that the web server can service it without reading any file.
The content being accessible through either an Ada generated variable or through a function,
it is mapped in memory (we avoid an open, read, close system call plus the gzip stuff) and
the server only has to return the buffer content.

Best regards,

Stephane

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Dmitry A. Kazakov]

On 2021-06-11 20:44, Stephane Carrez wrote:
> Thanks Dimitry for the clarification.
> 
> Different requirements leads to different solutions.
> 
> With ARE, I want to embed a Javascript file (such as jQuery), minify it with closure, compress it with gzip
> and make it available as raw content so that the web server can service it without reading any file.
> The content being accessible through either an Ada generated variable or through a function,
> it is mapped in memory (we avoid an open, read, close system call plus the gzip stuff) and
> the server only has to return the buffer content.

Same here. In the case of an integrated HTTP server I simply store the 
HTTP pages as a set of string constants and functions generating dynamic 
portions of. The HTTP server uses no external files. Most pages never 
exist in any moment of time, because the server generates portions of 
them and sends away chunks as soon as possible. For this reason I do not 
compress any pages. It would waste too much resources on a small 
embedded system and does not really matter for a large PC.

-- 
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Stephane Carrez]

Le vendredi 11 juin 2021 à 21:53:09 UTC+2, Dmitry A. Kazakov a écrit :
> On 2021-06-11 20:44, Stephane Carrez wrote: 
> > Thanks Dimitry for the clarification. 
> > 
> > Different requirements leads to different solutions. 
> > 
> > With ARE, I want to embed a Javascript file (such as jQuery), minify it with closure, compress it with gzip 
> > and make it available as raw content so that the web server can service it without reading any file. 
> > The content being accessible through either an Ada generated variable or through a function, 
> > it is mapped in memory (we avoid an open, read, close system call plus the gzip stuff) and 
> > the server only has to return the buffer content.
> Same here. In the case of an integrated HTTP server I simply store the 
> HTTP pages as a set of string constants and functions generating dynamic 
> portions of. The HTTP server uses no external files. Most pages never 
> exist in any moment of time, because the server generates portions of 
> them and sends away chunks as soon as possible. For this reason I do not 
> compress any pages. It would waste too much resources on a small 
> embedded system and does not really matter for a large PC.

Not the same. ARE takes a static content and converts it in an Ada source that you compile.

Compression can help even on embedded systems because it reduces the size of data transfer.
A jquery-3.4.1.js file is arround 273K. Minified by closure it is reduced to 89K.
If you also compress it, it reduces to 31K.

On slow networks these size reduction make a difference.

There is no waste of resource because the compression is made during the build process not at runtime.
I would say the opposite: what you get is smaller in size.

Best regards,

Stephane

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Stéphane Rivière]

> With ARE, I want to embed a Javascript file (such as jQuery), minify it with closure, compress it with gzip
> and make it available as raw content so that the web server can service it without reading any file.

Really cool ! :)

-- 
Be Seeing YouNumber Six

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Doctor Who]

On Fri, 11 Jun 2021 17:41:31 +0200, Doctor Who <doc@tardis.org> wrote:

>On Fri, 11 Jun 2021 15:51:52 +0200, Stéphane Rivière
><stef@genesix.org> wrote:
>
>>Hi Stephane,
>>
>>> I created a new tool to allow embedding any file in an Ada, C or Go binary.
>>
>>This is typically what I needed to improve my current AIDE v2 project
>>(Ada Instant Development Environment - source GNAT CE 2019 2020 2021 -
>>target Debian / Ubuntu with subtarget station (with GNATStudio, HAC,
>>libs, debug aware RTS, and goodies) or server (bare minimal).
>>
>>Many thanks !!!
>
>
>Your projects are very interesting, can I have the link for AIDE ?
>
>Thank you.


Thank you again , you have been very quick to reply........

https://stef.genesix.org/aide/aide.html

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Stéphane Rivière]

Hi Doctor,

> Your projects are very interesting, can I have the link for AIDE ?

Github is coming - watch https://github.com/sowebio in july
Alpha is here https://stef.genesix.org/pub/ada/aide

Don't read v2.14 but v0.14 :) The v1 was a completely different beast. I
just keep the numbering...

aide - binary
aide-2.14.zip - sources
v20-0.3.zip is the GP library used for AIDE

All this stuff is KISS¹ but very usable :)

sow - AIDE for Debian & Ubuntu Manual v34.pdf - read issues and to do
list at the end. You should *wait for v2.15* release to test it, even
the 'big view' already works. I have little bugs to fix for a 'full
flown experience' :)

sow - v20 Ada Library User Manual v28.pdf - API ref (a must read to
understand AIDE code) and more (methodology101 for kids and new coders).

HAC Ada Compiler User Manual v82.pdf manual for HAC (HAC is included in
AIDE. It's a very capable Ada subset interpreter, replacing Bash on all
our servers cluster) HAC is a Gautier de Montmollin project (refs in
doc). HAC is seven times faster than Bash, more productive and strongly
typed !

¹ According to the "Keep It Simple Stupid" theory

-- 
Be Seeing You
Number Six

Re: ANN: Ada Resource Embedder for C, Ada and Go

[Doctor Who]

On Sat, 12 Jun 2021 14:03:41 +0200, Stéphane Rivière
<stef@genesix.org> wrote:

>Hi Doctor,
>
>> Your projects are very interesting, can I have the link for AIDE ?
>
>Github is coming - watch https://github.com/sowebio in july
>Alpha is here https://stef.genesix.org/pub/ada/aide
>
>Don't read v2.14 but v0.14 :) The v1 was a completely different beast. I
>just keep the numbering...
>
>aide - binary
>aide-2.14.zip - sources
>v20-0.3.zip is the GP library used for AIDE
>
>All this stuff is KISS¹ but very usable :)
>
>sow - AIDE for Debian & Ubuntu Manual v34.pdf - read issues and to do
>list at the end. You should *wait for v2.15* release to test it, even
>the 'big view' already works. I have little bugs to fix for a 'full
>flown experience' :)
>
>sow - v20 Ada Library User Manual v28.pdf - API ref (a must read to
>understand AIDE code) and more (methodology101 for kids and new coders).
>
>HAC Ada Compiler User Manual v82.pdf manual for HAC (HAC is included in
>AIDE. It's a very capable Ada subset interpreter, replacing Bash on all
>our servers cluster) HAC is a Gautier de Montmollin project (refs in
>doc). HAC is seven times faster than Bash, more productive and strongly
>typed !
>
>¹ According to the "Keep It Simple Stupid" theory


Thank you very much.