[DVIPDFMx] 2010 engine plan

[Taco Hoekwater]

Hi,

Taco Hoekwater wrote:
> 
> For luatex itself, the two major plans for 0.60 are
>  2. consolidate binary lua module loading
> 
> Point 2 has unsolved issues with the use of the lua core, which
> really should become a dynamic, private dll to avoid problems between
> the lua core symbols use by such external modules  and luatex's
> embedded (patched) lua core symbols. I am not too thrilled about
> implementing this, so I have postponed it until now. (a longer, more
> technical email about this will follow in a few days)

Well, that took a few days longer than expected, but here is that
elaboration I promised.

The feature that loads C lua extension libraries uses dll-s/so-s which
are pre-build; they are assumed to be installed in the texmf tree
of the user's system. This part of the problem is solved already
by adding a new search type to kpathsea.

The still open problem is one of linking. Loading of precompiled
external lua libraries can be split between two cases: windows,
and other platforms.

The current situation on windows is as follows:

As windows users typically do not have a compiler handy, these files
are more likely then not downloaded from the web. Such dlls have had
to be pre-linked to a lua dll, and normally that is lua51.dll from
the standard lua distribution.

Luatex can load such dlls easily provided that lua51.dll exists on
the user's system as well, but these dlls will then use the functions
inside that lua51.dll and use the lua functions from that dll,
*not* the function inside luatex. This is a potential problem, because
the lua interpreter in luatex has local patches for luatex so that even
though it is still api-compatible, it is 100% implementation
compatible with stock lua.

So, I need to those external dlls to use the routines that are
compiled inside luatex, and, as far as I can make it out, the only
way to do that is to split of luatex's own lua interpreter into a
separate dll that is then used by both the external dll and the luatex
code proper. There are two (three) options:
1.  Somewhat sneakily name that newly created dll lua51.dll just like
    the 'normal' one. This will actually work, because the api is
    identical. It is possible to set up the search path such that
    luatex's dll is found before the system dll. On the downside,
    this will only work for as long as luatex's lua api really is
    100% compatible with the stock lua 5.1 dll
2.  Create a new name for the dll. This means that one can only load
    dlls that are compiled specifically for luatex (which may not be
    such a bad thing)
2a. Do point 2., but also create a proxy dll named lua51.dll that
    redirects to 2.


The current situation on unix-like systems is as follows:

The luatex binary is compiled with  -Wl,-E -ldl (small variations
depending on the platform). This means that the external .so file
is actually using the functions inside luatex automatically.

Therefore, there is no immediate need to take any action on these
platforms. However, the caveat at the end of point 1 above still
applies: this only works for as long as the luatex api is identical
to the stock lua 5.1 api.

There is just one small problematic thing and that is that -Wl,-E is
a bit too wild: it exports a great many more symbols than just the ones
from the lua api. This is fixable (at least for gcc) but could turn
into a configure-level nightmare.

So that is the current state. I welcome any comments and thoughts on
what will be the best way to proceed.

Best wishes,
Taco