Wireshark · Wireshark-dev: Re: [Wireshark-dev] Lua 5.3

[Peter Wu <peter@xxxxxxxxxxxxx>]

On Fri, Aug 19, 2016 at 05:38:57PM +0100, João Valverde wrote:
> 
> 
> On 08/19/2016 04:05 PM, João Valverde wrote:
> > 
> > 
> > On 08/19/2016 03:56 PM, João Valverde wrote:
> > > 
> > > 
> > > On 08/19/2016 02:54 PM, Peter Wu wrote:
> > > > On Mon, Aug 08, 2016 at 09:17:35PM +0100, João Valverde wrote:
> > > > > 
> > > > > 
> > > > > On 08/08/2016 05:58 PM, Pascal Quantin wrote:
> > > > > > Hi João,
> > > > > > 
> > > > > > 2016-08-08 18:52 GMT+02:00 João Valverde
> > > > > > <joao.valverde@xxxxxxxxxxxxxxxxxx
> > > > > > <mailto:joao.valverde@xxxxxxxxxxxxxxxxxx>>:
> > > > > > 
> > > > > >     Is moving to Lua 5.3 something we should look into?
> > > > > > 
> > > > > >     The 64 bit integer support seems really promising.
> > > > > > 
> > > > > > 
> > > > > > Hariel explained us that Lua 5.3 was a completely different
> > > > > > language (a
> > > > > > bit like what you have between Python 2.X and 3.X). You can find much
> > > > > > more info (from people knowing what they are taling about - so not me
> > > > > > ;)) in bug 10881.
> > > > > > 
> > > > > > Pascal.
> > > > > 
> > > > > 
> > > > > Thanks for that Pascal. The only sane way to approach the issue IMHO
> > > > > is to
> > > > > accept that this may and probably will break backward compatibility
> > > > > (not
> > > > > even think about supporting 5.1 or 5.2) and just consider whether
> > > > > that break
> > > > > is justified (hint: it is).
> > > > 
> > > > Why is it justified to break backwards compatibility and move from 5.2
> > > > to 5.3 without the ability to chose for 5.2? What is the killer feature
> > > > of 5.3 that makes it totally worth to possibly break older dissectors?
> > > > 
> > > > The disadvantage of C plugins is that it had to be recompiled for newer
> > > > versions. With a move from 5.2 to 5.3 and also removing GRegex and bitop
> > > > you make it quite likely to break Lua dissectors in some way.
> > > > 
> > > > I have once written a Lua library in C, interfacing with Libgcrypt for
> > > > which I studied the Lua manual. The API changes with 5.3 were not that
> > > > significant if I remember correctly (though you have to be careful with
> > > > providing a compatibility layer), but the ABI is certainly not
> > > > compatible.
> > > > 
> > > > In the recent proposed patches, you seem to have no issues with breaking
> > > > backwards compatibility. Have you developed Lua dissectors before?
> > > > Breaking things for the sake of "shiny, new, future" is not an
> > > > acceptable motivation, there must be something more appealing to justify
> > > > such breakage. Having 64-bit integer support, but taking away the bitop
> > > > library is a net loss without even considering the other factors.
> > > > 
> > > 
> > > Doesn't Lua 5.3 provide native bit operators? If so there is not net
> > > loss of functionality. That was my reasoning at least.
> > > 
> > > The language incompatibilities between 5.2 and 5.3 are minor. The
> > > wireshark API is exactly the same.
> > > 
> > > LPeg is more powerful and Lua-thonic than lrexlib, but there is a
> > > learning curve for that, no doubt. For anyone relying on lrexlib, it's a
> > > significant break. We can keep lrexlib, that's not a problem and it is
> > > orthogonal to the other changes.
> > > 
> > > As far as killer features go, besides the obvious, how about better
> > > UTF-8 support?
> > > 
> > > I don't have time for a more detailed answer right now but I'd like to
> > > say I think this change is entirely justified but I also completely
> > > understand disagreeing with that opinion.
> > 
> > I'm referring to the upgrade to Lua 5.3 here, i.e, breaking backward
> > compatibility, same as any other Lua script moving from 5.1/5.2 to 5.3.
> 
> Let me also ask, Peter, you're pushing back against building Lua from source
> and also pushing back against upgrading to 5.3.

I am against pulling the Lua source tree in the tree, reasons were
already given in https://code.wireshark.org/review/#/c/17172

I am not against adding Lua 5.3 compatibility, but would like to see
compatibility for existing dissectors. Keeping support for 5.2 (or even,
argh, 5.1) would allow distros/users to build with older versions if
they desire.

> I'm not seeing how that is a viable long-term option on Linux.

If 5.3 compatibility is added it should be no problem. After looking at
the actual changes (https://www.lua.org/manual/5.3/readme.html#changes
and also the linked incompatibilities), I see the following changes to
the Lua code:

 - 64-bit integer support. Nice!
 - Bitwise operators. Cool, but we would have to add a compatibility
   library to keep old dissectors working.
 - UTF-8 "support". This just adds some helper functions that can
   calculate lengths, converts between codepoints and chars and adds a
   %U format specifier. It adds no new data type so won't really affect
   the WSLUA API.

So you are right that there are not many changes in that area. On the
C library side we have also have no really window shattering changes
other than the integer stuff. Of course you would have to recompile any
C libraries, but I realize that it is probably a special case.

Overall I am positive with *adding* compatibility for 5.3, but at the
same time we should proceed carefully in order not to break existing
dissectors.

Things like replacing the regex library is not related to 5.3
compatibility and should be out of scope for above discussion.  Since it
breaks older dissectors in a more severe way, it would need some
convincing arguments and alternatives though.
-- 
Kind regards,
Peter Wu
https://lekensteyn.nl