Scons upgrade - small task for someone?

thetechnobear · June 21, 2022, 2:38pm

ok, so here’s a small task if someone is interested…

Task

upgrade EigenD version of Scons to latest Scon (4.3.0) - see, https://scons.org

Why

Its part of the Python 3 project for EigenD.

the basic build for scons is still working on python 3, so thats what Im using, and seems to have basic python 3.0 support…

however, Ive a feeling some of the packages (tools/Scons/packages) might struggle with later version of python 3.x, which are increasingly removing things that were just ‘depreciated’ in earlier 3.x versions.

it may also be that some of these tools in Scons, Ive just not used at this stage… and they may yet fail, e.g. when I come to create installers.

Scon current version also appears to be something like 2.5.1… so its quite a big jump to 4.x

What to do?

clone my EigenD repo, use python 3 branch.
make sure you can compile EigenD with python 3.

(i.e. make sure its working before you change anything ! )

THEN,

create a new branch off python3, to do your changes in !

THEN
understand how Scons has been packaged within EigenD
I think basically its everything under tools/Scons + tools/scons.py

EVERYTHING else is NOT scons
i.e. tools/*.py (excluding scons.py) and tools/pip_cmd are NOT part of scons…
so don’t touch them

really it should just be a matter of replacing old version with new version.
the only fun bit, is you likely can’t use any installer they provide… since its not a system install.
(and also it many need to be trimmed of ‘crap’ we don’t need… but Ive not checked this)

hopefully the above should not take too long !

now the tricky bit… (possibly)
THEN,
try to compile…
it may well be that scons has changed things, like how its tools are used etc.

really, at this point the key is determining HOW different it is, how much hassle its going to be… before committing lots of time to it!

side note: (a why I mentioned proceeding with caution on last step)

Its a little unclear to me, if scons as a project is still alive … its getting commits in GitHub, but its unclear if it has a real future , or if people have moved on from it.

on my side, I dont particularly like it, however EigenLabs have a lot of custom scons code (in tools/*.py) so it may be a reasonable amount of work to move away from it. more than I have time for at the moment. so it may be worth SOME effort to upgrade it…
but perhaps, at some point in the future we might move away from it…

so its a bit ‘in flux’ , but I lack time to do anything in this area at the moment!

NothanUmber · June 21, 2022, 8:19pm

Will give it a try. (might take some time, very sporadic free evenings atm.)

NothanUmber · June 21, 2022, 8:26pm

Just checked out EigenD with the python3 branch (under Linux) and tried “make”:

~/git/EigenD$ make
Traceback (most recent call last):
  File "/home/ferdinand/git/EigenD/tools/scons.py", line 59, in <module>
    import SCons.Script
  File "/home/ferdinand/git/EigenD/tools/packages/SCons/Script/__init__.py", line 377
    exec "%s = _SConscript.DefaultEnvironmentCall(%s)" % (name, repr(name))
         ^
SyntaxError: invalid syntax
make: *** [tools/Makefile:7: all] Fehler 1

Would you expect it to compile with the currently checked in Scons version (as I understand the text above) or would Scons have to be upgraded first anyways?
Edit: Ok, that Python 3.0 is probably meant literally… will try this (was on 3.9)

Edit2: Installing Python 3.0 is tricky as compiling this needs dev libraries that are long gone from usual repos. Got Python 3.5.3 installed (which is the oldest version that already works with OpenSSL 1.1.0) but get the same error. @thetechnobear Before I spend time trying to get Python 3.0 (from 2008) to run: Is that really the version you are working with and that you want me to install?

thetechnobear · June 22, 2022, 7:47am

What are you running on?

I’m using python 3.10.5 from python.org on Mac
…. On my python3 branch

Should work out of the box - Id would have told you if I believed there were additional steps
also worked on 3.8 when I was using Xcode python install.

I’ve not tried (nor plan to) any python prior to 3.10 now!

No idea about windows

I’ll try later on my intel mac , see if that’s ok.

edit: works ok on my intel mac too.

edit2: one thing, I mentioned in the main topic.
Im not compiling all pig_* modules at present, I want to (obviously) check some are working before converting all… to avoid compiling place a file called ‘ignore’ in these directories.
(for this purpose you can ignore all pig directories)

thetechnobear · June 22, 2022, 9:24am

ok, I now know what the issue is

while Im compiling everything with Python3, scons is actually running using python 2 !!!

the offending line is
tools/scons.py

#! /usr/bin/env python
#
# SCons - a Software Constructor

for me, the default python is set to python2
/Library/Frameworks/Python.framework/Versions/2.7/bin/python
(since this is what I use to run the released version of EigenD)

but this is just the python running scons, all my compiling etc is using python defined in our eigend setup.

indeed, if I change
tools/scons.py to /usr/bin/env python3
then I get the same error.

note: we really don’t want to edit scons.py in normal cases, since its part of scons… not an eigend file!
(that said, its a trivial file, at least in 2.5!)

k, so this actually is not that surprising, as I mentioned in the first post…
in the scons 2.5.1 it talks about python 3.0, so Im guessing its broken for later versions of python.

I just (incorrectly) thought it was ok with with 3.8/3.10 as it seemed to be working for me.
but appears my earlier assumptions that scons was going to die were correct!
all the more reason to move it to the later scons

ok, so what I would do is…
install python 2 from python.org - so that you can build, or at least see scons ‘working’
(this is just to verify your setup, and that we are seeing the same thing)

then you can try with scons 4…

as I said in OP, at first just verify… what ‘updating it means’,
e.g. can you just delete tools/packages/SCons , and replace it?, and replace scons.py

the reason I say this, is Im pretty familiar with the EigenD setup in tools/*.py
so if scons 4 needs some small changes, then I can make these if necessary.
(frankly, EigenD is not really using much of Scons, so Im hoping its fairly ‘compatible’)

NothanUmber · July 2, 2022, 7:58pm

Have moved to Windows now where I still have the original Eigenlabs runtime (with the Python2 executable to run the current version of Scons).
Would I still have to disable all the plugins? Currently I get:

tmp\obj\plg_arranger\src\arranger_native_python.obj (cl)
arranger_native_python.cpp
tmp\obj\plg_arranger\src\arranger_native_python.cpp: warning C4335: Mac file format detected: please convert the source file to either DOS or UNIX format
tmp\obj\plg_arranger\src\arranger_native_python.cpp(28): error C2664: 'const char *PyUnicode_AsUTF8AndSize(PyObject *,Py_ssize_t *)': cannot convert argument 2 from 'long *' to 'Py_ssize_t *'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(28): note: Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast
tmp\obj\plg_arranger\src\arranger_native_python.cpp(537): error C2664: 'long PyString_AsStringAndSize(PyObject *,char **,long *)': cannot convert argument 3 from '`anonymous-namespace'::pip_ssize_t *' to 'long *'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(537): note: Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast
tmp\obj\plg_arranger\src\arranger_native_python.cpp(553): error C2664: 'long PyString_AsStringAndSize(PyObject *,char **,long *)': cannot convert argument 3 from '`anonymous-namespace'::pip_ssize_t *' to 'long *'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(553): note: Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast
tmp\obj\plg_arranger\src\arranger_native_python.cpp(588): error C2664: 'long PyString_AsStringAndSize(PyObject *,char **,long *)': cannot convert argument 3 from '`anonymous-namespace'::pip_ssize_t *' to 'long *'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(588): note: Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast
tmp\obj\plg_arranger\src\arranger_native_python.cpp(1654): error C2059: syntax error: '.'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(1669): error C2143: syntax error: missing ';' before '}'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(3501): error C2059: syntax error: '.'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(3516): error C2143: syntax error: missing ';' before '}'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(3516): error C2059: syntax error: '}'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(3695): error C2059: syntax error: '}'
tmp\obj\plg_arranger\src\arranger_native_python.cpp(3695): error C2143: syntax error: missing ';' before '}'
scons: *** [tmp\obj\plg_arranger\src\arranger_native_python.obj] Error 2
scons: building terminated because of errors.

Edit: Changes I did:

set PI_PYTHON to my Python3 executable
for some reason the original popen call in generic_tools __get_python function always silently failed for me on Windows (status was always empty), so I changed it to this which seems to work:

    def __getpython(self):
        # print "environment PI_PYTHON" , self['PI_PYTHON']
        import subprocess
        process=subprocess.Popen([self['PI_PYTHON'],join(os.path.dirname(__file__),'detect.py')],stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        status, stderr = process.communicate()
        exit_code = process.wait()
        # print "status:"  , status, "stderr:", stderr, "exit code:", exit_code
        (exe,incpath,libpath,libs,linkextra,prefix) = [s.strip() for s in status.split(';')]
        # print((exe,incpath,libpath,libs,linkextra,prefix))

        self.Append(CPPPATH=[incpath])
        self.Append(LIBS=libs)
        self.Append(LIBPATH=libpath)
        self.Append(LINKFLAGS=linkextra)
        self.Replace(PI_PREFIX=prefix or None)

Edit 2: just verified that EigenD 2.2 still compiles on this environment (when leaving PI_PYTHON at Python2)

Edit 3: Ok, without all the plg_* packages I get further now (still with the old SCons) (python3.dll still has to be used as target instead of python26.dll in windows_tools.py. Also seems to need a 32 bit Python3 binary…). In piw_termparse.cpp I still run into the error I also get with the plg_* packages:

piw\src\piw_termparse.cpp(32): error C2664: 'const char *PyUnicode_AsUTF8AndSize(PyObject *,Py_ssize_t *)': cannot convert argument 2 from 'long *' to 'Py_ssize_t *'
piw\src\piw_termparse.cpp(32): note: Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast
scons: *** [tmp\obj\piw\src\piw_termparse.obj] Error 2
scons: building terminated because of errors.

Might be a 32 vs 64 bit issue?

Edit 4: when replacing the long * with Py_ssize_t * in the definition of PyString_AsStringAndSize in piw_termparse.cpp, lib_juce\epython.cpp, app_juceworkbench\epython.cpp etc. I get further again. (exclusive app_commander which is still Python2 style)
Now the problem (e.g. in lib_alpha2 or app_eigend2 is that the template seems to generate invalid cpp code:

tmp\obj\app_eigend2\eigend_native_python.obj (cl)
eigend_native_python.cpp
tmp\obj\app_eigend2\eigend_native_python.cpp: warning C4335: Mac file format detected: please convert the source file to either DOS or UNIX format
tmp\obj\app_eigend2\eigend_native_python.cpp(4757): error C2059: syntax error: '.'
tmp\obj\app_eigend2\eigend_native_python.cpp(4772): error C2143: syntax error: missing ';' before '}'
tmp\obj\app_eigend2\eigend_native_python.cpp(6177): error C2059: syntax error: '.'
tmp\obj\app_eigend2\eigend_native_python.cpp(6192): error C2143: syntax error: missing ';' before '}'
tmp\obj\app_eigend2\eigend_native_python.cpp(6192): error C2059: syntax error: '}'
tmp\obj\app_eigend2\eigend_native_python.cpp(6371): error C2059: syntax error: '}'
tmp\obj\app_eigend2\eigend_native_python.cpp(6371): error C2143: syntax error: missing ';' before '}'
scons: *** [tmp\obj\app_eigend2\eigend_native_python.obj] Error 2
scons: building terminated because of errors.

Edit 5: I think somehow pip_cmd\template must contain some Mac specific characters. But I cannot get rid of those - tried mac2unix+unix2dos, converting to Windows1252 encoding via VSC - still getting the “Mac file format detected” warning - and when removing the odd newlines in the generated cpp then the errors disappear… Hm… Enough for today.

thetechnobear · July 4, 2022, 8:34am

Got family here, so no time for this now …

Generally, I’m not going to be tackling windows until after macOS is fully working.

So, please don’t send PR for anything unless you already know it also works on macOS , since as above , I won’t accept until after macOS is fully working.

NothanUmber · July 4, 2022, 12:52pm

Ok, sure, have a good time!
Can switch to my old Macbook, which I can upgrade to Big Sur (which is the last OS version that Apple offers for it). I hope any changes that work there would also work with recent macOS and M1 Macs - but unless somebody with such machine would be willing to give it a try at some point there is of course no way to guarantee that.