EigenD : upgrade to python 3

Eigenharp
1

so, started this today.

Im going to use this topic to cover what Im up to:

you can follow my progress on my python3 branch on GitHub.

you can also see some (very!!!) rough notes Im taking whilst doing the move
… at the moment, its mainly about what language differences im finding between py2/py3
so that ‘across dev sessions’ I can remind myself of the errors likely to occur and the fix.

note: this does not include every change which you can obviously see using GitHub logs.

please note
Im certainly not a python expert, frankly, I dont even like the bl**dy thing, but needs must…
so if others more knowledge of python want to dive in and help please do :wink:

I’ll also post particular issues, I have in the hope some others might have an idea !

4 Likes
2

Happy you’ve pull up your T’s sleeves :wink:
Heat is everywhere…

My shameless call of awareness/help/participation… ( Eigenharp DevCon experts ) @gbevin, @GoneCaving , @NothanUmber , @dkah

2 Likes
3

yeah, its getting pretty heavy going for a non-python lover :wink:

look like Ive got the scons working on py3, and even the update lex/yacc/pip stuff to py3,
so build seems to be now working.

now Im onto looking at the native code spat out by the pip generator.
I need to make sure the generated (native) code from pip actually compiles…
… currently thats failing, due to changes in the embedded python api v3.

first discovery is it looks like PyInt and PyString no longer exist, so the template used to create code is invalid. - Ive temporarily trying to replace these with PyLong / PyBytes which seems like it should work.

however, looks like Im going to take a much better look at the whole template, as seems like quite alot of stuff might need ‘fixing’ in this area.

pretty scary stuff, as its python native code mixed in with yacc templates… and its been a very long time since I played with yacc and lex.

Im hoping though that I can just get a bit more familiar with the Python embedded api, and the current native code that is being spat out.
then it should become clear what issues where are… and so hopefully wont have to play too much with yacc. but that may be wishful thinking !

the ‘good news’ is once ive managed to get one EigenD module building, its likely to be straightforward for the others… given its all generated from the same lex/yacc stuff in pip files.

(though of course there is a big difference between compiling and not crashing :wink: )

also, it may put me in a better situation than last time I tried the Py3 port, as Ive a better idea whats going on with the whole pip->python->c++ layer, which was always a bit unclear to me!

anyway, had enough for today… will just keep chipping away at it when I get some spare moments.

1 Like
4

I’m not sure how much time I’m going to have in the near term, but have “watched” the repo for changes, and will clone the branch and keep track as time allows.

Python3 I think moved strings to unicode strings everywhere.

1 Like
5

Python3 I think moved strings to unicode strings everywhere.

yeah, Ive already tackled that in some areas…thats not really going to be a big issue.
you can use decode() etc to convert, though theres some cases where bytes/strings were used synonymously… so those need a bit of a tweak, e.g. conditional decoding.

generally, the python code is not too hard, as there are a lot of resources for P2->P3 , as its obviously been done by so many devs … so often googling will yield the answers.

and as above, Im taking notes, as I suspect most of the required changes will be the same across every EigenD module/scons file… so as I go thru, it gets easier.

so, I think the actually python code will be ok… I can get by on that I think.
the trickier bit is the native python api, as thats something Ive only ever fleetingly looked at,
and there is quite alot of this code generated from the pips.
(iirc, ,from last time… the python api is pretty simple, not too unlike other embedded language apis)

that said, of course, its C++ code, so im on more familiar territory :wink:

(I’m also hoping understand the pip layer/python integration better, will mean I can find out what caused the issue last time with the GIL causing deadlocks)

1 Like
6

Sounds good. I remember taking a bit of a look at the GIL deadlock around the same time, and while I’ve good grasp of python, I’d never had to work with multithreaded python&C++ code before.

7

Probably only for the simple stuff, but 2to3 looks promising, it comes with quite a number of automated “fixers” that generate applyable patches: 2to3 - Automated Python 2 to 3 code translation — Python 3.10.5 documentation
E.g. it does that bracket thing:

--- agent.py    (original)
+++ agent.py    (refactored)
@@ -214,7 +214,7 @@
         return async.success()

     def set_enclosure(self,enclosure):
-        print 'enclosure set to',enclosure
+        print('enclosure set to',enclosure)

Or iterators:

def quit(self):
-        for v in self.__subsystems.itervalues():
+        for v in self.__subsystems.values():

Such stuff.

Can apparently also be extended with custom fixers. So if a new pattern is found that should be replaced all over the place this might be a viable choice.

1 Like
8

yeah, Ive seen this, but I reticent to use automated conversions esp. with a code base Im not so familiar with… it only needs to get one thing wrong, and I could be chasing down the issue for months.

besides, as above, the python code is not really the big issue - there is not that much of it really, as its all written in a similar ‘style’, so really, updating that is not such a big deal… and its good opportunity to go thru the code anyway - running a tool like 2to3 teaches you nothing!
I also dont want to invest a huge amount of time in one tool, that’ll have a one time use… when its as quick to just do it manually :wink:
(of course I reserve the right to change my opinion, when im doing the 10th module :laughing: )

as I said, I think the bigger issue, is the generated code - and that needs to be gone thru very careful.

last thing I want… is to do all the work, then spend months trying to track down bugs.
frankly, there are no shortcuts for this - it has to be done thoughtfully and carefully.

9

Ok, makes sense.
Good thing with 2to3 is that it doesn’t automagically change things (if one doesn’t want) but can also just output it’s proposals as patch files. So when looking at a particular file manually it might not hurt to generate the matching patch file up front to see when 2to3 would propose - might make a few things easier. And can still be ignored / done in a different way.

1 Like
10

This is exciting! Thank you!!!

11

got a bit further today… pips are now compiling ok
so Py3 → C++ layer is there (but of course untested ! )

next thing to figure out is the modern equivalent of async/coroutines

    @async.coroutine('internal error')
    def __cancel_verb(self,subj,row,c):
        row = int(action.abstract_string(row))
        c = int(action.abstract_string(c)) if c else None

        if row not in self[5]:
            yield async.Coroutine.success()

up to 3.8, you could pretty much use @asyncio.coroutines aka decorators, but that is now depreciated,
Im not going to move to a depreciated form, and as its dead in 3.10
though, Im using 3.8 , as this is Apple’s base install… but dont want this break, when apple inevitably updates beyond 3.8.

so a bit of investigation of 3.10 co-routines and asyncio
https://docs.python.org/3/library/asyncio-task.html

I dont think EigenD is doing anything special, I assume its just using this to yield to the ‘slow thread’
but it’s obviously going to be important to get this right (*)

(*) Im assuming also as well as the modules having to have coroutinges and yielding, there will be some more code from async, that creates the task, that this is all running in… so that may be the tricky bit.

2 Likes
12

I think the annotation variant is already deprecated in Python 3.8, one should use

async def my_coroutine():
    await whatever

This should already work in 3.7+. The main difference in 3.10 is as you wrote that this and get_event_loop is now not only deprecated but unavailable, one should use the new constructs instead. The page you linked is a good reference (you can switch to 3.8 in the upper left list field and the relevant parts of the guide don’t change)

This could also be helpful:

So long story short I guess 3.8 should be good to go - what is deprecated there best shouldn’t be used anymore, what isn’t deprecated (asyncio related) doesn’t look as if it would go away anytime soon.

Edit: Found two exceptions from the statement above (same documents as above, scrolling down a little) for asyncio functions that got deprecated in versions after 3.8. So not changing the API anymore seems more an aim than a rule written in stone… (Directly waiting for coroutines was already discouraged since the introduction of tasks though, just not officially deprecated.)

Deprecated in 3.9:
The explicit passing of coroutine objects to asyncio.wait() has been deprecated and will be removed in version 3.11. (Contributed by Yury Selivanov and Kyle Stanley in bpo-34790.)

Deprecated in 3.10:
asyncio.get_event_loop() now emits a deprecation warning if there is no running event loop. In the future it will be an alias of get_running_loop(). asyncio functions which implicitly create Future or Task objects now emit a deprecation warning if there is no running event loop and no explicit loop argument is passed: ensure_future(), wrap_future(), gather(), shield(), as_completed() and constructors of Future, Task, StreamReader, StreamReaderProtocol. (Contributed by Serhiy Storchaka in bpo-39529.)

2 Likes
13

yeah, Ive got the internet … so no shortage of documentation/examples…
(and I dont have time to read a ton of articles, so Im doing my research pretty ‘dynamically’ :wink: )

really the python specific, or changes is not really the ‘effort’ it’s much more about how EigenD is using features… and how it fits together. e.g. I just found that EigenD has its own async python object… so I need to review with this in mind.

really, Im just posting there, to say what Im up to - in case someone wants to actually dig into the EigenD code itself - “roll up your sleeves”… if not, I’ll keep digging at it.

I don’t really care too much about python versions… I’ll try to aim for latest, just so I don’t have to review this again for a while - but I’ll see how it goes in practice, just getting to any Python 3 version will be a significant step forward - also by doing this, I’ll know more about how EigenD is using python so make further changes easier.

anyway for now, moved a bit further forward…
Im trying to get to a position where I can test a few of the changes that have been made, as my biggest ‘fear’ is changing a ton of code with no way to test.

I made good progress on this yesterday… as Ive got quite a lot of EigenD building now. Ive also made it so I can ‘disable’ building of plugin modules, so I can start to build/test one at a time.
(Im hoping I can use also use some of the more ‘esoteric’ eigenD command line tools, which I think EigenLabs used for developing testing)

I may also need to find a way to get a decent debugger attached to track down issues… but again, we will see.

overall, Im in the ‘deep end’ at the moment, whilst Ive got handle on whats changing, everything is a bit ‘up in the air’ … and so its really unclear what further changes are going to be necessary.
hopefully once I can test things, then it’ll be clearer how much more is necessary.

2 Likes
14

If there is an isolated task that would help you if I had a look at (optimally one where one doesn’t need to comprehend EigenD sources end-to-end) I can give it a try.
One interesting debugger I came across recently is this: https://pytrace.com/
It records all calls between a start and stop call and allows to step back and forth in the recording, looking at variable values etc. Might be helpful for understanding race conditions (in case they still occur with the instrumentation in place…)

15

yeah, unfortunately, thats not really possible at the moment, it’s pretty much ‘everything in the air’ at the moment… and see whats working/not. it’s going to be a while before this is ‘stable’ and so particular tasks are clear.

pytrace… does that work with embedded python and python objects that are using the Python C api?

thats the bit, that is a little unclear to me, am Im aiming to run EigenD in a C++ IDE/debugger, that has some kind of python support?
then we get into issue of build/compile supporting … EigenD is massive as a ‘build unit’ and its Scon implementation is pretty daunting - though I better understand it now, so not as bad as it was before.

Id really love to move this to CMake, then I could easily build/debug using things like CLion or VisualStudio, but thats a pretty tricky task at the moment.
though, now I kind of understand how the pip/yacc/lex stuff works perhaps its becoming a little more feasible… we will see, one day?

in many ways, perhaps this should be a priority… a decent/modern development environment would make EigenD much easier to support , develop and debug.
(it would help me a lot in some of the ‘darker corners’ of EigenD, like the lexical/database parts)

anyway for now, I think Im just going to have to stick to good ol’ command line tools, as I can’t really spend the hours/days trying new tools that may or may not work.
( I also need to keep the numbers of ‘changes’ down, so I dont get lost with everything being built on new sand)

1 Like
16

Yeah, pytrace should also work from embedded python, looking at debuggers and/or tracing mechanisms that optimally don’t have much impact on runtime behavior is something I can investigate independently (starting next week).

17

yeah, I’m not going to switch debuggers…
the thing is EigenD is mostly C++, not python - so I really don’t see a lot of use for a python tracing tool.
so, Im perfectly happy to trace that generate C code (from python) in a C++ debugger.

really, I wanna get rid of as much of this python dependancy as possible… its probably the worst thing about EigenD , so much extra ‘weight’ for very little benefit.
hence why I wanna get rid of Scon as well, and move to a build system without (yet more) python dependancies.

18

ok, so bit by bit , a little further :slight_smile:

so I now having things mostly compiling… modules and apps.

ive excluded the following modules… as I dont need for proof of concept… easy to do, once I know other modules are working… but need to focus now!

./plg_synth/ignore
./plg_tabulator/ignore
./plg_conductor/ignore
./plg_illuminator/ignore
./plg_language/ignore
./plg_ukbd/ignore
./plg_rig/ignore
./plg_convolver/ignore
./plg_simple/ignore
./plg_loop/ignore
./plg_strummer/ignore
./plg_sampler2/ignore
./plg_t3d/ignore
./plg_livepad/ignore
./plg_primitive/ignore
./plg_midi/ignore
./plg_keyboard/ignore
./plg_host/ignore
./plg_stk/ignore
./plg_midi_monitor/ignore
./plg_scale_illuminator/ignore
./plg_recorder/ignore
./plg_midi_device/ignore
./plg_arranger/ignore
./plg_finger/ignore
./plg_osc/ignore
./app_commander/ignore

so tools/app_eigend and some essential modules (e.g devices/audio) are compiling.
but that really is only step 1… and possibly the simplest of steps :wink:

most of the python changes , Im ‘reasonably’ confident with…
the main question marks are…

pi/async.py → pi/pisync.py

so, I noticed the other day, eigend was NOT using async, but its own decorator called async, and this appeared to creating some kind of clash…
so for now Ive rename pi/async.py to pi/pisync.py , and changed references where necessary.
but this is no guarantee it’ll work :slight_smile:
… but it compiles, and kind of looks right

piw ->TypeObject… tp_compare → tp_as_async

this one could be problematic… and will need a bit of digging I suspect… and its pretty much ‘commented out’ for now… so nothing is likely to work.

so, EigenD is defining its own python object types, and this includes iterating over some types.
the issue is python 2 used to so this with a simple compare function, returning -1,0,1
where python 3, uses iterators using PyAsyncMethods. which as the name implies not only is working as an iterator , but also handling ‘async’ functionality (eg. wait etc) - so to implement this, Im going to really need to understand now only the python3 functionality in this area , but also what EigenD is doing with its PiSync decorator… and is that even compatible, with this functionality.

unfortunately, thats going to probably take quite some experimenting.

next steps?
first steps is to get some of the basic command line tools working, this’ll basically verify the simplest functions of python 3 working from the EigenD build.

once thats working, I’ll try doing more complex things, like seeing if the pico ‘module’ will instantiate.

at some point, Im definitely going to hit the tp_as_async issue above… at that point, I think I’ll end up having to try to create a test app where I prove things like the iterators, and also the pisync functionality…
as I’ll need it running in a ‘simple test environment’ … as they’ll be no chance of seeing whats going on in a full EigenD type setup, which’ll throws huge numbers of errors :wink:
… thats the point I suspect progress will slow quite a bit!

edit: not sure if Ive mentioned but Im keeping (very rough) notes in the eigend repo
documentation/dev_notes_python.md

mostly, just so when things dont work… I can look over them to see if there is an obvious change that could cause issue… that said, they are not really complete, so shouldn’t be relied on!

2 Likes
19

creeping forward, as of today, eigend cmd line apps run ‘ok’, but do not function :wink:

seems like an initialisation problem - I think threads starting before either python is initialised, or more likely the threads… so this means Im now in the thick of the code of piagent , session, manager, glue and context… scary stuff, but likely where the main python 3 issues will be , and also be resolved.

the good news is, when I figure all this out, then its likely this is where the fun around pisync/coroutines will all be happening… so will likely help me get to grip with that too - or at least point me in the right direction.

anyway, just a matter of figuring out, why this bit has changed with python 3,
the code in this area seems ok, so it does seem to be more an ‘order of operation’ issue.
(it could even be this pisync is not working, and hence why their is a new ‘initialisation’ issue)

1 Like
20

a slight change in direction, Im going to target latest Python (currently 3.10.5), which users can install from python.org.
as Im going thru the pain now of upgrading python, seems wise to do as much of this effort now, rather than potentially more changes later.

also, whilst Apple is currently on 3.8, there is nothing to say they wont upgrade this in a future OS - so being ‘independent’ of OS updates is probably a good move!
finally, we will need Windows users to upgrade to Python 3 as well, so getting all users to install from python.org seems to be the more consistent approach.

1 Like