EigenD : upgrade to python 3

Probably only for the simple stuff, but 2to3 looks promising, it comes with quite a number of automated “fixers” that generate applyable patches: 2to3 - Automated Python 2 to 3 code translation — Python 3.10.5 documentation
E.g. it does that bracket thing:

--- agent.py    (original)
+++ agent.py    (refactored)
@@ -214,7 +214,7 @@
         return async.success()

     def set_enclosure(self,enclosure):
-        print 'enclosure set to',enclosure
+        print('enclosure set to',enclosure)

Or iterators:

def quit(self):
-        for v in self.__subsystems.itervalues():
+        for v in self.__subsystems.values():

Such stuff.

Can apparently also be extended with custom fixers. So if a new pattern is found that should be replaced all over the place this might be a viable choice.

1 Like

yeah, Ive seen this, but I reticent to use automated conversions esp. with a code base Im not so familiar with… it only needs to get one thing wrong, and I could be chasing down the issue for months.

besides, as above, the python code is not really the big issue - there is not that much of it really, as its all written in a similar ‘style’, so really, updating that is not such a big deal… and its good opportunity to go thru the code anyway - running a tool like 2to3 teaches you nothing!
I also dont want to invest a huge amount of time in one tool, that’ll have a one time use… when its as quick to just do it manually :wink:
(of course I reserve the right to change my opinion, when im doing the 10th module :laughing: )

as I said, I think the bigger issue, is the generated code - and that needs to be gone thru very careful.

last thing I want… is to do all the work, then spend months trying to track down bugs.
frankly, there are no shortcuts for this - it has to be done thoughtfully and carefully.

Ok, makes sense.
Good thing with 2to3 is that it doesn’t automagically change things (if one doesn’t want) but can also just output it’s proposals as patch files. So when looking at a particular file manually it might not hurt to generate the matching patch file up front to see when 2to3 would propose - might make a few things easier. And can still be ignored / done in a different way.

1 Like

This is exciting! Thank you!!!

got a bit further today… pips are now compiling ok
so Py3 → C++ layer is there (but of course untested ! )

next thing to figure out is the modern equivalent of async/coroutines

    @async.coroutine('internal error')
    def __cancel_verb(self,subj,row,c):
        row = int(action.abstract_string(row))
        c = int(action.abstract_string(c)) if c else None

        if row not in self[5]:
            yield async.Coroutine.success()

up to 3.8, you could pretty much use @asyncio.coroutines aka decorators, but that is now depreciated,
Im not going to move to a depreciated form, and as its dead in 3.10
though, Im using 3.8 , as this is Apple’s base install… but dont want this break, when apple inevitably updates beyond 3.8.

so a bit of investigation of 3.10 co-routines and asyncio
https://docs.python.org/3/library/asyncio-task.html

I dont think EigenD is doing anything special, I assume its just using this to yield to the ‘slow thread’
but it’s obviously going to be important to get this right (*)


(*) Im assuming also as well as the modules having to have coroutinges and yielding, there will be some more code from async, that creates the task, that this is all running in… so that may be the tricky bit.

2 Likes

I think the annotation variant is already deprecated in Python 3.8, one should use

async def my_coroutine():
    await whatever

This should already work in 3.7+. The main difference in 3.10 is as you wrote that this and get_event_loop is now not only deprecated but unavailable, one should use the new constructs instead. The page you linked is a good reference (you can switch to 3.8 in the upper left list field and the relevant parts of the guide don’t change)

This could also be helpful:

So long story short I guess 3.8 should be good to go - what is deprecated there best shouldn’t be used anymore, what isn’t deprecated (asyncio related) doesn’t look as if it would go away anytime soon.

Edit: Found two exceptions from the statement above (same documents as above, scrolling down a little) for asyncio functions that got deprecated in versions after 3.8. So not changing the API anymore seems more an aim than a rule written in stone… (Directly waiting for coroutines was already discouraged since the introduction of tasks though, just not officially deprecated.)

Deprecated in 3.9:
The explicit passing of coroutine objects to asyncio.wait() has been deprecated and will be removed in version 3.11. (Contributed by Yury Selivanov and Kyle Stanley in bpo-34790.)

Deprecated in 3.10:
asyncio.get_event_loop() now emits a deprecation warning if there is no running event loop. In the future it will be an alias of get_running_loop(). asyncio functions which implicitly create Future or Task objects now emit a deprecation warning if there is no running event loop and no explicit loop argument is passed: ensure_future(), wrap_future(), gather(), shield(), as_completed() and constructors of Future, Task, StreamReader, StreamReaderProtocol. (Contributed by Serhiy Storchaka in bpo-39529.)

2 Likes

yeah, Ive got the internet … so no shortage of documentation/examples…
(and I dont have time to read a ton of articles, so Im doing my research pretty ‘dynamically’ :wink: )

really the python specific, or changes is not really the ‘effort’ it’s much more about how EigenD is using features… and how it fits together. e.g. I just found that EigenD has its own async python object… so I need to review with this in mind.

really, Im just posting there, to say what Im up to - in case someone wants to actually dig into the EigenD code itself - “roll up your sleeves”… if not, I’ll keep digging at it.

I don’t really care too much about python versions… I’ll try to aim for latest, just so I don’t have to review this again for a while - but I’ll see how it goes in practice, just getting to any Python 3 version will be a significant step forward - also by doing this, I’ll know more about how EigenD is using python so make further changes easier.

anyway for now, moved a bit further forward…
Im trying to get to a position where I can test a few of the changes that have been made, as my biggest ‘fear’ is changing a ton of code with no way to test.

I made good progress on this yesterday… as Ive got quite a lot of EigenD building now. Ive also made it so I can ‘disable’ building of plugin modules, so I can start to build/test one at a time.
(Im hoping I can use also use some of the more ‘esoteric’ eigenD command line tools, which I think EigenLabs used for developing testing)

I may also need to find a way to get a decent debugger attached to track down issues… but again, we will see.

overall, Im in the ‘deep end’ at the moment, whilst Ive got handle on whats changing, everything is a bit ‘up in the air’ … and so its really unclear what further changes are going to be necessary.
hopefully once I can test things, then it’ll be clearer how much more is necessary.

2 Likes

If there is an isolated task that would help you if I had a look at (optimally one where one doesn’t need to comprehend EigenD sources end-to-end) I can give it a try.
One interesting debugger I came across recently is this: https://pytrace.com/
It records all calls between a start and stop call and allows to step back and forth in the recording, looking at variable values etc. Might be helpful for understanding race conditions (in case they still occur with the instrumentation in place…)

yeah, unfortunately, thats not really possible at the moment, it’s pretty much ‘everything in the air’ at the moment… and see whats working/not. it’s going to be a while before this is ‘stable’ and so particular tasks are clear.


pytrace… does that work with embedded python and python objects that are using the Python C api?

thats the bit, that is a little unclear to me, am Im aiming to run EigenD in a C++ IDE/debugger, that has some kind of python support?
then we get into issue of build/compile supporting … EigenD is massive as a ‘build unit’ and its Scon implementation is pretty daunting - though I better understand it now, so not as bad as it was before.

Id really love to move this to CMake, then I could easily build/debug using things like CLion or VisualStudio, but thats a pretty tricky task at the moment.
though, now I kind of understand how the pip/yacc/lex stuff works perhaps its becoming a little more feasible… we will see, one day?

in many ways, perhaps this should be a priority… a decent/modern development environment would make EigenD much easier to support , develop and debug.
(it would help me a lot in some of the ‘darker corners’ of EigenD, like the lexical/database parts)

anyway for now, I think Im just going to have to stick to good ol’ command line tools, as I can’t really spend the hours/days trying new tools that may or may not work.
( I also need to keep the numbers of ‘changes’ down, so I dont get lost with everything being built on new sand)

1 Like

Yeah, pytrace should also work from embedded python, looking at debuggers and/or tracing mechanisms that optimally don’t have much impact on runtime behavior is something I can investigate independently (starting next week).

yeah, I’m not going to switch debuggers…
the thing is EigenD is mostly C++, not python - so I really don’t see a lot of use for a python tracing tool.
so, Im perfectly happy to trace that generate C code (from python) in a C++ debugger.

really, I wanna get rid of as much of this python dependancy as possible… its probably the worst thing about EigenD , so much extra ‘weight’ for very little benefit.
hence why I wanna get rid of Scon as well, and move to a build system without (yet more) python dependancies.

ok, so bit by bit , a little further :slight_smile:

so I now having things mostly compiling… modules and apps.

ive excluded the following modules… as I dont need for proof of concept… easy to do, once I know other modules are working… but need to focus now!

./plg_synth/ignore
./plg_tabulator/ignore
./plg_conductor/ignore
./plg_illuminator/ignore
./plg_language/ignore
./plg_ukbd/ignore
./plg_rig/ignore
./plg_convolver/ignore
./plg_simple/ignore
./plg_loop/ignore
./plg_strummer/ignore
./plg_sampler2/ignore
./plg_t3d/ignore
./plg_livepad/ignore
./plg_primitive/ignore
./plg_midi/ignore
./plg_keyboard/ignore
./plg_host/ignore
./plg_stk/ignore
./plg_midi_monitor/ignore
./plg_scale_illuminator/ignore
./plg_recorder/ignore
./plg_midi_device/ignore
./plg_arranger/ignore
./plg_finger/ignore
./plg_osc/ignore
./app_commander/ignore

so tools/app_eigend and some essential modules (e.g devices/audio) are compiling.
but that really is only step 1… and possibly the simplest of steps :wink:

most of the python changes , Im ‘reasonably’ confident with…
the main question marks are…

pi/async.py → pi/pisync.py

so, I noticed the other day, eigend was NOT using async, but its own decorator called async, and this appeared to creating some kind of clash…
so for now Ive rename pi/async.py to pi/pisync.py , and changed references where necessary.
but this is no guarantee it’ll work :slight_smile:
… but it compiles, and kind of looks right

piw ->TypeObject… tp_compare → tp_as_async

this one could be problematic… and will need a bit of digging I suspect… and its pretty much ‘commented out’ for now… so nothing is likely to work.

so, EigenD is defining its own python object types, and this includes iterating over some types.
the issue is python 2 used to so this with a simple compare function, returning -1,0,1
where python 3, uses iterators using PyAsyncMethods. which as the name implies not only is working as an iterator , but also handling ‘async’ functionality (eg. wait etc) - so to implement this, Im going to really need to understand now only the python3 functionality in this area , but also what EigenD is doing with its PiSync decorator… and is that even compatible, with this functionality.

unfortunately, thats going to probably take quite some experimenting.

next steps?
first steps is to get some of the basic command line tools working, this’ll basically verify the simplest functions of python 3 working from the EigenD build.

once thats working, I’ll try doing more complex things, like seeing if the pico ‘module’ will instantiate.

at some point, Im definitely going to hit the tp_as_async issue above… at that point, I think I’ll end up having to try to create a test app where I prove things like the iterators, and also the pisync functionality…
as I’ll need it running in a ‘simple test environment’ … as they’ll be no chance of seeing whats going on in a full EigenD type setup, which’ll throws huge numbers of errors :wink:
… thats the point I suspect progress will slow quite a bit!

edit: not sure if Ive mentioned but Im keeping (very rough) notes in the eigend repo
documentation/dev_notes_python.md

mostly, just so when things dont work… I can look over them to see if there is an obvious change that could cause issue… that said, they are not really complete, so shouldn’t be relied on!

2 Likes

creeping forward, as of today, eigend cmd line apps run ‘ok’, but do not function :wink:

seems like an initialisation problem - I think threads starting before either python is initialised, or more likely the threads… so this means Im now in the thick of the code of piagent , session, manager, glue and context… scary stuff, but likely where the main python 3 issues will be , and also be resolved.

the good news is, when I figure all this out, then its likely this is where the fun around pisync/coroutines will all be happening… so will likely help me get to grip with that too - or at least point me in the right direction.

anyway, just a matter of figuring out, why this bit has changed with python 3,
the code in this area seems ok, so it does seem to be more an ‘order of operation’ issue.
(it could even be this pisync is not working, and hence why their is a new ‘initialisation’ issue)

1 Like

a slight change in direction, Im going to target latest Python (currently 3.10.5), which users can install from python.org.
as Im going thru the pain now of upgrading python, seems wise to do as much of this effort now, rather than potentially more changes later.

also, whilst Apple is currently on 3.8, there is nothing to say they wont upgrade this in a future OS - so being ‘independent’ of OS updates is probably a good move!
finally, we will need Windows users to upgrade to Python 3 as well, so getting all users to install from python.org seems to be the more consistent approach.

1 Like

Seems more then reasonable/wise :wink:

1 Like

yeah, I don’t want to have to do all of these things on each OS upgrade :wink:

anyway, more good progress today…

the command line function are now starting to work, the initialisation issue is fixed.

picodump is actually communicating with the pico happily :slight_smile:
this is an important step, since means that basically c+±>python3->c++ translation is working.
it also enabled me to verify some of the python 3 changes that Ive made.

pezload kind works, but then hangs…
given it enumerates usb devices,
this is quite possibly the issue I mentioned with tp_as_sync a couple of days ago.
(in many ways I hope it is, as its a nice limited case, to try to fix and test)

so probably getting pezload to work is next thing,

Im also starting eigend from command line, and checking out python errors its throwing.

3 Likes

Yep, I think this makes a lot of sense. It’s more likely to be consistent across platforms than whatever Apple comes up with!

1 Like

indeed, the more I looked at Apple’s setup (see this post) , the more inconsistent it looks… and as we have seen, they are happy to mess about with it, as its not a ‘core part’ of their OS… I think it’s possible Apple provide it as a useful ‘system admin scripting tool’ (*), rather than something for ‘application development’


(*) Ive worked at places that used python exclusively for this, they needed more than shell scripting, but weren’t programmers, and python fitted really well, as its got some really nice ‘hooks’ into the OS.

another day, more progress… more working…

fixed the python issues with eigend app.
this included removing the bug reporting and latest version check, as not sure the Eigenlabs infra is there anymore, and obviously nothing going on with development there.

its now stalling, Id assume due to some kind of deadlock issue which I need to get into.

so, now I think, im getting close to the deeper issues…
it might take me a bit of time to resolve these, but will hopefully then ‘unlock’ eigend main functionality ()
(
) such then attention turns to individual modules

so its quite positive…


one thing this effort has shown me is just how reliant EigenD is on python :frowning:
[ rant ]
I previously thought EigenLabs had just used python as a bit of ‘glue’ for the C++ code.
but it’s now clear, that there is a lot more ‘functionality’ held in python code than Id previously thought.
a lot of the belcanto stuff is all python, which then hooks into native C++ objects.
the worst part is… this python code, is then littered with custom python C++ objects.

if you’re a C++ dev, you have to dig into the python code to see how it all hangs together.
if your a python dev, the code is alien due to all the python C++ interfaces which you have to grok (C++) to see what they are doing.
… and to make it worst still, you have to understand how the python/c++ interface works thru using pip which is a custom yacc/lex things.

really, tis a lesson in why not to use (so many) different technologies…
perhaps they are the right tool, but you are creating an environment that requires very specific/uncommon skills from your developers.
worst still, sure you can teach a C++ programmer some python to do it… but they are hardly going to be expert python programmers (and this can be seen in the EigenD code base :wink: )

[ / rant ] :wink:


anyway…

we will have to see how it goes… its definitely looking once the basics are in place, theres going to be a lot of code thats changed , so going to need careful testing etc.

definitely has given me some motivation to really think about next steps for MEC :wink:

but in the meantime, Im gonna keep plugging away at EigenD/Python 3… and just hope it works out ok in the end!

4 Likes

Yeah. From my PoV working in Python is fine. Working in C++ is also fine. Working in a combination of both, in a multi-threaded app, is painful. And I totally agree with your comments about how that reduces the talent pool (it’s a point I have to repeatedly make to our devs in my day job; “yes, I know it’s the new shiny thing; think carefully about adopting it!”).