a topic to discuss python 2 and eigenD, and the way forward…
2 Likes
I’m wondering what your thoughts are on the Python 2 end of life on Jan 1, 2020. I’m wondering how this might effect older installations of EigenD and it’s tools on Mac and Windows. Specifically I’m wondering how it might impact users that still rely on Commander. I use Linux and Windows and am familiar with having multiple versions of python running on my machines, but new users, especially on OSX might end up with conflicts depending on how newer versions of the OS install packages.
I use a tau, and because there is no easy access to entering belcanto directly from the keyboard with classic and factory setups, not having a stable Commander is an issue for me. I can run scripts from command line, but this is a bit of a hack, and not something some folk will be happy with, especially I suspect OSX users.
I personally think it might be time to look at rewriting commander in a more up to date python, but not sure what the implications of that are for the other tools that may be relying on older python interpreters. Haven’t looked too deeply into it, but I don’t see any major hurdles at the outset. Be interested to see what someone who’s taken a much deeper dive than I has to say about this.
Any comment?
1 Like
ok, so this is worthy of its own discussion … partly so I can also ‘document’ what Ive already considered and found out… and detail some of the issues and idas so far.
disclaimer:
I should preface this with one difficulty I face is that I am no ‘python expert’ ,
I know how to code a bit of python, but as will become evident eigenD integration is a bit more involved than that - so digging in EigenD has been extending my knowledge a lot in this area.
(also I did the last ‘dig’ a quite a few months back, so some of this is a bit ‘rusty’)
Python 2 ‘end of life’ , yes im aware of it, but really I’ve no idea how quickly the end-of-life would materially affect EigenD, we are already using an older version of 2.7.
I’d say its not a short term issue, more mid-term (2-5 years?)
some fact-iods to scope the ‘problem’:
EigenD uses python in two areas:
- tying eigenD modules to the architecture
- as a UI for Eigen Commander, Eigen Browser
EigenD module integration
This is done with the low level python C api, it is critical to EigenD - and would be a massive amount of work to remove (i.e. nigh impossible!)
the good news is, it does not depend upon any other python modules - its is very ‘vanilla’.
I have made attempts to replace this with Python 3, and had some success BUT had some issues with the threading model causing a block - this is not entirely surprising, as python and multithreading is generally problematic - however, I believe mid-term i can resolve this - but its going to take a reasonable amount of playing/hacking to do
Python UI (commander/browser)
we are already facing issues with these on macOS, and they will not work with the 64 bit version of python. the reason for this, is they are dependent on a UI python module (or two?) which is no longer maintained (?).
these UI are not ‘plain’ python, they also rely upon EigenD libs for communication (discussed a bit more below)
the plan has been to rewrite these python UI in C++ using Juce (as is the case with Workbench) - this would not be a one for one rewrite - rather looking at the requirements, and deciding what the app should look like. (e.g. it might be some functionality might be migrated into Workbench?)
once we achieve this, we dramatically reduce EigenD dependency on Python which I believe is very important for long term support.
unfortunately, the re-write is not just a bit of C++ code (otherwise id have done it a long time ago!)
the issue is, these applications do not communicate with EigenD via a simple binary protocol (e.g. osc, sockets) , rather than have to interface via a python interface to talk to the eigenD agents within EigenD.
this requires a much deeper understanding of EigenD architecture, and also the low level Python interface.
so, the first step was to write a very simple C++ app that could communicate with EigenD via this architecture. this was ‘no mean feat’ but I did actually manage to get working. so in principle I have an idea how to get this working …
Im really not worried about the C++ / Juce code and building the UI, that is an area Im very comfortable with.
Obviously, there is still quite a lot of work to go from ‘proof of concept’ to building one or two full applications - and also it is likely i would (at least initially) scale back the functionality to try to reduce this workload.
so, the plan…
I don’t believe the end of life will cause us any short term issues ( probably be fine for a couple of years?)
but we have two pieces of work…
the most important is to get EigenD to Python 3, so that the main application continues to work.
Its hard to say how much effort this will take me - my best guess is , it’ll be one of those tasks where I’ll spend days banging my head against a brick wall, and then suddenly it will ‘all fall into place’ 
Id then turn to the UI, basically, I will attempt to sort out the architectural parts, and then turn to writing the actually UI. (my general strategy is always to tackle the ‘unknowns first’)
I don’t think this is too problematic…but its quite a lot of work.
(there is no point starting this till Ive moved over to python 3, so there may be some gotchas in that too)
Collaboration/spreading the workload
Id love it if others want to help out in these area - esp. if they have deep python experience.
unfortunately the lower level stuff needs a pretty good understanding of the EigenD codebase, so I guess I will tackle that.
however, once I get the main ‘framework’ for the UI apps in place, hopefully there would be opportunities for others with C++ skills to get involved.
Other options:
I really appreciate EigenD and its functionality, and for many users its suits their needs perfectly,
so Im committed to keeping it working.
however, what I personally have found, and noticed from other longer term users, is that many would like a much lighter way to interface Eigenharps to their other instruments (software and hardware). They are used to using midi/osc and dont really need the Eigenharp to provide much more than this.
this was why MEC was born,
the primary goal of MEC, is to provide an application and framework, that can take many forms and can live on many different types of hardware.
It is based on pure C++ and cmake (a modern build system) so is not only portable but the longevity of the toolchain is guaranteed, and being much simpler (focused) - it can be developed/maintained by developers which a far reduced skillset (compared to eigenD).
a second part of the MEC ‘vision’ is running on smaller/cheaper devices.
the idea here is simple, musicians could afford to dedicate something like a raspberry PI to being a Eigenharp to Midi/OSC/CV converter - the rPI essentially becomes a ‘dongle’ , so can be treated like hardware.
we could then have a standard disk image that eigenharp users could then use… plug n’ play.
we would never have to even update this if we didnt want to, it would work for as long as the rPI lives
this is very much do-able today - Ive already got MEC working on a number of platforms (rPI/organelle/bela),
we are very fortunately that this second part is ‘coming towards us’ very rapidly, technology in this area is still improving … e.g. the new raspberry PI 4 is considerably more powerful than the PI3, and we are seeing adoption of this technology in many areas (e.g. Organelle/Norns using raspberryPI and Bela using Beaglebone) … also the technology is getting smaller (PI Zero)
at the last 3eee event in Portugal, I was able to use the Eigenharp without a laptop on an Organelle-m , both as standalone, and also for sending CV and midi to a modular - so its possible today 
my hope is in 2020, I’ll have MEC covering the use-cases required for many more users, and also to perhaps have a ‘standardised’ distribution that is cheap enough that anyone can use it,
at that point Eigenharp uses will be able to choose to use Eigenharps ‘standalone’ or to use EigenD on a laptop (etc) when they require more features etc.
side note:
for completeness (but unrelated to this topic really) , I should talk about Stage.
so Stage is written in C++/Juce and it actually talks to EigenD via OSC,
however, the OSC messages implemented only cover the limited functionality of Stage - it does not cover the full functionality of EigenD - so could not be used for an Eigen Commander or Browser replacement.
(the reason this approach was take for stage, is due to the iOS, I believe Eigenlabs would not have been able to ship the necessary libraries required to link Stage via the python runtime)
7 Likes
apologies for the ‘wall of text’ above…
but I wanted to get down some ideas, as much for myself to remind me of what is ‘to do’ and what has already been done - and now we have a forum, its much easier to spread information for discussion, and also that i can update as we ‘move forwards’
but to even it up a bit, a picture of my 3eee setup
Eigenharp Alpha -> Organelle MEC/Orac -> (sound and CV via ES-8) -> Eurorack
Soundplane is connected to Eurorack directly to Bela Salt to emit CV.
this also works for Eigenharp Pico, but for some reason I have issues doing the same with the Alpha which I need to investigate more.
2 Likes
I have made a promise to myself to familiarize myself with the MEC/EigenLite source code once my current projects are done (hopefully around Jan - Feb). If I am able to grasp what is what and feel that I might actually be able to contribute something of value I’ll check in with you if/when I have a general overview.
I mostly write web applications and C# stuff these days, so my C++ skills aren’t all that great. But I really like the concept of a light EigenD alternative and would like to help move that along if I’m able to.
4 Likes
the main focus of MEC is to be headless, to act like an embedded device - but I could imagine adding a web front end to it - to help with configuration etc - so your knowledge in this area could be very useful 
i realised i did not answer @amplogik directly …
tl;dr - this should not be an issue
background…
macOS can also have many version of Python installed, and EigenD hardcodes (annoyingly at times ;)) the path to the python to use. this is one reason why we have been able to run the 32 bit and 64 bit version of EigenD side by side for quite a while now.
furthermore, EigenD factory install ships with its own version of python (its the main contents of the ‘EigenD runtime’ package), which would also be an option going forward - but something Ive avoided for the 64bit version so far.
my experience is not that many people are writing belcanto directly, mostly adding to talkers in workbench. EigenCommander has not been working on the latest version of macOS for quite a while, and few people have noticed, much less commented on it 
what I did to get around this was to actually interface a text editor to the command line tools, which was ‘good enough’ for my purpose.
but thats just a ‘guess’, perhaps others are sorely missing EigenCommander.
the good news is a new EigenCommander (unlike EigenCommander) could be built just on the rpc interface (like the command line tools) if you just want to quickly enter belcanto.
without the ‘agent integration’ (which is the problematic bit) you’d lack a few features of the current commander - but I suspect it would be still very useful!
( this is one reason, I considered extending workbench to have belcanto/commander functionality)
all that said i would not want to re-integrate any python UI code back into the main EigenD repository. I think strategically with EigenD we need to reduce its toolchain and dependancies.
so, having some UIs written in python (EB/EC) and others in C++ (workbench/stage) adds confusion and dependancies - I suspect Eigenlabs came to realise this, and thats why Stage and Workbench (the newer elements) were written in C++.
honestly, Id love to take the whole python dependancy out of EigenD, but its just impractical for the agent side, not only is it a huge amount of work, but also require the every agent to have its python code re-written in C++ - so reducing down to vanilla python with no modules/ui dependancies is the best we can achieve.(imho)
3 Likes
I just ran into issues on Raspberry Pi and python 2.7. Everything in armf has moved to 3, and there are issues with the older packages on the deadsnakes ppa and raspberry pi 5. So I had to build python 2.7 from source, and wouldn’t you know it, it won’t interface with tk/tcl current libraries properly on arm, so… you can launch EigenD, but you get a blank window, except for the plugin window, oddly enough.
So effectively, if someone got a Pi today, model 4 or 5, and a current raspbian image, they’d be pretty much stuck. I’m trying to work out what’s going on with the deadsnakes ppa packages and see if I can’t do a workaround, but that’s unlikely to be a popular solution for someone looking for a grab and go pi install.
This led me to thinking about what it would take to update the dependencies to python3.
Are you familiar with 2to3 (2to3 — Automated Python 2 to 3 code translation — Python 3.12.4 documentation)? I’m wondering how far we’d be able to get with tools like that, or even GPT helping in some translations.
In the day job, I’ve had to update a ton of legacy code to 3, and found those tools invaluable, as otherwise the sheer volume of old code makes something like an update to python 3 a Sisyphean task.
The part that give me pause is wondering if automation like those tools could cope with the python interface to talk to the eigenD agents. That area is well beyond my knowledge, and I wouldn’t even be really confident I’d know how to test.
I’m wondering if you’d tried tools like that to do some of the heavy lifting and if you had any success?
2 Likes
ok, so Ive given this a go a couple of times.
each time Ive done it ‘manually’
yes, Ive very aware of the conversion tools. however, Im very wary of using them.
the main issue is, its a LOT of code to convert, and if you run this over them.
you have no idea what it has done, it might then ‘compile’ , but have introduced thousands of small bugs.
this would not be so problematic, if it were not for the fact, that I dont know python, nor the python code in EigenD that well… and the EigenD codebase is very complex.
destabilising the code base like this is asking for a lot of grief.
GPT is similar issue, Ive been using GitHub co-pilot a lot recently, and whilst it sometimes be nothing short amazing, it also can produced absolute hot garbage.
worst still when it ‘fails’ , as above, it’ll produce code that compiles, but just just not do whats intended.
put another way… when coding, its much easier to write/debug your own code, rather than trying to fix code that someone else has written (esp. if you cannot ask them about it. and the latter is what you get when you use code converters or AI to do it.
tl;dr; Id not be surprised it something like 2to3 could do a conversion. ( * )
however, given my lack of python skills, id not be able to verify what its done.
so yeah, whilst as an outsider its tempting to say…
“here’s a tool that’ll do it for you, you can just quick do it”
in practice, as the one who would actually have to do it - I think it’ll not be the case, its still a whole ton of work, potentially more than DIY, and much less ‘controlled’.
so really my ‘manual approach’ has been not just about converting the code.
but rather, by doing so, trying to gain some more knowledge about the way python is used.
so that IF/WHEN it doesn’t work, Ive at least some ideas of where to look.
but yeah, either way its a herculean task, theres so much code, so many changes to make.
the worst part is I cannot really convert it one small bit at a time, test, rinse/repeat.
I have to do a lot of it all at once, which means testing/debugging later will be a nightmare.
so yeah… this is why I keep doing a bit, and then after a few weeks putting it back on the shelf.
I will say last time, I did this, I was very much aware this would happen again, and this time around took more notes, and starting document/coming on the effort
see : EigenD : upgrade to python 3
so whilst, I shelved (again) after a few weeks, at least if I want to pick it up again, Ive some idea of where I got to … so could restart a bit easier.
however , one issue is, frankly, my heart is not in it.
I do like EigenD, in particular I like Workbench… and also I totally recognise its flexibility, and also it has a huge functional scope.
but I really do not like it at a technical level…
- reliance on Python and C++, makes it require a very unique skillset to work on.
- complex codebase, thats undocumented ,
the underlying c++ codebase is, imho, overly complex for what it actually does, and how people actually use eigend. - outdated build system, that is intrinsically linked to python
- use of python libraries (for old ui’s) that are no longer maintained.
- inconsistencies in app communication
workbench, iOS app, python apps all use different inter app comms to talk to eigend
so unfortunately, this creates a nasty combination.
EigenD needs a huge amount of work to get it working with newer technologies (not just python), but its not something I really enjoy working on , or have a lot of faith in going forward.
this is why I keep picking it up and then dropping it.
I have a desire to get it updated, and working again, but then, due to amount of work involved, after a while think - why am I doing this?
(which is how I end up creating things like MEC/Meta Morph as ‘modern’ alternatives)
ho hum, perhaps I should dive back into the python conversion again.
but theres so many more interesting projects I have on my plate/horizon.
( * ) ok, even this might not be the case.
EigenD makes extensive use of python → C++ mappings, and also has a very particular threading model. in fact one issue Ive had in early testing, was down to different threading models of python 2 and 3, which is causing EigenD to hang during startup.
2 Likes
Thanks so much for the feedback.
I hear you on the hot garbage that GPT and related technologies can produce.
As I was tinkering, trying to figure out why things were running but the UI wasn’t loading, I was wading through some of the codebase, and was gonna try sorting out the UI load problem on Raspbian. Thought it might be worth my time to experiment with 2to3 on this, but realized just how much I didn’t know about how the application hangs together. Wondered if anyone else had tried it, and if they ran right into problems outta the gate.
I’ve honestly thought that EigenD needed an entire rewrite at times, based on your MEC engine, and a more modern and streamlined workbench written entirely in C or something else that doesn’t rely on JIT compliling.
The functionality of workbench is amazing, but it’s always been slow and someone over complicated, but the dependency on Python is aging poorly.
But as you so rightly point out, this is a huge codebase, and next to undocumented, so it’s not a simple project at all.
It’s a brilliant platform for so many things, so it’s a shame to see it kind of fade away as tech moves on.
The instrument customization ( keygroups, layout, ability to address VST settings from the keybed…), hosting AU/VSTs, even creating a basic synth from scratch… just doesn’t exist out there anywhere else.
It would be amazing to see that same level of control for other gear, sending NPRN messages, audio routings with pipewire or something…
I need to take some time and really sit down with the MEC library and have a good think.
I’m wondering if I could take the concept of EigenD and maybe make something similar, including workbench and modular agents written in something a bit more current and optimized.
Of course, the day job gets in the way. LOL.
I’m gonna experiment a bit this week and see how bad my knowledge of EigenD’s internals really is, and see if I can’t see a path forward. I’ve a good knowledge of python and C, and porting apps to python3 for work.
My day job tho concentrates mostly on datasec, financials, systems and networks, so I am a bit outta my element here.
Guess there’s no better way to learn tho, than to dive in.
1 Like
yeah, my basic feeling is that EigenD does contain a lot of functionality that often is not required… since it can be achieved in many other ways.
e.g you don’t really need things like vst hosting or its ‘synths’
however, whilst writing things like MEC / Meta Morph ( * ) it did become apparent how much functionality is actually within EigenD that we kind of take for granted.
the main thing is the way you can switch functionality using various key combos … stuff like midi settings, scales, key layouts, splits.
so, yeah, a ‘re-write’ or replacement is the approach I keep toying with.
I’ll be honest, for my own purposes, things like MEC and Meta Morph work pretty well.
but its hard to make them cover everyone else’s use-cases - this goes back to how much functionality EigenD has … so whilst most don’t use much of its functionality, we tend to all use different bits of it.
anyway this is why EigenLite exists, its a C++ interface that means other applications can be written, be that encapsulating Eigenharps inside a VCV module, or a VST or standalone app.
its a tricky situation for sure.
( * ) obviously my recent preference is VCV rack, as in many ways its quite similar to EigenD, in that everything is done at audio rate, and using floating point (rather than midi).
also it has a vast number of modules so its a wonderful playground.
on reflection, I will admit, its pretty difficult in MetaMorph / vcv for us to use some advanced use-cases e.g. function/talkers of eigend are quite tricky to implement.
though, I do suspect, that perhaps the way forward is just to create more modules that handle these use-cases simply, rather than requiring users to figure out the vcv patching required to implement them.
(e.g. having an octave switcher, rather than trying to use function buttons along with ‘generic’ vcv modules)
so yeah… overall, I think your experiencing the same dilemma as I have…
try to ‘fix’ eigend … or replace it?
… and does replacing it , really work for ‘everyone’, who may not ‘agree’ with your choices etc.
2 Likes
Well, I do suspect that a lot of people have investments in VSTs, and with flatpack now the standard on linux, that’s only going to increase, especially with yabridge and carla and tons of people moving to Arturia VSTs. So I think that VST3 hosting or communication is likely to be a growing thing in the Linux or small systems world.
That being said, yeah, I have enjoyed tinkering with VCV rack, but haven’t gotten too deep into it. I have a lot of classic desktop and keyboard synths around here, so I tend to lean on midi / mpe integration, and increasingly OSC stuff.
What would be amazing is something like pipewire for mac and windows hosts. If you don’t know it, it’s something that works a bit like workbench in EigenD, but treats all your applications audio applications, hardware and the like as “agents” where you can route between them. It’s light on the control side tho.
Then it’s not a matter of replacing with my choices, instead a set of gui tools and plugins so that people can choose what features they want to load at the time.
I think an redesigned EigenD / Workbench would fit that bill. But it’s a mountain of work, and likely a time sink.
Well, I’ll play around this week and try to get a better handle on how EigenD hangs together, and see if I can’t brainstorm some way to build a workbench lite app.
I really should play more with VCV rack, and look at the APIs for it. It might be easier just to lean on that going forward.
Hmmm… wonder if anyone’s built for armf or arm64… Apple silicon is a species of arm64, so I assume so…
I mean, it might be the best foundation for a replacement. I just haven’t spent enough time with it. I should probably do that.
BTW, got EigenD working properly on a Raspberry Pi 5 with ubuntu 24 arm,
and raspbian bookworm. Bit of a mess. Neededed a virtualenv and a customized python build in that environment.
If anyone else is reading this, interested, I can document the steps. You’ll need the deadsnakes versions of the old python2.7 source, and to force some old debian buster library sources. It’s definitely a weekend project, and pretty rough if you don’t know cross compiling with GCC and linux pretty well.
1 Like
I add my voice on that opinion as well
2 Likes
Rack is easy to develop for, and there’s a great community for help.
Linux ARM isn’t currently officially supported, but a community member has been able to build and run for Raspberry Pi.
3 Likes
I just started building the toolchain for a VCV Pi build last night. I’d like to know more about any issues or workarounds that community member ran into. You wouldn’t happen to have any links to posts about their process or remember who it was, would you?
The sources seem to assume a 64 bit architecture, but for various reasons related to EigenD and Raspbian’s sound architecture, I need to target arm32/armf.
So I am anticipating a bit of friction in some modules. Be good to have a reference, and not try to reinvent the wheel from scratch.
1 Like
I don’t know about building vcv on the rPI, but I did recently build it for the Percussa SSP which is 32bit armf.
the main issue is vcv has a lot of dependancies, so you have to make sure this all line up.
the build process does this, however, it failed in for me in a few cases, which I had to fix by hand.
anyway, Id guess for the rPI someone has already published details, or has a fork that will just work.
all that said, rPI 4 is 64 bit, and Id definitely not go back to a rPI 3 for vcv
as for the MetaMorph modules, I highly doubt that this would work on a rPI with VCV rack, you asking too much in terms of performance.
VCV rack is not very efficient for these kind of devices due to the way it does most of its processing at a sample level NOT sample block, which makes quite a few (FPU) optimisation impossible.
really the only way Ive used the Eigenharp with a rPI, is using the PI to interface to the Eigenharp, and then spit out midi/osc to something else for a sound engine.
(ofc, that could be another rPI , eurorack or whatever)
overall, Im pretty conservative when using rPI for audio work, they are great, but they not that powerful compared to modern desktop/laptops and even tablets/phones - and often apps are not optimised for them.
2 Likes
I’d agree about the power of the Pi, up until the 5. Even my experiments on 4 showed it wasn’t great for some stuff, but… good enough to replicate the organelle.
The 5 with NVMe storage is a very different beast. Hell, you can run Windows on it. LOL. Not that it performs great, but it’s on part with an i3 for most stuff.
I figured it’s worth a go. Worst, I just leave my Pi as a midi / osc host running EigenD, and pipe the stuff out to another host as you say. I set up an old Pi 3 that way, and it works pretty well. With the added power, higher RAM and fast storage options on a 5, I figured I’d experiment.
I’ve been building the dependencies on armf, and had no issues yet, but only about half way through.
Any gotcha’s that you encountered building for armf that I should consider? Just because I haven’t bumped into any yet, doesn’t mean they’re not waiting for me around a corner, holding a crowbar and plotting against me. LOL.
1 Like
iirc, the main changes I had to do were related to arm 32bit, so not relevant to rPI5/64bit.
I think that should work as vcv supports linux 64bit.
anyway… with all these things, I just build, fix as I go along - I don’t ‘track’ issues
the only ‘general tip’ is a common one…
make sure you set the governor (aka scheduler) to performance.
there are other ways to improve audio performance on linux/rpi. but honestly, you get diminishing returns, and they are all very specific … so you’ll have to dig into that yourself and see what works for you.
these days, if you google ‘linux audio’ etc, you’ll find plenty of resources.
also VCV Rack forum has topics on it.
as for meta morph, I not really going to ‘support’ this, though you should be able to compile yourself … thats the reason its open source
the only ‘issue’ may be if you are using a pico…
this needs picodecoder which is NOT open source, so you need to link the static lib, Ive not got one for linux 64bit, but the macOS aarch64 should work.
this is not a problem for tau/alpha these will work with no adjustment
2 Likes
I’m a Tau user. Of course if I could get my hands on an alpha… LOL. Always wanted to step up.
anyway, thanks for the advice.
Gonna work on it on the side, and if I run into a puzzle, I may ask anothet question or two.
So far this seems pretty straightforward. About 3/4 of the way through the dependencies and no probs so far. knock on wood
2 Likes
Lots of information on the VCV Community Forum:
Search results for ‘Raspberry Pi’ - VCV Community (vcvrack.com)
2 Likes