macOS (12.x) : Python Framework vs /usr/bin/python3

Eigenharp
1

this is a bit of a general question for those familiar with Python on macOS…

prior to macOS 12.x
we had /Library/Frameworks/Python.framework.
or might have been /System/Library/Frameworks/Python.framework
(need to double check, but thats not the point of this post)

this framework has now gone, BUT we have /usr/bin/python3

the question is…
when is it enough to just have the python3 executable, vs the framework?

background.

so EigenD compiles using the Python.framework.

and so in my current dev for python3, Im using the framework supplied by Xcode.

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python

now thats ok for development, but obviously I dont want users to have to install Xcode to run EigenD.
(and that doesnt work anyway due to rpath, but lets ignore that for now :wink: )

so Im wondering, is the (python) development model on to compile against the Xcode framework, but then users run against /usr/bin/python3?

ok, to be clear… I know we could (all) just download/install Python package from python.org,
but thats an extra step for users… so its not ideal.
ideally, id use whatever is in place on macOS i

(and no, I dont want to use an installer and supply python to install on users machines, thats approach of runtime packages was horrible from eigenlabs :wink: )

2

I see it here in /System/Library/Frameworks/Python.framework :wink:

Still have a macmini running 10.12.5 (Sierra) and then have old MBPs on 10.13.

1 Like
3

yeah this is all post macOS 12.3, as Apple removed it, and hence the issue created/discussed here

I admit, I was being lazy, I didnt go check … and my machine had it installed in both /Library and /System/Library
iirc, apple will install things in /System/Library, and 3rd party (e.g. python.org) will install in /Library
(or sometimes ~/Library)

again, why Apple moved from a ‘framework’ to this simple /usr/bin/python3 is a bit unclear to me… and really the post is above the implications of this.

if we can’t figure it out, then probably I’ll got down the route of getting users to install from python.org. which we have previously done for the 64bit version.
its not ideal, as an extra install step, however, there are some benefits to this route, as it could be the same/similar for Mac and Windows (*)

(*) as we will also need to update the windows to python3 too, and I highly doubt windows has a python install ‘out of the box’ :wink:

EDIT: Apple are currently supplying Python 3.8, having reviewed seeing some of the changes post 3.8… Ive decided we should go straight to the latest Python (3.10.5), and so will require users to install from python.org
this approach, means we will be less dependent on future Apple OS updates providing different versions of Python.

1 Like
4

I don’t know the specific situation for macOS. I guess this “framework” is essentially a Python distribution that comes with a number of pre-installed libraries (similar to Anaconda)?
Usually there is a (small) set of system libraries that are supposed to come with any Python 3 installation. And Python programs are then distributed as packages that have a list of their dependendencies in the bundled metadata (the wheels). And then there are package managers like pip that (optionally) download and install all these dependencies recursively. (Dependency resolution in pip is pretty basic, but there are more sophisticated solutions like poetry).
Unfortunately the packages installed to site-packages via pip have hardcoded paths in them, so one can only run a copy of a site-packages folder at exactly the same path location where it was installed to. That is what afaik Eigenlabs did - so they could just uncompress the packages to that specified location with the EigenD installer.
One could perhaps let the installer patch the hardcoded parts in the packages of the site-packages folder to whereever the user wants to install the stuff…
The probably cleanest alternative would be to provide the dependencies as wheels as part of the installer and then remote-control pip during installation of EigenD to install these local wheels to the desired target location. (Unless Apple has completely deviated from the standard, pip should be installed on every Mac by default together with Python3)

For Windows one would probably have to package a Python installer with EigenD anyways (which shouldn’t be installed as system Python but locally, so it doesn’t mess with potential user-side Python installations)

5

packages do not have hardcoded paths in them.
in previously releases, I copied the python pkgs supplied by EigenLabs (installed to the EigenD sub directories), into the system framework (/system/library) and machine framework (/library) and had no issue.

anyway this is not relevant, as we are not using any python packages for a while now.
(they were only really used for the python based UIs, eigencommander/eigenbrowser)

indeed apple provided pip3, but its unclear where it installs too.

anyway, I think Ive answered my own question :wink:

I’ve a feeling, I was wrong… NO python is installed with the base macOS installed anymore.
it only comes with Xcode !
I was confused as I assumed usr/bin/python3, not being within Xcode, would mean its base install.
but that appears to be false…

% /usr/bin/python3 

>>> import sys
>>> print (sys.path)
['', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload', '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages']

we can clearly see here that the python3 is using the Xcode supplied framework.
so, there is a dependancy there.

I can’t tell for sure, as I (unfortunately) have Xcode on all my macs,
but Id assume this means /usr/bin/python3 is installed as part of Xcode (or its command line equivalent) , along with the framework.

anyway, doesn’t really matter, this just supports my decision to go with an install from python.org.

this will be a system wide install… Im not going to start messing with an application (EigenD) having its own version… I’m sure that python has adequate ways (site packages etc) of handling different application requirements on a single machine…

whilst this kind of ‘anti social’ behaviour is common on windows, things like Linux regularly share common frameworks… and software is built to cope with it, so Id assume this is true with something as successful/widely used as Python :wink:

(and I dont want to get into speculation of Eigenlabs motives, as we dont have them on hand to explain/discuss their approach :wink: )

update: ok, even more info
perhaps /usr/bin/python3 is there in base install, but really is ‘bare bones’

for two reason

a) its not any kind of symbolic link to the python3 in Xcode, its a full binary file.
and actually on my machine, whilst the same version python source is used … its clearly a different build.
(different size and date)

b) sitepackages reveal an interesting path… (compared to sys.path)

>>> print(site.getsitepackages())
['/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages', '/Applications/Xcode.app/Contents/Developer/AppleInternal/Library/Python/3.8/site-packages', '/Library/Python/3.8/site-packages', '/AppleInternal/Library/Python/3.8/site-packages', '/AppleInternal/Tests/Python/3.8']

its still primarily dependent on Xcode, but also interestingly, it has

/Library/Python/3.8/site-packages

interesting, because this directory does not even exist.

so this leads me to some ideas on how apple intended this to be used.
basically this a more standard unix python install, rather than macOS approach (with frameworks)
/usr/bin/python3 was there to give some ‘compatibility’, without needing an Xcode install but likely mainly just for light scripting. if you wanted to add packages you were expected to add in /Library/Python/3.8/site-packages.

this contrasts with ‘3rd party’ python installs (e.g. python.org) , that will install as frameworks

% /usr/local/bin/python3
Python 3.10.5 (v3.10.5:f377153967, Jun  6 2022, 12:36:10) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import site
>>> print(site.getsitepackages())
['/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages']

so ironically, Apple are now taking an approach closer to unix, whilst python.org are still using the more ‘apple like’ approach of frameworks.
this is actually ‘useful’ as it means there are less likely to be clashes.

users should be careful though …
installing python seems to change the default zsh path (.zprofile) , so you’ll default to the python.org version.

all that said, I think Apple arent really using python for the system at all, so its really not going to matter.

phew - makes me glad to install from python.org, as I think otherwise, users pythons setups on macOS are going to be quite different… and I bet its even worst on windows :wink:

6

You can also install Python without registering it as system Python (essentially it is mainly whether PYTHONPATH is changed and the new bin folder is put into PATH before any other Python interpreter). So a happy side-by-side installation should be possible without affecting other applications that might rely on a specific system Python. At least for Windows.
For Linux (where side-by-side installations are more the exception and all packages are supposed to be version-matched) it is probably better to either create a package with dependencies (that can then be resolved by the package manager.) Or let the user install Python themselves. (My “side-by-side” approach under Linux when needed is usually using distrobox (containers))

7

k, so todays fun… been digging thru massive of c2python and back again…

the general mechanism seems to work…
things are now not blocking anyway more… so thats progress.

however, Ive found we have a real issue with strings.
generally Ive got string handling to work between c/python ok.

BUT, looks like in their infinite wisdom, eigenlabs thought it would be a good idea to pass byte data around as strings. well char * are just bytes arent they !
worked in python 2, as it thought the same.

the main issue, is eigend doesn’t differentiate, so all the api calls are using stdstr regardless of if its really strings or bytes… so you can’t even do something different for the two cases.

so, for the most part… all the eigend python code wants to treat these as strings.
and generally thats ok, I can encode them using the new apis.
but the issue is, that screws the bits that are trying to use as byte data (disguised as strings!)
… those would really benefit from being treated as (python) bytes, but that breaks alot of eigend code!

ho hum… I’ll no doubt for something out…

ideally, id introduce a new ‘bytes’ type in the pip files, but that is a massive undertaking, and even though Ive learnt alot today about the whole c2p/p2c layers… I think quite along way off having enough understanding to do something as major as introduce a new type into it
(and make it work with the associated elements like functors)

hmmm… need a break from it I think!

8

Autchhhh… sounds serious :face_with_monocle:, and no fun.

Let’s see what the “weather” brings tomorrow?! ?…

1 Like
9

Does EigenD do things with these “strings” that only works with Python strings and not bytes / bytearray? Otherwise, would it perhaps be possible just to replace every Python side str type hint with bytes (immutable) or bytearray (mutable) which should convert 1:1 from (const) std::string / (const) char*?
Not really into what Python 2 did string wise - that was before my Python times :wink: From your explanation it sounds as if Python 2 str "this is a string" is like Python 3 bytes b"this is a string"?

Edit: example for bytes vs bytearray vs str: Python Bytes, Bytearray - w3resource

10

@keymanpal , in fairness I did expect these kind of issues… and foresee others along the journey.
its just part of the ‘process’… which I guess I’, just venting about !

@NothanUmber , I’m experienced enough to have tried these kind of things :wink:
as I already said, the issue is…
as I have it now, with using unicode strings on python side, you break a bit of eigend C++ code.
but, if you move to using bytes on the python side, you break a ton of eigend python code.

of course, you could hack either approach to work, but the more you do this, the more likely you are to screw something else up… and increasing the debugging effort at the end.

(these posts obviously just give a ‘taste’ of the issues, and what Ive tried… as I dont want to spend hours going into details… as those hours are better spent working on the issues)

so this turns into ‘approach’…

my greatest fear about this work, is NOT getting EigenD to compile/run using python 3, thats just a matter of slogging away at it… and eventually it’ll get there.
no… the biggest risk, is changing a ton of code, and making the thing unstable…

in my (pretty extensive ;)) experience, this is the single most common ‘total failure’ for projects that end up having to be refactored, or have other major changes. basically, you destabilise the code base, in a way that is even more ‘disruptive’ than when you are writing a brand new project… and basically everyone involved, just gets fed up/demoralised, and walks away from it - often commissioning a ‘new project’.

it’s bad enough with commercial projects, buts its (almost) always fatal for private/open source projects - since its done in devs spare time.

… so yeah,Im being very particular, about HOW I make these changes,
so that WHEN things go pear shaped (which they will), I’ll hopefully have a route to 'complete the job.

but to be clear… even with my approach, Im still fear the outcome…

so, really, as per @keymanpal … Im really just posting, not due to lack of ideas, or seeking solutions,
rather just posting an update… and venting a bit :wink:

as I stated at the beginning of this topic , this is going to be a bit of a slog (*) to do it ‘right’,
so just taking users along with me on that journey.

anyway, Ive started the ‘proper solution’ to this, which is to start supporting ‘byte arrays’.
these can be ‘inserted’ where needed.
this approach, means I can make eigend more explicit about its use-cases.

Im hopeful, that this won’t actually be needed in too many places, as I think most of the time strings are ok - but it provides me with a universal solution/ that can scale… and be tested :wink:

Im also considering putting in a (temporary) check the string translation code, that will look for ‘binary’ content in strings… thus highlighting areas that need to be moved to byte arrays.
(this check can be later removed, as once we have EigenD running, it’ll be quickly evident where binary data is being passed around via strings)

as an added benefit, like all of this work, its also made me much more familiar with how EigenD does the python/c mapping layer… so if there are further issues. I’ll be better prepared…

so yeah… as @keymanpal said, another day… less dark clouds, I keep plodding forwards !

(*) its only been 4-5 days so far, and Id say, with about 4-6 hours each day… and whilst Im making good progress… I think, Im still at early stages, theres alot still to do , and there will be many more tricky things ahead.

2 Likes
11

I’m truly appreciative of your (and others) efforts and happy you can get some more knowledge along the way.

“Vent” away! its good :wink:

Have to go find better encouragement analogies /words…

2 Likes
12

lol

I was trying to decide this morning if your last saying… (Let’s see what the “weather” brings tomorrow?!) works better in Spain or the UK?

Spanish weather is pretty much the same everyday, so unlikely to change … but its generally sunny (optimistic?)

whereas UK weather is very unpredictable, sure… you may have dark clouds one day, and sunny the next… but on the other hand you can have weeks of drizzly horrible weather.

so not sure which Id want to think of here :wink:

2 Likes
13

good news, bytearray is working :slight_smile:

so functionally this means I can now…
use pezload to load firmware to pico, then picodump shows data from pico.
(oh, Ive not said explicitly, but not only is this using python3 but this is also running native apple silicon)

so one more step down the road…
next step - likely to start verifying other command line tools (Ive started on some already)

of course, occasionally I attempt to fire up the full EigenD… as it is starting ok now,
but fails with various python errors… but I likely wont chase these until the command line tools work.
simply because… the command line tools are ‘simpler’ to trace down issues… and build up the foundations of how EigenD et al works.

if you want to see more ramblings, and where im at you can find here:
https://github.com/TheTechnobear/EigenD/tree/python3/documentation
in particular the python_dev_*.md docs

4 Likes
14

one day to the next? I’m sure it’s like Ireland, where if you don’t like the weather, just wait 5 minutes :wink:

3 Likes
16

Don’t you just hate it when you ask a question after searching on one thread and find the exact answer on another!
Its amazing what you are doing - I think python is a great language until it gets beyond one file, but have had to do that sort of thing for a living many times.

1 Like