Aftertouch Handler Script Development Log

A script to handle lost aftertouch data using Python.

This feels very pretentious, as this is nothing more than a glorified filter script, but my hope is to add functions to it over time to make it very interesting. My ultimate goal is to package it into a raspberry pi zero w image, so you can plug your board into the micro usb port and use it over bluetooth: plug and play.

Feel free to chime in here for desired “gestures” for triggers and I’ll try to add it to the project. Also, I am pretty sure there are many here more skilled as programmers than me, so if you want to take ownership or hijack the thread, go ahead. For me, this is an exercise to build a new skillset rather than designing a product so I don’t have any ego about this. I just want people to play their boards and enjoy it as much as me! :slight_smile:

So I got the script working for FL studio, a simplified version albeit. Now I need to find an appropriate MIDI module to use for a universal version.

Surprisingly, even cutting off the range to 0-30 for aftertouch and scaling it in a dumb manner, it sounds really good. I was able to get great dynamics even with 4 fingers in the same shared aftertouch “zone”. Obviously there is the risk to blow out your other notes still while doing this, so I need to decide how to handle it. Maybe I could have the CC value on the touch pad change/mix handling, so blown out notes output a LFO function, just share the same values, or repeat the previous input pattern.

So until I provide a script and decide on the best behavior, I’d recommend all users just go into your DAW and scale the aftertouch input up to 30, with some smoothing to make it more organic. It sucks we can’t get the whole dynamic range in the same horizontal zone but it will do for now.

While I’m digging around in the MIDI messages, I’m going to try my hand at USB sniffing to see if there is any way to detect a module change and convert it into a program change MIDI message. It would be ideal if we could tie the aftertouch setting to a module, allowing us to set the desired setting. I’m an extreme noob at this so don’t hold your breath on that feature!

I’ve been testing with splitting the MPE channels to different SWAM instruments, practicing with the attenuated aftertouch. I’ve found there is enough dynamic range to play the instruments convincingly, but it’s far easier to use two fingers on each of your hands, placing them staggered so the 4 horizontal zones don’t overlap. If you are extremely careful, only using the tips of your fingers, you can get effectively 8 horizontal aftertouch zones. This doesn’t really work with the piano (or grand clavier) module, but the keys module works fine for this.

I’m still ambitious about this script, but I feel like adapting to the limitations is sufficient to make use of this instrument. Maybe I’ll discover something interesting while doing USB sniffing but as far as I can tell, nothing is going to get us to Sensel or Seaboard depth.

EDIT: Good news, the module placement and type is signaled on the MIDIIN2(Joue) port. For some reason MIDI-Ox is very picky about connecting to it, but after I routed it through a virtual cable (CopperLan), it connected consistently. I’m figuring out the message but it doesn’t appear to be similar to anything currently documented.

EDIT2: Okay, I guess I was mislead from peeps on various forums: this is a real, documented message type. So I used this resource to figure out when you add a pad, it sends a file header for it, likely waiting for an ACK (which I haven’t found any way to return in MIDI-Ox), and probably sending the contents of the pads.

The message is as follows:
[F0 7E 01 07 01 01 42 49 4E 20 00 00 00 00 49 46 4F 32]

F0 7E (Start of non realtime universal message) 01 (Sysex Channel) 07 01 01 (File dump, header and device ID) 42 49 4E 20 (BIN file-type) 00 00 00 00 (Zero file size) 49 46 4F 32 (File Name IFO2)

33 5F 32 35 30 33 31 32 30 30 F7

33 5F (Pad location, 31-33 corresponding to area 1, 2, and 3) 32 35 30 33 31 32 30 30 (Pad code, seems to be exclusive to each one but 32 35 30 seems to be the 4x4 drum pad code) F7 (end of message)

With the USB sniffer I should be able to get the full data stream for the pad bin but that won’t be necessary for my purposes. I think using the first 3 bytes following the area to detect the pad type, and the rest as an exclusive serial number is enough to trigger the aftertouch type. I leave my investigations here mostly as an exercise in anyone interested in other cool projects.

Woof, so using the same technique in tracking aftertouch, I’ve found the velocity interferes too! I never noticed it on drums or my SWAM instruments as velocity wasn’t as important, but using the Piano, it’s very noticeable. So I guess another thing to add to my “feature list” is to use the aftertouch drop-outs to properly scale velocity on parallel note triggers. Yes, you could just kill the dynamic range and only use velocity 0-30, but that makes the dynamics of percussive instruments like piano sound very off.

Okay, good news. I’ve split up some of the functions of the script to handle cases, and I’ve developed cases for “strikes.” The easiest one, which works well, handles chords, using the value of the largest note velocity within a timeframe. The next one, which works okay, handles strikes after the initial first press, using recent aftertouch data to infer probable velocity. Right now, though the instrument isn’t accurate, it’s natural enough to sound intentional. I’m not skilled enough to feel like my playing is being mangled, as it was before, so I’m happy with the results. There is quite a bit of fudging still, but the dynamics seem to be preserved. I’m going to work on aftertouch drops next (as in strikes cutting off aftertouch), then aftertouch scaling (increasing the values based on active aftertouch keys), and finally release an alpha version of the script.

I finished the aftertouch scaling and although it’s a bit too sensitive, I’m able to get dynamic aftertouch on 10 fingers or more. What’s left is a smoothing function to smooth out my janky, primitive algorithms, and an inertia function to handle complete drops of aftertouch (filtering out sudden drops that are followed by a note on event).

After that I have to wade into the mire of MIDI port handling in Python. More than likely you’ll have to create an appropriate virtual MIDI cable to patch the output into your DAW of choice. I tried a virtual MIDI to patch into the Joué Play app but I couldn’t get it to play nice in Windows. Stupid API.

I might get an alpha out by next week. I’ll need some help tuning the values to fit a wide range of playstyles, and find the many bugs. Is there anyone out there willing to try the script? If so, what OS are you using so I can use the right MIDI module?

I have been following and I’m always supportive :wink:
My Joué is ready for new adventures…

I’m on macOS and I was going to mention; do you know Gig Performer? www.GigPerformer.com, it has scripting, OSC and is on both Win and macOS

1 Like

Nice! I appreciate the support and bearing with… I looked at Gigperformer and it uses a proprietary script so that’s a no-go for me unfortunately.

I’ve never developed anything for MacOS but I suspect many more users are using it anyways so I’ll see if I can compile an executable for convenience sake.

One major feature which hasn’t been implemented yet is assigning “note groups,” in which notes are grouped that are most likely interfering with one another. Right now they are in one big group so annoyingly, increasing pressure on one group of notes increases the rest.

There are also a million other edge cases which will require tuning various factors and values to cover, joy! My hope is the values I leave to the end user to tweak will allow it to be more versatile but expect some real jank :slight_smile:

P.S. You know how you can get a little more dynamics on the aftertouch, no script required? A piece of paper towel underneath the pad. Really, it slips and slides but it seems to increase the pressure required to max it out. Now I’m thinking of playing around with some ooblek underneath it for some real shenanigans :smiley:

1 Like

Sunlight is the best cure to crappy code in my opinion, so I’m going to release a pre-alpha with the MIDO module and rpt-midi this week. I am not sure if/how to package it but that’s going to be part of my learning process. I’ll get a github eventually, but for now I’ll just upload it somewhere else.

I’m simply not musically skilled enough to put the code through its paces and feel confident I’m putting in what I’m getting out. Basically my entire evaluation is “huh, that’s neat!” The velocity/pressure interaction in particular is rough, with it going to a fallback value often. I think the exponential scaling method I used for the pressure correction works really well so I’m going to re-write it entirely with that model. My naming conventions are also hilariously inconsistent so that’s another thing I’m working on. Performance is okay; I get under 48ns for initial calls, with 2 ms on subsequent calls, likely because of the repetitions under timeit and not doing any memoization. I notice some slow down while running it in FL Studio, but that might be from my instruments or performance issue with the built-in interpreter.

I was playing around with my bluetooth RPI thingey and got the Joué Play app to send some sysex messages. For whatever reason my RPI didn’t receive them so it couldn’t respond, but it looks like it’s a call to the board to return the pad ID. This would make a full-fledged program-change signal doable from the script, based on the pad used. At the very least, forwarding a message to the DAW would be awesome so we can play dynamically. I have no idea why they haven’t implemented it within the Joué editor, or returned the Sysex messages on the Play port. I’m guessing they want to prevent errant MIDI messages from being sent to the board and overwriting pads…

Also if any of you feel adventurous and want to implement my jank for yourself, note I don’t have an event history recycler so if you held a note for too long you’ll probably crash the interpreter, lol.

class simpleAftertouchHandler():

    class noteData():
        def __init__(self, noteValue, channel, velocity, timestamp):
            self.noteValue = noteValue
            self.velocity = velocity
            self.channel = channel
            self.aftertouch = velocity
            self.lastUpdate = timestamp
            self.eventHistory = [{"eventType":MIDI_NOTEON,
                                       "eventValue":velocity,
                                       "timestamp":timestamp}]
        def correctVelocity(self, correctedVel, timestamp):
            self.velocity = correctedVel
            self.eventHistory.insert(0, {"eventType":VEL_UPDATE,
                                       "eventValue":correctedVel,
                                       "timestamp":timestamp})
            
        def updateAftertouch(self, aftertouchValue, timestamp):
            self.aftertouch = aftertouchValue
            self.lastUpdate = timestamp
            self.eventHistory.insert(0, {"eventType": MIDI_KEYAFTERTOUCH,
                                         "eventValue": aftertouchValue,
                                         "timestamp": timestamp})

        def correctAftertouch(self, correctedAT, timestamp):
            self.aftertouch = correctedAT
            self.lastUpdate = timestamp
            self.eventHistory.insert(0, {"eventType": AFT_UPDATE,
                                         "eventValue": correctedAT,
                                         "timestamp": timestamp})
                          
    def __init__(self):
        self.currentNotes = {}
        self.noteGroups = []
        self.currentTime = 0
        self.callingNote = 0
        self.callingChan = 0
        self.callingNoteID = ""
        self.callingEvent = ""
        self.callingValue = 0
        print("Handler started")

    #Strike interference, used for notes played within strike timescale, before aftertouch values can be used   
    def strikeInterference(self, inputVel):
        strikeNotes = dict(filter(lambda note:
                                  self.currentTime - strikeTime <
                                  note[1].eventHistory[-1]["timestamp"],
                                  self.currentNotes.items()))                       #Filter current notes by lookback time from event history
        strikeNotes = dict(filter(lambda note:
                                  note[1].eventHistory[0]["eventType"] == MIDI_NOTEON,
                                  strikeNotes.items()))                             #Filter strike notes by note on events only, eventually check multiple events
        maxVelocity = max(map(lambda note: note[1].velocity, strikeNotes.items()))  #Get loudest note played
        velFloor = maxVelocity                                                      #Loudest velocity serves as floor for following notes
        firstNote = list(iter(strikeNotes.items()))[0]                              #Get first note 
        firstNoteTime = firstNote[1].eventHistory[-1]["timestamp"]                  #Get first note time for velocity calculation
        strikeNotes = dict(filter(lambda note:
                                  (note[1].velocity/interFactor <= 127 - velFloor)
                                  or (note[0] == firstNote[0]),
                                  self.currentNotes.items()))                       #Remove any notes likely not interfering based on interference factor
        self.noteGroups.append(strikeNotes)                                         #Add notes to note group for future processing of aftertouch
        lastNoteTime = self.currentTime - firstNoteTime                             #Increase velocity based on average note time
        outputVel = max(min(lastNoteTime/strikeTime
                                       * strikeFactor, 127), velFloor)              #Based on last note time vs look back time and scaling by strike factor, increase from the floor
        return outputVel

    #Pressure strike interference takes largest pressure values to find available space for correct reading. Velocity is then corrected to correct reading based on space. If
    #there is no available space, then a fuzzed value based on aftertouch pressure is used
    #Only affected by more than 3 notes?
    def pressureStrikeInterference(self, inputVel):
        #TODO Check which note group it might be in to narrow down false positives
        pressureList = map(lambda note: note[1].eventHistory,
                                self.currentNotes.items())                          #Get event history of all current notes
        pressureList = map(lambda noteEvents:
                           list(filter(lambda event: event["eventType"] == MIDI_KEYAFTERTOUCH,
                            noteEvents)), pressureList)                             #Only select pressure
        pressureList = filter(lambda eventHistory:
                              len(eventHistory) > 0,pressureList)                   #Remove empty event history
        pressureList = map(lambda events: events[0], pressureList)                  #Select only last pressure value, todo, average last 4 values?
        pressureList = filter(lambda event:
                              event["timestamp"] > (self.currentTime - oldPressure),
                              list(pressureList))                                   #Filter any old pressure values todo, filter 0 values
        pressureValues = list(map(lambda eventDict: eventDict["eventValue"],
                             pressureList))                                         #Reduce list from dictionaries to just pressure values
        if (len(list(pressureValues)) > 1 and inputVel == 1):                       #Use fallback if no presure values
            if (sum(pressureValues)/interFactor < (127)):                           #Use sum as the interference factor if less than max, otherwise use largest combinations
                strikeLost = min((sum(pressureValues) / (1 - interFactor), 127))    #Assume the lost strike value is reduced within the range left from all aftertouch values
            else:
                largestSum = max(sum(sorted(pressureValues[-2:-1])), 127)           #Sum largest two values, with max being 127
                strikeLost = min(largestSum / (1 - interFactor), 127)               #Use sum as the lost strike value
            strikeLost = max(pressureValues) if strikeLost == 127 else strikeLost   #Use largest of pressure list if value is 127
            #Calculate probable strike value based on strike lost range, or use number of keys pressed if 1
            if (inputVel== 1 and strikeLost < 10):
                outputVel = self.fallbackValue()
            else:
                outputVel  = min(int(max((inputVel / min(127 - strikeLost, .0001)) * 127, strikeLost)), defaultMax)
        else:
            outputVel = self.fallbackValue()
        return outputVel
                             
            
    def fallbackValue(self):
        return min(len(self.currentNotes)* 30, defaultMax)
        
    #Pressure interference looks at all notes pressure data to find available space for correct reading. If there is no space left for any reading (e.g. values will always be 1-5)
    #then it will use the average of the largest readings
    def pressureInterference(self, inputPressure):
        otherNotes = dict(filter(lambda note: note[0] != self.callingNoteID,
                                 self.currentNotes.items()))                        #Filter out calling note from current notes
        pressureNotes = dict(filter(lambda note:
                              MIDI_KEYAFTERTOUCH
                                    in map(lambda event:
                                        event["eventType"],
                                           note[1].eventHistory),
                                            otherNotes.items()))                    #Wrap result into list and map pressure values into average
        pressureAvg = list(map(lambda note:                                         #Process notes into filtered list
                          avg(list(map(lambda pressureEvents:                       #Map filtered list into average function
                                  pressureEvents["eventValue"],                     #Return values of events
                                  list(filter(lambda event:                         #List of events are filtered
                                              event["eventType"] == MIDI_KEYAFTERTOUCH
                                              and event["eventValue"] != 0,         #Only return events of non-zero value and aftertouch
                                         note[1].eventHistory))[0:4]))),            #Return last 5 values matching the filter
                              pressureNotes.items()))                               #Process the map over the dictionary
        pressureLost = 0                                                            #Initialize the lost pressure value
        if (len(pressureAvg) > 0):                                                  #Only proceed if list is more than 0 items and sum is more than value ceiling
            for idx, pressureVal in enumerate(sorted(pressureAvg)[::-1]):           #Loop through list largest to smallest
                pressureLost += (pressureVal / interFactor) ** (idx + 1)            #Calculate lost space for pressure values, increasing the exponent for smaller (cut off) values
            outputPressure = min(int(inputPressure + pressureLost), 127)
        else:
            outputPressure = inputPressure
        return outputPressure

    #Pressure intertia looks at any notes with an aftertouch dropped flag and no note offs and returns an extrapolated aftertouch based on inertia weight, and a decay time. This is
    #returned in an array of MIDIdata messages to be used by the device output function
    def pressureIntertia(self, MIDIdata):
        pressureList

    #Simple moving average of corrected values
    def smoothAftertouch(self, inputAft):
        #Filter last events to aftertouch updates and look back time
        aftUpdates = list(filter(lambda event:
                                      event["eventType"] == MIDI_KEYAFTERTOUCH
                                      and event["timestamp"] > (self.currentTime - lookbackTime),
                                      self.currentNotes[self.callingNoteID].eventHistory))
        #Filter last events to aftertouch corrections and look back time
        aftCorrections =  list(filter(lambda event:
                                      event["eventType"] == AFT_UPDATE
                                      and event["timestamp"] > (self.currentTime - lookbackTime),
                                      self.currentNotes[self.callingNoteID].eventHistory))
        #Use the largest list
        usedValues  = aftUpdates if len(aftUpdates) > len(aftCorrections) else aftCorrections
        
        outputAft = int(avg([*list(map(lambda event: event["eventValue"], usedValues)), inputAft]))
        return outputAft
        
    def processMIDI(self, MIDIdata):
        self.currentTime = MIDIdata["timestamp"]
        self.callingNote = MIDIdata["note"]
        self.callingChan = MIDIdata["channel"]
        self.callingEvent = MIDIdata["type"]
        self.callingNoteID = f'{MIDIdata["note"]}{MIDIdata["channel"]}'
        if (self.callingEvent == MIDI_NOTEON):
            self.callingValue = MIDIdata["velocity"]
            self.currentNotes[self.callingNoteID] = self.noteData(self.callingNote, #Add to note list
                                                                  self.callingChan,
                                                                  self.callingValue,
                                                                  self.currentTime)
            if (len(self.currentNotes) > 1):
                correctedVel = self.strikeInterference(MIDIdata["velocity"])        #Correct strike interference
                if (len(self.currentNotes)>3):
                    correctedVel = self.pressureStrikeInterference(MIDIdata["velocity"])#Correct strike based on pressure
                self.currentNotes[self.callingNoteID].correctVelocity(correctedVel,
                                                                      self.currentTime)#Record corrected velocity
                MIDIdata["velocity"] = correctedVel                                 #Update MIDIdata

        elif(self.callingEvent in {MIDI_KEYAFTERTOUCH, MIDI_CHANAFTERTOUCH}):
            self.callingValue = MIDIdata["pressure"]
            self.currentNotes[self.callingNoteID].updateAftertouch(self.callingValue,
                                                                   self.currentTime)#Update note list
            
        
            if (len(self.currentNotes) > 0):
                #print("Pressure before",MIDIdata["pressure"])
                correctedAft = self.pressureInterference(MIDIdata["pressure"])      #Get corrected aftertouch
                correctedAft = self.smoothAftertouch(correctedAft)                  #Smooth corrected aftertouch
                self.currentNotes[self.callingNoteID].correctAftertouch(correctedAft,
                                                                        self.currentTime)#Record corrected aftertouch
                MIDIdata["pressure"]= correctedAft                                  #Update aftertouch in MIDI data
                #MIDIdata["pressure"] = self.smoothCurrentPressure(MIDIdata)
                #print("Pressure after",MIDIdata["pressure"])
                

        #cc stuff here

        elif(self.callingEvent == MIDI_NOTEOFF):
            del self.currentNotes[self.callingNoteID]                               #Remove from note list
            noteGroups = {}                                                         #Remove from note group
            self.noteCount = len(self.currentNotes)                                 #Update count of playing notes

        else:
            print("Unhandled MIDI message")
            
        return MIDIdata

Thanks for your work on this after our earlier conversation. Sorry that I’ve had not time to respond recently, I will take a look next time I have time. Cheers.

1 Like

After testing with this far more convenient GUI, I’m not convinced the aftertouch handling is working after all, or maybe I broke it. I need to do further testing before packaging it into an executable. If you want to try it out, you’ll need to install Python and the modules: mido, PySimpleGui. There is obviously no warranty with this script and I am not responsible for any jankiness that explodes your expensive stereo monitors etc.

#By Ph0ton copyright lolol

#Libraries and Modules and Stuff
import PySimpleGUI as sg
import mido
import time

#Constants
VEL_UPDATE = 999
AFT_UPDATE = 998
MIDI_NOTEON = "note_on"
MIDI_NOTEOFF = "note_off" 
MIDI_KEYAFTERTOUCH = "polytouch" 
MIDI_CHANAFTERTOUCH = "aftertouch" 

default_strikeTime = .01                                                            #Time in seconds which correspond to notes played by the same strike
default_interFactor = .5                                                            #Interference factor scales how much the last note played interferes with the next group velocity or aftertouch
default_strikeFactor = .3                                                           #Strike to velocity factor scales velocity based on time interval
default_oldPressure = 30                                                            #Used to filter old notes (in seconds)
default_valuePerNote = 15                                                           #Baseline value lost from pressing note
default_lookbackTime = .1
default_lowerCeiling = 30                                                           #Maximum value before it will interfere with other notes
default_maxVal = 80                                                                 #Default value if all else fails!

#Math stuff

def avg(values):
    return sum(values)/len(values) if len(values) != 0 else 0

#Simple least squares fit
#Use lambda and map to get sums of values needed for least squares equation)
def sumsCalc(listX, listY):
    return map(lambda values: sum(values), [listX, listY, map(lambda x: x ** 2, listX), map(lambda x,y: x * y, listX, listY)])

#Using sumsCalc values, get the slope of the least squares line
def leastSquaresSlope(n, sumX, sumY, sumX2, sumXY):
    return ((n * sumXY) - (sumX * sumY))/((n * sumX2) - sumX ** 2)


#Using leastSquaresSlope value and sumsCalc values, get the intercept of the least squares line
def leastSquaresInter(n, slope, sumX, sumY):
    return (sumY - (slope * sumX))/n

#Using the associated functions, predict the value on the least squares line
def leastSquaresPredict(listX, listY, xIn):
    [n, [sumX, sumY, sumX2, sumXY]] = len(listX), sumsCalc(listX, listY)
    slope = leastSquaresSlope(n, sumX, sumY, sumX2, sumXY)
    intercept = leastSquaresInter(n, slope, sumX, sumY)
    return slope * xIn + intercept

#Connect MIDI in port, MIDI out

#GUI stuff

paraLayout = [[sg.Text("Strike Time",size=(20,1)),sg.Input(key="strikeInput", size=(4,1))],
                [sg.Text("Strike Factor (0-1)",size=(20,1)),sg.Input(key="strikeFactInput", size=(4,1))],
                [sg.Text("Interference Factor (0-1)",size=(20,1)),sg.Input(key="interInput", size=(4,1))],
                [sg.Text("Interference Value (1-127)",size=(20,1)),sg.Input(key="lowerValueInput", size=(4,1))],
                [sg.Text("Interference Floor (1-127)",size=(20,1)),sg.Input(key="floorInput", size=(4,1))],
                [sg.Text("Lookback Time",size=(20,1)),sg.Input(key="lookBackInput", size=(4,1))],
                [sg.Button("Update Parameters", key="parameterUpdate")]]

optLayout = [[sg.Checkbox("Show MIDI in", key="midiIN", enable_events=True, default=False, size=(12,1))],
                [sg.Checkbox("Show MIDI out", key="midiOUT", enable_events=True, default=False,size=(12,1))]]

handLayout = [[sg.Checkbox("MIDI", key="processMIDI", enable_events=True, default=True,size=(12,1))],
                 [sg.Checkbox("Velocity", key="processVel", enable_events=True, default=True,size=(12,1))],
                 [sg.Checkbox("Aftertouch", key="processAft", enable_events=True,  default=True,size=(12,1))]]

def default_layout(inPorts, outPorts):

    return [[sg.Frame("MIDI Port Options",layout=[
    [sg.Text("Joue MIDI In Port",size=(20,1)),sg.Combo(inPorts, key="inPortCombo",size=(25,1))],
    [sg.Text("Joue MIDI Out Port",size=(20,1)),sg.Combo(outPorts, key="outPortCombo",size=(25,1))],
    [sg.Button("Open Ports", key="portsButton"),sg.Text("Status: Idle", key="statusText")]])],
            [sg.Column(layout=[[sg.Frame("Parameters",layout=paraLayout)]]),
             sg.Column(layout=[
                 [sg.Frame("Options",layout=optLayout)],
                 [sg.Frame("Handling",layout=handLayout)]])],
            [sg.Output(size=(200,20))]]

default_title = "Joue MIDI Processor"
default_buttonState = {"portsButton":1}

class gui():
    
    def __init__(self):
        self.midi = MIDIparser()
        self.handler = self.midi.getHandler()
        self.MIDIports = self.midi.getMIDIports()
        #print(MIDIports[0])
        #print(MIDIports[1])
        self.layout = default_layout(self.MIDIports[0], self.MIDIports[1])
        self.window = sg.Window(default_title, self.layout, size=(400,500),finalize=True)
        self.buttonState = default_buttonState
        #Update input values
        self.window["strikeInput"].Update(value=default_strikeTime)
        self.window["strikeFactInput"].Update(value=default_strikeFactor)
        self.window["interInput"].Update(value=default_interFactor)
        self.window["lowerValueInput"].Update(value=default_valuePerNote)
        self.window["floorInput"].Update(value=default_lowerCeiling)
        self.window["lookBackInput"].Update(value=default_lookbackTime)
        self.main()

    def main(self):
        while True:                                                                     #UI loop
            event, values = self.window.read()
            if (event == "portsButton"):                                                 #Port button pressed selected
                if ("" in [values["inPortCombo"],values["outPortCombo"]]):               #Check port intent
                    sg.popup_timed("All ports must be selected",button_type=5, no_titlebar=True)
                elif(self.buttonState[event] == 1):
                    if (self.midi.openPorts(values["inPortCombo"], values["outPortCombo"])):
                        self.window["statusText"].Update("MIDI transform running")      #Update status
                        self.window[event].Update("Close Port")                         #Change text
                        self.window["inPortCombo"].Update(disabled = True)              #Disable combobox
                        self.window["outPortCombo"].Update(disabled = True)             #Disable combobox
                        self.buttonState[event] = 0                                     #Update state
                    else:
                        self.window["statusText"].Update("Error, ports in use")         #Fallback state
                else:
                    self.window["statusText"].Update("Idle")                            #Update status
                    self.midi.closePorts()                                              #Close midi ports
                    self.window[event].Update("Open Port")                              #Change text
                    self.window["inPortCombo"].Update(disabled = False)                 #Enable combobox
                    self.window["outPortCombo"].Update(disabled = False)                #Enable combobox
                    self.buttonState[event] = 1                                         #Update state
            elif (event == "parameterUpdate"):
                inputVals = list(values.values())[2:8]
                try:
                    inputVals = list(map(float,inputVals))
                except:
                    sg.popup_timed("Only input numbers",button_type=5, no_titlebar=True)
                else:
                    #List of inputs: strikeTime, strikeFactor, interFactor, valuePerNote, maxVal, lookbackTime
                    print("Parameters updated")
                    self.handler.parameterUpdate(*inputVals)
            elif (event in {"processMIDI","processVel","processAft","midiIN","midiOUT"}):
                self.midi.updateProcessOptions(event, values[event])                    #Update processing options
            elif (event == sg.WIN_CLOSED):                                              #Close window
                break
            else:
                print("Event not handled")                                              #Bad GUI event
        self.window.close()

class MIDIparser():

    def __init__(self):
        self.handler = joueHandler()
        self.lastUpdate = 0
        self.procOpt = {"processMIDI":True,
                        "processVel":True,
                        "processAft":True,
                        "midiIN":False,
                        "midiOUT":False}
        print("Init okay")

    def getMIDIports(self):
        MIDIinPorts = mido.get_input_names()
        MIDIoutPorts = mido.get_output_names()
        return [MIDIinPorts, MIDIoutPorts]

    def getHandler(self):
        return self.handler

    def openPorts(self, inPortName, outPortName):
        print(inPortName," ",outPortName)
        try: 
            self.inPort = mido.open_input(inPortName)
            print("In port okay")
            try:
                self.outPort = mido.open_output(outPortName)
            except:
                print("Out port problem")
                self.inPort.close()
                pass
            else:
                self.inPort.callback = self.transformMIDI
                print("Out port okay")
                return True
        except:
            return False

    def closePorts(self):
        self.inPort.close()
        self.outPort.close()

    def updateProcessOptions(self, option, value):
        self.procOpt[option] = value
        
    def transformMIDI(self, msg):
        if (self.procOpt["midiIN"]): print("MIDI input | Type: ",msg.type," Props: ",msg)
        if (self.procOpt["processMIDI"]):
            if (msg.type in {"note_on", "note_off"}):
                processedMIDI = self.handler.processMIDI(
                                                    {"type":msg.type,
                                                     "note": msg.note,
                                                     "channel":msg.channel,
                                                     "velocity": msg.velocity,
                                                     "pressure": 0,
                                                     "timestamp": time.time()})
                msg.velocity = processedMIDI["velocity"] if self.procOpt["processVel"] else msg.velocity
                
            elif (msg.type in {"aftertouch", "polytouch"}):
                processedMIDI = self.handler.processMIDI(
                                                    {"type":msg.type,
                                                     "note": msg.note,
                                                     "channel":msg.channel,
                                                     "velocity": 0,
                                                     "pressure": msg.value,
                                                     "timestamp": time.time()})
                msg.value = processedMIDI["pressure"] if self.procOpt["processAft"] else msg.value
        if (self.procOpt["midiOUT"]): print("MIDI output | Type: ",msg.type," Props: ",msg)
        self.outPort.send(msg)
            

    def __del__(self):
        self.inPort.close()
        self.outPort.close()

class joueHandler():

    class noteData():
        def __init__(self, noteValue, channel, velocity, timestamp):
            self.noteValue = noteValue
            self.velocity = velocity
            self.channel = channel
            self.aftertouch = velocity
            self.lastUpdate = timestamp
            self.eventHistory = [{"eventType":MIDI_NOTEON,
                                       "eventValue":velocity,
                                       "timestamp":timestamp}]
        def correctVelocity(self, correctedVel, timestamp):
            self.velocity = correctedVel
            self.eventHistory.insert(0, {"eventType":VEL_UPDATE,
                                       "eventValue":correctedVel,
                                       "timestamp":timestamp})
            
        def updateAftertouch(self, aftertouchValue, timestamp):
            self.aftertouch = aftertouchValue
            self.lastUpdate = timestamp
            self.eventHistory.insert(0, {"eventType": MIDI_KEYAFTERTOUCH,
                                         "eventValue": aftertouchValue,
                                         "timestamp": timestamp})

        def correctAftertouch(self, correctedAT, timestamp):
            self.aftertouch = correctedAT
            self.lastUpdate = timestamp
            self.eventHistory.insert(0, {"eventType": AFT_UPDATE,
                                         "eventValue": correctedAT,
                                         "timestamp": timestamp})
                          
    def __init__(self):
        self.currentNotes = {}
        self.noteGroups = []
        self.currentTime = 0
        self.callingNote = 0
        self.callingChan = 0
        self.callingNoteID = ""
        self.callingEvent = ""
        self.callingValue = 0
        self.parameterUpdate(default_strikeTime,
                        default_strikeFactor,
                        default_interFactor,
                        default_valuePerNote,
                        default_maxVal,
                        default_lookbackTime)
        self.defaultMax = default_maxVal
        self.oldPressure =default_oldPressure
        print("Handler started")

    #Updates parameters
    def parameterUpdate(self, strikeTime, strikeFactor, interFactor, valuePerNote, maxVal, lookbackTime):
        self.strikeTime = strikeTime
        self.strikeFactor = strikeFactor
        self.interFactor = interFactor
        self.valuePerNote = valuePerNote
        self.maxVal = maxVal
        self.lookbackTime = lookbackTime

    #Strike interference, used for notes played within strike timescale, before aftertouch values can be used   
    def strikeInterference(self, inputVel):
        strikeNotes = dict(filter(lambda note:
                                  self.currentTime - self.strikeTime <
                                  note[1].eventHistory[-1]["timestamp"],
                                  self.currentNotes.items()))                       #Filter current notes by lookback time from event history
        strikeNotes = dict(filter(lambda note:
                                  note[1].eventHistory[0]["eventType"] == MIDI_NOTEON,
                                  strikeNotes.items()))                             #Filter strike notes by note on events only, eventually check multiple events
        maxVelocity = max(map(lambda note: note[1].velocity, strikeNotes.items()))  #Get loudest note played
        velFloor = maxVelocity                                                      #Loudest velocity serves as floor for following notes
        firstNote = list(iter(strikeNotes.items()))[0]                              #Get first note 
        firstNoteTime = firstNote[1].eventHistory[-1]["timestamp"]                  #Get first note time for velocity calculation
        strikeNotes = dict(filter(lambda note:
                                  (note[1].velocity/self.interFactor <= 127 - velFloor)
                                  or (note[0] == firstNote[0]),
                                  self.currentNotes.items()))                       #Remove any notes likely not interfering based on interference factor
        self.noteGroups.append(strikeNotes)                                         #Add notes to note group for future processing of aftertouch
        lastNoteTime = self.currentTime - firstNoteTime                             #Increase velocity based on average note time
        outputVel = max(min(lastNoteTime/self.strikeTime
                                       * self.strikeFactor, 127), velFloor)         #Based on last note time vs look back time and scaling by strike factor, increase from the floor
        return outputVel

    #Pressure strike interference takes largest pressure values to find available space for correct reading. Velocity is then corrected to correct reading based on space. If
    #there is no available space, then a fuzzed value based on aftertouch pressure is used
    #Only affected by more than 3 notes?
    def pressureStrikeInterference(self, inputVel):
        #TODO Check which note group it might be in to narrow down false positives
        pressureList = map(lambda note: note[1].eventHistory,
                                self.currentNotes.items())                          #Get event history of all current notes
        pressureList = map(lambda noteEvents:
                           list(filter(lambda event: event["eventType"] == MIDI_KEYAFTERTOUCH,
                            noteEvents)), pressureList)                             #Only select pressure
        pressureList = filter(lambda eventHistory:
                              len(eventHistory) > 0,pressureList)                   #Remove empty event history
        pressureList = map(lambda events: events[0], pressureList)                  #Select only last pressure value, todo, average last 4 values?
        pressureList = filter(lambda event:
                              event["timestamp"] > (self.currentTime - self.oldPressure),
                              list(pressureList))                                   #Filter any old pressure values todo, filter 0 values
        pressureValues = list(map(lambda eventDict: eventDict["eventValue"],
                             pressureList))                                         #Reduce list from dictionaries to just pressure values
        if (len(list(pressureValues)) > 1 and inputVel == 1):                       #Use fallback if no presure values
            if (sum(pressureValues)/self.interFactor < (127)):                      #Use sum as the interference factor if less than max, otherwise use largest combinations
                strikeLost = min((sum(pressureValues) / (1 - self.interFactor), 127))#Assume the lost strike value is reduced within the range left from all aftertouch values
            else:
                largestSum = max(sum(sorted(pressureValues[-2:-1])), 127)           #Sum largest two values, with max being 127
                strikeLost = min(largestSum / (1 - self.interFactor), 127)          #Use sum as the lost strike value
            strikeLost = max(pressureValues) if strikeLost == 127 else strikeLost   #Use largest of pressure list if value is 127
            #Calculate probable strike value based on strike lost range, or use number of keys pressed if 1
            if (inputVel== 1 and strikeLost < 10):
                outputVel = self.fallbackValue()
            else:
                outputVel  = min(int(max((inputVel / min(127 - strikeLost, .0001)) * 127, strikeLost)), self.defaultMax)
        else:
            outputVel = self.fallbackValue()
        return outputVel
                             
            
    def fallbackValue(self):
        return min(len(self.currentNotes)* 30, self.defaultMax)
        
    #Pressure interference looks at all notes pressure data to find available space for correct reading. If there is no space left for any reading (e.g. values will always be 1-5)
    #then it will use the average of the largest readings
    def pressureInterference(self, inputPressure):
        otherNotes = dict(filter(lambda note: note[0] != self.callingNoteID,
                                 self.currentNotes.items()))                        #Filter out calling note from current notes
        pressureNotes = dict(filter(lambda note:
                              MIDI_KEYAFTERTOUCH
                                    in map(lambda event:
                                        event["eventType"],
                                           note[1].eventHistory),
                                            otherNotes.items()))                    #Wrap result into list and map pressure values into average
        pressureAvg = list(map(lambda note:                                         #Process notes into filtered list
                          avg(list(map(lambda pressureEvents:                       #Map filtered list into average function
                                  pressureEvents["eventValue"],                     #Return values of events
                                  list(filter(lambda event:                         #List of events are filtered
                                              event["eventType"] == MIDI_KEYAFTERTOUCH
                                              and event["eventValue"] != 0,         #Only return events of non-zero value and aftertouch
                                         note[1].eventHistory))[0:4]))),            #Return last 5 values matching the filter
                              pressureNotes.items()))                               #Process the map over the dictionary
        pressureLost = 0                                                            #Initialize the lost pressure value
        if (len(pressureAvg) > 0):                                                  #Only proceed if list is more than 0 items and sum is more than value ceiling
            for idx, pressureVal in enumerate(sorted(pressureAvg)[::-1]):           #Loop through list largest to smallest
                pressureLost += (pressureVal / self.interFactor) ** (idx + 1)       #Calculate lost space for pressure values, increasing the exponent for smaller (cut off) values
            outputPressure = min(int(inputPressure + pressureLost), 127)
        else:
            outputPressure = inputPressure
        return outputPressure

    #Pressure intertia looks at any notes with an aftertouch dropped flag and no note offs and returns an extrapolated aftertouch based on inertia weight, and a decay time. This is
    #returned in an array of MIDIdata messages to be used by the device output function
    def pressureIntertia(self, MIDIdata):
        pressureList

    #Simple moving average of corrected values
    def smoothAftertouch(self, inputAft):
        #Filter last events to aftertouch updates and look back time
        aftUpdates = list(filter(lambda event:
                                      event["eventType"] == MIDI_KEYAFTERTOUCH
                                      and event["timestamp"] > (self.currentTime - self.lookbackTime),
                                      self.currentNotes[self.callingNoteID].eventHistory))
        #Filter last events to aftertouch corrections and look back time
        aftCorrections =  list(filter(lambda event:
                                      event["eventType"] == AFT_UPDATE
                                      and event["timestamp"] > (self.currentTime - self.lookbackTime),
                                      self.currentNotes[self.callingNoteID].eventHistory))
        #Use the largest list
        usedValues  = aftUpdates if len(aftUpdates) > len(aftCorrections) else aftCorrections
        
        outputAft = int(avg([*list(map(lambda event: event["eventValue"], usedValues)), inputAft]))
        return outputAft

        
    def processMIDI(self, MIDIdata):
        self.currentTime = MIDIdata["timestamp"]
        self.callingNote = MIDIdata["note"]
        self.callingChan = MIDIdata["channel"]
        self.callingEvent = MIDIdata["type"]
        self.callingNoteID = f'{MIDIdata["note"]}{MIDIdata["channel"]}'
        if (self.callingEvent == MIDI_NOTEON):
            self.callingValue = MIDIdata["velocity"]
            self.currentNotes[self.callingNoteID] = self.noteData(self.callingNote, #Add to note list
                                                                  self.callingChan,
                                                                  self.callingValue,
                                                                  self.currentTime)
            if (len(self.currentNotes) > 1):
                correctedVel = self.strikeInterference(MIDIdata["velocity"])        #Correct strike interference
                if (len(self.currentNotes)>3):
                    correctedVel = self.pressureStrikeInterference(MIDIdata["velocity"])#Correct strike based on pressure
                self.currentNotes[self.callingNoteID].correctVelocity(correctedVel,
                                                                      self.currentTime)#Record corrected velocity
                MIDIdata["velocity"] = correctedVel                                 #Update MIDIdata

        elif(self.callingEvent in {MIDI_KEYAFTERTOUCH, MIDI_CHANAFTERTOUCH}):
            self.callingValue = MIDIdata["pressure"]
            self.currentNotes[self.callingNoteID].updateAftertouch(self.callingValue,
                                                                   self.currentTime)#Update note list
            
        
            if (len(self.currentNotes) > 0):
                #print("Pressure before",MIDIdata["pressure"])
                correctedAft = self.pressureInterference(MIDIdata["pressure"])      #Get corrected aftertouch
                correctedAft = self.smoothAftertouch(correctedAft)                  #Smooth corrected aftertouch
                self.currentNotes[self.callingNoteID].correctAftertouch(correctedAft,
                                                                        self.currentTime)#Record corrected aftertouch
                MIDIdata["pressure"]= correctedAft                                  #Update aftertouch in MIDI data
                #MIDIdata["pressure"] = self.smoothCurrentPressure(MIDIdata)
                #print("Pressure after",MIDIdata["pressure"])
                

        #cc stuff here

        elif(self.callingEvent == MIDI_NOTEOFF):
            del self.currentNotes[self.callingNoteID]                               #Remove from note list
            noteGroups = {}                                                         #Remove from note group
            self.noteCount = len(self.currentNotes)                                 #Update count of playing notes

        else:
            print("Unhandled MIDI message")
            
        return MIDIdata


#Run Window
mainWindow = gui()

I’m completely overhauling the pressure handling in this script; using large changes in aftertouch values and the lack of note off messages to infer that there are other notes changing the value. I decided to take this route after looking at the outputs in the 4x4 matrix values, where I can easily duplicate interference events. There are partial overlaps, where two notes can output pressure at the same time, but it’s far more common for the other notes aftertouch to drop. It turned out my script was accidentally handling this gracefully in the smoothing function, as it basically ignored the default behavior of a newly triggered note to completely override old pressure values (causing it to drop to zero). But ultimately this behavior was unintentional, and it caused a side effect of stale aftertouch values staying in newly triggered notes.

On the other hand, the strike interference correction works really well, where it uses all the pressure values to infer true velocity (e.g. the harder and more notes you press in the same zone, the smaller the range of the velocity of the new note). There is an unfortunate side effect that pressing harder in the same zone will basically reduce any new velocity values to 1, so it just uses a max velocity. It’s okay, but annoying when you repeat the same strikes and they get harder based on pressure across the instrument. This is a hard limitation unfortunately, but I could apply some fuzzing function to make it sound more natural until there is enough room in the zone for a good range of velocities.

There will never be fully accurate inferences of lost pressure or velocity values, but my intention with this script is to make the uncertainty sound natural, and the corrections to fail gracefully. Apologies for not compiling a mac OS executable, but I don’t want to waste anyone’s time managing a bunch of garbage utilities :slight_smile: I’ll continue to post the code until it gets to a more developed point.

1 Like

Okay, I’ve used some of my past findings and current findings to come up with a new model of pressure handling. Basically, each horizontal strip on the Joué can be split into 3-4 sub-zones. Each of those sub-zones seem to have a pressure range of 0-127 values but functionally overlap with adjacent sub-zones with sufficient pressure. This means that a note can have aftertouch interfered with completely or partially in any given zone. For either kind of interference, they can be attenuated by notes within the same zone so it’s possible to infer the real pressure of interfered aftertouch if you split the input over time with frames. It won’t be exact and it will be completely wrong if there are too many notes within a zone or sub-zone; you can’t get blood from a stone.

There is no way to know the vertical location of a note unless you use cc74 in absolute mode. This is too limiting so I’m going to basically guess based on detecting drops. I’m not sure if I can get good performance on doing a linear regression on every input so this is going to be a lot of trial and error, hence the time it’s taken. Hope to provide some code with the next update.

Hey all,

After some false starts I’m back to work refactoring my code and implementing a new model. My various physical modeling efforts (really, just some inertia equations, cylinder-piston displacement algorithms, and least squares regression) were insufficient to explain the behavior. I could intuit what was happening but not mathematically reproduce it.

I had to turn back to the principles behind the circuit operation and I found some valuable information here:

To sum it up, Joué has a cheaper/simpler multiplexer that pulls adjacent zones (cells as they call them) to ground by default. Since there are current limits on the board, the voltage driving the sensed zone drops according to the number of adjacent zones and lack of resistance (e.g. amount of pressure applied on each adjacent zone). Then when driving subsequent zones, they are attenuated by the same principle.

With reasonable approximations to the range of resistance, pressure curve, and algorithm that converts voltage to aftertouch values, it’s possible to “solve” the true values from a single reference frame (i.e. scan down the board) and previous known values. It creates a line which all solutions for pressure are compatible with the given aftertouch values, and using regression, find the most likely magnitude along that line.

For dropped notes, the search space becomes n-dimensional and it becomes extremely difficult to find a reasonable solution. The fallback, as I said before, will use an inertia function to just guess the values and narrow the space. With any luck, a musical approximation will be sufficient, and it will be seamless.

Performing these calculations in real-time might be too slow, or maybe not. If I were a proper programmer I could implement some actual machine learning with some data from you guys but lets not have perfect get in the way of good enough. :slight_smile:

I put all the information out here for the novice instrument builder to learn from the mistakes of this device and possibly some workarounds for other devices (if any). I do not have an ETA for my next revision but I’ll keep you all in the loop. :smiley:

2 Likes

Quick update:

As I’ve said in the past, I’ve used this project to expand my programming skills. Thankfully I hit a wall when designing the script, which allowed me to learn about design patterns. Now that I have a primitive understanding, I’ll sum up the model I’ve been developing:

First a JoueManager object receives a midi message, and decides if it should create a new activeNote or not. When it creates one, it wraps the new object in a midi data adapter for a note manager object (which translates and decides on how to handle midi events) and a node adapter for a circuit manager object. When the note manager updates the midi data, the activeNote decides on state changes and notifies the circuit manager on the change. This is viewed in data of resistance and voltage on that side. The circuit manager then decides whether or not to link nodes together in a circuit, or create a new circuit. This in turn uses the measured voltage/resistance to deduce the true values within the circuit. The circuit updates the nodes, which then notifies the note manager of all the respective changes to the updated midi data. Finally the JoueManager sends out the events of linked notes, solving the interference issues.

The velocity seems to be a bit tougher because I think it uses a capacitor charge time (not sure what to call this in a non-jargon way) to infer literally how fast you hit a note, rather than looking at the pressure (so how fast resistance changes rather than magnitude). I’ll try different models for this but the first version will fudge it.

Sorry folks for the treating of this project so preciously. I really do want to improve my coding skills so I’m doing basically everything the hard way. If you simply knock down the velocity and pressure values you’ll get something good enough without this little application. You just need to play the Joué like it’s a small insect and you’ll get all the dynamics you need :stuck_out_tongue:

I’m very glad that this project is helping you with programming and the logic of problem solving and information processing.

I’m sorry I havent responded much. Truth is I have lots of different MPE controllers so it is quite common for me to get sidetracked by other ones, and to use each device to its strengths rather than weaknesses. And I’ve not run into very many other Joue users online who really want the best MPE out of it, so I’m not too surprised that other people havent joined in this discussion.

1 Like

Yeah; mostly it’s just people complaining and ostensibly setting it aside (for good reason). If you have a stable of instruments there is no reason to restrain yourself to just one of them. However, as long as there is one person that might use this script then I’ll be satisfied with this work :slight_smile:

If nothing else, my hope is the code may be salvaged by folks for other random projects!

Just a quick update, I’m still here. I built a pretty decent framework in Idle but then got sloppy and didn’t save it as I was used to auto-saving in my other IDEs so it’s gone. I haven’t made any progress since but I’ll be starting up again.