Question for MIDI builders: Arduino frame loss issue

[caj]

2 hours ago, RAc said:

yes, actually, that appears to be a fairly hot track to follow up on as the problem is definitely related to a rapid succession of messages being sent out. Also, I added the code that adds the very short microdelay directly after transmitting each message, and I have not seen those packet losses since, so it may be related to back-to-back messages.

 

I have also realized that the problem aggrevates when I sample the key board less frequently. This also hints at the possibility that too narrow a spacing between outbound messages may be part of the problem, as then there are more messages to be pumped out in a single loop invocation. As you correctly observe, The Arduino does not have to do a whole lot; in the loop, I basically scan the sensor, then the keyboard, and as I have all of the information about velocity and keys to send, I pump out all resulting messages in a loop. It looks as if that is a contributing factor to the issue.

 

Thanks again for the input!  

 

That sounds like exactly the problem I had with an SPI library for an NFC reader.  Sending two messages in immediate succession caused a problem because the library didn't have a baked-in delay after a message was sent, and so a signal pin would bang-bang down and back up too quickly for the reader to catch that a new message was coming.  

 

Your experience led me to review my own code, because a Teensy should be more than fast enough to experience the same problem.  It turns out that when I process an Anglo bellows change I have to note-off everything that's playing in one direction and note-on everything in a second direction, and that should cause missed notes.  However, I still have Serial debug messages being printed after each note on message, which produces enough of a delay.  I guess I should replace them with a delayMicroseconds when I take them out.

[RogerT]

On 11/25/2024 at 6:23 PM, caj said:

However, I still have Serial debug messages being printed after each note on message, which produces enough of a delay.  I guess I should replace them with a delayMicroseconds when I take them out.

Sounds like a workaround.

BTW..fascinating thread. 
A year or two ago someone was selling midified melodeons in the UK, but the s/w was seriously flaky, with errant notes sounding and odd behaviours (on my test rig, at least). I was never allowed to look at the code. This has piqued my interest. I don't particularly like midi instruments…but the underlying ideas and s/w are interesting (yeah, retired programmer here). 
You can correct me but 'realtime' (i was told, for trading systems at least) is stuff that happens within 50ms.

[RAc]

1 hour ago, RogerT said:

You can correct me but 'realtime' (i was told, for trading systems at least) is stuff that happens within 50ms.

From my understanding (I may legally call myself an embedded rt developer w/ close to 30 years of experience in the field), there is no "hard" limit that separates rt from non rt. The only limitation required by real time systems is that there is a ("small") guaranteed cap on turnaround time for real time components.

 

Nevertheless, we probably agree that midi messages *should* in theory easily be handled using the computing power of even smaller computing devices available in 2024. We also agree, I am sure, that the delta between theory and practice exists exclusively in practice, never in theory (adage courtesy of @JimLucas).

[Steve Schulteis]

1 hour ago, RogerT said:

You can correct me but 'realtime' (i was told, for trading systems at least) is stuff that happens within 50ms.

 

"Real-time" simply means that a result must arrive before some deadline in order to be considered correct. It's up to the developer to determine what the deadline should be, and it will depend on the context. 50 ms may be fine for trading (I wouldn't know), but it would be much too slow for generating MIDI events in response to button presses.

[caj]

On 12/3/2024 at 12:56 PM, RogerT said:

You can correct me but 'realtime' (i was told, for trading systems at least) is stuff that happens within 50ms.

 

If we're talking about the latency between the player making a gesture and the instrument firing off a MIDI message, you probably want an overall response less than 20ms, or 1/50 sec.  

 

Every year I have student design teams designing unusual MIDI instruments with all manner of weird sensors with different latencies, and in general we find that responses lower than 1/50 sec start to feel muddy, and 1/100 sec is well below the perceptible latency of a realistic musical interface.  Of course, this depends on the specific kind of gesture you're measuring, be it a key press or bowing or whatnot, the travel of the key, etc.

 

So for example, the HX711 chip commonly used to read load cells has a 10Hz sampling mode and an 80Hz mode, and you absolutely need to have it run in 80Hz mode.   80Hz doesn't break any speed records, but it's more than fast enough for simulating bellows very responsively.

 

[RAc]

9 hours ago, caj said:

 

If we're talking about the latency between the player making a gesture and the instrument firing off a MIDI message, you probably want an overall response less than 20ms, or 1/50 sec.  

 

Every year I have student design teams designing unusual MIDI instruments with all manner of weird sensors with different latencies, and in general we find that responses lower than 1/50 sec start to feel muddy, and 1/100 sec is well below the perceptible latency of a realistic musical interface.  Of course, this depends on the specific kind of gesture you're measuring, be it a key press or bowing or whatnot, the travel of the key, etc.

 

So for example, the HX711 chip commonly used to read load cells has a 10Hz sampling mode and an 80Hz mode, and you absolutely need to have it run in 80Hz mode.   80Hz doesn't break any speed records, but it's more than fast enough for simulating bellows very responsively.

 

Thanks, caj, here is the math from the other side:

 

Scottish reels (that is sort of like a benchmark test) are frequently played at 240 bpm. Apt players can manage to squeeze in (pun intended) at least 2 notes per beat (not even considering ornamentation), so we are talking 8 notes per second (240/60 ~ 4 notes per second times two) at minimum. Each note played causes two events to happen - note on and note off. So we are talking ~62ms (1/16 of a second), in which a MIDI event needs to be processed (see below). For a MIDI concertina, we also need to take into account that both hands can play simultaneously, so we are down to potentially < 30ms.

 

Which brings up back to the original question - because "processing" does not only include the sampling of the buttons and forwarding the event to the host, but THE ENTIRE chain up to the speakers must be able to handle that throughput. That means that the 30ms include the travel of the MIDI events over USB to the host and the rendition of the event to the host's audio engine.

 

I have tested several setups for USB hosts, and I am fairly confident by now that the packet loss is to a high degree owed to a bottleneck on the USB host side. On my Acer laptop running Fluidsynth as the USB host, I have decent latency and practically no packet losses (albeit with the delay included in the USBSend command). Against a Windows tablet running Simplepiano, playing is practically impossible; as I press and release a button reasonably fast, I hear the corresponding note only after the release. (Note that this in not an inherent problem of Windows but rather the framework WUP which is not compatible with real time applications). Interstingly enough, there do not appear to be too many packet losses here.

 

It becomes really interesting when I use a Raspberry 5 (running Linux) as the USB host. The nice thing aboout doing that is that I can turn off all unwanted features on the Pi, so the machine does not do anything but pump incoming MIDI messages to audio.

 

The Pi 5 does not have audio hardware but 4 USB connectors and two HDMI connectors. When I use HDMI audio, there is decent latency and occassional packet losses. When I use a good USB headset, it is slightly worse, but when I use a cheapo USB-audio adapter, the turnaround is worse than with the Windows/WUP setup - I get both a very high latency AND packet losses galore. So apparently the USB controller of the Pi shares the CPU between the USB Phys, meaning that you share USB between the MIDI input and the audio output which cumulates the turnaround times.

 

Sorry for being too technical, but the sobering summary of these findings is that if you want a decent and halfway realistic MIDI instrument, you need to ensure that the entire chain from the button press to the audio output meets the deadlines computed above. Thus, you can expect the most reliable MIDI concertina to be one where the entire chain is realized in the intrument itself (ie the audio is part of the instrument with no comm interface in between); the second best is one where all the components are under your control.  

[alex_holden]

56 minutes ago, RAc said:

The Pi 5 does not have audio hardware but 4 USB connectors and two HDMI connectors. When I use HDMI audio, there is decent latency and occassional packet losses. When I use a good USB headset, it is slightly worse, but when I use a cheapo USB-audio adapter, the turnaround is worse than with the Windows/WUP setup - I get both a very high latency AND packet losses galore. So apparently the USB controller of the Pi shares the CPU between the USB Phys, meaning that you share USB between the MIDI input and the audio output which cumulates the turnaround times.

 

How about trying an audio DAC hat that uses I2S rather than USB? I would also be tempted to use hardware UARTs on both ends to send old-fashioned raw MIDI, bypassing the two USB stacks in the MIDI chain.

 

With a bit of Googling I found someone talking about getting lower latency audio on the Pi using a custom firmware based on Circle rather than Linux. Here is an example of someone who already built a MIDI synth this way:

https://github.com/dwhinham/mt32-pi

[RAc]

Thanks for the investigation, Alex, much appreciated!

 

I guess UARTs would work, but USB already provides a built-in interface to relay MIDI messages. If I did decide to go UART, I would need to write the audio conversion layer on the peer device myself or try to integrate existing conversion libs. Either sounds like an awful lot of work which I do not feel like investing into a hobby project (I already write software for my daily income, so in my spare time, I'd rather practice concertina than write even more software, the project already has eaten up a lot of time). Besides, if I did decide to go down the road, I would probably also abandon MIDI altogether and replace it with a protocol that compresses messages better than MIDI, and that is even more work... also, the bottleneck in usb is not the serial speed (usb 2.0 supports 480kbps which is multitudes higher than you can hope for with a serial UART, so even with a protocol overhead of 1000%, you should be able to pipe a single MIDI message over a usb 2.0 wire in 13 microseconds if my math serves me right).

 

Thanks also for the pointer to the i2s audio interface - I came across that before, and I may need to go that road in case HDMI also proves a dead end.

 

Thanks again for participating in the brain storm, very enlightening!

 

Edited yesterday at 09:00 PM by RAc

[caj]

16 hours ago, RAc said:

Scottish reels (that is sort of like a benchmark test) are frequently played at 240 bpm. Apt players can manage to squeeze in (pun intended) at least 2 notes per beat (not even considering ornamentation), so we are talking 8 notes per second (240/60 ~ 4 notes per second times two) at minimum. Each note played causes two events to happen - note on and note off. So we are talking ~62ms (1/16 of a second), in which a MIDI event needs to be processed (see below). For a MIDI concertina, we also need to take into account that both hands can play simultaneously, so we are down to potentially < 30ms.

 

 

Well, the total delay felt by the user depends on whether these things are read in serial or in parallel, synchronously or asynchronously.

 

For example, if I have a noisy keyboard where I can read all the keys in under 1ms, but I debounce the keys over multiple iterations of the main loop, then I might have a 10ms delay added to every key --- but this won't produce a 30ms delay when the user hits 3 keys.  It will just be 10ms and never more than that, unlikely to be noticed by the player.

 

Likewise, the 80Hz HX711 I'm using is for the bellows measurement, and that doesn't incur any compounding delays for the button presses on either hand.  

 

 

[RAc]

7 hours ago, caj said:

For example, if I have a noisy keyboard where I can read all the keys in under 1ms, but I debounce the keys over multiple iterations of the main loop, then I might have a 10ms delay added to every key --- but this won't produce a 30ms delay when the user hits 3 keys.  It will just be 10ms and never more than that, unlikely to be noticed by the player.

True, but this only applies to notes played simultaneously over overlapping - ie chords. Note also that in this case, the MIDI protocol will enforce distinct messages for each note; there is no such thing as a "chord" message. I agree that for MIDI over USB, the three distinct messages may in this use case (assuming a good technical solution) pass the entire control chain fast enough to not pose a problem.

 

Sorry for possibly not being clear, but my point is this: If a player plays very fast melodic phrases with distinct notes in fast succession, he (or she of course) must expect the entire control chain - ie the key sampling, the generation of the event, the passing of the event to the sound generator and the acoustic rendition of the message - for individual key presses to be completed in less than 30ms to meet the computed deadline.

 

All I can say from my empirical observations is that this is apparently not guaranteed (not even for significantly slower playing; I do not claim to be able to play scottish reels in combat speed reliably). In some manifestations of the control chain (eg with a WUP based MIDI controller) this appears to not hold.

 

I do not think that asynchronous processing is an option that significantly relieves the problem as the async processing must naturally also meet the deadline - the musician can only use the instrument meaningfully if the sound generation starts op stops very shortly (ideally with 0 latency) after the key press.