MIDI-OX User Forum (http://www.midiox.com/cgi-bin/yabb/YaBB.pl)
MIDI-OX >> MIDI-OX Bug List >> Sysex bug
(Message started by: Troub on Nov 2nd, 2005, 3:00pm)

Title: Sysex bug
Post by Troub on Nov 2nd, 2005, 3:00pm
Using Sonar 4 pentium 4 2.4 with 1gig ram
Using MIDI-OX and Yoke.
Input USB Turtle beach midi port
routed to Sonar 4
Output sonar four is Yoke 1
Input to Midi-OX is Yoke 1
Output to Korg M1 is Usb Turtle beach midi port

I have sysex files for my M1 stored in sonar 4
They work fine without MIDI-OX
but after hooking it up I can't successfully send a correct file.
There seems to be extreme data corruption.
So I compared the input and output windows of MIDIOX
and wow...

Sysex from Sonar 4 in the input window is different than
the output window...???

I would think the input and output would be identical.
obviously some manipulation of sysex is going on??

Steve :o

Title: Re: Sysex bug
Post by Jamie OConnell on Nov 3rd, 2005, 3:23pm

on 11/02/05 at 15:00:08, Troub wrote:
Sysex from Sonar 4 in the input window is different than
the output window...???


In what way is it different?  You can capture the SysEx output in a 2nd MIDI-OX instance if you attach it to the 1st instance via MIDI Yoke 2.  In the 2nd Instance, Choose View |SysEx...  Then SysEx | Receive Manual Dump...  Then start the dump in SONAR.

Title: Re: Sysex bug
Post by Troub on Nov 3rd, 2005, 11:56pm
I have the following setup:
Korg M1 >> USB Turtle Beach input
USB Turtle Beach Input >> Sonar 4 IN
Sonar 4 Out >> Yoke 1
Yoke 1 >> MIDI-OX In
MIDI-OX Out >> USB Turtle Beach Output
USB Turtle Beach Out >> Korg M1.

When I dump from Sonar 4 the sysex goes out to the Korg and tries to load but there are data corruption errors.

When I look at the input and output windows in MIDI-OX I see the same data coming in and going out but The differences are isolated internal corruptions.

The length's are the same roughly but the internal Hex codes are only similar. They match for 32 bytes or so then some garbage is thrown in... then they match again for awhile.

They are most different at the begining of the dump and more similar at the end but by that time everything is offset incorrectly for the Korg to read correctly.

I assumed from the help files that I could just put MIDI-OX on each side of SONAR to monitor input and output to and from SONAR... I was trying the output side first.

OF COURSE, the sysex dump to the Korg works fine untill I put Midi-OX into the mix.

Is it possible to contact you directly and discuss?

Steve

Title: Re: Sysex bug
Post by Jamie OConnell on Nov 4th, 2005, 11:52pm

Quote:
They match for 32 bytes or so then some garbage is thrown in... then they match again for awhile.


What type of garbage? Is it illegal SysEx data (outside the range of 00 - 7F) or just packaged differently from the input buffers? MIDI-OX might package the data differently when it is sending it to the output port. MIDI Yoke often will package data differently as it passes it along. It should always be the same data in the same order, but there are no rules as to how much or how little data should be placed into a SysEx buffer (obviously it shouldn't overflow). This certainly is true when input buffers are sized differently from output buffers as well. Differing MIDI Interface driver manufacturers will package SysEx messages differently as well.  

If you have some short examples of mismatched input vs. output data feel free to post it in a message here.  You can copy data from either monitor to the clipboard and then paste it into a message.

You definitely should be able to place different instances of MIDI-OX on either side of SONAR.

Sorry, but my current schedule does not permit time even for paid consulting. This forum is an attempt to provide at least some help to people (plus others may benefit).


Title: Re: Sysex bug
Post by Troub on Nov 9th, 2005, 12:33am
The Data is in the proper sysex range but in hex it's actually different. I could understand if it were just nulls that were interjected due to packaging and timing issues but the data is actually altered, and of course my M1 won't accept the data when it has come through the MidiOX. I'll post some samples in the next one but they're LARGE so I'll try and get excerpts that START at the same location and SHOULD be the same. I don't really NEED help or MIDIOX. just thought that it was strange and you would be interested if it is a bug.

Title: Re: Sysex bug
Post by Troub on Nov 9th, 2005, 12:38am
TIMESTAMP IN PORT STATUS DATA1 DATA2 CHAN NOTE EVENT              
00013929   1  --     F0  Buffer:  8192 Bytes   System Exclusive      
SYSX: 00 63 63 35 00 22 00 01 00 00 00 00 00 00 00 00 00 00
SYSX: 63 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 32
SYSX: 00 00 00 00 00 00 00 63 00 63 00 63 00 00 00 00 00 00
SYSX: 53 6F 00 66 74 20 48 6F 72 6E 10 73 00 00 5B 7F 00 00
SYSX: 00 00 00 00 60 40 00 00 00 60 40 00 00 00 03 00 00 00
SYSX: 04 02 00 09 00 0C 00 00 0B 00 28 28 14 14 00 00 00 1F
SYSX: 63 0D 03 00 00 0F 00 03 04 21 00 16 10 37 2E 00 00 7B
SYSX: 05 02 00 00 00 00 00 63 63 26 02 36 58 1E 00 00 63 02
SYSX: 00 63 2F 08 39 08 36 1C 00 32 36 00 41 00 00 05 00 63
SYSX: 54 54 55 53 0A 00 00 00 00 00 00 00 00 00 00 00 00 00
SYSX: 00 63 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SYSX: 32 00 00 00 00 00 00 00 63 00 00 63 00 63 00 00 00 00
SYSX: 00 00 48 65 6C 6C 73 42 00 65 6C 6C 73 01 00 29 01 7F
SYSX: 15 00 00 00 24 60 00 42 00 00 60 42 00 00 00 00 00 00
SYSX: 00 00 02 00 00 06 00 0C 00 0A 00 19 40 19 1E 1E 00 00
SYSX: 1F 34 00 00 50 0A 68 01 00 00 00 1E 00 28 37 2E 00 00
SYSX: 03 7B 1D 10 00 63 34 63 00 00 00 00 00 00 4A 00 00 00
SYSX: 00 00 63 63 63 63 00 63 63 00 40 00 00 00 00 00 00 00
SYSX: 63 0D 51 00 00 34 2F 00 00 00 00 00 10 00 00 00 62 71
SYSX: 00 00 04 00 42 37 4C 00 00 00 00 00 63 5E 63 58 00 50
SYSX: 00 63 40 00 00 00 00 00 00 00 63 58 00 00 00 4C 00 00
SYSX: 00 00 00 44 72 6F 00 70 20 20 20 20 20 20 08 00 20 34
SYSX: 7F 00 00 00 00 00 00 7C 41 00 00 60 00 41 00 00 00 00
SYSX: 00 00 00 00 02 12 03 00 21 00 00 06 02 0A 0A 13 13 00
SYSX: 08 00 3F 04 48 00 00 00 40 00 0B 0C 3A 00 00 27 00 63
SYSX: 00 00 00 00 59 00 00 00 00 00 00 00 00 75 00 00 63 00
SYSX: 63 62 00 63 00 63 63 63 00 52 00 59 00 3E 00 55 00 63
SYSX: 00 63 00 4F 00 50 00 01 00 41 01 08 01 00 00 00 00 00
SYSX: 00 00 00 00 63 00 00 00 00 00 00 00 00 00 00 00 00 00
SYSX: 00 00 00 32 00 00 00 00 00 00 00 63 00 63 00 00 63 00
SYSX: 40 40 40 00 00 5A 65 70 68 79 72 20 40 20 20 00 01 00
SYSX: 2E 7F 00 1F 00 00 00 15 60 3D 00 00 00 60 3D 00 00 00
SYSX: 00 00 00 00 00 02 00 06 00 00 07 00 02 17 32 32 00 32
SYSX: 32 00 00 1F 21 00 12 14 2A 45 00 7F 00 4E 00 00 00 00

Title: Re: Sysex bug
Post by Troub on Nov 9th, 2005, 12:38am
Ok, Now from the Output window

TIMESTAMP IN PORT STATUS DATA1 DATA2 CHAN NOTE EVENT              
00013929   1   1     F0  Buffer:  8192 Bytes   System Exclusive      
SYSX: 66 06 54 4A 60 6C 06 22 6C 7A 08 02 60 6C 64 04 64 78
SYSX: 11 50 03 72 78 50 05 4A 33 04 56 02 60 04 64 02 66 64
SYSX: 10 7E 03 56 10 7E 6C 07 72 10 40 03 22 12 1D 7A 0B 10
SYSX: 18 72 04 56 33 1C 62 05 4A 1C 40 04 67 22 20 7A 03 56
SYSX: 28 7C 4C 04 56 28 66 03 72 28 1D 58 05 22 30 74 02 56
SYSX: 3B 34 50 04 22 38 72 02 02 56 40 70 00 00 00 60 28 00
SYSX: 00 00 3C 0D 10 18 55 3C 07 10 24 3C 06 10 2A 30 3C 12
SYSX: 10 48 3C 05 55 10 54 3C 04 10 60 3C 6A 12 10 78 3C 04
SYSX: 10 04 5D 3C 04 10 10 3C 10 10 3B 28 3C 06 10 34 3C 04
SYSX: 03 10 40 70 00 00 F7 00 00 00 00 00 00 00 00 00 00 00
SYSX: 00 63 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SYSX: 32 00 00 00 00 00 00 00 63 00 00 63 00 63 00 00 00 00
SYSX: 00 00 48 65 6C 6C 73 42 00 65 6C 6C 73 01 00 29 01 7F
SYSX: 15 00 00 00 24 60 00 42 00 00 60 42 00 00 00 00 00 00
SYSX: 00 00 02 00 00 06 00 0C 00 0A 00 19 40 19 1E 1E 00 00
SYSX: 1F 34 00 00 50 0A 68 01 00 00 00 1E 00 28 37 2E 00 00
SYSX: 03 7B 1D 10 00 63 34 63 00 00 00 00 00 00 4A 00 00 00
SYSX: 00 00 63 63 63 63 00 63 63 00 40 00 00 00 00 00 00 00
SYSX: 63 0D 51 00 00 34 2F 00 00 00 00 00 10 00 00 00 62 71
SYSX: 00 00 04 00 42 37 4C 00 00 00 00 00 63 5E 63 58 00 50
SYSX: 00 63 40 00 00 00 00 00 00 00 63 58 00 00 00 4C 00 00
SYSX: 00 00 00 44 72 6F 00 70 20 20 20 20 20 20 08 00 20 34
SYSX: 7F 00 00 00 00 00 00 7C 41 00 00 60 00 41 00 00 00 00
SYSX: 00 00 00 00 02 12 03 00 21 00 00 06 02 0A 0A 13 13 00
SYSX: 08 00 3F 04 48 00 00 00 40 00 0B 0C 3A 00 00 27 00 63
SYSX: 00 00 00 00 59 00 00 00 00 00 00 00 00 75 00 00 63 00
SYSX: 63 62 00 63 00 63 63 63 00 52 00 59 00 3E 00 55 00 63
SYSX: 00 63 00 4F 00 50 00 01 00 41 01 08 01 00 00 00 00 00
SYSX: 00 00 00 00 63 00 00 00 00 00 00 00 00 00 00 00 00 00
SYSX: 00 00 00 32 00 00 00 00 00 00 00 63 00 63 00 00 63 00
SYSX: 40 40 40 00 00 5A 65 70 68 79 72 20 40 20 20 00 01 00
SYSX: 2E 7F 00 1F 00 00 00 15 60 3D 00 00 00 60 3D 00 00 00
SYSX: 00 00 00 00 00 02 00 06 00 00 07 00 02 17 32 32 00 32
SYSX: 32 00 00 1F 21 00 12 14 2A 45 00 7F 00 4E 00 00 00 00
SYSX: 00 00 03 00 01 64 39 00 00 39 00 08 00 00 63 3C 00 00
SYSX: 00 00 00 00 00 63 59 00 00 00 00 63 00 37 48 00 1C 00

Title: Re: Sysex bug
Post by Jamie OConnell on Nov 9th, 2005, 3:40pm
Nulls definitely mean something in SysEx and cannot be inserted or sorted without changing meaning.

As you say, theses are excerpts and not the full dumps. So they are taken rather out of context. The second one is longer than the first but neither are the full 8192 bytes. The second one shows an imbedded F7 followed by something other than F0, so that is indicative of a problem of some sort. I'd have to see the full dumps to really get any sort of handle on what's going on, but my guess is that there is some unexpected data in the input buffers that is confusing MIDI-OX.

Title: Re: Sysex bug
Post by Troub on Nov 9th, 2005, 8:42pm
These were both starting at the beginning, not out of context. I just grabbed the first section of both. However they are too long to post here... they are part of a 50-60KByte Total system load for an M1.
Nothing prior...
I can email you complete dumps if you're curious. Just let me know where to send them.


Title: Re: Sysex bug
Post by jwpgroup on Dec 11th, 2006, 6:54am
I'm pretty sure that there is a bug with Sysex and Midi-USB (Midi-OX 7.0.0.365). The problem manifiests itself as a data corruption in the outgoing data stream which inserts a '00' every 240 bytes followed by the next two bytes being shifted and the third 'lost'. Choosing packets of 240 bytes in length shows just this corruption. Packets of a different length result in data loss as well.

We are currently working as a third-party developer coding Midi-USB routines and have used Midi-OX as part of our testing (it's great BTW!). This bug has taken us a couple of days to pin down as we where convinced it must be our software/hardware - but it isn't - it's Midi-OX.  

We finally got to the bottom of the problem by sending a file from Midi-OX and crosslinking the real world Midi I/O on our product (so the file would return to another version of Midi-OX). We thought it was our hardware (so we checked further), but the same data corruption problem shows with two other midi-usb converters we have here (from other manufacturers), so our hardware is fine.

To finish the test we then sent the file using Sysex97 (a simple utility we found on the web) and the outgoing dump was not corrupted in any way. QED (unfortunatly) Midi-OX corrupts outgoing USB-Midi Sysex data.

Take a Midi-USB converter and couple the input to the output and then send a Sysex file  to see the issue.

Let me know if I can help with further diagnosis.

John



Title: Re: Sysex bug
Post by Jerry Jorgenrud on Dec 12th, 2006, 2:18pm
Here's how I test these sorts of things.

MIDI-OX:1 -> hardware out A -> MIDI Cable -> Hardware In B -> MIDI-OX:2

Open the Sysex view in OX:2 and select Sysex/Receive Manual Dump.

Open the Sysex view in OX:1 and select File/Send Sysex File.

Back in OX:2 when the file is finished I OK the manual dump box and the received sysex appears in the Display Window. In the Command window I load the same sysex file. If the byte count is not the same then something is wrong. If the byte count is the same then I run Sysex/Compare Windows to make sure the bytes are identical and not just the same count.

If there seems to be an issue I'll couple OX:1 to OX:2 with MIDIYoke and run the same tests. If MIDI-OX has an issue it should show up there as well as with a hardware port.

For an even more stressful test I'll use only one instance of MIDI-OX so it is both sending and receiving at the same time.

I do this with a variety of test sysex files, from short 7 byte files to large files in excess of 100k. Some of the files have multiple sysex strings inside and some are single sysex. The largest single sysex is about 35k. The largest file takes about 10 minutes to make the round trip.

To test your issue I used a USB MIDI port with output cabled to input. I usually run 256 byte buffers but you felt 240 was a magic number so I tried that as well as the 256 byte ones. I tried running with and without a delay after F7 and with a delay between buffers in settings ranging from 1 to 60 milliseconds. (Less than 1 and my interface chokes.) I also tried using only one instance of MIDI-OX.

All of the files were sent and received without incident.

You don't mention that you're using MIDIYoke somewhere in your setup but if you are you should upgrade to the newest version just released. It fixed a rare sysex corruption issue.

Otherwise I'm hard-pressed to offer any suggestions ...

Title: Re: Sysex bug
Post by jwpgroup on Dec 13th, 2006, 5:12pm
Hi Jerry

thanks for that.

This scenario...

"MIDI-OX:1 -> hardware out A -> MIDI Cable -> Hardware In B -> MIDI-OX:2  Open the Sysex view in OX:2 and select Sysex/Receive Manual Dump".

Is exactly what I've done to prove the problem (no other item in the chain), except I'm using external USB-MIDI devices. I'm 100% positive it's a bug as I've now tested it with our hardware, a 'T' box USB-Midi converter and a M-audio MIDIsport. They all fail in exactly the same way with a file sent from MIDI-OX and are completly fine with the same file via Sysex97.

TBH, I wasn't prepared to believe it at first as the company we are working for (a well known name in the MIDI interface field) where convinced they had use MIDI-OX for similar transfers in the past. However... after further discussions on this specific point we concluded that they had used the programme with a computer with a internal plug-in midi port and not over USB, so that must be where the problem lies.

"If there seems to be an issue"...

Trust me, this is worth investigation. Three independent class compliant devices (on XP) from different companies can't all be wrong and fail (in the same way), where a different sysex Tx programme (sysex97) has no problem.

The buffer situation... "I usually run 256 byte buffers but you felt 240 was a magic number"...  is only relevent to the loss of data. The first corruption occurs at 240 bytes in (& then on) and sending data in chunks of this length seems to stop bytes getting 'lost' at the corruption point (the file is still corrupted though). Just so you know, with buffers the same as the corruption length the timing delay can be zero and you still get all the file (well you do on our kit & the 'T' box - the MIDIsport seems to need time to think!).

As covered above, I'm not using anything but MIDI-OX and a single channel converter with the output coupled to the input. Bear in mind that this can be one of three different makes (two of which are on the market) , so I don't think it's a hardware issue.  

Here is the first bit of the file....

Sent first;

F0 00 20 13 0B 7F 6E 18 70 1F 65 18 70 1F 5D 65 18 70 1F 65 18 70 3B 1F 65 18 70 1F 65 00 76 00 20 61 18 70 1F 65 0E 18 70 1F 65 28 06 00 22 40 40 00 00 40 64 00 00 00 40 48 01 00 40 70 00 01 00 40 00 00 00 00 10 08 01 00 40 18 01 00 22 40 40 01 00 40 60 01 00 00 40 00 02 00 40 20 00 02 00 40 40 02 00 40 10 60 02 00 40 00 02 00 22 40 20 02 00 40 40 02 04 00 40 60 02 00 40 00 00 03 00 40 20 03 00 40 00 40 03 00 40 60 03 00 22 40 00 03 00 40 20 03 44 00 40 40 03 00 40 60 00 03 00 40 00 04 00 40 00 20 04 00 40 40 04 00 00 40 60 04 00 40 00 00 00 00 00 00 00 00 00 00 08 00 00 00 00 04 00 40 01 20 04 00 40 00 00 00 20 00 00 00 00 00 40 04 04 00 40 60 04 00 40 00 48 05 00 40 7F 5F 2D 69 08 00 10 4F 61 02 00 2D 59 69 4C 04 1F 65 4C 64 2B

Received;

F0 00 20 13 0B 7F 6E 18 70 1F 65 18 70 1F 5D 65 18 70 1F 65 18 70 3B 1F 65 18 70 1F 65 00 76 00 20 61 18 70 1F 65 0E 18 70 1F 65 28 06 00 22 40 40 00 00 40 64 00 00 00 40 48 01 00 40 70 00 01 00 40 00 00 00 00 10 08 01 00 40 18 01 00 22 40 40 01 00 40 60 01 00 00 40 00 02 00 40 20 00 02 00 40 40 02 00 40 10 60 02 00 40 00 02 00 22 40 20 02 00 40 40 02 04 00 40 60 02 00 40 00 00 03 00 40 20 03 00 40 00 40 03 00 40 60 03 00 22 40 00 03 00 40 20 03 44 00 40 40 03 00 40 60 00 03 00 40 00 04 00 40 00 20 04 00 40 40 04 00 00 40 60 04 00 40 00 00 00 00 00 00 00 00 00 00 08 00 00 00 00 04 00 40 01 20 04 00 40 00 00 00 20 00 00 00 00 00 40 04 04 00 40 60 04 00 40 00 48 05 00 40 7F 5F 2D 69 08 00 10 4F 61 02 00 2D 59 69 00 4C 04 65 4C 64 2B

The seventh byte from the end shows the corruption.

Hope this helps!

Let me know if I can assist further with the diagnosis.

All the best

John


Title: Re: Sysex bug
Post by Jerry Jorgenrud on Dec 15th, 2006, 7:00pm
Hi John,

To follow up, you said:

[I did everything you did] "except I'm using external USB-MIDI devices"

and

"they had used the programme with a computer with a internal plug-in midi port and not over USB, so that must be where the problem lies"

It seems you missed the part where I said:

"To test your issue I used a USB MIDI port"

In fact, like you, I used an M-Audio MIDISport. I don't see how it can be a USB issue anyway. Applications don't do anything different, they don't even know if the MIDI interface is USB, serial, PCI, joystick, virtual, whatever. I spent a couple more hours on three machines trying to get corrupted data out of the MIDISport but it just isn't happening.

I had never heard of SysEx97 but I found that and all I can say is that shows you how simple it really is to send a sysex file. I poked around a bit and noticed that he uses what I would call a large amount of large buffers -- 30 of them at 8192 bytes if I'm right. Maybe MIDI-OX would work better in your situation if you sized the buffers bigger.

You also might try just "Actions -> Send -> SysEx File" rather than loading the file up in SysEx View. That brings the complexity all the way down to ReadFile, MemCopy, here's your buffer.

I didn't grasp this at first but recently started to wonder if perhaps when you say "USB-MIDI" you mean the type of USB-MIDI that is only implemented on Win XP (partially) and that MS says you don't need to write your own USB drivers for? I suppose that's what you mean when you say "class compliant"?

There are eight black boxes between my side:

http://msdn2.microsoft.com/en-us/library/ms789663.aspx

and yours (scroll down)

http://msdn2.microsoft.com/en-us/library/ms789375.aspx

Title: Re: Sysex bug
Post by jwpgroup on Dec 16th, 2006, 6:16am
Hi Jerry

I'm honestly not trying to be antagonistic here :-)

The problem we have (which is 100% repeatable) was first raised by our customer during his beta testing (different setup, different computer, different location). He sees exactly what I see.

Between the two of us (we provided the USB-MIDI microcontroller code - compliant to the USB-MIDI 1.0 spec, he designed the hardware & wrote all the other code), it took us three days to establish that the problem (for both parties) lay in the sysex Tx from MIDI-OX (Rx is fine).

Initially, like you and our customer, I was not prepared to believe it was a MIDI-OX issue, because we all think it's really great (honestly). Hand on heart, we could not have done this dev work without this application. So... It had to be our stuff right - after all we where the ones with beta software.

Anyways eventually after a lot of testing and head scratching (& really as a last resort) in a moment of desperation (!) I tried the two other MIDI-USB things we have here (the M-Audio MIDIsport UNO and the Tbox). I really was astonished when they both failed in exactly the same way.

To then backup this testing (& get a working transfer), I searched the web and found sysex97. I completly agree that it's not anything compared to MIDI-OX, but at this point I needed something (pre-written) to do a sysex transfer as a test that our code was not the problem. It worked, so that's why I mention it.

So, let's look at this a bit more...

"In fact, like you, I used an M-Audio MIDISport."

I really don't understand why you are not seeing the failure we are. I'm running XP Pro SP2 on a Dell M60 laptop (in case hardware interaction is the issue). I don't know what our customer was using but I suspect it was a reasonably powered box.  

"I don't see how it can be a USB issue anyway. Applications don't do anything different"

I only raised this because our customer was convinced he had done sysex transfers reliably in the past (with MIDI-OX). The only difference in our new scenario is the use of USB as opposed to a plug-in MIDI port.  

"I had never heard of SysEx97 but I found that and all I can say is that shows you how simple it really is to send a sysex file."

AS I mentioned above, I'm not holding this up as perfect (far from it), solution. It does work here (& with our customer) though, where MIDI-OX doesn't. :-(

"Maybe MIDI-OX would work better in your situation if you sized the buffers bigger."

I tried all that and it made no difference. It doesn't explain why you don't see it with the same hardware.

"You also might try just "Actions -> Send -> SysEx File"

That is what I'm doing.

"I didn't grasp this at first but recently started to wonder if perhaps when you say "USB-MIDI" you mean the type of USB-MIDI that is only implemented on Win XP (partially) and that MS says you don't need to write your own USB drivers for? I suppose that's what you mean when you say "class compliant"?"

Boy (now we've worked on this) do I know why you say that like that ;-)  Our contract was to provide routines to enable a single-chip microcontroller to bi-directionally communicate MIDI over USB to a Windows XP and/or MAC OSX PC without any custom drivers in accordance with the USB Device Class Definition for MIDI devices 1.0. no more, no less.

It's a published ratified spec and we meet it (as does the MIDIsport UNO and the 'T' box). The client did not want a custom driver, they wanted an out of the box solution, which I have to say works really well (apart from this issue with MIDI-OX).

Clutching at straws here, I'll PM you the Sysex dump we are using to see if that fails with you. I'd be happy to discuss this on the phone or do a log-me-in so you can see the problem.

Let me know if you'd like to take this further.

Thanks again

John




Title: Re: Sysex bug
Post by Jerry Jorgenrud on Dec 16th, 2006, 11:50am
" I'm honestly not trying to be antagonistic here"

Well neither am I. There's nothing like a good bug. The sysex code has been undisturbed for so long moths fly out when I open the files.

"I'm not holding this up as perfect (far from it)"

Again I'm not disparaging the program, just trying to point out that it IS a simple procedure.

"I'm running XP Pro SP2 on a Dell M60 laptop"

I'm trying this out for the most part with the same OS and a Dell something or other laptop.

" Boy (now we've worked on this) do I know why you say that like that"

:D I know one fellow who built a "class compliant" USB device that worked perfectly except on Russian and Polish localized versions of XP. So you still can't sleep all that soundly.  :P

But the main reason I bring that up is that the MIDI Sport I'm using requires its own drivers. I'll need to find one of those "class compliant" devices and test it out. As far as I can tell we've eliminated everything else.

"I'll PM you the Sysex dump"

Thanks, and for anyone keeping score that also works fine here.

I'm concerned about what might be going on in those MS black boxes. They might be doing some direct memory access, for example, that makes assumptions about the buffers. But I'll need to get some gear here that doesn't work in order to make any progress on tracking that down.

It would be useful if you could also let me see a failed dump and note the size and number of the buffers if that is at all possible.

Thanks for your report and your willingness to help with the diagnosis.

Title: Re: Sysex bug
Post by jwpgroup on Dec 16th, 2006, 3:30pm
Hi Jerry

"So you still can't sleep all that soundly."

Don't! You can imagine how I felt when this 'bug' surfaced. :o

We delivered what we considered to be final code over a month ago and this came to light only when the final stages of 'in the field reprogramming' where being tested (just before product launch).  The code runs on a flash based microcontroller, so it can be field 'updated' to a later version over sysex (well assuming it contains 100% what you send of course!). The subtle changes that affect the MIDI-OX transfer (but not Sysex97) sure had us foxed for a while ;-)

"the MIDI Sport I'm using requires its own drivers"

Funny you should say that. We had two (outwardly identical) 'T' boxes here when we where in the early stages of coding (when we looked at several manufacturers devices at USB level). One was 'compliant' whilst the other needed drivers, so you need to be sure you get a later version of whatever you buy. I vaguely remember looking at another M-audio interface (which also wasn't 'compliant'), so I didn't analyse it any further. Maybe that is what you have.

At one point I had about half a dozen units from different manufacturers here so I could have given it a much fuller test (although three failures do look pretty conclusive TBH).

"I'll need to get some gear here that doesn't work in order to make any progress on tracking that down"

The kit I've definitly got it to fail with is... (based on a web search and using American sites in view of your locale)....

http://www.pgmusic.com/th_tbox.htm

This box seems faster than the M-audio unit and (assuming you use 240 byte buffers (to avoid data loss) will pass the whole file with just the data corruption with 0 delays. I'd get this if you want a simple clean box (unbiased view!)

http://www.zzounds.com/item--MDOUNO

Is the M-audio device I have. Note you don't need to load any drivers on XP (although a disk is supplied for W95/W98/2000).

I've found the M-audio box does drop data when the errors happen, but it maybe that this could be fixed by increasing the delays in MIDI-OX (I knew I'd found the problem at that point so was less bothered about getting a file of the correct length!).

Returning to the 'issue'...

The 'compliant' thing is a big thing here. People don't want the faff of having to load drivers - plug n play USB on OSX or XP is the order of the day. If only everyone knew how flawed and patched this whole USB area is (well in XP anyway)!  they might take a different view ;-)

"I'm concerned about what might be going on in those MS black boxes. They might be doing some direct memory access, for example, that makes assumptions about the buffers"

I think there must be something like that happening and after what we've just been through to get this stuff to work properly I share your concern. ;-)

"It would be useful if you could also let me see a failed dump and note the size and number of the buffers if that is at all possible."

I've PM'd you another file. I just created this, so I can make it happen repeatedly when I want to!

This was Tx/Rx'd with 240 byte buffers and zero delay. You'll see 357 bytes that don't match and the same repeating corruption. Any other buffer length leads to data loss at the point of corruption.

Don't forget that Sysex97 always works (might be some clue). You'll have to ignore the revolting music clip when you run it of course  :P

One (unrelated) thing I should mention is I've had several lock-ups on the second version of MIDI-OX when I compared a received file to one loaded (to check for diffrences). If you see that behaviour you might want to clear the moths from that code area as well ;-)

"Thanks for your report and your willingness to help with the diagnosis."

Hey thanks for looking at it.  The customer wants to recommend MIDI-OX for in-the-field sysex code changes, so I have to admit to a certain degree of 'need' in my request for you to find the bug.  It will help us all in the long run.

Happy hunting (& let me know if I can do anything more).

All the best

John

Title: Re: Sysex bug
Post by Jerry Jorgenrud on Dec 16th, 2006, 7:13pm
Thanks for the kit info. It looks like a late-model T box for me.

" One (unrelated) thing"

I'll keep an eye out. I hate to keep throwing MS under the bus but we used their rich edit control and have lived to regret it. It works well with the moderately sized data you normally run into but anomalies develop at higher loads. Fortunately we've gotten away with it so far because most people's requirements are light enough to not be an issue. But I wish it worked better and at times it's embarassing to see the control struggling. On the other hand I never wanted to write a full-featured edit control from scratch and still don't.

"It will help us all in the long run"

We live to serve has always been our motto here  8)

Title: Re: Sysex bug
Post by Jamie OConnell on Dec 17th, 2006, 7:30pm
FWIW, I can't cause a failure with the new MIDI Yoke at all, doing transfers, even when specifying 0 delays in MIDI-OX.
I created a SysEx message from the excerpt above and tagged on an F7 to the end (248 Bytes) and then copied it 7 times.

What sort of delays were you using in MIDI-OX?
Configuration:

MIDI-OX (1)->MIDI Yoke 2->MIDI-OX (2)

[SysEx|Receive Manual Dump...]

Settings
(For Both MIDI-OX instances)
Delay [ 0 ] Milliseconds between buffers
[x] Delay after F7 [ 0 ] Milliseconds
Input Buffers: [ 256 ] Bytes x [ 16 ] Buffers
Output Buffers: [ 256 ] Bytes x [ 16 ] Buffers

I also cannot get the EDIROL UM-880 (USB-MIDI) to fail:

MIDI-OX (1)->[2:UM-880]->cable->[UM-880:1]->MIDI-OX (2)

Settings
(For Both MIDI-OX instances)
Delay [ 0 ] Milliseconds between buffers
[x] Delay after F7 [ 0 ] Milliseconds
Input Buffers: [ 256 ] Bytes x [ 16 ] Buffers
Output Buffers: [ 256 ] Bytes x [ 16 ] Buffers

I then copied the message 8 more times (17856 bytes) and sent that. At this point I needed to insert a 10 millisceond delay between buffers to avoid a WINMM "driver busy" message: "[x] Delay after F7 [ 10 ] Milliseconds".

Regardless, I always received exactly the same data as sent (Copying and Pasting the original MIDI message and using the Compare Windows function).

It's possible that all of the USB devices you tested use the same driver code (and or circuits), so it wouldn't be that surprising if several of them exhibited that same failure. It remains to be explained why this SysEx97 program doesn't fail? I wonder what sort of delays or processing overhead it might insert to allow these USB devices to function correctly.


Title: Re: Sysex bug
Post by jwpgroup on Dec 17th, 2006, 11:30pm
Hi Jamie

"What sort of delays were you using in MIDI-OX?"  

With two of the three tested units (ours & the Tahong 'T' box) no delay is necessary (as long as you use a buffer length of 240 bytes). With these two items you get the entire file but with three byte (100% repeatable) corruptions every 240 bytes.

Different buffer lengths cause data loss when the corruption occurs, irrespective of delay setting.

The third test unit (the M-audio unit), finds it hard to keep up and a delay is necessary.

"I also cannot get the EDIROL UM-880 (USB-MIDI) to fail"

Correct me if I'm wrong ;-), but this unit requires a specific driver written by Roland, so I'm not surprised it works.  Jerry & I currently think this is a Microsoft 'class compliant' - MIDI-OX interaction issue, so you need a product that doesn't need drivers but depends on the OS.
 
"It's possible that all of the USB devices you tested use the same driver code (and or circuits)"

That's definetly not the case in the outside world (i.e. leaving aside the OS). This whole 'bug' came to light because we where testing USB 'class compliant' routines which I (and my partner in crime Dave) had personally written, running on a completly different single chip micro to that used in either of the other units. No doubt about that.

Sadly (from a MIDI-OX point of view) this problem has now shown itself on three different micros from three different manufacturers (one of which I'm involved with as a programmer/developer). IMHO, It's fairly heavy proof of a 'bug' when one also considers that there is no issue at all when the code is transmitted with Sysex97.

I think all this points to an interaction 'issue' between MIDI-OX & the 'budled' MS 'class' drivers.

"It remains to be explained why this SysEx97 program doesn't fail?"

That is indeed something to wonder about, but it doesn't.

" I wonder what sort of delays or processing overhead it might insert to allow these USB devices to function correctly."

I've pretty much tried everything I can to get MIDI-OX to work with zero success I'm afraid. It's a really weird bug for sure.  It's not delay related - the corruption is always the same and Sysex97 is way faster than MIDI-OX.

Just so you know, I've PMd Jerry the code examples - let me know if you'd like those as well.

Thanks for looking Jamie. I really appreciate how responsive you guys are being over this.

All the best

John




Title: Re: Sysex bug
Post by Jamie OConnell on Dec 19th, 2006, 1:00pm

on 12/17/06 at 23:30:12, jwpgroup wrote:
Jerry & I currently think this is a Microsoft 'class compliant' - MIDI-OX interaction issue, so you need a product that doesn't need drivers but depends on the OS.

"It's possible that all of the USB devices you tested use the same driver code (and or circuits)"

That's definetly not the case in the outside world (i.e. leaving aside the OS). This whole 'bug' came to light because we where testing USB 'class compliant' routines which I (and my partner in crime Dave) had personally written, running on a completly different single chip micro to that used in either of the other units. No doubt about that.


It sounds like you are saying that these devices don't need drivers?  But that can't be: it is impossible.  All MIDI devices need to use drivers.  It is just a question of whether the driver is written completely by the manufacturer (UM-880), partly by the manufacturer and partly by MS (using the boiler plate 'port class') or completely by MS (still using 'port class').  I still suspect that all devices that have this problem are using the class boiler plate driver code.  

MIDI-OX does nothing unusual with the buffers it ships.  It simply uses midOutLongMessage() to hand the buffers to an open driver (marshalled via MMSYSTEM and WINMM).  Proof of what comes out of MIDI-OX can be seen by the MIDI Yoke loopback tests described above.  Logic dictates that there must be some sort of problem with the MS 'port class' SysEx implementation.

So that I can attempt to reproduce your results could you specify your MIDI-OX SysEx settings as I have done in the previous message?

Title: Re: Sysex bug
Post by jwpgroup on Dec 19th, 2006, 2:37pm
OK...
 
Configuration:

MIDI-OX (1)->External Device (out)- External Device (in) ->MIDI-OX (2)  

Where 'External device' is a unit that ships with no drivers and depends on the USB Device Class definition for MIDI devices 1.0 as supported under XP (i.e. what we could refer to as MS bundled 'class' drivers).


[SysEx|Receive Manual Dump...]

Settings
(For Both MIDI-OX instances)
Delay [ 0 ] Milliseconds between buffers
[x] Delay after F7 [ 0 ] Milliseconds
Input Buffers: [ XXX ] Bytes x [ YY ] Buffers
Output Buffers: [ XXX ] Bytes x [ YY ] Buffers

Where XXX can be any length (although values not=240 cause data loss as well as corruption).

Where YY needs to be great enough for the buffers not to overflow.

I've P.M'd you the before and after, 100% repeatable result.

"It sounds like you are saying that these devices don't need drivers?"

What I'm saying is the devices do not load drivers of their own per se, instead they interface via the USB audio device class drivers.

http://www.usb.org/developers/devclass_docs/midi10.pdf

"I still suspect that all devices that have this problem are using the class driver code."  Oh yes :-)
 
"Logic dictates that there must be some sort of problem with the MS 'port class' SysEx implementation" ... and the way it makes memory usage assumptions and MIDI-OX interacts within that framework.

The trouble is... we both know who is the more likely to be bothered about identifing (& fixing it). ;-)

John






Title: Re: Sysex bug
Post by Jerry Jorgenrud on Dec 19th, 2006, 4:12pm
Hey Jamie,

I haven't been able to reproduce this problem under any circumstances, as mentioned earlier in the thread, but I do have one of these "class compliant" devices coming so I hope to actually see this thing in person. In the mean time I have sent to John a small sysex sender that I bolted together that uses a different memory management scheme than MIDI-OX. He tells me that it works fine. So now he has two programs that work OK and MIDI-OX that doesn't. :-/ It doesn't seem possible, does it? If you look at the errors in the example John sent they look like corrupt USB MIDI data packets and not anything to do with our buffers. How could they be? Plus there's no question that our code is and has been for a long time completely "legal" and non-controversial. Nevertheless we don't appear to be compatible with whatever MS is doing in their XP USB class driver. I say that with the caveat that I haven't seen this happening for myself to verify. Not that I'm implying doubt, John. ;)

Title: Re: Sysex bug
Post by Jerry Jorgenrud on Dec 21st, 2006, 6:38pm
Hi Jamie and John,

My new "class compliant" MIDISport arrived today and I want to remove all suspense by saying that John is not the total blithering idiot he appeared to be when he showed up at our door, hat in hand (and hand on heart if I remember correctly). I was able to reproduce his problem exactly. To set things up I had my old USB MIDISport cabled connecting input to output. I used two instances of MIDI-OX and pushed data until I was sure there were no problems. I did have to play around with the buffers a little to get things running smoothly on the nastiest tests. My test machine is a Dell laptop I rescued out of the trash. I think the model number probably starts with POS. It's only USB 1.1 and the CPU seems to go an hiatus every once in a while. But I did get things moving and I was satisfied that all of my test sysex was making it through reliably. I unplugged the old MIDISport and plugged in the new one. XP "found" the hardware and then informed me the new hardware was ready for use. No (additional) drivers necessary. I started testing and it was a train wreck right from the start. Significant data corruption. I viewed the files in Hex Workshop which has an industrial strength compare feature and it showed me all the places where data was lost, inserted, and switched around. !!! When I set the buffer sizes to 240 I was able to generate the identical "failed" file that John showed us. Then I ran that little sysex loader I mentioned sending to John and his file came through without problems. One run out of about a dozen or so had a problem. It had dropped three bytes. Since that's the size of the data part of a USB MIDI packet, I'm thinking it was a dropped packet due to the MIDISport or the Dell and not part of the original problem. But it will bear keeping an eye on. So what's the secret? (I saw a couple of you lean forward.) Unfortunately, the solution for the sysex loader may not be the best solution for MIDI-OX. The loader creates a heap and uses HeapAlloc() to get guaranteed contiguous memory (which seems to be the main issue) big enough to fit the whole file and the MIDIHDR, reads the file right in to the mhdr->lpData buffer, prep and bang. One shot. Comes through smooth as silk. Ponder that Mr. O'Connell. :)

Title: Re: Sysex bug
Post by Jamie OConnell on Dec 23rd, 2006, 7:30pm
FWIW, I never thought John was an idiot :D. I always believed it was happening as he described, but I suspected a problem with the MS drivers (and still do) as MIDI-OX works fine with other Device drivers (Yoke and UM-880). I am now leaning towards suspecting it has a problem with where the SysEx buffers are allocated from. As I recall, MIDI-OX uses the original Win3.1 MMSystem requirements: it allocates buffers using GlobalAlloc() with the DDE_SHARE flag. In Win9x this has the effect of allocating SysEx buffers from low memory (All buffers being used must fit within 64K of low unpageable memory). Apparently this is no longer a requirement in Win32 and in fact may be the cause of the corruption, although it is completely legal to still use GlobalAlloc(). We could try using HeapAlloc() (for non-Win9x OS's) and see if it fixes things for these drivers. I imagine it's still important to get memory that can't be paged out, but I am willing to believe almost anything these days ;D

Title: Re: Sysex bug
Post by Jerry Jorgenrud on Jan 4th, 2007, 7:21am
Just a bump/update for anyone following this issue.

To recap, the problem is the transmission of corrupt sysex messages when using a MIDI USB device that does not require an additional driver to be installed when installing it on Win XP. In other words the box probably had the words "class compliant" on it somewhere, you are running it on Win XP, and when you first plugged it in XP did not ask for a driver installation disk. John reported the problem and I was able to duplicate it.

I was also able to duplicate the problem using other sysex utilities such as Bome's SendSX http://www.bome.com/midi/sendsx/ and to verify that the problem is fairly well known even though we had never heard of it here before.

Additionally, I was able to determine that the problem does NOT exist if the size of the output sysex buffer in MIDI-OX is greater than the size of the sysex you are trying to send. In John's case the size of the sysex message -- from F0 to F7 -- was fairly large at around 37k bytes. Setting a buffer length greater than 37k allowed the sysex to be sent correctly. I wasn't able to test this "fix" with other test utilities that don't allow the buffer size to be set by the user, such as SendSX.

We are guessing that the USB audio class driver supplied with Win XP requires that a single sysex message fit in one buffer and will not tolerate splitting a sysex message across buffers. If so, and we are only speculating on what goes on inside the dark module, this is an undocumented requirement on the part of this driver and indeed directly contradicts the documentation regarding sysex buffers. If I had not signed away my right to express my opinion about the performance of MS software as part of a licensing agreement ... I would probably declare this to be a bug in usbaudio.sys, the MS supplied class driver. But since I have signed away those rights, I won't say that.

In the meantime, if I had this problem sending sysex I would increase my sysex output buffer size in MIDI-OX until I stopped having the problem.

A couple of points.

1) Most sysex messages are not very big. SysExFiles that ARE big tend to be composed of many smaller sysex. It's only the bytes between (and including) F0 and F7 -- one sysex message -- that need to fit in one buffer. I have one file that is 96k but the longest individual sysex in the file is only 24 bytes. So 24 byte buffers would work fine with this file.

2) If you do need to set a higher size for the output buffers, type the number in rather than trying to use the "spinner". In addition to the "spinner" problem, the box is not wide enough to see the whole number very clearly if you need to go to five figures. We've never had to do that before. A new version of MIDI-OX will be available shortly that will make this easier.

3) The maximum size you can type in is 65k (65,535). Well you can type a bigger number in but it won't do you any good.

4) The most likely problem you could run into is if the hardware attached to your interface or the interface itself needs a firmware update. These are more likely to be longer sysex and if they are corrupted in their transmission, you might end up creating functionally useless objects.

We're still discussing what, if any, additional steps we might take to work around this issue.

Title: Re: Sysex bug
Post by jwpgroup on Jan 4th, 2007, 12:45pm
;D  ;D ;D Ah ha... forgive me if I now bask in the warm glow of a public declaration by my esteemed collegues that I'm not a blithering idiot!!   ;)

A couple of observations/questions about that last post Jerry...

"I have one file that is 96k but the longest individual sysex in the file is only 24 bytes. So 24 byte buffers would work fine with this file."

Does MIDI-OX 'automatically' swap buffers for each F0-F7 section then? ... If that isn't the case, I'm intrigued how multi F0-F7 combinations make the file pass muster.

(abridged) "the most likely problem occurs during a firmware update where you might end up creating functionally useless objects."

Just an observation on that. It would be extremly unlikely that any manufacturer (worth their salt) would not apply some kind of checksum for such a system critical file (I know we do) and also try (where possible) to load any incoming update into a buffer area before 'copying' it over the old firmware. It's just too risky not to take that approach.

I'm not saying (worse case) that a complete trash of the device can't/won't happen  :), just trying to reassure people that in most cases a failure to update / rejection of the file would be a more 'normal' outcome of the corruption issue.

John








Title: Re: Sysex bug
Post by Jerry Jorgenrud on Jan 4th, 2007, 4:01pm

on 01/04/07 at 12:45:32, jwpgroup wrote:
;D  ;D ;D Ah ha... forgive me if I now bask in the warm glow of a public declaration by my esteemed collegues that I'm not a blithering idiot!!   ;)


;D I guess you didn't see that before ...Well if everyone keeps using the words John and idot so closely together it will start hitting the top 20 in Google :D

" Does MIDI-OX 'automatically' swap buffers for each F0-F7 section then? ... If that isn't the case, I'm intrigued how multi F0-F7 combinations make the file pass muster."

That's a good catch, John. We move to a new buffer at F7 if "Delay after F7" is selected in the configuration options, even if the delay is "0". I neglected to mention that part of making that 96k file work with 24 byte buffers. If "Delay after F7" is not checked, then you can expect the same kind of data corrruption as we are talking about. The point I didn't drive home is that our problem driver :-X is expecting an F0 at the beginning of the buffer and an F7 by the time it gets to the end of it. No more no less. The problem is that while it does need to start waiting for that F7 as soon as it encounters the F0, it needs to sustain that vigil across as many buffers as it takes. And when it doesn't encounter an F7 by the end of the buffer it needs to stop hacking a fur ball.

I'm working with a beta version of the next release and things are in flux in that area. I've already forgotten about that silly old version that you poor folks have to deal with.

"a more 'normal' outcome of the corruption issue"

Once again I didn't make my point, which is that, no matter how much confidence I had in the manufacturer of the device, I would make sure that I could deliver an uncorrupted file before attempting a critical operation like upgrading the firmware. I trust the manufacturer of the air bags in my car, too, but I'm not going to test that trust by slamming into a bridge abutment at high speed if I can help it. :P Of course it's a melodramatic comparison and your point that probably the only thing that would happen is a failed update is a more balanced view. Assuming one values balance then that's a good thing.;)

Out of curiosity, how do you let the user know that the update failed?

Thanks for spotting those two areas that needed clarification. :)

Now then, John is not an idiot, John was never an idiot, we never thought John was an idiot ...

Title: Re: Sysex bug
Post by jwpgroup on Jan 4th, 2007, 6:39pm
"Out of curiosity, how do you let the user know that the update failed?"

TBH, that depends entirely on the product :). If you've got a couple of seven segment displays you might set "FL" (for fail) to the display. If you've just got a bunch of LEDs then you might make them 'lock' into a repetitive flashing sequence or perhaps just turn off all the LEDs once done (correctly). Obviously if you've got something with a LCD display you just say "it's done". Usually the manual defines the procedure.

"no matter how much confidence I had in the manufacturer of the device..."

Actually, confidence or not (!) this is not always a totally 'safe' thing to do :o. The 'typical' approach to this (assuming and that's a big assumption) that you've got enough spare flash memory to use for the task, is to load the entire 'new' firmware to the spare flash area, checksum it (to make sure it's not been corrupted by MIDI-OX (joke) :P ) and then (& only then) do you copy it back over the original firmware.

When you do the actual 'copy over' you need the copy routine to reside and run from RAM because you can't read n' write to the same flash area at the same time. If, during the couple of seconds the copy over process takes, you then trip over the power cord, just be sure to keep the original packing materials to hand, because as you initmated before...

"you WILL end up creating a functionally useless object"

It'll then have to be reprogrammed back at the manufacturer, because you will have trashed it (for further 'in the field' upgrades). :-[

My last post was just clarifying that the file would (normally) be checked before it was used (so one would 'know' if the sysex was corrupt before it was used in vengance).

Caveat... Some of the above explanation can vary. you might be able to go for a simple bootstrap loader that will always be present (so the power glitch would drop you back to that), but you won't be able to pull that off with anything other than a serial interface. With USB, the code is just too verbose (without any product firmware) to have it reside on it's own (in a typical single-chip micro).

I'll stop now before we get lost in techno-babble ;)

You get the idea. :)

John

Title: Re: Sysex bug
Post by Jerry Jorgenrud on Jan 4th, 2007, 7:04pm

on 01/04/07 at 18:39:08, jwpgroup wrote:
You get the idea.  :)

Thanks John, I think now I know exactly what position to assume if I ever decide to do something like update my MIDISport :o
That would be a praying position for you smirkers in the back row.

Title: Re: Sysex bug
Post by jwpgroup on Jan 4th, 2007, 7:07pm
Nah.

Just buy a 'T' box instead. I did tell you it was slow!  ;D

John

Title: Re: Sysex bug
Post by Peter L Jones on Jan 8th, 2007, 2:02pm
(Comment from a non-Windows internals programmer... ;)) I take it the USB stack and the USB-MIDI device definition are both fairly hairy and that implementing an alternative Class Compliant driver to the Microsoft-supplied one would be, eh, "non-trivial"..?

Title: Re: Sysex bug
Post by Jerry Jorgenrud on Jan 9th, 2007, 9:04am
Peter L Jones wrote (Comment from a non-Windows internals programmer... ;))
Jerry writes (Reply from a non-internal Windows programmer, err ... ::)

I don't go near the kernel so it all looks pretty hairy to me :P but my understanding is that there is a lot of your typical voodoo going on down there in regard to interpretations of the USB spec and the MIDI Spec and fitting that into the Windows multimedia architecture, which is starting to look as though it was designed by Gaudi http://www.brodyaga.ru/images2002/Sagrada%20Familia%20DSCN1719_brodyaga_ru.JPG. Not that there's anything wrong with that, of course. It's fascinating to look at but the upkeep is a bit of a headache :-/

I've seen mention of a project known as "tunnelling under usbaudio", trying to keep the good parts whilst working around the less useful ones, which should show you how desperate they are not to start from scratch.

I support the goal of "class compliance". When they get it finished :-X it should be a good thing. It should mean cheaper interfaces that work better because every manufacturer won't have to hire their own surly DDK man for one thing.

I'm confident that we've identified the problem with the sysex corruption, though, and have a way to work around it. A stand-alone solution has passed heavy testing and now it needs to find it's way into MIDI-OX itself. One issue that we haven't really resolved yet, though, is if we do work around the problem, and others do as well, it reduces the pressure on MS to do the right thing and fix the problem :-X which would be a lot better solution than any "patch" we could apply.

Maybe someone else more familiar with internals can address your question in regard to what makes it so fearsome that grown men don't want to go in there and slay this bug.

Title: Re: Sysex bug
Post by TK on Jan 19th, 2007, 10:17pm
I must say that I haven't read the whole thread, and I'm not sure of it's redundant, but maybe my input could be interesting for you.

For the case that you are using a PIC18F4550 or similar derivative for your USB-MIDI interface - forget it! The EUSART peripheral has a bug which leads to the effect, that additional 00 are inserted into the data stream. I've proven this via hardware with a scope, see also http://forum.microchip.com/tm.aspx?m=85120, it's a silicon bug, which has been confirmed by Microchip.

Corrupted SysEx streams when using USB-Midi under Windows: from the user point of view (I've zero windows programming knowledge) this seems to be windows related and heavily timing dependent, the same happens when data is sent from a Java application to a USB-MIDI device, but also USB interfaces with dedicated drivers (like MIDIsport 2x2) are affected. Again I've proven this with a scope to ensure that MIDI-Ox doesn't fool me. This was about 2 years ago, so please forgive me if things have changed in the meantime. However, my workaround for this Java issue is (and please don't laugh!) to use a routing through MIDI-Yoke and MIDI-Ox to overcome this problem

Best Regards, Thorsten.

Title: Re: Sysex bug
Post by Jerry Jorgenrud on Jan 20th, 2007, 10:55am
For those who don't know, Thorsten is the highly respected inventor of the MIDIbox series of DIY MIDI projects ( http://www.ucapps.de/ ) whose interesting hardware always reminds me that I nearly flunked soldering. Here I'll quote Thorsten from his web site to show that he is very familiar with the problems with the USB class driver that we were late to discover and that are the subject of this thread:

The Microsoft driver isn't multiclient capable.
The Microsoft driver isn't able to send one SysEx string which is distributed over multiple buffers without errors.
The Microsoft driver doesn't allow the USB module to change the device name.

Thank you Thorsten. To this list I would add:

Upon closing, the driver emits 32 "sustain off" messages, two for each channel. (It may have been intending to send reset messages?)

Multi-port devices incorrectly enumerate on multi-device systems. When multiple USB MIDI devices are added to a system and each use the class driver, the device names have a number added after the generic name. The generic name is "USB Audio Device" on English language Windows, additional devices become "USB Audio Device (2)", (3), etc. I'm not sure what specific series of events cause this, but we have observed that it's possible to have two devices with the same name -- two "USB Audio Device (2)", for example. It's bad enough with the 2, 3, etc. situation trying to remember which hardware the numbers refer to, but when they can have the exact same name -- that's just shameful.

Aside from this problem with the class driver, I've also observed, although not with a scope, the same phenomena Thorsten describes here regarding spontaneous insertion of extra MIDI data in a stream. I have intimate knowledge of six USB MIDI interfaces now and foiur have exhibited this problem of inserting nulls, while one has the additional problem of dropping bytes and another one inserts other bytes in addition to nulls. The only USB NIDI device that has worked flawlessly  (maybe I just didn't catch it at the right time) is made by Roland.

As Thorsten notes on his web site, Sun have been informed of their Java MIDI bug but seem uninterested in fixing it. MS have also been informed of the class driver bugs and have never acknowledged them. My own belief is that these companies do not actually understand MIDI, and in particular SysEx, and when they get a bug report along these lines, they just throw it in the "crazy kook" pile.

Buyer beware when it comes to things USB MIDI.

Thanks for stopping by, Thorsten




Title: Re: Sysex bug
Post by jwpgroup on Jan 21st, 2007, 5:41pm
"Upon closing, the driver emits 32 'sustain off' messages, two for each channel."

Just to clarify that... it's logical to send a sustain off for each of the sixteen channels, what's illogical is to then repeat the 'sustain off' instead of sending a more sensible 'all notes off' command (as the second batch of sixteen - one per channel).

"The only USB MIDI device that has worked flawlessly  (maybe I just didn't catch it at the right time) is made by Roland."

But not with the MS class driver right?! (just checking!)  ;)

AIUI, everything (that depends on the class driver), fails when you split a SySex file over multiple buffers.  :(

John

Title: Re: Sysex bug
Post by Jerry Jorgenrud on Jan 22nd, 2007, 8:13am
Right, not with the class driver. The Roland was unique in that you could use either the class driver or their own driver (via a switch on the unit). What I meant by flawless was that it didn't do anything else wrong, like insert nulls. But  it still had the problems the others did when using the class driver. And it had no problems at all when using the Roland driver.

Title: Re: Sysex bug
Post by jjones on Jun 25th, 2010, 10:13pm
Korg T2
windows XP
Cakewalk UM-1G USB Midi interface

I've downloaded the T2 sounds from the Korguk.com website to my PC running XP.
I've downloaded the MidiOx software, a new driver for the Cakewalk interface and I've changed (increased and decreased) the buffer numbers as well as tried other mentioned solutions for the program, but still no luck in getting the sounds to dump into the T2.  
All of the combination names have transferred, but no sounds.
Within PROG A, A00 Aeroglide, A01 Piano 16' and A02 Brass 1 sounds have successfully transferred onto the T2.  The remaning sounds look like jiberish ( *    <    /    , ") with no sounds.
I was hoping that someone could send me the "little sysex" file that was sent to John in previous posts.  
Thanks,
Julie
225-755-1925



MIDI-OX User Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB 2000-2003. All Rights Reserved.