Author |
Topic: Sysex bug (Read 48383 times) |
|
jwpgroup
Member
MIDI-OX Rules!
Gender: 
Posts: 12
|
 |
Re: Sysex bug
« Reply #20 on: Dec 19th, 2006, 2:37pm » |
 Quote  Modify
|
OK...
Configuration:
MIDI-OX (1)->External Device (out)- External Device (in) ->MIDI-OX (2)
Where 'External device' is a unit that ships with no drivers and depends on the USB Device Class definition for MIDI devices 1.0 as supported under XP (i.e. what we could refer to as MS bundled 'class' drivers).
[SysEx|Receive Manual Dump...]
Settings
(For Both MIDI-OX instances)
Delay [ 0 ] Milliseconds between buffers
[x] Delay after F7 [ 0 ] Milliseconds
Input Buffers: [ XXX ] Bytes x [ YY ] Buffers
Output Buffers: [ XXX ] Bytes x [ YY ] Buffers
Where XXX can be any length (although values not=240 cause data loss as well as corruption).
Where YY needs to be great enough for the buffers not to overflow.
I've P.M'd you the before and after, 100% repeatable result.
"It sounds like you are saying that these devices don't need drivers?"
What I'm saying is the devices do not load drivers of their own per se, instead they interface via the USB audio device class drivers.
http://www.usb.org/developers/devclass_docs/midi10.pdf
"I still suspect that all devices that have this problem are using the class driver code." Oh yes 
"Logic dictates that there must be some sort of problem with the MS 'port class' SysEx implementation" ... and the way it makes memory usage assumptions and MIDI-OX interacts within that framework.
The trouble is... we both know who is the more likely to be bothered about identifing (& fixing it). 
John
|
|
 IP Logged |
|
|
|
JerryJorgenrud
New Member
MIDI-OX Rules!
Posts: 0
|
|
Re: Sysex bug
« Reply #21 on: Dec 19th, 2006, 4:12pm » |
Quote Modify
|
Hey Jamie,
I haven't been able to reproduce this problem under any circumstances, as mentioned earlier in the thread, but I do have one of these "class compliant" devices coming so I hope to actually see this thing in person. In the mean time I have sent to John a small sysex sender that I bolted together that uses a different memory management scheme than MIDI-OX. He tells me that it works fine. So now he has two programs that work OK and MIDI-OX that doesn't.  It doesn't seem possible, does it? If you look at the errors in the example John sent they look like corrupt USB MIDI data packets and not anything to do with our buffers. How could they be? Plus there's no question that our code is and has been for a long time completely "legal" and non-controversial. Nevertheless we don't appear to be compatible with whatever MS is doing in their XP USB class driver. I say that with the caveat that I haven't seen this happening for myself to verify. Not that I'm implying doubt, John.
|
|
IP Logged |
|
|
|
JerryJorgenrud
New Member
MIDI-OX Rules!
Posts: 0
|
|
Re: Sysex bug
« Reply #22 on: Dec 21st, 2006, 6:38pm » |
Quote Modify
|
Hi Jamie and John,
My new "class compliant" MIDISport arrived today and I want to remove all suspense by saying that John is not the total blithering idiot he appeared to be when he showed up at our door, hat in hand (and hand on heart if I remember correctly). I was able to reproduce his problem exactly. To set things up I had my old USB MIDISport cabled connecting input to output. I used two instances of MIDI-OX and pushed data until I was sure there were no problems. I did have to play around with the buffers a little to get things running smoothly on the nastiest tests. My test machine is a Dell laptop I rescued out of the trash. I think the model number probably starts with POS. It's only USB 1.1 and the CPU seems to go an hiatus every once in a while. But I did get things moving and I was satisfied that all of my test sysex was making it through reliably. I unplugged the old MIDISport and plugged in the new one. XP "found" the hardware and then informed me the new hardware was ready for use. No (additional) drivers necessary. I started testing and it was a train wreck right from the start. Significant data corruption. I viewed the files in Hex Workshop which has an industrial strength compare feature and it showed me all the places where data was lost, inserted, and switched around. !!! When I set the buffer sizes to 240 I was able to generate the identical "failed" file that John showed us. Then I ran that little sysex loader I mentioned sending to John and his file came through without problems. One run out of about a dozen or so had a problem. It had dropped three bytes. Since that's the size of the data part of a USB MIDI packet, I'm thinking it was a dropped packet due to the MIDISport or the Dell and not part of the original problem. But it will bear keeping an eye on. So what's the secret? (I saw a couple of you lean forward.) Unfortunately, the solution for the sysex loader may not be the best solution for MIDI-OX. The loader creates a heap and uses HeapAlloc() to get guaranteed contiguous memory (which seems to be the main issue) big enough to fit the whole file and the MIDIHDR, reads the file right in to the mhdr->lpData buffer, prep and bang. One shot. Comes through smooth as silk. Ponder that Mr. O'Connell.
|
|
IP Logged |
|
|
|
Jamie OConnell
Administrator
Gender:
Posts: 2027
|
|
Re: Sysex bug
« Reply #23 on: Dec 23rd, 2006, 7:30pm » |
Quote Modify
|
FWIW, I never thought John was an idiot  . I always believed it was happening as he described, but I suspected a problem with the MS drivers (and still do) as MIDI-OX works fine with other Device drivers (Yoke and UM-880). I am now leaning towards suspecting it has a problem with where the SysEx buffers are allocated from. As I recall, MIDI-OX uses the original Win3.1 MMSystem requirements: it allocates buffers using GlobalAlloc() with the DDE_SHARE flag. In Win9x this has the effect of allocating SysEx buffers from low memory (All buffers being used must fit within 64K of low unpageable memory). Apparently this is no longer a requirement in Win32 and in fact may be the cause of the corruption, although it is completely legal to still use GlobalAlloc(). We could try using HeapAlloc() (for non-Win9x OS's) and see if it fixes things for these drivers. I imagine it's still important to get memory that can't be paged out, but I am willing to believe almost anything these days 
|
| « Last Edit: Dec 23rd, 2006, 7:31pm by Jamie OConnell » |
IP Logged |
--Jamie
Music is its own reward.
|
|
|
JerryJorgenrud
New Member
MIDI-OX Rules!
Posts: 0
|
|
Re: Sysex bug
« Reply #24 on: Jan 4th, 2007, 7:21am » |
Quote Modify
|
Just a bump/update for anyone following this issue.
To recap, the problem is the transmission of corrupt sysex messages when using a MIDI USB device that does not require an additional driver to be installed when installing it on Win XP. In other words the box probably had the words "class compliant" on it somewhere, you are running it on Win XP, and when you first plugged it in XP did not ask for a driver installation disk. John reported the problem and I was able to duplicate it.
I was also able to duplicate the problem using other sysex utilities such as Bome's SendSX http://www.bome.com/midi/sendsx/ and to verify that the problem is fairly well known even though we had never heard of it here before.
Additionally, I was able to determine that the problem does NOT exist if the size of the output sysex buffer in MIDI-OX is greater than the size of the sysex you are trying to send. In John's case the size of the sysex message -- from F0 to F7 -- was fairly large at around 37k bytes. Setting a buffer length greater than 37k allowed the sysex to be sent correctly. I wasn't able to test this "fix" with other test utilities that don't allow the buffer size to be set by the user, such as SendSX.
We are guessing that the USB audio class driver supplied with Win XP requires that a single sysex message fit in one buffer and will not tolerate splitting a sysex message across buffers. If so, and we are only speculating on what goes on inside the dark module, this is an undocumented requirement on the part of this driver and indeed directly contradicts the documentation regarding sysex buffers. If I had not signed away my right to express my opinion about the performance of MS software as part of a licensing agreement ... I would probably declare this to be a bug in usbaudio.sys, the MS supplied class driver. But since I have signed away those rights, I won't say that.
In the meantime, if I had this problem sending sysex I would increase my sysex output buffer size in MIDI-OX until I stopped having the problem.
A couple of points.
1) Most sysex messages are not very big. SysExFiles that ARE big tend to be composed of many smaller sysex. It's only the bytes between (and including) F0 and F7 -- one sysex message -- that need to fit in one buffer. I have one file that is 96k but the longest individual sysex in the file is only 24 bytes. So 24 byte buffers would work fine with this file.
2) If you do need to set a higher size for the output buffers, type the number in rather than trying to use the "spinner". In addition to the "spinner" problem, the box is not wide enough to see the whole number very clearly if you need to go to five figures. We've never had to do that before. A new version of MIDI-OX will be available shortly that will make this easier.
3) The maximum size you can type in is 65k (65,535). Well you can type a bigger number in but it won't do you any good.
4) The most likely problem you could run into is if the hardware attached to your interface or the interface itself needs a firmware update. These are more likely to be longer sysex and if they are corrupted in their transmission, you might end up creating functionally useless objects.
We're still discussing what, if any, additional steps we might take to work around this issue.
|
|
IP Logged |
|
|
|
jwpgroup
Member
MIDI-OX Rules!
Gender:
Posts: 12
|
|
Re: Sysex bug
« Reply #25 on: Jan 4th, 2007, 12:45pm » |
Quote Modify
|
Ah ha... forgive me if I now bask in the warm glow of a public declaration by my esteemed collegues that I'm not a blithering idiot!!
A couple of observations/questions about that last post Jerry...
"I have one file that is 96k but the longest individual sysex in the file is only 24 bytes. So 24 byte buffers would work fine with this file."
Does MIDI-OX 'automatically' swap buffers for each F0-F7 section then? ... If that isn't the case, I'm intrigued how multi F0-F7 combinations make the file pass muster.
(abridged) "the most likely problem occurs during a firmware update where you might end up creating functionally useless objects."
Just an observation on that. It would be extremly unlikely that any manufacturer (worth their salt) would not apply some kind of checksum for such a system critical file (I know we do) and also try (where possible) to load any incoming update into a buffer area before 'copying' it over the old firmware. It's just too risky not to take that approach.
I'm not saying (worse case) that a complete trash of the device can't/won't happen , just trying to reassure people that in most cases a failure to update / rejection of the file would be a more 'normal' outcome of the corruption issue.
John
|
|
IP Logged |
|
|
|
JerryJorgenrud
New Member
MIDI-OX Rules!
Posts: 0
|
|
Re: Sysex bug
« Reply #26 on: Jan 4th, 2007, 4:01pm » |
Quote Modify
|
on Jan 4th, 2007, 12:45pm, jwpgroup wrote:| Ah ha... forgive me if I now bask in the warm glow of a public declaration by my esteemed collegues that I'm not a blithering idiot!! |
|
I guess you didn't see that before ...Well if everyone keeps using the words John and idot so closely together it will start hitting the top 20 in Google
" Does MIDI-OX 'automatically' swap buffers for each F0-F7 section then? ... If that isn't the case, I'm intrigued how multi F0-F7 combinations make the file pass muster."
That's a good catch, John. We move to a new buffer at F7 if "Delay after F7" is selected in the configuration options, even if the delay is "0". I neglected to mention that part of making that 96k file work with 24 byte buffers. If "Delay after F7" is not checked, then you can expect the same kind of data corrruption as we are talking about. The point I didn't drive home is that our problem driver  is expecting an F0 at the beginning of the buffer and an F7 by the time it gets to the end of it. No more no less. The problem is that while it does need to start waiting for that F7 as soon as it encounters the F0, it needs to sustain that vigil across as many buffers as it takes. And when it doesn't encounter an F7 by the end of the buffer it needs to stop hacking a fur ball.
I'm working with a beta version of the next release and things are in flux in that area. I've already forgotten about that silly old version that you poor folks have to deal with.
"a more 'normal' outcome of the corruption issue"
Once again I didn't make my point, which is that, no matter how much confidence I had in the manufacturer of the device, I would make sure that I could deliver an uncorrupted file before attempting a critical operation like upgrading the firmware. I trust the manufacturer of the air bags in my car, too, but I'm not going to test that trust by slamming into a bridge abutment at high speed if I can help it.  Of course it's a melodramatic comparison and your point that probably the only thing that would happen is a failed update is a more balanced view. Assuming one values balance then that's a good thing.
Out of curiosity, how do you let the user know that the update failed?
Thanks for spotting those two areas that needed clarification.
Now then, John is not an idiot, John was never an idiot, we never thought John was an idiot ...
|
|
IP Logged |
|
|
|
jwpgroup
Member
MIDI-OX Rules!
Gender:
Posts: 12
|
|
Re: Sysex bug
« Reply #27 on: Jan 4th, 2007, 6:39pm » |
Quote Modify
|
"Out of curiosity, how do you let the user know that the update failed?"
TBH, that depends entirely on the product . If you've got a couple of seven segment displays you might set "FL" (for fail) to the display. If you've just got a bunch of LEDs then you might make them 'lock' into a repetitive flashing sequence or perhaps just turn off all the LEDs once done (correctly). Obviously if you've got something with a LCD display you just say "it's done". Usually the manual defines the procedure.
"no matter how much confidence I had in the manufacturer of the device..."
Actually, confidence or not (!) this is not always a totally 'safe' thing to do  . The 'typical' approach to this (assuming and that's a big assumption) that you've got enough spare flash memory to use for the task, is to load the entire 'new' firmware to the spare flash area, checksum it (to make sure it's not been corrupted by MIDI-OX (joke) ) and then (& only then) do you copy it back over the original firmware.
When you do the actual 'copy over' you need the copy routine to reside and run from RAM because you can't read n' write to the same flash area at the same time. If, during the couple of seconds the copy over process takes, you then trip over the power cord, just be sure to keep the original packing materials to hand, because as you initmated before...
"you WILL end up creating a functionally useless object"
It'll then have to be reprogrammed back at the manufacturer, because you will have trashed it (for further 'in the field' upgrades). 
My last post was just clarifying that the file would (normally) be checked before it was used (so one would 'know' if the sysex was corrupt before it was used in vengance).
Caveat... Some of the above explanation can vary. you might be able to go for a simple bootstrap loader that will always be present (so the power glitch would drop you back to that), but you won't be able to pull that off with anything other than a serial interface. With USB, the code is just too verbose (without any product firmware) to have it reside on it's own (in a typical single-chip micro).
I'll stop now before we get lost in techno-babble
You get the idea.
John
|
| « Last Edit: Jan 4th, 2007, 6:42pm by jwpgroup » |
IP Logged |
|
|
|
JerryJorgenrud
New Member
MIDI-OX Rules!
Posts: 0
|
|
Re: Sysex bug
« Reply #28 on: Jan 4th, 2007, 7:04pm » |
Quote Modify
|
on Jan 4th, 2007, 6:39pm, jwpgroup wrote:| You get the idea. |
|
Thanks John, I think now I know exactly what position to assume if I ever decide to do something like update my MIDISport
That would be a praying position for you smirkers in the back row.
|
|
IP Logged |
|
|
|
jwpgroup
Member
MIDI-OX Rules!
Gender:
Posts: 12
|
|
Re: Sysex bug
« Reply #29 on: Jan 4th, 2007, 7:07pm » |
Quote Modify
|
Nah.
Just buy a 'T' box instead. I did tell you it was slow!
John
|
|
IP Logged |
|
|
|
Peter L Jones
Expert
Hit it
Gender:
Posts: 978
|
|
Re: Sysex bug
« Reply #30 on: Jan 8th, 2007, 2:02pm » |
Quote Modify
|
(Comment from a non-Windows internals programmer... ) I take it the USB stack and the USB-MIDI device definition are both fairly hairy and that implementing an alternative Class Compliant driver to the Microsoft-supplied one would be, eh, "non-trivial"..?
|
|
IP Logged |
"...Playing fast around the drums is one thing. But to play with people for others, to listen to, that's something else. That's a whole other world." -- Tony Williams
|
|
|
JerryJorgenrud
New Member
MIDI-OX Rules!
Posts: 0
|
|
Re: Sysex bug
« Reply #31 on: Jan 9th, 2007, 9:04am » |
Quote Modify
|
Peter L Jones wrote (Comment from a non-Windows internals programmer... )
Jerry writes (Reply from a non-internal Windows programmer, err ... 
I don't go near the kernel so it all looks pretty hairy to me but my understanding is that there is a lot of your typical voodoo going on down there in regard to interpretations of the USB spec and the MIDI Spec and fitting that into the Windows multimedia architecture, which is starting to look as though it was designed by Gaudi http://www.brodyaga.ru/images2002/Sagrada%20Familia%20DSCN1719_brodyaga_ ru.JPG. Not that there's anything wrong with that, of course. It's fascinating to look at but the upkeep is a bit of a headache
I've seen mention of a project known as "tunnelling under usbaudio", trying to keep the good parts whilst working around the less useful ones, which should show you how desperate they are not to start from scratch.
I support the goal of "class compliance". When they get it finished it should be a good thing. It should mean cheaper interfaces that work better because every manufacturer won't have to hire their own surly DDK man for one thing.
I'm confident that we've identified the problem with the sysex corruption, though, and have a way to work around it. A stand-alone solution has passed heavy testing and now it needs to find it's way into MIDI-OX itself. One issue that we haven't really resolved yet, though, is if we do work around the problem, and others do as well, it reduces the pressure on MS to do the right thing and fix the problem which would be a lot better solution than any "patch" we could apply.
Maybe someone else more familiar with internals can address your question in regard to what makes it so fearsome that grown men don't want to go in there and slay this bug.
|
| « Last Edit: Jan 9th, 2007, 12:20pm by JerryJorgenrud » |
IP Logged |
|
|
|
TK
New Member
MIDI-OX Rules!
Posts: 1
|
|
Re: Sysex bug
« Reply #32 on: Jan 19th, 2007, 10:17pm » |
Quote Modify
|
I must say that I haven't read the whole thread, and I'm not sure of it's redundant, but maybe my input could be interesting for you.
For the case that you are using a PIC18F4550 or similar derivative for your USB-MIDI interface - forget it! The EUSART peripheral has a bug which leads to the effect, that additional 00 are inserted into the data stream. I've proven this via hardware with a scope, see also http://forum.microchip.com/tm.aspx?m=85120, it's a silicon bug, which has been confirmed by Microchip.
Corrupted SysEx streams when using USB-Midi under Windows: from the user point of view (I've zero windows programming knowledge) this seems to be windows related and heavily timing dependent, the same happens when data is sent from a Java application to a USB-MIDI device, but also USB interfaces with dedicated drivers (like MIDIsport 2x2) are affected. Again I've proven this with a scope to ensure that MIDI-Ox doesn't fool me. This was about 2 years ago, so please forgive me if things have changed in the meantime. However, my workaround for this Java issue is (and please don't laugh!) to use a routing through MIDI-Yoke and MIDI-Ox to overcome this problem
Best Regards, Thorsten.
|
| « Last Edit: Jan 19th, 2007, 10:25pm by TK » |
IP Logged |
|
|
|
JerryJorgenrud
New Member
MIDI-OX Rules!
Posts: 0
|
|
Re: Sysex bug
« Reply #33 on: Jan 20th, 2007, 10:55am » |
Quote Modify
|
For those who don't know, Thorsten is the highly respected inventor of the MIDIbox series of DIY MIDI projects ( http://www.ucapps.de/ ) whose interesting hardware always reminds me that I nearly flunked soldering. Here I'll quote Thorsten from his web site to show that he is very familiar with the problems with the USB class driver that we were late to discover and that are the subject of this thread:
The Microsoft driver isn't multiclient capable.
The Microsoft driver isn't able to send one SysEx string which is distributed over multiple buffers without errors.
The Microsoft driver doesn't allow the USB module to change the device name.
Thank you Thorsten. To this list I would add:
Upon closing, the driver emits 32 "sustain off" messages, two for each channel. (It may have been intending to send reset messages?)
Multi-port devices incorrectly enumerate on multi-device systems. When multiple USB MIDI devices are added to a system and each use the class driver, the device names have a number added after the generic name. The generic name is "USB Audio Device" on English language Windows, additional devices become "USB Audio Device (2)", (3), etc. I'm not sure what specific series of events cause this, but we have observed that it's possible to have two devices with the same name -- two "USB Audio Device (2)", for example. It's bad enough with the 2, 3, etc. situation trying to remember which hardware the numbers refer to, but when they can have the exact same name -- that's just shameful.
Aside from this problem with the class driver, I've also observed, although not with a scope, the same phenomena Thorsten describes here regarding spontaneous insertion of extra MIDI data in a stream. I have intimate knowledge of six USB MIDI interfaces now and foiur have exhibited this problem of inserting nulls, while one has the additional problem of dropping bytes and another one inserts other bytes in addition to nulls. The only USB NIDI device that has worked flawlessly (maybe I just didn't catch it at the right time) is made by Roland.
As Thorsten notes on his web site, Sun have been informed of their Java MIDI bug but seem uninterested in fixing it. MS have also been informed of the class driver bugs and have never acknowledged them. My own belief is that these companies do not actually understand MIDI, and in particular SysEx, and when they get a bug report along these lines, they just throw it in the "crazy kook" pile.
Buyer beware when it comes to things USB MIDI.
Thanks for stopping by, Thorsten
|
|
IP Logged |
|
|
|
jwpgroup
Member
MIDI-OX Rules!
Gender:
Posts: 12
|
|
Re: Sysex bug
« Reply #34 on: Jan 21st, 2007, 5:41pm » |
Quote Modify
|
"Upon closing, the driver emits 32 'sustain off' messages, two for each channel."
Just to clarify that... it's logical to send a sustain off for each of the sixteen channels, what's illogical is to then repeat the 'sustain off' instead of sending a more sensible 'all notes off' command (as the second batch of sixteen - one per channel).
"The only USB MIDI device that has worked flawlessly (maybe I just didn't catch it at the right time) is made by Roland."
But not with the MS class driver right?! (just checking!)
AIUI, everything (that depends on the class driver), fails when you split a SySex file over multiple buffers. 
John
|
|
IP Logged |
|
|
|
JerryJorgenrud
New Member
MIDI-OX Rules!
Posts: 0
|
|
Re: Sysex bug
« Reply #35 on: Jan 22nd, 2007, 8:13am » |
Quote Modify
|
Right, not with the class driver. The Roland was unique in that you could use either the class driver or their own driver (via a switch on the unit). What I meant by flawless was that it didn't do anything else wrong, like insert nulls. But it still had the problems the others did when using the class driver. And it had no problems at all when using the Roland driver.
|
|
IP Logged |
|
|
|
jjones
New Member
MIDI-OX Rules!
Posts: 1
|
|
Re: Sysex bug
« Reply #36 on: Jun 25th, 2010, 10:13pm » |
Quote Modify
|
Korg T2
windows XP
Cakewalk UM-1G USB Midi interface
I've downloaded the T2 sounds from the Korguk.com website to my PC running XP.
I've downloaded the MidiOx software, a new driver for the Cakewalk interface and I've changed (increased and decreased) the buffer numbers as well as tried other mentioned solutions for the program, but still no luck in getting the sounds to dump into the T2.
All of the combination names have transferred, but no sounds.
Within PROG A, A00 Aeroglide, A01 Piano 16' and A02 Brass 1 sounds have successfully transferred onto the T2. The remaning sounds look like jiberish ( * < / , ") with no sounds.
I was hoping that someone could send me the "little sysex" file that was sent to John in previous posts.
Thanks,
Julie
225-755-1925
|
|
IP Logged |
|
|
|
|
User Forum ::
Powered by YaBB
Home 
Help 
Search 
Members 
Login 
Register
Reply
Notify of replies
Send Topic
Print