11-17-2004

WOO breakthrough on the EW-flash linkup. The "gum" issue was a minor one after all; seems the sample flash file had it appending all new messages to a variable instead of overwriting the previous ones, meaning that the variable would take up more and more memory at a linear rate. I fixed this by rewriting all of the "incoming += (new item)" lines to "incoming = (new item)" or the equivalent, and it was pretty stable after that.

JJ's code ended up slotting in place fairly neatly, with few modifications needed. Most of what we did last night had to do with adjusting his code to parse the incoming OSC packets. As they come out of EW, the packets have a name (defined in the OSC block in EW as their 'OSC_Command'), and two arguments, both of the input coordinates for each point. The arguments always come as X first and Y second, so it was not necessary to append or modify that data to differentiate them. In the flash, JJ has the movie generating an empty movie clip for each distinct packet, then drawing a circle at the proscribed coordinates. He also has the movie assigning the clip names based on the incoming packet names, meaning that 1) we might be able to 'smooth' the motion between updates and 2) the number of points drawn in flash grows automatically with the number of points sent from EW. I tested this by adding two more transmit subpatches into EW to make a total of five; the flash faithfully drew five circles without any modification.

To do:

Increase the number of transmitted points in EW, while checking for stability. Mostly grunt work; I'll probably use the EW 4.0.0 beta to take advantage of their new interface.

Run EW on the sample video file to check for climbing memory usage, indicative of a memory leak or something similar. This may cause stability issues during prolonged usage... something to keep in mind.

Saturday we plan to try a rough prototype of the system with a live camera feed. I expect some degree of offset in where the points draw in relation to the shadow, due to the necessary difference between the projection angle and the camer viewing angle. This is something that can be worked out in later tuning; right now, we just want to test the system from stem to stern.

11-15-2004

Picked up a response this morning on the discussion board post:

extracting the matrix into indevidual values and then exporting it to flash
is the way to go.
i wasnt able to prase a message with more then one value in it to flash, but
something like
/xball_01 = 34
/yball_01 = 234
/xball_02 = 33
/yball_02 = 212
and then merging it in flash works, and might even be simpler then bringing
an array (=matrix) into flash and then spliting it into indevidual values.
anyways, i know of no other way to send multiple values to flash thru osc.
sorry for the lousy spelling.
post back if you find a way to send a couple of values as a single string
and then spliting it in flash into a nice array :)

So... he's mostly affirming my approach with the existing tools.

In the meantime, I tossed the Java Runtime Environment 5 on my machine and fired up flosc. Installation was *very* smooth and easy, no problems starting and running the thing either. So far, I've been able to connect to it from both EW and flash, and using the demo .fla file I was able to get the mouse coordinates to show up in flash.

I also figured out how to increase the number of inputs available on a StringToOSC or ScalarToOSC block, allowing me to bundle multiple data elements in a single OSC packet. I haven't found any naming options for them, but they do consistently come in in the same order. Besides, I can always concatenate some labels in front of the coordinate strings if this becomes a problem.

On to the bad news: even three tracked testing points were enough to gum up the system in sending out to flash. Tracking the test video runs this workstation's processor at about 65-70% capacity, if flosc is also running (which doesn't take much when idle). When flash is displaying the coordinate data as well, I can watch the processor usage ramp up to 100%, at which point the flash demo sort of gets "gummed up" and pretty much stops responding. It isn't a hard crash, though... if I stop EW, the system eventually catches up, though the data that streams out of EW seems to get lost. I'll have to do some more work to determine whether it is flosc or flash gumming up the works.

The best remedy I can think of is to relocate the flash client to another machine. All we have to do is point flash at the IP address of the flosc server and let it run. I think this would work well in the final setup too, because it means we could keep the bulk of the processing with the camera itself, and nothing more than an ethernet cable would need to run to the rendering system (or we could go wireless if we are feeling bold).

11-11-04

Nothing much yet, looking at the SDKs and documentation for developing new blocks if it comes to that. Posted this on the eyesweb discussion board just now:

Hi all,

I'm trying to extract the points generated by the LKTracker to send their
coordinates through OSC to flosc, and then on to flash. Has anyone attempted
this before?

Ideally, I'd like to find some way to send the entire MoCapDataList for each
frame to OSC.

Barring that, I know that MoCap.Item.Extract lets me pull out values for an
individual point. Is there a way to send out that matrix through OSC?

I've been able to use Math.Matrix.GetEntry to pick out single values (i.e.
just the X or the Y coordinate), but they are meaningless outside the
context of each other. Is there a way to concatenate these values together
in a string which can then be transmitted with StringToOSC? I'd be willing
to incorporate a subpatch like this for every point tracked by the
LKTracker, if that's what it takes.

Thanks in advance, any tips/suggestions appreciated,

-swong


I'll have to see if I get any hits on that. I'd rather not have to learn C++ just for this one thing.

11-3-2004

I, for one, welcome our Republican Overlords.

Moving right along,

Had a bit of success extracting the matrix data from the mocap list coming off of the LK tracker; a mocap.item.extract pointing to the point label coupled with a math.matrix.display block shows me what sort of data is coming off of the LK block. Swapping in a pair of item extractors for the matrix display block lets me isolate the X and Y data to separate "channels." Tying in a string displayer for the label output on the mocap.item.extract lets me see what point is being tracked by that extractor block. In all, three data elements make up a point's location: the point's label, the X, and the Y; out of context, none of those means anything, so they all have to pull from the same input and they all have to go out together. I'd like to find a way to concatenate them together and send them out as a single string with OSC.

Trust me, it all makes sense on the screen.

This approach is a bit inelegant. Doing it this way will require me to have a subpatch (that whole mechanism in the preceding paragraph) for every point being tracked, some 40 in all at the moment. I'm keeping my eyes out for a block that will somehow export the whole list at once, though the only stuff I've found so far ports it as a somewhat complex data structure to java or C++. In the meantime, I may have to go with the inelegant solution of writing a new block in C++ that will concatenate two strings together. I'll try posting on one of the discussion boards; I'd be surprised if someone didn't write something like this a long time ago.

11-1-2004

Bit more work on the Lucas-Kanade tracking patch, with some findings.

Threshold, as noted before, seems to deal with the threshold necessary to do a real-time reset of the tracking system and assign new points. Lowering the threshold basically increases the sensitivity of whatever mechanism causes a reset. Lowering the value, then, means that new tracking points are assigned more frequently to the test loop, resulting in a more accurate, more up-to-date edge definition.

This increased definition comes with the price of an increased computational load, though. Dropping the threshold all the way to .05 definitely results in sluggish playback, and that’s just with a loop with my hand in it. I’m running the system at around .2 now, with reasonable response times and fairly accurate tracking.

Ultimately, though, I don’t think the LK tracking algorithm, at least in this implementation, will ever be really good at tracking overly rapid motion. For the slower stuff, like where a user moves their hands with spread fingers, it seems good enough to isolate the gaps between the fingers.

Moving right along, I dug up a promising post on the discussion newsgroup, originally posted back in April 2004:

From: Anders Bo Pedersen To: eyesweb.discussion
Re: LKtracker problem

Use the MoCap.Item.Extract and then the Math.Matrix.Extract

Best
Anders

"trimard_" wrote in message
news:MDU4PpG$DHA.1828@lab0.infomus.dist.unige.it...
> Is there a way to extract the coordinates of the points made by the
> LKtracker? so that i can assign a move image action?
> thnx
>
>

-strapped this configuration into the LK Tracking example, the MoCap.Item.Extract fishing its input right off of the LK block’s output stream (it’s a copy of what is going on to the display stream).

-spliced Output.MoCap.Draw2DItems into the system near the end of the output stream, taking the output of the LK Tracker and the dot-drawing “view” block, then dropping in a second display block to show the visual output of that sucker.

The result: the dots in this output are replaced with little crosses and named labels. The implications:

1) The dots are tracked as discrete data objects, and aren’t the result of some creative visual filtering (kind of obvious in retrospect, but I had to be sure).

2) Since they are labeled, there may be a means of tracking semi-persistent data on the Flash side of things. We could, for example, somehow buffer previous frames in Flash, and look at where, say, point_6 was three frames back, and calculate some rudimentary motion data from it. We don’t have to, but it is another option there if we want it (which is A Good Thing).

3) There is no point three.

So, the data is there. MoCap.Item.Extract only seems to want to let me extract the data for one point at a time; I’ll have to go in and look at a way of pulling the whole list for every frame.

10-24-2004

The threshold levels don't seem to make *that* much of a difference.

Neither does that alpha filter slider. I've been tracing the linkages and I think the threshold has to do with how often the system does a live "reset" to pick new points to track.

Looked up alpha filter in the eyesWeb documentation; it had this to say:

"This block performs alpha-filtering on the input stream according to the following formula:

y(sub i) = y(sub i-1) + (alpha)(x(sub i) - y(sub i-1))

where x(sub i) is the current input, y(sub i) and y(sub i-1) are respectively the current and previous outputs, and (alpha) is the value of the Alpha parameter. Note that if the current input is not valid, accordingly the output will be set as invalid."

The ramifications are staggering. I hope whoever is reading this is as ecstatic as I am about that.

10-21-2004

Poking around some more with Lucas-Kanade feature tracking – adjusted it to draw points instead of stripes, and played around with some other values. The input image the algorithm is working off of is based on the background subtraction method shown in a few other examples, so I tweaked the threshold levels to match what I’ve been using in the other files (more on this in a bit).: 200 levels, minimum at 0, maximum at 200.

This seems to be as clean as I can get the background subtraction for now. These settings seem to filter out most of the noise, but don’t distinguish between screen black and shadow black. From a graphic standpoint, I think grays darker than 60% are a bad idea, because they are too easily mistaken for shadows. We may see different results on switching to a rear-capture model; all in due time.

Since all but one of my testing files uses some sort of test pattern on the screen that I have since been unable to filter out, I have been using the remaining “clean” test: my hand’s silhouette waving around on the screen. The feature tracker plots a set of points on the screen and attempts to track them in successive frames of input. At intervals, or if the image changes too much, it assigns new points.

Advantages:

“state-neutral” – doesn’t need a beginning reference frame to function, meaning that it can run continuously without having to stop. The tracking points seem to conform to the edges of the shadow pretty well at the right settings (more on this in a bit), though they don’t work out to a perfect edge.

Disadvantages:

Need to think about this some more – I’m concerned that I can’t get enough tracking points running simultaneously to calculate more complex shadows.

To do: put in some more complex shadows. See about extracting the tracking point position data to a set of coordinates. See about skewing and stretching the input frame to match the projector output proportions (is this called normalization in eyesWeb?)

Notes on settings:

For background subtraction: on the subtraction patch: 200 levels, minimum at 0, maximum at 200.

For point tracking, setting the threshold to somewhere between .22 and .3 seems to give fairly clean results without any too many jittery artifacts, while maintaining a good number of tracking points. Have to look up what this is actually adjusting, though.

10-20-2004

Grabbed and packed some more test footage.

Playing around with Lucas-Kanade feature tracking patch – possibly feeding the locations of the tracked points out through OSC? Motion keyed but doesn’t need reference colors to work from. May experiment later with just feeding point data to a screen readout.

10-15-2004

Showed up to generate testbed footage – found a brand new combination lock on the multimedia cabinet in AE1203 – can’t run the projector without that computer or a laptop. Going to try making some stand-in footage in the cubicle so I have something to work with. (I really could use the clean edges on shadows)

Started pulling out some of the clips on eyesweb. May have to do some work to reduce some minor flickering in the video capture.

Fuzzy edges noticed around shadows – this may be because the projector is so close to the screen. Also noticed: in the test conditions, the umbra is nearly the same color as projected black. This may make isolating the shadow from an all-black region difficult.

Reinstalled eyesWeb 3.3.0 with the upgrade patch and the gesture processing library. It seems to work again.

Doing yet more reading on patches and other eyesweb notes.

10-13-2004

Demo for Phil – seemed impressed by progress. Retrieved DV cam with working firewire.

Played around for a while with different included eyesWeb patches. Probed into some of the motion analysis packages which generate blobs of color around detected motion – I think this is an algorithmic effect, not keeping track those blobs as a set of data, but instead generating them by comparing a sequence of frames for differences based on a threshold value.

Encountered some critical eyesweb errors at one point. Reboot didn’t help, may need to reinstall eyesWeb.

10-13-2004

Demo for Phil – seemed impressed by progress. Retrieved DV cam with working firewire.

Played around for a while with different included eyesWeb patches. Probed into some of the motion analysis packages which generate blobs of color around detected motion – I think this is an algorithmic effect, not keeping track those blobs as a set of data, but instead generating them by comparing a sequence of frames for differences based on a threshold value.

Encountered some critical eyesweb errors at one point. Reboot didn’t help, may need to reinstall eyesWeb.

10-11-2004

Put together the flash test patterns for testing, I’ll have to go into a dark room with a projector later to finish off the test videos.

Also fished a thread off of the eyesweb.discussion group... I’ll put it in “floscDiscussion.doc” on the server.

10-8-2004

Note to self: generate test footage to calibrate eyesweb. Test patterns (rendered black areas should be distinct from shadows to eyesweb), sample lighting conditions, off camera angles.

Sample test – rendered black letters on white background: “If you can read this, the threshold is too low” and that sort of thing.

10-5-2004

Got FW running on my workstation with camera belonging to undergrad program on very temporary loan. Grad program camera should be available by the end of the week for longer term usage. We will be borrowing that and a FW cable for the duration.

*note- figure out a good way to mount the camera to point at a background screen installed in cubicle to aid test work

Eyesweb took the input signal nicely. Most of the sample patches worked well enough – was able to run some of the basic silhouette and background extraction patches outright. A smooth, matte background screen (a piece of paper) worked best, which makes sense.

Ambient lighting didn’t seem to make too much of a difference considering that we are able to go in and adjust the threshold levels to compensate. There is minimal delay in the more basic patches, which is encouraging. I also spent some time playing around with other individual blocks to manipulate the screen image.

*note – look for some kind of shear type block so that we can place the camera at an off-angle and compensate for it.