Hackaday-Rendering A 3D Environment From Kinect Video

7/23/2014 Rendering a 3D environment from Kinect video
http://hackaday.com/2010/11/15/rendering-a-3d-environment-from-kinect-video/ 1/14
Hackaday
Fresh hacks every day
Home
Hackaday Projects
Our Videos
Submit a Tip
Forums
About
July 23, 2014
Rendering a 3D environment from Kinect video
November 15, 2010 By Mike Szczys 76 Comments
[Oliver Kreylos] is using an Xbox Kinect to render 3D environments from real-time video. In other words, he takes the
video feed from the Kinect and runs it through some C++ software he wrote to index the pixels in a 3D space that can
be manipulated as it plays back. The image above is the result of the Kinect recording video by looking at [Oliver] from
his right side. Hes moved the viewers playback perspective to be above and in front of him. Part of his body is missing
and there is a black shadow because the camera cannot see these areas from its perspective. This is very similar to the
real-time 3D scanning weve seen in the past, but the hardware and software combination make this a snap to
reproduce. Get the source code from his page linked at the top and dont miss his demo video after the break.
[Thanks Peter]
Virtual Reality Gets Real with 3 Kinect Cameras Holograms With The New Kinect
Virtual Physical Reality With Kintinuous And An Oculus Rift
The Race Is On To Build A Raspi Kinect 3D Scanner 3D Printed Stick Shift Handle
Google+
Mike Szczys
Filed Under: Kinect hacks, video hacks Tagged With: 3d, c++, Kinect, render, scanning
Comments
1. Michael says:
November 15, 2010 at 12:05 pm
I wonder if you took 3-4 of theses and synchronized them, putting one on each wall of a room, you could get a
higher quality environment.
Follow 308
Reply Report comment
2. Daniel Magliola says:
Hmmmm, this would be interesting to see with TWO Kinects, thatd fill all the blanks, right?
3. Michael Bradley says:
I love it!!!! Can you take 3 of these and point them at the center of the room so as to build a complete 3D image
without shadows???
4. mixadj says:
Thats aweswome
5. twistedsymphony says:
this is probably one of the cooler Kinect hacks Ive seen I was apprehensive about the device at first, but now
I want one just to fool around with.
6. spyder_21 says:
Its a nice start towards the right direction. I might buy one soon if cool stuff like this comes out. Would not buy it
to play stupid kinecct games.
7. xeracy says:
@Michael Bradley 3x Kinect == Cheap Mocap? Im looking at getting one for some form of live production
visuals.
8. spiritplumber says:
This is incredible. The Kinect is going to open up a lot of avenues of research.
9. Garak says:
Really really cool
Will using more than one kinect sensors work? My understanding is that it projects a grid of IR dots over its
field of view and uses them for the measurement. Will intersecting grids confuse the sensors?
10. Roon says:
I really want to see someone do something with 4 of these, you could do so much
11. xeracy says:
@Garak i imagine this could be done my quickly turning the dots on/off and sampling them in a continuous
cycle.
12. turn.self.off says:
ok, how long until someone builds a esper machine out of all this?!
@xeracy, I think so, and this guy did a great job. I am impressed with how when he rotates it, how much
information is available around the corners, ie: the front of his face, when the cam is to the side.
When he rotated, I had flash back to The Matrix, the first scene when the girl is in the air, and all stops, camera
rotates, and she continues. Just imaging, that was done with several still cameras all positioned, etc. with this,
just freeze, rotate, and continue!!!
14. Drake says:
Next stop 3D PORN!
Anywho
Would interfacing the kinect with a wii be HaD worthy?
I may have a crack at it later this week if so
15. xeracy says:
@Michael Bradley and the kicker? ITS ALL REAL TIME! I really wish i was skilled enough to do this on my
own.
16. macpod says:
Wow, I did not realize the system was that precise. I thought in situations as this that the depth stepping
increments would be closer to a foot or so in distance if not more.
Judging from the coffee mug and torso shots however, it seems the distance granularity is much smaller! Now I
want one
17. IssacBinary says:
Just use real time photoshop content aware fill to fill in the gaps ;)
18. Oren Beck says:
Hopefully, someone will write a Parser environment for Kinect data. To make files that could somehow be
rendered into Skeinforge Etc parameter/object details.
Enlisting the commercial solid print bureaus like oh-Shapeways and their peers in a scheme of printed object
credits for prize funding might kickstart the ideas.
It would be way cool to have 3D busts of my Grandkids..
19. Filespace says:
i would like to see this guy manipulate a virtual object in real-time if even a ball perhaps
20. jc says:
Im wondering what would it look like if you added a mirror in the kinects vield of vision?
21. Whatnot says:
Sorry guys but you likely cant do more than one at the same time, it projects a IR pattern as part of gathering
data, that would interfere and fail with more than one in a room.
22. Sci says:
Now someone just needs to hook this up to a cheap 3D display for live holographic video calls.
My bets are on something like the DLP project+spinning mirror combo. Or possibly a spinning LCD if someone
can get all the power & signal connections to it.
23. rizla says:
In response to the can it use more than one kinetic. If you changed the frequency of the IR, would you be able
incorporate more kinetics without having them step over each other?
24. TheZ says:
Quick Thought: You can use different IR wavelengths. You would have to replace the IR LEDs. And code how
to detect them.
>Sorry guys but you likely cant do more than one at the same time, it projects a IR pattern as part of gathering
data, that would interfere and fail with more than one in a room.
25. chbab says:
It seems that your shape evolved did a huge work in this direction for its player projection as its silhouette is very
clean compared to what you see here It opens the door to augmented reality stuff with a cheap device :)
26. Removed says:
kind of reminds me of that software from movie Deja Vu
27. rasz says:
IR projector in Kinect means you cant use more than one at the same time. You can sync them like ToF
cameras.
But you could use few more normal cameras and use Kinect depth info to reconstruct/simulate/cheat the whole
scene.
There are algorithms that reconstruct 3D scene from ONE video feed http://www.avntk.com/3Dfromfmv.htm
Having few at different angles + one with 3D data should speed things up.
28. da13ro says:
Not sure if the kinect requires any reference points for calibration but instead of different wavelengths (which I
imagine would be difficult/impractical) couldnt you setup a shutter system, solid state. Block the IR of other units,
sample data on one and cycle. Would slow down your available refresh rate.
Sweet hack mate, very impressed.
29. rasz says:
^^^CANT sync them like ToF cameras.
I like the idea about different IR wavelengths, but i think Kinect uses laser instead of led.
I guess you could use two Kinects directly in front of each other just making sure IR dots dont end up at each
other cameras that would give you almost 90% of 3D and texture data.
30. cornelius785 says:
@TheZ
I dont know about different wavelengths for multiple kinects. Alot of it depends if the kinect can differentiate
between different wavelenghts AND being able to hack the firmware to do stuff appropriately. Hasnt all the
hacking been on the computer side of just controlling it and getting useful information back? Id either go with
very narrow filters or synchronize all the kinect together and some multiplexing. If it is possible, Im sure someone
will figure out.
Wasnt it in Hitch Hikers Guide to the galaxy that they mention the progression of user interfaces as: physical
button -> touch interface -> wave hand and hope it works? Isnt the third stage upon now? Im wondering how
long I have to wait before I can control my Mythtv box with hand gestures in the air.
31. aarong11 says:
Hmm, instead of replacing the IR LEDs, wouldnt it be possible to place IR filters in front of both the LED and
Detector of different Kinects? That way, each Kinect should only detect the wavelength of IR light it was
emitting.
32. sarrin32 says:
you could put on a gimp suit where each joint is a different colour (forearms, hands, thighs etc). The computer
could use the colour coding to identify each joint, do measurements etcthen it could do motion capture.add
the motion capture to a real time or post calculated 3d scene with digital actors.hurray. How long till we get
kinect to bvh converters?
33. Eamon says:
The trick would be to calculate SIFT points on some frames, and use those to track objects as they move. This is
the basic mechanism behind current reconstruction techniques, whether they use one or two cameras. The depth
map would improve the fidelity of the representation, and should provide shortcuts that would let this run faster.
34. blue carbuncle says:
Keep up the good work everyone! Kinect is coming along nicely :)
35. Mike says:
Outstanding work
36. jim says:
I swear Ive imagined doing this for years, and how cool the glitches and shadows in some set ups would look.
37. Colecago says:
That is pretty amazing. Picture does not do it justice, video is awesome.
38. qn4 says:
In regards to those arguing against using multiple Kinects at once, one could consider putting something along the
lines of the shutter glasses (used for many of the current 3D displays) over the IR projectors, and dropping the
(depth) frames not associated with the currently projecting Kinect. Im sure that a bit of crafty software design
could interpolate the two 15Hz (normally 30Hz IIRC) streams fairly well, too.
Better yet if the exposure time is less than 1/60th of a second (30Hz/2) and the sync can be intentionally offset
39. NatureTM says:
After this seeing this, Im absolutely getting one. Very cool.
40. qn4 says:
da13ro beat me to it I really have to refresh the page sometimes before posting things. Still, this thing is full of
awesome capabilities, and its great to see that so many skilled people are making use of it.
41. Martin says:
Could you polarise the IR coming from two kinects at 90 degrees to each other. Then use filters on the cameras
to block the other set.
42. macw says:
polarization would work just fine provided that there was enough light remaining after the fact for the camera to
work properly.
So would strobing them on and off alternately its a very common procedure when you have multiple sensors
operating on the same band (ultrasonic distance sensors are a notable case, since theres not much ability to
reject returns).
Filtering for wavelength might work, provided that you had physical bandpass filters on the camera. The depth
sensor is monochrome and would react basically the same to any frequency its responsive to (different brightness
but because its an uncontrolled environment you cant rely on that to differentiate two sources).
I would guess that polarization would be the cheapest and quickest to implement, with bandpass filters being not
that much more complex. Time-division multiplexing would only be a good idea where you absolutely cannot
modify the kinect hardware in any way for some reasonotherwise its just a waste of effort.
I do really want to see what happens if you put a mirror in the path, though. Im imagining a window in the feed
through which you can look and see the other side of your room, just as if the mirror were actually a window into
an alternate dimension :P
43. Torwag says:
about the IR grid stuff I would assume they modulate the IRs by some frequency to avoid influence of other IR
emitters. Either that, or one has to modulate the IR by themself. After that it would be relatively easy to use
several units simultaenoulsy by giving each of them a different modulation frequency and using an electrical filters
or an FFT algorithm to isolate the indvidual frequencies from each other.
No need for different wavelength, optical filters, etc.
44. TR says:
Its unlikely that they modulate the IR output (as in pulse the laser/LED whichever it is) because the camera would
have to be able to capture at least that fast. >120Hz for US incandescent lights. So, a camera that captures video
at greater than say 300-400 FPS to adequately figure out whats noise and whats signal. Doubt they used anything
like that. I think the polarization would be the best bet without opening up the connect. Would try it if I had
another connect and time. Maybe someone can try using two pairs of the free 3D theater glasses. One glasses
lens for each projector and depth camera.
45. curious says:
Could you hook up multiple kinects to capture the other angles of the room and have a full 3d map?
I just looked at this guys youtube page, OMG, this guy is on it! I thought I was fast with code (only uControllers)
this guy rocks! He did some augmented vr, addressed the mirror question, etc..
47. qwed88 says:
Im not wanting live video like that, but if it could fix a Kinect to a rotating base and scan environments in 3d!
This would be functional I could use this.
Or software written to take slices so an object could be rotated in front of it and scanned. This would make a
relatively cheap 3d scanner.
Seems as if one could use this with a program like Zbrush to sculpt with your hands.
48. qwed88 says:
As a follow up to my comment I just saw another video of his, with a cg model of a creature sitting on his desk
moving and all in real time. Hes really not that far from the Zbrush idea.
Imagine a Kinect above your monitor as your sculpting with your hands the model on the monitor?
49. EquinoXe says:
November 16, 2010 at 12:15 am
re 2 kinect: in theory should be easy.
get 4 linear polarization filters (2 for each kinect).
Polarize kinect 1 @ 135 and kinect 2 @ 45
(place a polarizing filter on depth cam as well as upon IR source)
now both kinects cant see each other but they can see their own beam.
50. yeah says:
November 16, 2010 at 3:14 am
@EquinoXe
Polarization is normally not sustained under diffuse reflection, so even though the light would be polarized, the
light coming back from the scene wouldnt. I guess a shutter system as suggested above could do the trick, but
then youd have to hope that the kinect doesnt use any type of temporal coherence.
Newer Comments
Leave a Reply
The Hackaday Prize
COUNTDOWN FOR ENTRIES
28:
Days
01:
Hours
Enter your comment here...
27
Minutes
33
Seconds
Never Miss a Hack

Search
Search this website Search
Categories
Select Category
In case you missed it
Talking BeagleBoard with [Jason Kridner]
Leave a Comment
HOPE X: Interviews with Ellsberg and Snowden
40 Comments
HOPE X: Commodore 64s Are Back, Baby
14 Comments
HOPE X: Hackaday Shirt Gets Hacked at Hacker Convention
52 Comments
HOPE X: Lock Picking and Lock Sport
7 Comments
More Posts from this Category
Our Columns
Retrotechtacular: Were Gonna Have Manual Transmissions the Way
My Old Man Told Me!
25 Comments
Hackaday Links: July 20, 2014
7 Comments
THP Hacker Bio: hackersbench
18 Comments
Developed on Hackaday: Discovering Shenzhen and its Companies
21 Comments
Retrotechtacular: AT&Ts Hello Machine
33 Comments
More Posts from this Category
Recent comments
Jim Thompson on Cold War Clock is all Tubes
Kip on Retrotechtacular: Were Gonna Have Manual Transmissions the Way My Old Man Told Me!
trandi on POV Display Does it on the Cheap
matseng on POV Display Does it on the Cheap
Andrew on Smart Hat Puts Your Head in the Game
Bogdan on USB Rotary Phone: A Lync to the Past
Remy Dyer on VoCore, The Tiny Internet Of Things Thing
Slingrock on Building a keyboard from scratch
reelyActive on HOPE X: Creating Smart Spaces With ReelyActive
rasz_pl on HOPE X: Creating Smart Spaces With ReelyActive
Find Hacks by Date
November 2010
S M T W T F S
Oct Dec
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30
Return to top of page
Copyright 2014 Genesis Framework WordPress Log in
Hackaday, Hack A Day, and the Skull and Wrenches Logo are Trademarks of Hackaday.com
Our Policies

Hackaday-Rendering A 3D Environment From Kinect Video

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Hackaday-Rendering A 3D Environment From Kinect Video

Transféré par

Droits d'auteur :

Formats disponibles

7/23/2014 Rendering a 3D environment from Kinect video

Vous aimerez peut-être aussi