Fluid Simulation Integration with Blender

Dear readers, as promised I will now follow up with the integration of 3D objects in our tracked footage. It took a while to continue this series because my beloved father unfortunately passed away.

Today I will create some fluid simulations for our kitchen scene. Last time we tracked only the relevant part of the footage which produced a tracked camera from frame 431 to frame 618. First we will now cut down the background footage. Then we modify the tracked camera to start a frame 0.


Img01Change the camera animation in Blender
In Blender change the timeline to the dope sheet and navigate to frame 430 (one before our relevant first frame). Zoom in so that you can see the key frames. Now select the menu “SELECT – Before current frame”. With the mouse over the dope sheet hit “x” to delete all key frames from 0 – 430.

Now navigate to frame 431 and select the menu “SELECT – After current frame”.  Hit “g” and type “431-“ which will move all the selected key frames to frame 0. Make sure to type the minus character at the end!

Switch back to the timeline and set start and end  accordingly (0/187).

Img02Use Adobe Media Encoder to cut down our background footage
Open Adobe Media Encoder and open preferences (STRG+,). Select Appearance/Display Format. Make sure that the frame rate is set accordingly to the frame rate of the background footage. (In my case 25 FPS).
Close preferences and drag in your footage. Choose AVI uncompressed and name the file. In my case “SHOOT27_431-618.avi”. Since Adobe Media Encoder gives us no direct way to show the current frame we are at we need to work with the time code provided. Since I recorded my footage without resetting the time code of my camera the clip starts with the time code: 11:40:51:14 which translates to 11 hours, 40 minuts, 51 seconds and 15 frames. The last number defines the frame in the case of 25FPS the value ranges from 0-24.

Img03Since we do want to encode starting at frame 431 we need to add this to the current starting frame 14. 431+14 = 445. Click on the time code to change it and change the last number to 445. Img04

As soon as you hit <ENTER> to confirm the value the time code calculates correctly to: 11:41:08:20. Now set the IN-POINT by clicking on the button right next to the time code.

Similar to the start frame we navigate to the required end frame. We move the current position to the first frame and again change the number 14 to 618+14 = 632. After the current position changed we can set our OUT-POINT by clicking on the 2nd button from left next to the time code.  Make sure that the output video has the same aspect ration and the same key frame rate as the original footage and encode the video.

Create 3D Model and set background footage

Img-07Now lets create our 3D Model of a drinking glass and a plane below where we do want to simulate a wet surface later. I won’t explain the creating a simple model for a drinking  glass in this tutorial. Align the 3D Models with the 3D Model you got from PF-Track. In my case this is the part of the kitchen with the window.

Switch the renderer to CYCLES in the top dropdown on blender (see image). Next we do want to load in our background footage so we can move the PF-Track model of our kitchen to another layer. With the mouse in the 3D View hit “n” to show the right toolbar. Img-08 

Scroll down to “Background Images” and select the checkbox. Then load in the background movie and set the settings like illustrated in the screen shot.

Now we will set the material for our two objects. Select the drinking glass, then  the material tab and create a new material by clicking on the button “+ New Material”.


Name the material glass. Select “Glass BSDF” shader and set the color to pure white (1.0/1.0/1.0. Default is 0.8/0.8/0.8). Leave the refraction index (IOR) at 1.310. You can find a list of materials with their corresponding refraction indices here.



Since we also want to render the background footage so that it’s reflection is caught in the class we need to change the “world”-settings. In the “world-tab” open up the section “surface” and click on the “use nodes”-Button.  Then click on the small button right to the color and select “Image-Texture” from the pop up menu. Select the AVI-file we created earlier, set the amount of frames, start frame and AutoRefresh. Make sure you set Vector to “Texture Coordinate | Window”.

1. Img-08-1 2. Img-08-2 3. Img-08-3

Create the fluid simulation system

Create a fluid domain

Img13-FluidDomainCreate a cube around the surface and the drinking glass. Switch to wireframe to see the objects within. This will be our water simulation domain. Create a materialWater” for this object. Again use the Glass BSDF shader but change the refraction index (IOR) to 1.301 and the color to pure white (1/1/1). Select the cube and switch to the tab “Physics” and click on “Fluid”. Set the type to “Domain”.

TIME: The timing of the simulation is very important. There are textboxes for Start and End.  This indicates Start and End Time in SECONDS! So in our case start with 0 and end with 186 (frames) / 25 (FPS) = 7.44 (~7.4). Set the SPEED setting to 1 which indicates normal speed.

DOMAIN SIZE: To create a realistic water simulation the simulator needs to know the size the domain cube represents in the real world. Under section “Fluid World” find the setting “Real World Size”. This value indicates the longest side of the cube in meters. So a value of 0.4 represents 40 cm.

SLIP-TYPE: You can find these settings under the section “Fluid Boundary”.   The slip-type determines the stickiness of the surface of the boundary (surface adhesion). You can change the surfaces Smoothing Options (0 = off, 1=Standard,…) and Subdivision (the resolution of the surface for the calculations: 1=off, 2=1 subdivide, 3=2 subdivides,…). Be careful! A high value of the resolution increases the calculation of the simulation significant.

PARTICLES: To create a more realistic simulation use particles. To be able to use particles (splash when hit boundary/obstacles) you need to set the subdivision (Boundary settings) to at least 2. Tracer allows to define how much particles already exist at the beginning od the simulation.

Create fluid obstacles

Although it is possible to use the objects we already created as obstacles in the simulation it sometimes can be more effective to create simplified versions of the objects as obstacles to reduce calculation time.  To illustrate this duplicate the objects drinking glass and the “wet” surface and name them “Drinking glass obstacle” and “kitchenette surface obstacle” or similar.

Make sure you turn rendering off for these obstacle objects. In the “Physics”-Tab select “Fluid” and choose “Obstacle” as type.

Since we have our surface set as obstacle we can modify our visible kitchenette surface and modify it to a wet ground. For that subdivide the visible plane several times and use extraction tools to shape it like a little water surface (see an example in the next section below the glass).

Create InFlow object

Img11-InflowSphereSince we do want the water mysteriously appear in the middle of the air and fill the glass we need an InFlow-Object to indicate where the water will come from. Create a small sphere which you also need to hide from rendering. This sphere must reside inside the fluid domain!

Img11-InflowSettingsIn the “Physics”-tab activate “Fluid” and set the type to “In-Flow”.  We initialize the volume with the volume of the object (sphere) so we set the “Volume initialization” to “Volume”. In my case I want to have velocity along the positive x axis so I set the X velocity to 0.6.

We also do not want to poor constantly water into the glass so we enable the InFlow object only for a brief moment in time. For that we navigate to frame 0 in our time line. Activate the “Enabled”-Checkbox and right click it to  insert a key frame. Move forward to frame 18. Deactivate the “Enabled”-Checkbox and set another key frame here.

Bake the fluid simulation

To bake the fluid simulation simply switch to the fluid domain object and hit the “Bake”-Button in the Physics-Tab. This will take a time. After that you will see the simulation when you scrub through the timeline and also when you render it:

Img14-BakedWater Img15-RenderNoNodes

Preparing render for After Effects composition

Finally we do not want to render the background footage directly but composite it later in After Effects. To do that we need to extract the background footage. Since we want to keep all the reflections we cannot simply remove the background render in the world-tab.

PassINdexFor the the drinking glass and the fluid domain set the PASS INDEX to 1 (object tab). For the water surface set the PASS INDEX to 2. Switch to “Nodes”-View and and select “World” in the bottom toolbar. Use the ID-Mask nodes to isolate a alpha map for the objects with the pass index 1 and two. Then use the “Set Alpha” node to isolate the object from the rendered image. We now can take the result from ID 2 and modify it with RGB curves and make it slightly transparent by using an Alpha Over. We place the result of the isolation in the foreground (lower image input) and the the upper image to black (0/0/0) with Alpha 0. We use the factor to make the water surface transparent. In our case I set this to 0.427. We then combine the water surface and the class with the water again with an Alpha Over and take the result as final render.



Now we can finally render our animation and integrate it with the original footage in After Effects. I hope you have enjoyed this tutorial.


Blender 3D Integration with PFTrack

imageI recently bought a DJI Ronin to be able to do smooth shots. Not only a smooth shot is much more enjoyable for the eye it also is much easier to track if you do 3D-Integration. While shooting smooth shots with the DJI Ronin still requires a lot of practice the shots are a lot smoother as if they were shot plain handed.

Today I will show you how to import a moving shot into PF-Track. Then we will of course track the camera but also to position a 3D-Model (glass) on the kitchenette.  In the image on the right I have visualized my shot where my camera moves from left to right.

I structured the post with the following topics

  • PFTrack application
    • Creating the required nodes in PFTrack
    • Configure “Photo Survey” node, match points and solve camera
    • Configure “Photo Mesh” and create the mesh
    • Exporting mesh and camera data to Blender
  • Blender application
    • Importing mesh and camera data into Blender
    • Verification (optional) but recommended

I am using PFTrack 2015.05.15 for this. Create a new project PFTrack first. Change to project view by clicking on the “PRJ” Button in the left lower corner (image).  Click the “Create”-Button, fill out Name, Path,… and click “Confirm”. Enable filebrowser and project media browser by clicking on the corresponding icons in the top of the application (image). Import the footage by dragging it into the “Default”-Folder or create your own project structure. 

Creating the required nodes in PFTrack

Drag your shot to the node window. In the lower left corner enable the “nodes-menu” (image).  Click on “Create” to create an “Photo Survey”-node. Setup “Photo Mesh”- and “Export”- nodes with the same procedure. Your node tree should look like this:

Configure “Photo Survey” node, match points and solve camera

imageDouble click the “Auto Track” node. Since the calculations take quite some time we should only calculate what is necessary. Since I have a much longer recording (switching the camera on/off while you are holding the heavy DJI Ronin is quite a challenge) I only need to track a small portion. In my case from frame 431 to 618. Open “Parameters” of the “Photo Survey”-node and setStart frame” and “End frame” in the “Point Matching” section. Finally hit “AutoMatch” (image) and wait until the calculations are done.

After the points have been tracked click on the “Solve all” button (image) in the “Camera Solver” section. If you enable the split view (see buttons on the right corner) you will end up with a point cloud and a solved camera:

Configure “Photo Mesh” node and create the mesh

After solving the camera we need to create depth maps for each frame and create a mesh. Note that you won’t get a perfect mesh but it will suffice to help place things in the 3D world in Blender later. Switch to the “Photo Mesh” nodes. If you do not require all points set the bounding box accordingly in the “Scene” section. To do this click the “Edit” button.  If you hover now over the planes of the bounding box in the 3D view they will highlight and can be moved by dragging them with the mouse. Once you are finished hit “Edit” again.

Let’s create the depth maps next. Depending on your requirement set the Depth Maps resolution to “Low”, “Medium” or “High”. Be aware that a higher resolution results in a much longer calculation. I left the variation % at the default of 20 and set my resolution to “Medium”. Now hit the “Create” button in the “Depth Mapssection. This will take a while.

After building the depth maps we can create the mesh. Note that you also could create a smaller portion of the mesh by setting the bounding box in the “Mesh” section.  Create the mesh simply by hitting the “Createbutton in the “Meshsection. And finally we should have our mesh:

Exporting mesh and camera data to Blender

PFTrack offers to export the mesh and camera data in various formats: “Open Alembic”, “Autodesk FBX 2010 (binary)”. Also you can export the mesh without camera to “Wavefront OBJ” and “PLY”. “Open Alembic” export fails on my windows pc and I have not been able to use that so far.

For Blender we we should have two options: “Autodesk FBX 2010 (binary)” and “Wavefront OBJ”.

Unfortunately we have two issues with the FBX format. First of all Blender can only import “Autodesk FBX 2013 (binary)”. Therefore we need an extra step in converting the fbx-file with Autodesks FBX Converter 2013.2. This allows us to import the cameras and the mesh, but the camera rotations are completely messed up Sad smile.  I do not know if this is a bug in Blender or PFTrack but it does not help to make a smooth workflow. So what is the solution?

imageThe solution is to split up camera- and mesh export. So first we export the mesh as “Wavefront OBJ”. Since Blender uses the z-axis for up/down we change the default settings for the coordinate system toRighthanded” and “Z up”. Then we name an output file (f.e. Kitchen-z-up.obj) and click on the button “Export Mesh”.

To export the camera data we use the previously created “Export” node that is connected to the “Photo Survey” node. In the parameters of the “Export” node we select the formatCollada DAE”. Choose what to export in the TABs on the right side. Since I won’t be needing the point cloud I removed the point cloud from the export. Make sure that the camera is selected and “Separate Frame” is not checked. If checked PFTrack would create a separate camera for each frame. Since we do want to render an animation later leave that unchecked. Name the output file (f.e. KitchenNoPC.dae) and hit the “Export Scenebutton.


So we end up with two files. One (Kitchen-z-up.obj) contains our model and the other (KitchenNoPC.dae) our animated tracked camera.

Importing mesh and camera data into Blender

Start up Blender (I am using Version 2.74).  Open user preferences (STRG+ALT+U) and select the “AddOns”-Tab. Select the categoryImport-Export” and make sure that “Import-Export Wavefront OBJ format– AddOn is selected.

First make sure that our render settings are set correctly. We set resolution (should match with footage) and frame rate.
(It is crucial to set the frame rate correctly before you import the animated camera !! Otherwise the camera will be out of sync even if you change the frame rate later!!)

imageSelect “File/Import/Wavefront (.obj)” from the menu. Navigate to mesh obj file you created with PFTrack (f.e. “Kitchen-z-up.obj”) Make sure that you change the import settings in the left lower corner as shown in this image:

Then click on “Import” to import the object.


Select “File/Import/Collada (Default) (.dae)” from the menu. Navigate to the exported collada camera track file (f.e. KitchenNoPC.dae) and clickImport COLLADA”.

This will import two objects: An empty object “CameraGroup_1” and the animated cameraImageCamera01_2” (names can vary of course). Although the position of the camera after the import looks correct, the position of the camera will rotate 90 degrees on the global x-axis once you scrub through the timeline. I assume that the Pixelfarm team meant to parent the “ImageCamera01_2” to the “CameraGroup_1” because the empty object is rotated 90 degrees on the x-axis.

imageSo simply select the animated cameraImageCamera01_2”. In the object settings (image) select parent and choose the empty object CameraGroup_1”.

And we are almost finished. Since PFTrack exports the camera over the full length of the shot you might want to define the animation range in the timeline window like so:

Finally we need to fix the field of view of the camera which is also not correctly imported/exported by PFTrack. In PFTrack double click on the “Photo Survey” node and you can find the camera settings in the camera tab:

So back in Blender select the animated camera (“ImageCamera01_2”) and switch to the camera settings. Change the sensor to “Horizontal” and set the width to the film back value from PFTrack. In this case “14.75680”.
(!! Make sure your render settings are set to the same aspect ratio as your footage !!).

Then change the focal length of the camera to the value from PFTrack. In this case “12.851”.

imageVerification (optional but recommended): To see if everything is correct I recommend to load the original footage as background and see if it matches correctly. Mistake with f.e.  the frame rate settings happen easily. To do this select the animated camera again. With the mouse cursor in the 3D View hitN” to show the settings of the selected object in the right bar in the 3D View.

Find “Background Image” setting and check it. Then hit the “Add Image” button.

Then selectMovie Clip” instead of “Image”. Uncheck the option “Camera Clip”. ClickOpen” and navigate to the footage and click “Open Clip”.

Then click “Front” and set the Opacity to 0.500.
Now you can scrub through the time line and see if everything lines up perfectly.


Next thing of course is to create some 3D objects and place them on the table. For the final render we simply move the mesh to another layer and mix the original footage with our CGI objects in After Effects. Pay attention to things like lightning and reflection. Maybe a topic for another post Smile. So long.