I propose an idea about a theoretical imaging sensor that features adaptive frameless sampling schema. At the moment it probably isn't technologically feasible but who knows, in a few years it could be already practical.
Main features
Primary idea of Adaptive Sampling Sensor (or ASS?) is that it should sample each image sensor element temporally unrelated to other elements. Data from each element should be collected then and only then if collected charge rises above determined noise floor. This means that element has gathered enough light to actually contain any useful information. Second important feature should be that element does not output data if change between two samples has been smaller than determined value. Pre-requisite for implementing these features is that the time from last data output must be constantly measured because this allows to calculate the "average" value for that time span. Without measuring the time for each element, more time and less light would give the same value as less time and more light.
This theoretical sensor would send data out only when: 1) enough photons have been gathered to rise above noise floor; 2) there has been a significant change in element value. Each element "fires" unrelated to other elements and is "adaptive" to incoming light levels. No light = no changed data = no output.
ASS type sensor mimicks the way eyes work. Signal sampling is unrelated to other cells and signal is generated only when there is change. Human eyes must constantly jitter in order to refresh the image because without change, cells refuse to send signals to brain. Reptilians, who don't have the jitter feature, only see moving objects. To see the whole surround they must constantly move their eyes or themselves.
Main benefits
Main benefits of ASS type sensor are channeling data flow to where it is most needed. For the same bandwidth, more useful information can be gathered. As sampling is not temporally fixed, more samples can be gathered for moving objects and thus reduce stutter and/or motion blur. As fixed temporal sampling goes out of the window, so do frames. Frames (or better to say individual slices of time) can be reconstructed from the feed but it is not necessary if theoretical display supports rendering of temporally adaptive data.
This scheme resembles in some ways the way some video codecs work. Main difference is that codec algorithms can only be applied to already temporally sampled (frame-based) data. This means that data throughput bottleneck happens before encoding and only benefit is reduced storage need. Applying data discarding straight on sensor means that data feed is either reduced or contains relatively more useful information. Raw sensor feed is similar to constant quality variable data rate encoding logic where no data means no storage space wasted.
Tuesday, October 30, 2012
Wednesday, October 24, 2012
Time to go frameless?
For more than a hundred years, the idea of cinema has been fixed on sampling moments of time. Technical advances have made it possible to increase the sampling frequency but the idea is still the same. When comes the time to leave fixed quantization behind and step up with cinema 2.0 as it could be called. The analogue with video codecs could maybe describe the idea: constant size and variable quality vs. variable size and constant quality. At the moment cinema is fixed on constant size and variable quality domain because of strict temporal sampling and constant frame size but the quality of image degrades every time we move the camera or point it on something that moves. Motion blur steals information from us and there is no way to get it back.
Cinema 2.0
Second generation cinema would mean that we take a step away from fixed sampling and make a move into the world of adaptive sampling and constant quality. Eliminating the baked-in motion blur would be the first objective. Let our eyes do the work and decide when visuals move too fast to pick details. World doesn't render motion blur for our eyes, our brain does it. And so should it be.
Cinema 2.0
Second generation cinema would mean that we take a step away from fixed sampling and make a move into the world of adaptive sampling and constant quality. Eliminating the baked-in motion blur would be the first objective. Let our eyes do the work and decide when visuals move too fast to pick details. World doesn't render motion blur for our eyes, our brain does it. And so should it be.
Thursday, October 18, 2012
Interesting times ahead
One thing ends, another begins. In the words of Dave Stewart: "Now show me frame two"
Trying to wrap my head around the question, whether there is any potential in starting a company that specializes in some specific supporting areas for vfx. There are companies that do that but none that I am aware of in the scandinavia or baltic states. Probably I just don't know them. It seems that most companies try to be end-to-end solution to vfx or animation which has it's advantages but also shortcomings. To specialize on some specific time-consuming piece of image manipulation takes both time, opportunity and practice. And there is a very practical limit that comes from the number of people working in the company - the less people, the more generalists they have to be if end-to-end workflow is the goal. That in turn equals lower efficiency and quality.
Two main problems seem to be lack of specialists in the area and possible lack of work due to not being an end-to-end facility. First problem could be solved by slowly building a capable core team of skilled specialists by investing into learning and knowledge. Second is a bit more difficult but with fast, cost-effective and high-quality work, existing post and vfx houses might see the that outsourcing some self-contained parts of work can leverage the quality of their work and let them do what THEY do best.
Keeping a highly specialized artist in the team is not cost-effective for most small(er) vfx houses. It works with some fields (modellers, animators) but not so well for others. Hiring freelancers for specific projects can solve the problem but freelancers need to be coordinated, workload distributed and so on. Not too bright either. Doing things by assigning team members to special work they are not too comfortable with usually results in 3x more time=budget spent and mediocre results. Plus it keeps them away from doing things they do better. In the end, this might even make the company avoid projects that require some special work or thy to come up with different workarounds that aren't necessarily any cheaper or less time consuming.
I see some light in the end of the tunnel but it is yet difficult to tell if it is a choo-choo train or bright future.
Trying to wrap my head around the question, whether there is any potential in starting a company that specializes in some specific supporting areas for vfx. There are companies that do that but none that I am aware of in the scandinavia or baltic states. Probably I just don't know them. It seems that most companies try to be end-to-end solution to vfx or animation which has it's advantages but also shortcomings. To specialize on some specific time-consuming piece of image manipulation takes both time, opportunity and practice. And there is a very practical limit that comes from the number of people working in the company - the less people, the more generalists they have to be if end-to-end workflow is the goal. That in turn equals lower efficiency and quality.
Two main problems seem to be lack of specialists in the area and possible lack of work due to not being an end-to-end facility. First problem could be solved by slowly building a capable core team of skilled specialists by investing into learning and knowledge. Second is a bit more difficult but with fast, cost-effective and high-quality work, existing post and vfx houses might see the that outsourcing some self-contained parts of work can leverage the quality of their work and let them do what THEY do best.
Keeping a highly specialized artist in the team is not cost-effective for most small(er) vfx houses. It works with some fields (modellers, animators) but not so well for others. Hiring freelancers for specific projects can solve the problem but freelancers need to be coordinated, workload distributed and so on. Not too bright either. Doing things by assigning team members to special work they are not too comfortable with usually results in 3x more time=budget spent and mediocre results. Plus it keeps them away from doing things they do better. In the end, this might even make the company avoid projects that require some special work or thy to come up with different workarounds that aren't necessarily any cheaper or less time consuming.
I see some light in the end of the tunnel but it is yet difficult to tell if it is a choo-choo train or bright future.
Monday, October 1, 2012
3Delight and Blender
That AtomKraft thing for AE didn't really cut it afterall. Maybe I need some more time to figure out how all things work in this horrible mess that AE is for such work.
But 3Delight exporter for Blender, coded by Matt Ebb, is still great! Fiddled with it some more and compiled a new glass shader because the example shaders for glass are.. not very glass-looking. For some reason Blender now can't load shader names in existing scenes, displays text "loading shaders...". When I start Blender up and add material in new scene, everything is fine. Must investigate what is wrong.
Overall, 3Delight is a whole new world. Displacement is great and it should support pTex also. Should download Mari trial or some pTex sample files and try to get pTextures working. Shader writing is interesting, tried to figure out what I must do to compile C++ DSO functions. It seems that it isn't very difficult with cmake. Rendertime Python code evaluation is also something that can be very useful, for example for creating new geometry.
pTex seems to have catched wind very well with prMan, 3Delight, Arnold, vRay, Mari, 3D-Coat, Mudbox, 3DS Max, Maya, Modo, Nuke already supporting it and the list goes longer every day. Hope that Blender also adds pTex support to Cycles renderer and it's painting tools.
But 3Delight exporter for Blender, coded by Matt Ebb, is still great! Fiddled with it some more and compiled a new glass shader because the example shaders for glass are.. not very glass-looking. For some reason Blender now can't load shader names in existing scenes, displays text "loading shaders...". When I start Blender up and add material in new scene, everything is fine. Must investigate what is wrong.
Overall, 3Delight is a whole new world. Displacement is great and it should support pTex also. Should download Mari trial or some pTex sample files and try to get pTextures working. Shader writing is interesting, tried to figure out what I must do to compile C++ DSO functions. It seems that it isn't very difficult with cmake. Rendertime Python code evaluation is also something that can be very useful, for example for creating new geometry.
pTex seems to have catched wind very well with prMan, 3Delight, Arnold, vRay, Mari, 3D-Coat, Mudbox, 3DS Max, Maya, Modo, Nuke already supporting it and the list goes longer every day. Hope that Blender also adds pTex support to Cycles renderer and it's painting tools.
Thursday, September 27, 2012
3Delight render engine in AE and Nuke
Just found out that company Jupiter Jazz has released their AtomKraft toolkit for both After Effects and Nuke. I read about AtomKraft sometime last year but as it was still in beta or whatever phase I forgot about it. Now I rediscovered it and guess what! It offers 3Delight engine in AE for only 99$ and in Nuke for 399$. They also have a free watermarked version for AE and free 2-thread version for Nuke. Kind of cheap really considering what it can do.
So what can it do?
Basically it can:
This whole thing made me think that maybe it is time to try to render our next animation project in 3Delight through Aftereffects. Sounds a bit odd but it's probably worth a try. Only thing to do now is get Blender Alembic exporter working and simply stuff things together in AE...
So what can it do?
Basically it can:
- read obj, Alembic and RIB files
- read Renderman Shading Language shaders and compile them render-time
- provide almost all features of 3Delight including GI, SSS, primitives, micropolygon displacement, motion blur, dof, HDRI lighting, image based shadows, and so on
- use Cortex tools
- do camera projections (in After Effects!)
- texture baking
- render passes
- proxy geometry
This whole thing made me think that maybe it is time to try to render our next animation project in 3Delight through Aftereffects. Sounds a bit odd but it's probably worth a try. Only thing to do now is get Blender Alembic exporter working and simply stuff things together in AE...
Monday, September 24, 2012
Trip to Dolomites
We really enjoyed the trip even though we had to retreat a bit early due to snowfall and cold weather. Sella towers are a beautiful place for climbing and really easy to access. At first we thought we would sleep in the refuge that is located on Sella pass but when we got there we decided to sleep in tent. Finding a place for tent there wasn't a very easy task but eventually we found a place that had been used before and was well hidden also :)
Due to our general lack of shape we managed to climb only two longer routes, the Steger route on First Sella tower and the Kasnakoff route on Second Sella. Steger was nice and easy, on Kasnakoff we made some quirky variation through rotten rock that deviated from the IV+ variant that is recommended in newer books. In addition we climbed some single-pitch routes on the Citta dei Sassi boulder field nearby.
 |
| Kasnapoff route on Second Sella - variation pitches |
Tuesday, August 28, 2012
LUT for viewing CF raw
After some messing around I finally made a LUT that makes raw cineform files look like both transforms discussed in previous post (sensor > XYZ and XYZ > sRGB) have been applied.
EDIT: lut file link is broken at the moment, try to fix it soon...
Download the LUT file (.cube format) here:
Register the LUT file by doubleclicking on it (Windows), this installs it to Firstlight LUT directory.
To apply the lut, make sure that input and output curves are set to the same setting and that primary corrections are disabled. For some reason enabling primary corrections makes image slightly pink in highlights.
This lut works nicely in NLEs and video players but for some reason Resolve does not show any Cineform corrections. It seems that Resolve is able to reference raw file directly and bypasses Cineform renderer (confirmed in bmcuser forum by David Newman from Cineform)
WB correction, dammit!
Previous LUT makes image viewable but it still looks a bit too green. This is due to missing white balancing step which I tried to get working and got working (kind of) but I'm not sure I made it correctly.
DNG files contain metadata called as-shot-neutral which should be the value which is considered white (according to which illuminant? must dig more). In BMCC files, values read something like 0.63 1.00 0.79. If we take them as RGB values, it doesn't make any sense because RGB white should be 1.00 1.00 1.00. I figured it must be in either sensor values or in XYZ. Messing around with it, I thought that applying sensor > XYZ transform to this colour would give me white in XYZ space that I could then use to grade raw data in XYZ space by setting it's white point to my values. It works, kind of, because it removes the green tint but colors still seem a bit off.
Some sources (dcRaw developer for example) suggest that WB correction values to be applied before debayering because it prevents some debayer artifacts. If I could do it, I would try it... Seems that I must code my own test engine to try out my genius ideas... maybe even make it work on CUDA...
EDIT: lut file link is broken at the moment, try to fix it soon...
Register the LUT file by doubleclicking on it (Windows), this installs it to Firstlight LUT directory.
To apply the lut, make sure that input and output curves are set to the same setting and that primary corrections are disabled. For some reason enabling primary corrections makes image slightly pink in highlights.
This lut works nicely in NLEs and video players but for some reason Resolve does not show any Cineform corrections. It seems that Resolve is able to reference raw file directly and bypasses Cineform renderer (confirmed in bmcuser forum by David Newman from Cineform)
WB correction, dammit!
Previous LUT makes image viewable but it still looks a bit too green. This is due to missing white balancing step which I tried to get working and got working (kind of) but I'm not sure I made it correctly.
DNG files contain metadata called as-shot-neutral which should be the value which is considered white (according to which illuminant? must dig more). In BMCC files, values read something like 0.63 1.00 0.79. If we take them as RGB values, it doesn't make any sense because RGB white should be 1.00 1.00 1.00. I figured it must be in either sensor values or in XYZ. Messing around with it, I thought that applying sensor > XYZ transform to this colour would give me white in XYZ space that I could then use to grade raw data in XYZ space by setting it's white point to my values. It works, kind of, because it removes the green tint but colors still seem a bit off.
Some sources (dcRaw developer for example) suggest that WB correction values to be applied before debayering because it prevents some debayer artifacts. If I could do it, I would try it... Seems that I must code my own test engine to try out my genius ideas... maybe even make it work on CUDA...
BMCC RAW to Cineform RAW
EDIT: useful links
http://lclevy.free.fr/cr2/
http://www.guillermoluijk.com/tutorial/dcraw/index_en.htm
One of the hot topics in BMCuser forum is the Cineform Conversion and particularly the raw dng to Cineform raw possibility. As Cineform can convert dpx files to cineform raw files (with dpx2cf command line utility), this could be one way to deal with large file sizes that working with raw files tends to bring upon.
NOTE: Older Cineform versions don't seem to support dng conversion, you need release from at least april/may of 2012! Otherwise you get the "DPX file not found" error.
DNG sequence to Cineform RAW file
Converting dng sequence to cineform is pretty easy. Use dpx2cf tool with, for example, following command:
dpx2cf.exe Frame*.dng shot1_cf.avi -q5
This reads all dng files from shown source (* marks frame numbers) and converts it into cineform file. Additional settings like frame rate etc can be set, see settings by simply running the program in command prompt.
EDIT: there was a typo in command example, there should not have been a " mark after shot1_cf.avi.
Cineform RAW looks green?
After conversion it is immediately apparent that something is wrong. Image looks very green and strange. This is because Cinefrom converter does not apply any correction on the image and we see pure raw (debayered ofcourse) image. As sensors are usually more sensitive to green light, image looks green.
Raw processing software (like Lightroom or Camera Raw) applies camera calibration correction before showing the image to user and so we don't really know what the sensor actually sees. Because dng and other raw formats contain these profiles in their metadata, raw conversion software usually knows how to perform this correction. Cineform converter on the other hand does not use this info, puts raw sensor values straight into the file and also shows us these values without correction.
Reading DNG metadata
To obtain metadata from dng file, we need some software that can extract that info from the file header. For example a handy set of tools called EXIFUtils. With this tool we can extract all the interesting metadata that BMCC dng files have to offer.
For example:
Camera
image-type : Main
main-width : 2432
main-len : 1366
main-bits-sam : 12
main-comp : None
photo-int : Color Filter Array (CFA)
strip-off : 9856
orient : Upper Left
sample-pix : 1
row-strip : 1366
strip-cnt : 4983168
planar-conf : Chunky
cfa-rpt-pat-dim : 2 2
cfa-rpt-pattern : 01000201
tag-9216 : 00000001
dng-version : 01020000
dng-model : Blackmagic Cinema Camera
dng-lin-table : 54 55 55 56 57 58 59 59 60 61 62 63 63 64 65
66 66 67 68 69 70 70 71 72 73 74 74 75 76 77
78 78 79 80 81 81 82 83 84 85 85 86 87 88 89
89 90 91 92 93 93 94 95 96 96 97 98 99 100
100 101 102 103 104 104 105 106 107 107 108
109 110 111 111 112 113 114 115 115 116 117
118 119 119 120 121 122 122 123 124 125 126
126 127 128 129 130 130 131 132 133 134 134
135 136 137 137 138 139 140 141 141 142 143
144 145 145 146 147 148 149 149 150 151 152
152 153 154 155 156 156 157 158 159 160 160
161 162 163 164 164 165 166 167 167 168 169
170 171 171 172 173 174 175 175 176 177 178
179 179 180 181 182 182 183 184 185 186 186
187 188 189 190 190 191 192 193 194 194 195
196 197 197 198 199 200 201 201 202 203 204
205 205 206 207 208 209 209 210 211 212 212
213 214 215 216 216 217 218 219 220 220 221
222 223 223 224 225 226 227 227 228 229 230
231 231 232 233 234 235 235 236 237 238 238
239 240 241 242 242 243 244 245 246 246 247
248 249 250 250 251 252 253 253 254 255 256
257 257 258 259 260 261 261 262 263 264 265
265 266 267 268 268 269 270 271 272 272 273
274 275 276 276 277 278 279 280 280 281 282
283 283 284 285 286 287 287 288 289 290 291
291 292 293 294 295 295 296 297 298 298 299
300 301 302 302 303 304 305 306 306 307 308
309 310 310 311 312 313 313 314 315 316 317
317 318 319 320 321 321 322 323 324 325 325
326 327 328 328 329 330 331 332 332 333 334
335 336 336 337 338 339 339 340 341 342 343
343 344 345 346 347 347 348 349 350 351 351
352 353 354 354 355 356 357 358 358 359 360
361 362 362 363 364 365 366 366 367 368 369
369 370 371 372 373 373 374 375 376 377 377
378 379 380 381 381 382 383 384 384 385 386
387 388 388 389 390 391 392 392 393 394 395
396 396 397 398 399 399 400 401 402 403 403
404 405 406 407 407 408 409 410 411 411 412
413 414 414 415 416 417 418 418 419 420 421
422 422 423 424 425 426 426 427 428 429 429
430 431 432 433 433 434 435 436 437 437 438
439 440 441 441 442 443 444 444 445 446 447
448 448 449 450 451 452 452 453 454 455 455
456 457 458 459 4
dng-bl-lvl-dim : 1 1
dng-bl-level : 256
dng-wh-level : 60074
dng-def-crop-origin : 16 8
dng-def-crop-size : 2400 1350
dng-color-matrix-1 : 1.31 -0.50 0.01 -0.42 1.44 0.05 0.07 0.22 0.
73
dng-color-matrix-2 : 1.01 -0.27 -0.08 -0.49 1.34 0.11 -0.06 0.33
0.54
dng-camera-calib-1 : 1.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00
dng-camera-calib-2 : 1.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00
dng-as-shot-neutral : 0.63 1.00 0.79
dng-base-exp : 2.40
dng-calib-illum-1 : 17
dng-calib-illum-2 : 21
tag-c763 : 0859011400000000
tag-c764 : 24.00
We get a lot of info about file size, black levels, crops etc. And some color matrices.
DNG color matrix
These color matrices are for transforming between sensor values and XYZ values. When I first tried applying the matrix, image still looked strange and green until I got that these matrices are meant for converting from XYZ to sensor values. To get sensor->XYZ conversion, we need an inverted matrix!
Inverted matrix for conversion sensor values -> XYZ:
0.863 0.305 -0.033
0.257 0.793 -0.058
-0.16 -0.268 1.39
After applying this matrix, image still looks a bit strange, that's because we still need to apply XYZ -> sRGB transform to properly view it on computer or broadcast monitor.
Matrix for conversion from CIE-XYZ to Linear D65 sRGB:
3.24081 -1.53731 -0.498587
-0.969243 1.87597 0.0415551
0.0556384 -0.204007 1.05713
After these conversions we get the nice linear image in sRGB space that we can grade.
Sample images (with sRGB gamma applied for viewing purposes!)
These samples are images after conversion to sRGB and gamma transform. It is obvious that they have some differences compared to the way Resolve or Lightroom present the dng-s. Values seem to be more evenly distributed and image is somewhat flatter. There also is some green cast in first shots. This is because no G/M tint correction has been applied yet.
Cineform debayer algorithms
Following are samples from shot 1 with different debayer settings left-right, top-bottom:
Bilinear demosaic
Matrix 5x5 adaptive
CF Advanced smooth
CF Advanced detail 1
CF Advanced detail 2
CF Advanced detail 3
http://lclevy.free.fr/cr2/
http://www.guillermoluijk.com/tutorial/dcraw/index_en.htm
One of the hot topics in BMCuser forum is the Cineform Conversion and particularly the raw dng to Cineform raw possibility. As Cineform can convert dpx files to cineform raw files (with dpx2cf command line utility), this could be one way to deal with large file sizes that working with raw files tends to bring upon.
NOTE: Older Cineform versions don't seem to support dng conversion, you need release from at least april/may of 2012! Otherwise you get the "DPX file not found" error.
DNG sequence to Cineform RAW file
Converting dng sequence to cineform is pretty easy. Use dpx2cf tool with, for example, following command:
dpx2cf.exe Frame*.dng shot1_cf.avi -q5
This reads all dng files from shown source (* marks frame numbers) and converts it into cineform file. Additional settings like frame rate etc can be set, see settings by simply running the program in command prompt.
EDIT: there was a typo in command example, there should not have been a " mark after shot1_cf.avi.
Cineform RAW looks green?
After conversion it is immediately apparent that something is wrong. Image looks very green and strange. This is because Cinefrom converter does not apply any correction on the image and we see pure raw (debayered ofcourse) image. As sensors are usually more sensitive to green light, image looks green.
Raw processing software (like Lightroom or Camera Raw) applies camera calibration correction before showing the image to user and so we don't really know what the sensor actually sees. Because dng and other raw formats contain these profiles in their metadata, raw conversion software usually knows how to perform this correction. Cineform converter on the other hand does not use this info, puts raw sensor values straight into the file and also shows us these values without correction.
Reading DNG metadata
To obtain metadata from dng file, we need some software that can extract that info from the file header. For example a handy set of tools called EXIFUtils. With this tool we can extract all the interesting metadata that BMCC dng files have to offer.
For example:
Camera
image-type : Main
main-width : 2432
main-len : 1366
main-bits-sam : 12
main-comp : None
photo-int : Color Filter Array (CFA)
strip-off : 9856
orient : Upper Left
sample-pix : 1
row-strip : 1366
strip-cnt : 4983168
planar-conf : Chunky
cfa-rpt-pat-dim : 2 2
cfa-rpt-pattern : 01000201
tag-9216 : 00000001
dng-version : 01020000
dng-model : Blackmagic Cinema Camera
dng-lin-table : 54 55 55 56 57 58 59 59 60 61 62 63 63 64 65
66 66 67 68 69 70 70 71 72 73 74 74 75 76 77
78 78 79 80 81 81 82 83 84 85 85 86 87 88 89
89 90 91 92 93 93 94 95 96 96 97 98 99 100
100 101 102 103 104 104 105 106 107 107 108
109 110 111 111 112 113 114 115 115 116 117
118 119 119 120 121 122 122 123 124 125 126
126 127 128 129 130 130 131 132 133 134 134
135 136 137 137 138 139 140 141 141 142 143
144 145 145 146 147 148 149 149 150 151 152
152 153 154 155 156 156 157 158 159 160 160
161 162 163 164 164 165 166 167 167 168 169
170 171 171 172 173 174 175 175 176 177 178
179 179 180 181 182 182 183 184 185 186 186
187 188 189 190 190 191 192 193 194 194 195
196 197 197 198 199 200 201 201 202 203 204
205 205 206 207 208 209 209 210 211 212 212
213 214 215 216 216 217 218 219 220 220 221
222 223 223 224 225 226 227 227 228 229 230
231 231 232 233 234 235 235 236 237 238 238
239 240 241 242 242 243 244 245 246 246 247
248 249 250 250 251 252 253 253 254 255 256
257 257 258 259 260 261 261 262 263 264 265
265 266 267 268 268 269 270 271 272 272 273
274 275 276 276 277 278 279 280 280 281 282
283 283 284 285 286 287 287 288 289 290 291
291 292 293 294 295 295 296 297 298 298 299
300 301 302 302 303 304 305 306 306 307 308
309 310 310 311 312 313 313 314 315 316 317
317 318 319 320 321 321 322 323 324 325 325
326 327 328 328 329 330 331 332 332 333 334
335 336 336 337 338 339 339 340 341 342 343
343 344 345 346 347 347 348 349 350 351 351
352 353 354 354 355 356 357 358 358 359 360
361 362 362 363 364 365 366 366 367 368 369
369 370 371 372 373 373 374 375 376 377 377
378 379 380 381 381 382 383 384 384 385 386
387 388 388 389 390 391 392 392 393 394 395
396 396 397 398 399 399 400 401 402 403 403
404 405 406 407 407 408 409 410 411 411 412
413 414 414 415 416 417 418 418 419 420 421
422 422 423 424 425 426 426 427 428 429 429
430 431 432 433 433 434 435 436 437 437 438
439 440 441 441 442 443 444 444 445 446 447
448 448 449 450 451 452 452 453 454 455 455
456 457 458 459 4
dng-bl-lvl-dim : 1 1
dng-bl-level : 256
dng-wh-level : 60074
dng-def-crop-origin : 16 8
dng-def-crop-size : 2400 1350
dng-color-matrix-1 : 1.31 -0.50 0.01 -0.42 1.44 0.05 0.07 0.22 0.
73
dng-color-matrix-2 : 1.01 -0.27 -0.08 -0.49 1.34 0.11 -0.06 0.33
0.54
dng-camera-calib-1 : 1.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00
dng-camera-calib-2 : 1.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00
dng-as-shot-neutral : 0.63 1.00 0.79
dng-base-exp : 2.40
dng-calib-illum-1 : 17
dng-calib-illum-2 : 21
tag-c763 : 0859011400000000
tag-c764 : 24.00
We get a lot of info about file size, black levels, crops etc. And some color matrices.
DNG color matrix
These color matrices are for transforming between sensor values and XYZ values. When I first tried applying the matrix, image still looked strange and green until I got that these matrices are meant for converting from XYZ to sensor values. To get sensor->XYZ conversion, we need an inverted matrix!
Inverted matrix for conversion sensor values -> XYZ:
0.863 0.305 -0.033
0.257 0.793 -0.058
-0.16 -0.268 1.39
After applying this matrix, image still looks a bit strange, that's because we still need to apply XYZ -> sRGB transform to properly view it on computer or broadcast monitor.
Matrix for conversion from CIE-XYZ to Linear D65 sRGB:
3.24081 -1.53731 -0.498587
-0.969243 1.87597 0.0415551
0.0556384 -0.204007 1.05713
After these conversions we get the nice linear image in sRGB space that we can grade.
Sample images (with sRGB gamma applied for viewing purposes!)
These samples are images after conversion to sRGB and gamma transform. It is obvious that they have some differences compared to the way Resolve or Lightroom present the dng-s. Values seem to be more evenly distributed and image is somewhat flatter. There also is some green cast in first shots. This is because no G/M tint correction has been applied yet.
Cineform debayer algorithms
Following are samples from shot 1 with different debayer settings left-right, top-bottom:
Bilinear demosaic
Matrix 5x5 adaptive
CF Advanced smooth
CF Advanced detail 1
CF Advanced detail 2
CF Advanced detail 3
Friday, August 24, 2012
BMCC sample clips from Afterglow footage
Some sample footage from John Brawley's Afterglow footage have been released. They are DNG files containing camera raw data and are available for download through this post from John's blog HERE.
So, here is my take on grading the raw footage:
Graded in Lightroom because I wanted to try new Lightroom and see what I can do with dng files in there. Actually I'm not too sure I could make the same grade in Resolve because I'm not that used to Resolve yet.
Footage itself is great, some indoor shots, outdoor shot and low-light shots which show how much or little grain you get. Sensor white balance is I think 5600K and evening shots were very warm orangy due to that. I changed WB in Lightroom quite a bit, down to 3100-3200K. For shot 3, balcony, image looked too cool so I pushed WB up to 7200K. For first scenes I lowered the exposure and highlight values to get face back to normal exposure. It's rather amazing what these raw files are capable of! Very nice detail, dynamic range and little noise. Even the last two shots that were shot in low light situation and needed heavy WB correction stayed together very well and noise is not a big issue there.
Overall look - definitely impressive for 3000$ camera!
So, here is my take on grading the raw footage:
Graded in Lightroom because I wanted to try new Lightroom and see what I can do with dng files in there. Actually I'm not too sure I could make the same grade in Resolve because I'm not that used to Resolve yet.
Footage itself is great, some indoor shots, outdoor shot and low-light shots which show how much or little grain you get. Sensor white balance is I think 5600K and evening shots were very warm orangy due to that. I changed WB in Lightroom quite a bit, down to 3100-3200K. For shot 3, balcony, image looked too cool so I pushed WB up to 7200K. For first scenes I lowered the exposure and highlight values to get face back to normal exposure. It's rather amazing what these raw files are capable of! Very nice detail, dynamic range and little noise. Even the last two shots that were shot in low light situation and needed heavy WB correction stayed together very well and noise is not a big issue there.
Overall look - definitely impressive for 3000$ camera!
Monday, August 20, 2012
Something to listen
Since my mobile phone decided that some buttons are useless and should not work, I changed my service provider to get cheaper calls and free new phone. New phone will still be phone sized not a shiny bread board that talks but it has some new bells and whistles my previous phone didn't have. One of them is music player that, in addition to music, can also spend my time with some educational listening. So I decided to download...
The entire FXGuide audio archive
Every single piece of audio from FX Podcast, Red Centre and VFX Show are now downloaded totalling 10.4 GB worth of mp3 files. It should be about 360 hours of listening, more than enough for every bus, train or plane trip in foreseeable future.
When this phone decides to stop working, phones with nice screens and quality video players probably cost next to nothing. Then I will download all the FXGuide TV episodes!
The entire FXGuide audio archive
Every single piece of audio from FX Podcast, Red Centre and VFX Show are now downloaded totalling 10.4 GB worth of mp3 files. It should be about 360 hours of listening, more than enough for every bus, train or plane trip in foreseeable future.
When this phone decides to stop working, phones with nice screens and quality video players probably cost next to nothing. Then I will download all the FXGuide TV episodes!
Thursday, August 16, 2012
Image debayering - what and how?
A recent thread about whether Blackmagic camera will have an antialiasing filter or not has gotten me thinking about the process of debayering/demosaicing in general and it's applications for different purposes. Currently topics discussed in that tread are related to whether or not BMCC will have an AA filter, what would be the consequences of having it or not and what is the purpose of such filter in general. I suggest reading that thread, lots of interesting stuff being written!
So, what is debayering / demosaicing?
Almost all still cameras and most video cameras these days have the so-called CMOS sensor. It means that light is captured on single sensor (as opposed to CCD systems which have separate sensor for each color) and to get color information out of it, a special filter, called Color Filter Array (CFA) is placed on the sensor. This filter has alternating pattern of red, green and blue sites in checkerboard (mostly) or some other pattern (hexagonal patterns are sometimes used). Because human eye is most sensitive to green light and thus green gives us lightness information, CFA has as much green sites as red and blue combined. In 2x2 checker, two diagonal patches would be green and other ones red and blue. Bayer pattern is essentially a hard-wired 4:2:2 subsampling schema that operates on RGB channels (in GRB order in this case).
CMOS sensor itself is basically black and white, it does not "see" color but only the intensity of light. So through filter we get intensities of green, red and blue light in a certain pattern. To get full RGB raster from this intensity map, debayering operation is performed. In it's simplest form, average values of adjascent sites can be calculated to make full image for each color. Simple averaging is easy but tends to give a very low quality image. For better results, "smarter" algorithms can be used that take into account value gradients, similarities and what not. A lot of these algorithms are patented and secret because among digital image makers, better image is worth a lot of money. Some information about different algorithms can be found on Wikipedia
What about BMCC
One topic that got me thinking about the difference between combating moire and aliasing on DSLRs and digital film cameras was the AA filter from company called Mosaic Engineering. They offer a filter that you can attach in front of Canon 7D or 5D sensor and get rid of ugly color moire issues. In the thread mentioned before, some people argued that not having such a filter in front of the BMCC sensor would turn up similar ugly patterns as on most DSLRs. I can't say that I totally get the logic behind this. DSLRs generally skip lines to pull HD image from 5-6K sensor without downscaling that is some expencive processing. Skipping lines makes images vulnerable to moire because when pattern frequency and line skipping frequency clash, moire jumps up.
In the Pool Shark clip from John Brawley, chair with double fibre mesh backing made people jump because "BMCC has moire!" although this pattern is visible for anyone who sees such chair in person. Fibre mesh on the back of chair is somewhat similar to sensor that skips elements. Some spots are "blind", some "see". When this blind-see pattern sees another high-contrast pattern, these blind-see spots can either line up or not. When they don't some spots see another pattern, some dont and new pattern is formed that has maximums at lined up parts and minimums where patterns "cancel" each other out. Like two sine waves summed. This happens with sensors that have gaps between image elements, whatever the reason for this is (skipping, gaps due to sensor construction etc).
BMCC sensor does not (?) have gaps between sensor elements, at least I hope so. Gapless sensor catches all light that falls on it and thus should effectively prevent most moire patterns. It is natural antialiasing where for example a very thin diagonal line leaves mark on all pixels it crosses, although it doesn't cover any of them entirely. The intensity of pixel gives us impression that line crosses it but only barely and thus resulting image does not show aliasing. Aliased image would be produced if we tested for stick only in the dead centers of elements. Most pixels would show nothing and some would show everythin, thus a jagged aliased image.
What happens with this naturally antialiased bayer image during demosaicing is totally another story and this is where most of the ugly colored 1-2 pixel wide edges appear. They are the result of incorrect calculations in predicting what RGB values in this point should be. From totally black and white source image projected on bayer sensor, demosaicing can produce a rather rainbow like result where edges shimmer with purple, yellow, blue and every other color imaginable.
New understanding!
During writing the above text, it clicked that even with gapless sensor, patterns can introduce moire because for example on one element, 100% value is captured but on other two only 50% each because light patch falls on both of them. This leads to bright values on every third element and half-bright on every other two and so a new pattern that didn't exist in projected image appears as moire
This is where AA filter comes to play and cuts off higher frequencies that produce most of such moire.
So, what is debayering / demosaicing?
Almost all still cameras and most video cameras these days have the so-called CMOS sensor. It means that light is captured on single sensor (as opposed to CCD systems which have separate sensor for each color) and to get color information out of it, a special filter, called Color Filter Array (CFA) is placed on the sensor. This filter has alternating pattern of red, green and blue sites in checkerboard (mostly) or some other pattern (hexagonal patterns are sometimes used). Because human eye is most sensitive to green light and thus green gives us lightness information, CFA has as much green sites as red and blue combined. In 2x2 checker, two diagonal patches would be green and other ones red and blue. Bayer pattern is essentially a hard-wired 4:2:2 subsampling schema that operates on RGB channels (in GRB order in this case).
CMOS sensor itself is basically black and white, it does not "see" color but only the intensity of light. So through filter we get intensities of green, red and blue light in a certain pattern. To get full RGB raster from this intensity map, debayering operation is performed. In it's simplest form, average values of adjascent sites can be calculated to make full image for each color. Simple averaging is easy but tends to give a very low quality image. For better results, "smarter" algorithms can be used that take into account value gradients, similarities and what not. A lot of these algorithms are patented and secret because among digital image makers, better image is worth a lot of money. Some information about different algorithms can be found on Wikipedia
What about BMCC
One topic that got me thinking about the difference between combating moire and aliasing on DSLRs and digital film cameras was the AA filter from company called Mosaic Engineering. They offer a filter that you can attach in front of Canon 7D or 5D sensor and get rid of ugly color moire issues. In the thread mentioned before, some people argued that not having such a filter in front of the BMCC sensor would turn up similar ugly patterns as on most DSLRs. I can't say that I totally get the logic behind this. DSLRs generally skip lines to pull HD image from 5-6K sensor without downscaling that is some expencive processing. Skipping lines makes images vulnerable to moire because when pattern frequency and line skipping frequency clash, moire jumps up.
In the Pool Shark clip from John Brawley, chair with double fibre mesh backing made people jump because "BMCC has moire!" although this pattern is visible for anyone who sees such chair in person. Fibre mesh on the back of chair is somewhat similar to sensor that skips elements. Some spots are "blind", some "see". When this blind-see pattern sees another high-contrast pattern, these blind-see spots can either line up or not. When they don't some spots see another pattern, some dont and new pattern is formed that has maximums at lined up parts and minimums where patterns "cancel" each other out. Like two sine waves summed. This happens with sensors that have gaps between image elements, whatever the reason for this is (skipping, gaps due to sensor construction etc).
BMCC sensor does not (?) have gaps between sensor elements, at least I hope so. Gapless sensor catches all light that falls on it and thus should effectively prevent most moire patterns. It is natural antialiasing where for example a very thin diagonal line leaves mark on all pixels it crosses, although it doesn't cover any of them entirely. The intensity of pixel gives us impression that line crosses it but only barely and thus resulting image does not show aliasing. Aliased image would be produced if we tested for stick only in the dead centers of elements. Most pixels would show nothing and some would show everythin, thus a jagged aliased image.
What happens with this naturally antialiased bayer image during demosaicing is totally another story and this is where most of the ugly colored 1-2 pixel wide edges appear. They are the result of incorrect calculations in predicting what RGB values in this point should be. From totally black and white source image projected on bayer sensor, demosaicing can produce a rather rainbow like result where edges shimmer with purple, yellow, blue and every other color imaginable.
New understanding!
During writing the above text, it clicked that even with gapless sensor, patterns can introduce moire because for example on one element, 100% value is captured but on other two only 50% each because light patch falls on both of them. This leads to bright values on every third element and half-bright on every other two and so a new pattern that didn't exist in projected image appears as moire
This is where AA filter comes to play and cuts off higher frequencies that produce most of such moire.
Thursday, July 26, 2012
AfterEffects has wrong DCDM transform?
Yesterday I tried out different things in AfterEffects color management workflow and found out that the DCDM XYZ transform for DCP generation seems to be wrong. Images generated with this setting have a slight magenta cast when viewed with inverse transform in Stereoscopic Player or EasyDCP Player. Images transformed with OpenDCP are fine in players and look identical to originals. Must do some more fiddling and see if error is between chair and monitor or in AE...
Spheron camera does 3D measurements..?
Happened to read about the Spheron camera today and found out that it also does 3D measurements. As their homepage is rather vague about the technology behind it (or rather the description is easy to overlook) it took me some time to find out how it is done. It seems that for 3D measurements you have to shoot two panoramas at different heights and from these two images, simple parallax measurements give you the position in 3D space.
Sounds easy but raised a few questions. How do you measure the position of objects near the "poles" as approaching the poles, separation between the two images disappears? Do you have to measure the set and mark the distance between two points on the image to give the scene it's scale?
Sounds easy but raised a few questions. How do you measure the position of objects near the "poles" as approaching the poles, separation between the two images disappears? Do you have to measure the set and mark the distance between two points on the image to give the scene it's scale?
Wednesday, July 25, 2012
Thoughts on BMCC part 3
Random thoughts on Blackmagic Cinema Camera.
Lenses and stuff
BMCC camera has canon EF lens mount which means that it accepts all EF, EF-S and ZE lenses. With Nikon to Canon adapter, Nikon lenses should be also possible. Camera has electronic contacts to drive iris and focus (?, said to have focus control through LANC). Iris adjustments are made through menu or by pressing the IRIS button on camera which sets iris so that no pixel is clipped.
As the camera sensor is about 15.8x8.9 mm in size, it has a crop factor of approximately 2.3x in relation to full-frame (Vistavision size) sensor. This is larger than for example Canon 550D or 7D, both of which have crop factor of 1.6x. Compared to 35mm and 16mm film cameras, BMCC is placed somewhere inbetween as Super16 measures 12.52x7.41mm and 35mm is something like 22x12mm. BMCC is somewhat like oversized Super16 and resolution should also be similar to that of 16mm film scan.
A lot of people seem to complain about the not-so-great bokeh abilities due to small sensor but I can't really get the logic behind it. As this camera accepts the EF lenses and focal plane is at the same distance as DSLRs, only effect the smaller sensor has is cropping part of the image. In my eyes it makes the bokeh effect stronger not weaker because the relative size of bokeh rings increases. Anyway, how many shots will be having need for strong artistic bokeh? Not too many I think, because you wouldn't see a damn thing this way...
The last missing feature
Although the camera looks great it still has one missing feature in my opinion and that is genlock input. It could probably be added with firmware update by somehow utilizing the LANC or audio input socket. Don't know exactly if this is technically possible but it shouldn't be very difficult to do as BNC cables with 3.5mm "mini genlock" jack exist and in the end it's only electricity and a bunch of wires.
Why would I need genlock input? I would like to use this camera for stereoscopic shooting and for that, genlock is a must. LANC could be used to start recording at the same time but without constant syncing, cameras would drift out of sync.
Lenses and stuff
BMCC camera has canon EF lens mount which means that it accepts all EF, EF-S and ZE lenses. With Nikon to Canon adapter, Nikon lenses should be also possible. Camera has electronic contacts to drive iris and focus (?, said to have focus control through LANC). Iris adjustments are made through menu or by pressing the IRIS button on camera which sets iris so that no pixel is clipped.
As the camera sensor is about 15.8x8.9 mm in size, it has a crop factor of approximately 2.3x in relation to full-frame (Vistavision size) sensor. This is larger than for example Canon 550D or 7D, both of which have crop factor of 1.6x. Compared to 35mm and 16mm film cameras, BMCC is placed somewhere inbetween as Super16 measures 12.52x7.41mm and 35mm is something like 22x12mm. BMCC is somewhat like oversized Super16 and resolution should also be similar to that of 16mm film scan.
A lot of people seem to complain about the not-so-great bokeh abilities due to small sensor but I can't really get the logic behind it. As this camera accepts the EF lenses and focal plane is at the same distance as DSLRs, only effect the smaller sensor has is cropping part of the image. In my eyes it makes the bokeh effect stronger not weaker because the relative size of bokeh rings increases. Anyway, how many shots will be having need for strong artistic bokeh? Not too many I think, because you wouldn't see a damn thing this way...
The last missing feature
Although the camera looks great it still has one missing feature in my opinion and that is genlock input. It could probably be added with firmware update by somehow utilizing the LANC or audio input socket. Don't know exactly if this is technically possible but it shouldn't be very difficult to do as BNC cables with 3.5mm "mini genlock" jack exist and in the end it's only electricity and a bunch of wires.
Why would I need genlock input? I would like to use this camera for stereoscopic shooting and for that, genlock is a must. LANC could be used to start recording at the same time but without constant syncing, cameras would drift out of sync.
Thoughts on BMCC part 2
Thoughts on Blackmagic Cinema Camera
Dynamic range and encoding options
Previous post discussed the overall story of BMCC and 2.5K bayer pattern CMOS sensor, lets continue in the same line. According to Blackmagic, this camera will have 13 stops of latitude which is great! Why is it great? Because the more latitude you have, the less blown out highlights and black shadow splots you will have. You can always add your clippings and smooth rollouts to your liking but important thing is that you will have the option to get every drop this camera is able to capture.
For recording raw sensor data, CinemaDNG file format is used. It is an Adobe initiated open format for raw sensor data that supports different bayer patterns and bunch of other features. One 2.5K raw file will be about 5MB with 12 bits per pixel. BMCC will use log encoding for raw files and 12 bits are plenty for that. There is a nice discussion about log versus linear in Cineform insider blog. Log encoding preserves more information in shadow areas and is thus more suitable for digital cameras that tend to need some under-exposure to avoid rather ugly looking clipped highlights. On the downside, as log image does not linarly match captured light intensity values but instead contains the derived log-encoded values, it must be linearized with correct operation that suits the log encoding curve. Such curves are usually provided by camera manufacturers in form of simple reversible 1D LUT. When this camera will be released, Davinci Resolve will probably get an update with ACES IDT transform for BMCC camera.
Other encoding options besides CinemaDNG are Prores 422HQ and DNxHD, both in either log encoding for grading or rec709 for fast stuff. Prores is mostly nice feature for FCP users because it suits so damn well with it and Resolve. DNxHD should make both Avid and general Windows/Linux user happy. It is possible to decode Prores on Windows too (Quicktime does that) and with ffmpeg, it is possible to encode Prores on Windows although FCP is said to complain about the files (some missing metadata) although it plays them.
Recording media
Recording media for BMCC is SSD drives which is really nice because they are much cheaper per GB and more readily available than some exotic memory card format. And they are fast too. One 256GB SSD is said to hold about 30 min of RAW CinemaDNG sequences or 2,5h of Prores or DNxHD. SSDs must be formatted in Apple file system but it can be done on Windows too with certain programs. It is said that camera itself cannot format the drive, so no quick-buying these during shoots...
Probably the camera also accepts standard 2.5" HDD drives which are dirt cheap and should be fast enough for Prores/DNxHD 220Mbit/s speed (less than 30MB/s). One 500GB HDD should be able to record almost 5 hours of footage! Someone should try this out!
Dynamic range and encoding options
Previous post discussed the overall story of BMCC and 2.5K bayer pattern CMOS sensor, lets continue in the same line. According to Blackmagic, this camera will have 13 stops of latitude which is great! Why is it great? Because the more latitude you have, the less blown out highlights and black shadow splots you will have. You can always add your clippings and smooth rollouts to your liking but important thing is that you will have the option to get every drop this camera is able to capture.
For recording raw sensor data, CinemaDNG file format is used. It is an Adobe initiated open format for raw sensor data that supports different bayer patterns and bunch of other features. One 2.5K raw file will be about 5MB with 12 bits per pixel. BMCC will use log encoding for raw files and 12 bits are plenty for that. There is a nice discussion about log versus linear in Cineform insider blog. Log encoding preserves more information in shadow areas and is thus more suitable for digital cameras that tend to need some under-exposure to avoid rather ugly looking clipped highlights. On the downside, as log image does not linarly match captured light intensity values but instead contains the derived log-encoded values, it must be linearized with correct operation that suits the log encoding curve. Such curves are usually provided by camera manufacturers in form of simple reversible 1D LUT. When this camera will be released, Davinci Resolve will probably get an update with ACES IDT transform for BMCC camera.
Other encoding options besides CinemaDNG are Prores 422HQ and DNxHD, both in either log encoding for grading or rec709 for fast stuff. Prores is mostly nice feature for FCP users because it suits so damn well with it and Resolve. DNxHD should make both Avid and general Windows/Linux user happy. It is possible to decode Prores on Windows too (Quicktime does that) and with ffmpeg, it is possible to encode Prores on Windows although FCP is said to complain about the files (some missing metadata) although it plays them.
Recording media
Recording media for BMCC is SSD drives which is really nice because they are much cheaper per GB and more readily available than some exotic memory card format. And they are fast too. One 256GB SSD is said to hold about 30 min of RAW CinemaDNG sequences or 2,5h of Prores or DNxHD. SSDs must be formatted in Apple file system but it can be done on Windows too with certain programs. It is said that camera itself cannot format the drive, so no quick-buying these during shoots...
Probably the camera also accepts standard 2.5" HDD drives which are dirt cheap and should be fast enough for Prores/DNxHD 220Mbit/s speed (less than 30MB/s). One 500GB HDD should be able to record almost 5 hours of footage! Someone should try this out!
Thoughts on Blackmagic Cinema Camera
Blackmagic Cinema Camera is to be released in a few days, on 27th of July to be exact. I have been following the discussions about this camera in bmcuser.com forum and here are my two cents on this topic.
Since 2008 when Vincent Laforet shot the famous Reverie short that started the whole DSLR film mania, little has happened camera-wize on the low-budget end of nice looking filmmaking. Hacks for Canon cameras and Panasonic GH2 have opened some additional possibilities with higher bit-rates and other rather insignificant features but nothing really exiting has happened. It seems that Canon has dropped the ball in DSLR development because new camera models have brought absolutely nothing new feature or quality wise. Probably it is so because of things like C300 and C500 but they are 5x more expensive than say a 5dmkII.
In may 2012, Blackmagic Design, company best known for video capture cards, converters etc, announced their plans to make a 2.5K CMOS sensor camera that shoots CinemaDNG raw for only 2995$. And not only announced, they showed a working prototype also! RED once also promised 3K for 3K$ but this utopia never realized and Scarlet became the somewhat lighter twin brother for Epic with a much more heftier price tag than 3K$.
So, only three months ago Blackmagic came out of nowhere with it's new camera and surprised everyones socks off. What's so interesting about this camera besides the relatively cheap price?
2.5K CMOS sensor
BMCC has an approximately micro-4/3 sized sensor that measures 15.81 x 8.88 mm with 2432x1366 elements. I say elements because it is not absolutely correct to call these pixels as they don't give you RGB data for each one. As a CMOS sensor, it is basically a single sensor overlaid with Bayer filter. This means that each element "sees" either green, red or blue depending on the position. Most common Bayer filter has twice as many green elements as red or blue because green is most helpful for luminance recording and human eye is also most sensitive to green color.
One could argue that Bayer sensor is similar to 4:2:2 chroma subsampling but this is wrong because they are different schemes for completely different purposes. Chroma subsampling means throwing chroma samples in YCrCb color space which takes place after all the other image reconstruction processes have been done. Bayer pattern with it's distribution of green, red and blue cells contains information about real RGB values and captures as much information as possible with given sensor.
Constructing full RGB image from Bayer pattern is called debayering. Raw image from Bayer sensor is basically monochromatic, it does not have colors as such because each element registered only the light level it received through Bayer filter. Debayering algorithms use clever schemes to interpolate the missing values for all RGB colors and thus reconstruct full RGB raster image. Very basic algorithms do simple interpolation between elements of single color, advanced algorithms use all surrounding elements regardless of "color" and use local gradients to choose, which direction should be chosen for interpolation. It might be called inverse gradient-strength weighted interpolation or something in that direction...
So, as this camera has a Bayer pattern sensor, it does not really have the 2.5K image color-resolution wise. But depending on the debayer algorithm, it comes close. It will be a wonderful fullHD camera for sure and possibly even beyond that. Filming for 2K digital cinema will probably be no problem because even Avatar was up-scaled from 1920 pixels to fill the 1998 pixel wide DCP container. Avatar was filmed mostly with Sony F35 cameras that output true 1920x1080 RGB image (debayered and downscaled from 3.5K sensor) but debayered and downscaled BMCC image probably comes pretty damn close.
Next post will be about dynamic range and encoding.
Since 2008 when Vincent Laforet shot the famous Reverie short that started the whole DSLR film mania, little has happened camera-wize on the low-budget end of nice looking filmmaking. Hacks for Canon cameras and Panasonic GH2 have opened some additional possibilities with higher bit-rates and other rather insignificant features but nothing really exiting has happened. It seems that Canon has dropped the ball in DSLR development because new camera models have brought absolutely nothing new feature or quality wise. Probably it is so because of things like C300 and C500 but they are 5x more expensive than say a 5dmkII.
In may 2012, Blackmagic Design, company best known for video capture cards, converters etc, announced their plans to make a 2.5K CMOS sensor camera that shoots CinemaDNG raw for only 2995$. And not only announced, they showed a working prototype also! RED once also promised 3K for 3K$ but this utopia never realized and Scarlet became the somewhat lighter twin brother for Epic with a much more heftier price tag than 3K$.
So, only three months ago Blackmagic came out of nowhere with it's new camera and surprised everyones socks off. What's so interesting about this camera besides the relatively cheap price?
2.5K CMOS sensor
BMCC has an approximately micro-4/3 sized sensor that measures 15.81 x 8.88 mm with 2432x1366 elements. I say elements because it is not absolutely correct to call these pixels as they don't give you RGB data for each one. As a CMOS sensor, it is basically a single sensor overlaid with Bayer filter. This means that each element "sees" either green, red or blue depending on the position. Most common Bayer filter has twice as many green elements as red or blue because green is most helpful for luminance recording and human eye is also most sensitive to green color.
One could argue that Bayer sensor is similar to 4:2:2 chroma subsampling but this is wrong because they are different schemes for completely different purposes. Chroma subsampling means throwing chroma samples in YCrCb color space which takes place after all the other image reconstruction processes have been done. Bayer pattern with it's distribution of green, red and blue cells contains information about real RGB values and captures as much information as possible with given sensor.
Constructing full RGB image from Bayer pattern is called debayering. Raw image from Bayer sensor is basically monochromatic, it does not have colors as such because each element registered only the light level it received through Bayer filter. Debayering algorithms use clever schemes to interpolate the missing values for all RGB colors and thus reconstruct full RGB raster image. Very basic algorithms do simple interpolation between elements of single color, advanced algorithms use all surrounding elements regardless of "color" and use local gradients to choose, which direction should be chosen for interpolation. It might be called inverse gradient-strength weighted interpolation or something in that direction...
So, as this camera has a Bayer pattern sensor, it does not really have the 2.5K image color-resolution wise. But depending on the debayer algorithm, it comes close. It will be a wonderful fullHD camera for sure and possibly even beyond that. Filming for 2K digital cinema will probably be no problem because even Avatar was up-scaled from 1920 pixels to fill the 1998 pixel wide DCP container. Avatar was filmed mostly with Sony F35 cameras that output true 1920x1080 RGB image (debayered and downscaled from 3.5K sensor) but debayered and downscaled BMCC image probably comes pretty damn close.
Next post will be about dynamic range and encoding.
Tuesday, July 24, 2012
ACES workflow and Resolve
Last two days I have tried to wrap my head around the ACES thing and how it works in Davinci Resolve. ACES was introduced in Resolve 8.something and is now an alternative to native YRGB math. I probably first read about ACES from project Mango blog because Mango was shot with Sony F65 which is meant to support ACES workflow. For Mango (or Tears of Steel as they now call it), they converted the thing from SLog to ACES openexr files and from there on to rec709 which they use for compositing etc. At first I didn't pay much attention to this ACES thing but when I found it from Resolve I decided to dig a little deeper and find out what this whole thing is about.
Basically, ACES is a color gamut that is meant to be so wide that you cannot possibly go beyond it even with extreme grading or transforms. As I understand it, it expresses colors in XYZ space which is device-independent and thus allows to describe all possible colors. Images in ACES colorspace should be scene-referred (as opposed to output-referred/display-referred) which means that color values describe actual light levels in the scene that the camera or other captude device received.
EDIT: ACES has RGB primaries associated with it and thus is a theoretically limited color space. It, however, contains all colors visible to human eye and so is unlimited in any practical terms. ACES values are relative scene exposure values - they express real light level ratios in filmed scene.
To achieve the conversion to scene-referred values, something called Input Device Transform is performed. It is a transformation that is specific to imaging device (for example the sensor of certain camera model) and converts the values from recorded camera data to scene exposure values. This transform must be camera and recording method specific because cameras tend to apply different log or gamma curves to map the most valuable data to limited bit depths. IDT reverses these curves and sensor response to calculate real light levels that hit the sensor. It is a bit unclear to me if this should happen in IDT or is some additional 1D or 3D transform needed...
After transforming camera data to scene referred values, something called Reference Rendering Transform is performed. I haven't quite figured out how it works in Resolve but as I understand it, it is basically the emulation of ideal film print that has the pleasing qualities of film. How this transform is constructed and what are the pleasing qualities that it tries to achieve are, is fuzzy to me at the moment. It is meant to replace the current practice of release print film stock emulation LUTs that are used in grading.
EDIT: Reference Rendering Transform applies manipulations that make image look perceptually pleasing. As films stocks have been developed to look pleasing also with some colors being more saturated than others, comparison of RRT with ideal film stock is logical.
For viewing the "ideal film stock" on screen, Output Device Transform is performed. ODT transforms the image to whatever output device you are viewing the image on. Without RRT and ODT you can not display ACES data. ODT is device specific and maps the ideal image to device gamut and gamma. For example if you view the image on P3 DCI projector the ODT makes sure you get the right mapping to the gamut used in projector. Same with rec709 or any other way of displaying images.
In addition to IDT and ODT, operations called Input Device Calibration Transform and Output "thesamething" can be performed. Calibration transforms are meant to level the differences between different models of same device. For example two different Alexa cameras might have slightly different sensor response curves and IDCT is meant to remove that difference between cameras. These transforms are device specific and are not shared. Probably they won't be used much because to construct such a transform you have to run your cameras and output devices through some heavy measurements.
Basically, ACES is a color gamut that is meant to be so wide that you cannot possibly go beyond it even with extreme grading or transforms. As I understand it, it expresses colors in XYZ space which is device-independent and thus allows to describe all possible colors. Images in ACES colorspace should be scene-referred (as opposed to output-referred/display-referred) which means that color values describe actual light levels in the scene that the camera or other captude device received.
EDIT: ACES has RGB primaries associated with it and thus is a theoretically limited color space. It, however, contains all colors visible to human eye and so is unlimited in any practical terms. ACES values are relative scene exposure values - they express real light level ratios in filmed scene.
To achieve the conversion to scene-referred values, something called Input Device Transform is performed. It is a transformation that is specific to imaging device (for example the sensor of certain camera model) and converts the values from recorded camera data to scene exposure values. This transform must be camera and recording method specific because cameras tend to apply different log or gamma curves to map the most valuable data to limited bit depths. IDT reverses these curves and sensor response to calculate real light levels that hit the sensor. It is a bit unclear to me if this should happen in IDT or is some additional 1D or 3D transform needed...
After transforming camera data to scene referred values, something called Reference Rendering Transform is performed. I haven't quite figured out how it works in Resolve but as I understand it, it is basically the emulation of ideal film print that has the pleasing qualities of film. How this transform is constructed and what are the pleasing qualities that it tries to achieve are, is fuzzy to me at the moment. It is meant to replace the current practice of release print film stock emulation LUTs that are used in grading.
EDIT: Reference Rendering Transform applies manipulations that make image look perceptually pleasing. As films stocks have been developed to look pleasing also with some colors being more saturated than others, comparison of RRT with ideal film stock is logical.
For viewing the "ideal film stock" on screen, Output Device Transform is performed. ODT transforms the image to whatever output device you are viewing the image on. Without RRT and ODT you can not display ACES data. ODT is device specific and maps the ideal image to device gamut and gamma. For example if you view the image on P3 DCI projector the ODT makes sure you get the right mapping to the gamut used in projector. Same with rec709 or any other way of displaying images.
In addition to IDT and ODT, operations called Input Device Calibration Transform and Output "thesamething" can be performed. Calibration transforms are meant to level the differences between different models of same device. For example two different Alexa cameras might have slightly different sensor response curves and IDCT is meant to remove that difference between cameras. These transforms are device specific and are not shared. Probably they won't be used much because to construct such a transform you have to run your cameras and output devices through some heavy measurements.
Random rants and progressbar
This blog is meant to replace sometimes failing memory and pieces of paper that tend to get lost or buried under other very-important-stuff. Main focus will probably be related to animation, vfx, grading, digital cinema and information that I find to be useful some way or another. Occasional posts about more earthly themes also possible...
Subscribe to:
Posts (Atom)