developer

How to measure video quality perception

Leandro Moreira developer, distributed systems, pattern, tests

Update 3 (05/16/2020): Wrote an updated guide to use VMAF through FFmpeg.

Update 2 (01/06/2016): Fixed reference video bitrate unit from Kbps to KBps

Update 1 (10/16/2016): Anne Aaron presented the VMAF at the Demuxed 2016.

When working with videos, you should be focusing all your efforts on best quality of streaming, less bandwidth usage, and low latency in order to deliver the best experience for the users.

This is not an easy task. You often need to test different bitrates, encoder parameters, fine tune your CDN and even try new codecs. You usually run a process of testing a combination of configurations and codecs and check the final renditions with your naked eyes. This process doesn’t scale, can’t we just trust computers to check that?

bit rate (bitrate): is a measure often used in digital video, usually it is assumed the rate of bits per seconds, it is one of the many terms used in video streaming.

screen-shot-2016-10-08-at-9-30-26-am
same resolution, different bitrates.

codec: is an electronic circuit or software that compresses or decompresses digital content. (ex: H264 (AVC), VP9, AAC (HE-AAC), AV1 and etc)

We were about to start a new hack day session here at Globo.com and since some of us learned how to measure the noise introduced when encoding and compressing images, we thought we could play with the stuff we learned by applying the methods to measure video quality.

We started by using the PSNR (peak signal-to-noise ratio) algorithm which can be defined in terms of the mean squared error (MSE) in decibel scale.

PSNR: is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise.

First, you calculate the MSE which is the average of the squares of the errors and then you normalize it to decibels.


MSE = ∑ ∑ ( [n1[i]-n2[i]] ) ^ 2 / m * n
*n1 is the original image, n2 the comparable image, m and n are the image size
PSNR = 10 log₁₀ ( MAX ^ 2 / MSE )
*MAX is the maximum possible pixel value of the image

view raw

math.math

hosted with ❤ by GitHub

For 3D signals (colored image), your MSE needs to sum all the means for each plane (ie: RGB, YUV and etc) and then divide by 3 (or 3 * MAX ^ 2).

To validate our idea, we downloaded videos (720p, h264) with the bitrate of 3400 kbps from distinct groups like News, Soap Opera and Sports. We called this group of videos the pivots or reference videos. After that, we generated some transrated versions of them with lower bitrates. We created 700 kbps, 900 kbps, 1300 kbps, 1900 kbps and 2800 kbps renditions for each reference video.

Heads Up! Typically the pivot video (most commonly referred to as reference video), uses a truly lossless compression, the bitrate for a YUV420p raw video should be 1280x720x1.5(given the YUV420 format)x24fps /1000 = 33177.6KBps, far more than what we used as reference (3400KBps).

We extracted 25 images for each video and calculate the PSNR comparing the pivot image with the modified ones. Finally, we calculate the mean. Just to help you understand the numbers below, a higher PSNR means that the image is more similar to the pivot.

700 kbps 900 kbps 1300 kbps 1900 kbps 2800 kbps 3400 kbps
Soap Op. 35.0124 36.5159 38.6041 40.3441 41.9447 ∞
News 28.6414 30.0076 32.6577 35.1601 37.0301 ∞
Sports 32.5675 34.5158 37.2104 39.4079 41.4540 ∞
screen-shot-2016-10-08-at-9-15-24-am
A visual sample.

We defined a PSNR of 38 (from our observations) as the ideal but then we noticed that the News group didn’t meet the goal. When we plotted the News data in the graph we could see what happened.

The issue with the video from the News group is that they’re a combination of different sources: External traffic camera with poor resolution, talking heads in a studio camera with good resolution and quality, some scenes with computer graphics (like the weather report) and others. We suspected that the News average was affected by those outliers but this kind of video is part of our reality.

kitbcrnx2uuu4
The different video sources are visible in clusters. (PSNR(frames))

We needed a better way to measure the quality perception so we searched for alternatives and we reached one of the Netflix’s posts: an approach toward a practical perceptual video quality metric (VMAF). At first, we learned that PSNR does not consistently reflect human perception and that Netflix is creating ways to approach this with the VMAF model.

They created a dataset with several videos including videos that are not part of the Netflix library and put real people to grade it. They called this score of DMOS. Now they could compare how each algorithm scores against DMOS.

netflix
FastSSIM, PSNRHVS, PSNR and SSIM (y) vs DMOS (x)

They realized that none of them were perfect even though they have some strength in certain situations. They adopted a machine-learning based model to design a metric that seeks to reflect human perception of video quality (a Support Vector Machine (SVM) regressor).

The Netflix approach is much wider than using PSNR alone. They take into account more features like motion, different resolutions and screens and they even allow you train the model with your own video dataset.

“We developed Video Multimethod Assessment Fusion, or VMAF, that predicts subjective quality by combining multiple elementary quality metrics. The basic rationale is that each elementary metric may have its own strengths and weaknesses with respect to the source content characteristics, type of artifacts, and degree of distortion. By ‘fusing’ elementary metrics into a final metric using a machine-learning algorithm – in our case, a Support Vector Machine (SVM) regressor”

Netflix about VMAF

The best news (pun intended) is that the VMAF is FOSS by Netflix and you can use it now. The following commands can be executed in the terminal. Basically, with Docker installed, it installs the VMAF, downloads a video, transcodes it (using docker image of FFmpeg) to generate a comparable video and finally checks the VMAF score.


# clone the project (later they'll push a docker image to dockerhub)
git clone –depth 1 https://github.com/Netflix/vmaf.git vmaf
cd vmaf
# build the image
docker build -t vmaf .
# get the pivot video (reference video)
wget http://www.sample-videos.com/video/mp4/360/big_buck_bunny_360p_5mb.mp4
# generate a new transcoded video (vp9, vcodec:500kbps)
docker run –rm -v $(PWD):/files jrottenberg/ffmpeg -i /files/big_buck_bunny_360p_5mb.mp4 -c:v libvpx-vp9 -b:v 500K -c:a libvorbis /files/big_buck_bunny_360p.webm
# extract the yuv (yuv420p) color space from them
docker run –rm -v $(PWD):/files jrottenberg/ffmpeg -i /files/big_buck_bunny_360p_5mb.mp4 -c:v rawvideo -pix_fmt yuv420p /files/360p_mpeg4-v_1000.yuv
docker run –rm -v $(PWD):/files jrottenberg/ffmpeg -i /files/big_buck_bunny_360p.webm -c:v rawvideo -pix_fmt yuv420p /files/360p_vp9_700.yuv
# checks VMAF score
docker run –rm -v $(PWD):/files vmaf run_vmaf yuv420p 640 368 /files/360p_mpeg4-v_1000.yuv /files/360p_vp9_700.yuv –out-fmt json
# and you can even check VMAF score using existent trained model
docker run –rm -v $(PWD):/files vmaf run_vmaf yuv420p 640 368 /files/360p_mpeg4-v_1000.yuv /files/360p_vp9_700.yuv –out-fmt json –model /files/resource/model/nflxall_vmafv4.pkl

view raw

using_vmaf.sh

hosted with ❤ by GitHub

You saved around 1.89 MB (37%) and still got the VMAF score 94.


{
"aggregate": {
"VMAF_feature_adm2_score": 0.9865012294519826,
"VMAF_feature_motion_score": 2.6486005151515153,
"VMAF_feature_vif_scale0_score": 0.85336751265595612,
"VMAF_feature_vif_scale1_score": 0.97274233143291644,
"VMAF_feature_vif_scale2_score": 0.98624814558455487,
"VMAF_feature_vif_scale3_score": 0.99218556024841664,
"VMAF_score": 94.143067486687571,
"method": "mean"
}
}

view raw

vmaf_result.json

hosted with ❤ by GitHub

Using a composed solution like VMAF or VQM-VFD proved to be better than using a single metric, there are still issues to be solved but I think it’s reasonable to use such algorithms plus A/B tests given the impractical scenario of hiring people to check video impairments.

A/B tests: For instance, you could use X% of your user base for Y days offering them the newest changes and see how much they would reject it.

Functor, Pointed Functor, Monad and Applicative Functor in JS

Leandro Moreira developer, functional, javascript, pattern

function-machine


// This post will briefly explain (omiting, skipping some parts) in code what is
// Functor, Pointed Functor, Monad and Applicative Functor. Maybe by reading the
// code you will easily grasp these functional concepts.
// if you only want to run this code go to:
// https://jsfiddle.net/leandromoreira/buq5mnyk/
// or https://gist.github.com/leandromoreira/9504733c7f8c6361c46270ea953d8409
// This code requires you to have require.js loaded (or you can load ramda instead :P)
requirejs.config({
paths: {
ramda: 'https://cdnjs.cloudflare.com/ajax/libs/ramda/0.13.0/ramda.min'
},
});
require(['ramda'], function(_) {
// First let's create a Container that is a type that holds (wraps) a value, a useful abstraction to handle state.
var Container = function(x) {
this.__value = x;
}
// of is a method to create Container of x type
Container.of = function(x) {
return new Container(x);
};
console.log("should be 3", Container.of(3))
// We can improve this building block (Container) by providing a way to handle the wrapped value,
// this is basically a Functor, which is a type that implements map (it is mappable) and obeys some laws.
// By the way a Pointed Functor is a functor with an of method.
Container.prototype.map = function(f) {
return Container.of(f(this.__value));
}
var c4 = Container.of(4)
var inc = function(x) {
return x + 1
}
var c5 = c4.map(inc)
// We first created a container of 4 then we map a increase over it resulting in a container of 5
console.log("should be 5", c5)
// Maybe is a functor that checks if the value is null/undefined
// it is useful to avoid erros like "Cannot read property x of null"
Container.prototype.isNothing = function() {
return (this.__value === null || this.__value === undefined);
};
// Now our map will also check weather it's valid or not.
Container.prototype.map = function(f) {
return this.isNothing() ? Container.of(null) : Container.of(f(this.__value));
};
var address = function(person) {
return person.address;
};
var upperCase = function(t) {
return t.toUpperCase()
}
// Although we're passing an invalid value to the container it won't broke
console.log("should be null without errors", Container.of(null).map(address).map(upperCase))
// but when we do pass the right parameter it produces the expected output
console.log("should be HERE", Container.of({
name: "Diddy",
address: "here"
}).map(address).map(upperCase))
// this is good but a failing error with no message can make things worst 😦
// This functions maps any function a functor
var map = _.curry(function(ordinaryFn, functor) {
return functor.map(ordinaryFn);
});
var aFunctor = Container.of(2)
var sum6 = function(x) {
return x + 6
}
// given an ordinary function and an functor it produces another functor
var plus6 = map(sum6)
var y = plus6(aFunctor)
console.log("should be a Functor of 8", y)
// Either is a functor that can return two types either Right (normal flow) or Left (some error occorred).
// Now here what is great is that we can say what was the error.
var Left = function(x) {
this.__value = x;
};
Left.of = function(x) {
return new Left(x);
};
Left.prototype.map = function(f) {
return this;
};
var Right = function(x) {
this.__value = x;
};
Right.of = function(x) {
return new Right(x);
};
Right.prototype.map = function(f) {
return Right.of(f(this.__value));
}
console.log("should be 10", Right.of(8).map(inc).map(inc))
console.log("should be unchaged 8", Left.of(8).map(inc).map(inc))
var nonNegative = function(x) {
if (x < 0) {
return Left.of("you must pass a positive number")
} else {
return Right.of(x)
}
}
console.log("should be 10", nonNegative(9).map(inc))
console.log("should be an error message", nonNegative(–4).map(inc))
// IO is a functor that holds functions as values, and instead of mapping the value
// it'll map functions and compose them like a array of functions.
var IO = function(f) {
this.__value = f;
};
IO.of = function(x) {
return new IO(function() {
return x;
});
};
IO.prototype.map = function(f) {
return new IO(_.compose(f, this.__value));
};
var composedLazyFunctions = IO.of(3).map(inc).map(inc).map(inc)
console.log("this is a lazy composed function", composedLazyFunctions)
console.log("this is the execution of that composed function", composedLazyFunctions.__value())
var readFile = function(filename) {
return new IO(function() {
return "read file from " + filename
});
};
var print = function(x) {
return new IO(function() {
return x
});
};
// Cat will be a composed function that produces and IO of an IO :X
var cat = _.compose(map(print), readFile)
var catGit = cat('.git/config')
console.log("it should be an IO of IO IO(IO())", catGit)
// This creates an awkward situation where if we want the real value we need to
// catGit.__value().__value() how about create a join that unwraps the value.
IO.prototype.join = function() {
return this.__value()
};
console.log("should be 'read file from .git/config'", catGit.join().join())
// Notice that we still need to call join twice, and if we join every time we map?
// this is what we know was chain
var chain = _.curry(function(ordinaryFn, functor) {
return functor.map(ordinaryFn).join();
});
var complexSum = function(initialNumber) {
return new IO(function() {
var x = initialNumber * 4
var y = x * 4
return (y + 42) – x * 4
});
};
var incIO = function(x) {
return new IO(function() {
return x + 1
});
};
var doubleIO = function(x) {
return new IO(function() {
return x * 2
});
};
var cleverMath = _.compose(
chain(doubleIO),
chain(incIO),
chain(incIO),
complexSum
);
var multiplier = Math.floor((Math.random() * 552) + 7)
var ordinaryValue = Math.floor((Math.random() * 98134123) – 12)
var cleverMathResult = cleverMath(ordinaryValue * multiplier)
console.log("should be 88", cleverMathResult.join())
// Monads are pointed functors that can flatten 🙂
// Now let's finish with an Applicative Functor which is a pointed functor with an ap(ply) method
Container.prototype.ap = function(other_container) {
return other_container.map(this.__value)
}
console.log("should be Container(4)", Container.of(inc).ap(Container.of(3)))
})
// Please consider to read these links bellow
// http://www.leonardoborges.com/writings/2012/11/30/monads-in-small-bites-part-i-functors/
// https://drboolean.gitbooks.io/mostly-adequate-guide/content/ch8.html

view raw

functor_monad_applicative_javascript.js

hosted with ❤ by GitHub

Functional Programing 101 :: WWH

Leandro Moreira developer, functional, javascript, pattern , , ,

function-machine

WWH: What? Why? How?

  1. What: a quick (hopefully, useful to real world) guide to functional programing using JavaScript strongly based on most adequate book.
  2. Why: it might empower you to write more robust programs: reusable, shorter, easier to reason about, less prone to error among others.
  3. How: by providing a quick textual introduction (WWH) followed by a simple code example and when possible a real code example.

Intro :: concepts

Functional Programing

What: a way to build code in which you use functions as the main design tool.

Why: might lead to code that’s easier to test, debug, parallelize, and understand.

How: thinking about what programs should do instead of how, using functions as the major unit to solve problems on computer.

First Class Functions

What: “functions are like any other data type and there is nothing particularly special about them – they may be stored in arrays, passed around, assigned to variables.”

Why: use functions to compose programs in a style that you can easily reason about, maintain, reuse and grow.

How: just create and use functions to solve problems.


// set to vars
var hi = function(name) {
return 'Hi ' + name
}
// rebound to other var
var greeting = hi
// an array of fns
var greetings = [hi, greeting]
// passed as arguments
var execIn1Second = function(fn) {
setTimeout(fn, 1000)
}

view raw

js.js

hosted with ❤ by GitHub

Pure Functions

What: “a function that, given the same input, will always return the same output and does not have any observable side effect.”

Why: with pure functions we can easily cache, debug, test and parallelize the processing of them. There is no state to understand / set up.

How: write functions that does not have side effect. Although we’ll eventually write programs that mutate values, we can certainly try to minimize it. (And when we do need to mutate values, we can use functions to help us)


// impure – because you can change promo value (side effect)
// and for the same input X it can produce a different output.
var promo = 40
var isPromo = function(price) {return price === promo}
// pure
var isPromo = function(price) {return price === 40}

view raw

pure.js

hosted with ❤ by GitHub

Basic toolbox :: currying

What: “You can call a function with fewer arguments than it expects. It returns a function that takes the remaining arguments.”

Why: you can promote the reusability to function level, you can use them to compose programs that expects another function

How: build a function with n parameters that returns n functions instead of the immediate result.


// an ordinary function
var sum = function(a, b){return a+b}
sum(1, 3) // 4
// a curried version of that sum
var curriedSum = function(a){
return function(b){return a+b}
}
var plusOne = curriedSum(1) // function(b) {return a+b}
plusOne(3) // 4
plusOne(1) // 2 -> plusOne is useful outside it's original scope
// a simple function to curry any other function
// we'll use this in future
// DO NOT USE THIS IN PRODUCTION
var curry = function(uncurriedFn){
var argumentsCall = []
return function curriedFn(){
var args = Array.prototype.slice.call(arguments)
if (args.length > 0) {
argumentsCall = argumentsCall.concat(args)
if (uncurriedFn.length == argumentsCall.length) {
return uncurriedFn.apply(this, argumentsCall)
}
}
return curriedFn
}
}
// an usage example of curry
var curriedAjax = curry(function(method, path){
return $.ajax(path, {method: method})
})
var ajaxGET = curriedAjax('GET')
var allUsers = ajaxGET('/users')
var allStars = ajaxGET('/stars')

view raw

curry.js

hosted with ❤ by GitHub

Medium toolbox :: composing

What: is the act of creating your programs using lots of functions.

Why: this promotes the reuse at a great level and forces you to think about what instead of how.

How: chain functions to produce a new callable function.


// a simple function to compose a function
// DO NOT USE THIS IN PRODUCTION
// you can skip the reading of this function if you want
var compose = function(){
var fns = Array.prototype.slice.call(arguments)
return function composed(){
var thisCall = this
var args = Array.prototype.slice.call(arguments)
fns.reverse().forEach(function(fn){
if (Object.prototype.toString.call(args) !== '[object Array]') args = [args]
args = fn.apply(thisCall, args)
})
return args
}
}
// take a list of Strings and return a list of up case strings
var toUpperCase = function(list) {
return list.map(function(x){return x.toUpperCase()})
}
// take a list of Strings and return a list of length
var length = function(list) {
return list.map(function(x){return x.length})
}
// take a list of Integer and return the sum
var sum = function(list) {
var sum = 0
list.forEach(function(x){sum += x})
return sum
}
// chain all these functions and produce a new
var sumCharsOnList = compose(sum, length, toUpperCase)
sumCharsOnList(["NX", "PS4", "XboxOne"])

view raw

composition.js

hosted with ❤ by GitHub


// brief explanation of what happens here
var sumCharsOnList = compose(sum, lenght, toUpperCase)
sumCharsOnList(["NX", "PS4", "XboxOne"])
// –> first the list ["NX", "PS4", "XboxOne"] is passed to the function toUpperCase
// –> toUpperCase returns another list and it's the input for the lenght function
// –> the lenght function return a list of integers which will become the input for sum function
// –> sum function will reduce the list summing all the integers and returning the sum

view raw

explanation.js

hosted with ❤ by GitHub

Example :: motivational

What: a better example to motivate you to go further with functional programing.

Why: most near real world examples are great to motivate you to learn something.

How: since you can see all the concepts together, I think you’ll notice the value.

You can see the example running at https://jsfiddle.net/swmrmgur/2/ and check the commented code down bellow.

Screen Shot 2016-04-27 at 2.17.19 PM


// _ is an instance of Ramda
// $ is an instance of jQuery
// This is an app extracted from the book
// mostly-adequate-guide/content/ch6.html
// it's query the flickr API's and mount
// a lots of images on the page
// IMPURE functions ahead
// takes a function (callback) and returns another
// function which takes an url
var getJSON = _.curry(function(callback, url) {
$.getJSON(url, callback);
})
// takes a selector and returns a functions
// which you can pass an html
var setHtml = _.curry(function(sel, html) {
$(sel).html(html);
})
// PURE, good and reliable functions ahead
// just create an image for a given url
var img = function(url) {
return $('<img />', {
src: url,
});
};
// just create an url for a given query
var url = function(t) {
return 'http://api.flickr.com/services/feeds/photos_public.gne?tags=&#39; +
t + '&format=json&jsoncallback=?';
};
// _.prop is a curried function which takes
// a property name and then an object
// ex: var getPrice = _.prop('price')
// getPrice({price: 3, x: "y"}) // returns 3
// takes an object extracts its media and
// then from the media it extracts the m property
var mediaUrl = _.compose(_.prop('m'), _.prop('media'));
// it takes a media url and create an image
var mediaToImg = _.compose(img, mediaUrl);
//it takes an object and extracts items from it
// map each one to an image
var images = _.compose(_.map(mediaToImg), _.prop('items'));
// it takes images and set
// them on the document.body
var renderImages = _.compose(setHtml('body'), images);
// given a string, it'll create an url and
// gets the json and render the images
var app = _.compose(getJSON(renderImages), url);
// given a query it'll create a page
app('cats');
// I strongly recommend you to read again from app to mediaUrl

view raw

flickr.js

hosted with ❤ by GitHub

Advanced toolbox & conclusion

I hope you might see the benefits you can have from using one or other technique from functional programming but for sure there are other benefits not shown here, I strongly recommend you to read the INCREDIBLE free book (gitbook) “Professor Frisby’s Mostly Adequate Guide to Functional Programming”, in fact, most of the ideas and examples here are from it.

There are advanced techniques to deal with data mutation with less pain, to handle errors and exceptions without try and catch and more abstractions that can help you and you can read them on the book.

And don’t use the handcrafted curry and compose built here (they’re far from production-ready), instead use a library like Ramda, which provides many basic functions like: map, filter and other all of them already curried, or lodash-fp.

Yeah, there no monado here. A special thank to Daniel Martins and Juarez Bochi, they helped a lot.

cover