Most of JSFEAT
methods relies on custom data structures. There are just few provided at the moment but I'm sure its number will increase with new functionality.
The core and starting structure for any project is most likely matrix_t
:
var my_matrix = new jsfeat.matrix_t(columns, rows, data_type, data_buffer = undefined);
matrix_t
is quite flexible structure, it can be used as image representation or regular matrix for mathematics. columns
and rows
is the same as defining width
and height
for image. But let's look at data_type
argument. It allows you to describe matrix_t
underlaying data.
To construct data_type
you need to use library internal signatures. NOTE: at the moment most of methods support only single channel operations due to performance reasons.
// single channel unsigned char var data_type = jsfeat.U8_t | jsfeat.C1_t; // 2 channels 32 bit integer var data_type = jsfeat.S32_t | jsfeat.C2_t; // 3 channels 32 bit float var data_type = jsfeat.F32_t | jsfeat.C3_t;
Let's put all together:
var columns = 320, rows = 240, data_type = jsfeat.U8_t | jsfeat.C1_t; var my_matrix = new jsfeat.matrix_t(columns, rows, data_type); // we can access following properties: // my_matrix.cols, my_matrix.rows // my_matrix.data - underlaying array for data holding // to check what type is used for data you can use bit mask: // if(my_matrix.type & jsfeat.U8_t) ... // my_matrix.channel = 1/2/3/4
You can resize matrix_t
at any time using resize
method. NOTE: this operation will delete current data array if new size is larger.
my_matrix.resize(new_cols, new_rows, new_channels);
You also can access matrix_t
buffer
property that is a wrapper for data
and implemented with data_t
structure.
This is just a wrapper for JavaScript ArrayBuffer
. But provides easier way to interpret underlaying data:
// you can provide preallocated ArrayBuffer var my_data = new jsfeat.data_t(size_in_bytes, buffer = undefined); var my_data_buffer = my_data.buffer; // ArrayBuffer var u8_data = my_data.u8; // Uint8Array var i32_data = my_data.i32; // Int32Array var f32_data = my_data.f32; // Float32Array
// you can also provide pre-allocated data_t for matrix var columns = 320, rows = 240, data_type = jsfeat.U8_t | jsfeat.C1_t; var data_buffer = new jsfeat.data_t(columns*rows); var my_matrix = new jsfeat.matrix_t(columns, rows, data_type, data_buffer);
A structure to wrap several matrix_t
instances. Each data
entry is 2x smaller then previous:
var levels = 3, start_width = 640, start_height = 480, data_type = jsfeat.U8_t | jsfeat.C1_t; var my_pyramid = new jsfeat.pyramid_t(levels); // this will populate data property with matrix_t instances my_pyramid.allocate(start_width, start_height, data_type); var level_0 = my_pyramid.data[0]; // cols = 640, rows = 480 var level_1 = my_pyramid.data[1]; // cols = 320, rows = 240 var level_2 = my_pyramid.data[2]; // cols = 160, rows = 120 // with build method you can draw input source to levels // skip_first_level is true by default my_pyramid.build(source_matrix_t, skip_first_level = true);
2D point with coordinates, level and score properties:
// all arguments are zero by default var my_point = new jsfeat.keypoint_t(x = 0, y = 0, score = 0, level = 0);
Used for ransac
based motion estimator.
var model_size = 4; // minimum points to estimate motion var thresh = 3; // max error to classify as inlier var eps = 0.5; // max outliers ratio var prob = 0.99; // probability of success var params = new jsfeat.ransac_params_t(model_size, thresh, eps, prob);
JSFEAT
has very simple and experimental linked pool based cache system. At the moment I'm not sure if it is really needed since most JavaScript engines have their own powerful caching. But running some tests I noticed that repeatedly calling methods that need temporary Array
(s) allocation significantly increase its execution time. So replacing allocation with pooled buffers helps to improve performance in some cases. How it works:
var size_in_bytes = 640; var temp_buffer = jsfeat.cache.get_buffer(size_in_bytes); var temp_u8 = temp_buffer.u8; // Uint8Array 640 entries // but you also can get other data types // Int32Array but length will be 640/4 = 160 entries var temp_i32 = temp_buffer.i32; // since all buffers comes from data_t instance // you can also use it to construct matrix_t var columns = 320, rows = 240, data_type = jsfeat.U8_t | jsfeat.C1_t; var my_matrix = new jsfeat.matrix_t(columns, rows, data_type, temp_buffer.data); // be careful because you always should provide enough space for matrix
NOTE: you always need to put buffer back to cache pool after you are done:
jsfeat.cache.put_buffer(temp_buffer);
At the moment the number of temporary buffers is fixed to 30 entries. I cant hardly imagine scenario where you may need more buffers. If the whole idea having cache
pool will prove its usefulness we can extend it to different data types and add auto extending functionality.
JSFEAT
has it's own jsfeat.math
module, it contains different math utilite methods that can be useful.
Calculates gaussian kernel coefficients using specified options:
var kernel_size = 5, sigma = 0, kernel_array = [], data_type = jsfeat.F32_t; jsfeat.math.get_gaussian_kernel(kernel_size, sigma, kernel_array, data_type); // you can provide zero sigma to determinate it automatically from kernel_size // or set desired value
NOTE: setting data_type
to jsfeat.U8_t
will result in integer based kernel that can be used to filter unsigned char
image avoiding floating-point operations.
Generic sorting method that allows you to specify start and end sorting indices for the source array aswell as provide custom comparison method. Current implementation was derived from *BSD system qsort()
.
// sort numeric array var arr = [10,2,1,0,0,4,6,1,3,8,5,3]; var cmp_numeric = function(a, b) { return (a < b); } jsfeat.math.qsort(arr, 0, arr.length-1, cmp_numeric); // sort array of points var point_arr = [ new jsfeat.keypoint_t(1,10), new jsfeat.keypoint_t(5,1), new jsfeat.keypoint_t(33,10), new jsfeat.keypoint_t(0,0), new jsfeat.keypoint_t(8,6), new jsfeat.keypoint_t(3,0)]; // define comparison method to sort points by ascending of y coordinates // and if y's are equal x's should ascend. var cmp_points = function(a, b) { return (a.y < b.y) || (a.y < b.y && a.x < b.x); } jsfeat.math.qsort(point_arr, 0, point_arr.length-1, cmp_points);
Calculate median value of provided Array
, NOTE: input Array
instance will be modified.
var arr = [10,2,1,0,0,4,6,1,3,8,5,3]; var median = jsfeat.math.median(arr, 0, arr.length-1);
Calculates the perspective transform from 4 pairs of the corresponding points. The function calculates the 3x3 Matrix
jsfeat.math.perspective_4point_transform(mat:matrix_t, src_x0, src_y0, dst_x0, dst_y0, src_x1, src_y1, dst_x1, dst_y1, src_x2, src_y2, dst_x2, dst_y2, src_x3, src_y3, dst_x3, dst_y3);
Various generalized matrix operations.
Transposes a matrix.
jsfeat.matmath.transpose(At:matrix_t, A:matrix_t);
Performs matrix multiplication.
jsfeat.matmath.multiply(C:matrix_t, A:matrix_t, B:matrix_t);
Post multiply the nrows x ncols
matrix A
by the transpose of the mrows x ncols
matrix B
to form the nrows x mrows
matrix C
, i.e. C = A*B'
.
jsfeat.matmath.multiply_ABt(C:matrix_t, A:matrix_t, B:matrix_t);
Post multiply the transpose of the nrows x ncols
matrix A
by the nrows x mcols
matrix B
to form the ncols x mcols
matrix C
, i.e. C = A'*B
.
jsfeat.matmath.multiply_AtB(C:matrix_t, A:matrix_t, B:matrix_t);
Post multiply an nrows x ncols
matrix A
by its transpose. The result is an nrows x nrows
square symmetric matrix C
, i.e. C = A*A'
.
jsfeat.matmath.multiply_AAt(C:matrix_t, A:matrix_t);
Pre multiply an nrows x ncols
matrix A
by its transpose. The result is an ncols x ncols
square symmetric matrix C
, i.e. C = A'*A
.
jsfeat.matmath.multiply_AtA(C:matrix_t, A:matrix_t);
Quick inverse of 3x3 matrix. Can operate inplace.
jsfeat.matmath.invert_3x3(from:matrix_t, to:matrix_t);
Quick 3x3 matrix multiplication. Can operate inplace.
jsfeat.matmath.multiply_3x3(C:matrix_t, A:matrix_t, B:matrix_t);
Calculate 3x3 matrix determinant.
var determinant = jsfeat.matmath.mat3x3_determinant(M:matrix_t);
Solves the system of linear equations Ax = B
using Gaussian elimination with optimal pivot element chosen. NOTE: input matrix_t
instances will be modified and result output in matrix B
.
// A and B modified and result output in B jsfeat.linalg.lu_solve(A:matrix_t, B:matrix_t);
Solves the system of linear equations Ax = B
using Cholesky factorization. The matrix must be symmetrical and positively defined. NOTE: input matrix_t
instances will be modified and result output in matrix B
.
// A and B modified and result output in B jsfeat.linalg.cholesky_solve(A:matrix_t, B:matrix_t);
This routine decomposes an rows x cols
matrix A
, into a product of the three matrices U
, W
, and V'
, i.e. A = UWV'
, where U
is an rows x rows
matrix whose columns are orthogonal, W
is a 1 x cols
matrix, and V
is an cols x cols
orthogonal matrix.
// U - the left orthogonal matrix // W - vector of singular values // V - the right orthogonal matrix // options - jsfeat.SVD_U_T and/or jsfeat.SVD_V_T to return transposed U and/or V jsfeat.linalg.svd_decompose(A:matrix_t, W:matrix_t, U:matrix_t, V:matrix_t, options);
Solves the system of linear equations Ax = B
using Singular value decomposition (SVD) method; the system can be over-defined and/or the matrix A
can be singular.
// A - left-hand side of the system // B - right-hand side of the system // X - output solution jsfeat.linalg.svd_solve(A:matrix_t, X:matrix_t, B:matrix_t);
This routine calculates the pseudo-inverse of the matrix A
.
// inverts matrix A to matrix Ainvert jsfeat.linalg.svd_invert(Ainvert:matrix_t, A:matrix_t);
Computes eigenvalues and eigenvectors of a symmetric matrix.
jsfeat.linalg.eigenVV(A:matrix_t, EigenVectors:matrix_t, EigenValues:matrix_t); // you can ask for Vectors or Values only jsfeat.linalg.eigenVV(A:matrix_t, null, EigenValues:matrix_t); jsfeat.linalg.eigenVV(A:matrix_t, EigenVectors:matrix_t, null);
The functions in this module estimate various geometrical transformations between two point sets.
VIDEO STABILIZER DEMO ORB MATCHING DEMO
This kernel calculates the affine transform from corresponding points. The function calculates the 3x3 Matrix
// create affine kernel // you can reuse it for different point sets var affine_kernel = new jsfeat.motion_model.affine2d(); var affine_transform = new jsfeat.matrix_t(3, 3, jsfeat.F32_t | jsfeat.C1_t); var count = 33; var from = []; var to = []; for(var i = 0; i < count; ++i) { // you can use keypoint_t structure // or just provide object with x and y properties from[i] = { "x":Math.random()*320, "y":Math.random()*240 }; to[i] = { "x":from[i].x + 5, "y":from[i].y+5 }; } affine_kernel.run(from, to, affine_transform, count); // you can also calculate transform error for each point var error = new jsfeat.matrix_t(count, 1, jsfeat.F32_t | jsfeat.C1_t); affine_kernel.error(from, to, affine_transform, error.data, count);
This kernel calculates perspective transform between point sets. Result is 3x3 Matrix
// create homography kernel // you can reuse it for different point sets var homo_kernel = new jsfeat.motion_model.homography2d(); var homo_transform = new jsfeat.matrix_t(3, 3, jsfeat.F32_t | jsfeat.C1_t); var count = 33; var from = []; var to = []; for(var i = 0; i < count; ++i) { // you can use keypoint_t structure // or just provide object with x and y properties from[i] = { "x":Math.random()*320, "y":Math.random()*240 }; to[i] = { "x":from[i].x + 5, "y":from[i].y+5 }; } homo_kernel.run(from, to, homo_transform, count); // you can also calculate transform error for each point var error = new jsfeat.matrix_t(count, 1, jsfeat.F32_t | jsfeat.C1_t); homo_kernel.error(from, to, homo_transform, error.data, count);
RANdom SAmple Consensus. [for more info see: http://en.wikipedia.org/wiki/RANSAC]
// this class allows you to use above Motion Kernels // to estimate motion even with wrong correspondences var ransac = jsfeat.motion_estimator.ransac; // create homography kernel // you can reuse it for different point sets var homo_kernel = new jsfeat.motion_model.homography2d(); var transform = new jsfeat.matrix_t(3, 3, jsfeat.F32_t | jsfeat.C1_t); var count = 333; var from = []; var to = []; for(var i = 0; i < count; ++i) { // you can use keypoint_t structure // or just provide object with x and y properties from[i] = { "x":Math.random()*320, "y":Math.random()*240 }; to[i] = { "x":from[i].x + Math.random()*5, "y":from[i].y+Math.random()*5 }; } // each point will be marked as good(1) or bad(0) var mask = new jsfeat.matrix_t(count, 1, jsfeat.U8_t | jsfeat.C1_t); var model_size = 4; // minimum points to estimate motion var thresh = 3; // max error to classify as inlier var eps = 0.5; // max outliers ratio var prob = 0.99; // probability of success var params = new jsfeat.ransac_params_t(model_size, thresh, eps, prob); var max_iters = 1000; var ok = ransac(params, homo_kernel, from, to, count, transform, mask, max_iters);
Least Median of Squares. Similar to above algorithm but uses median error value to filter wrong matches.
// this class allows you to use above Motion Kernels // to estimate motion even with wrong correspondences var lmeds = jsfeat.motion_estimator.lmeds; // create homography kernel // you can reuse it for different point sets var affine_kernel = new jsfeat.motion_model.affine2d(); var transform = new jsfeat.matrix_t(3, 3, jsfeat.F32_t | jsfeat.C1_t); var count = 333; var from = []; var to = []; for(var i = 0; i < count; ++i) { // you can use keypoint_t structure // or just provide object with x and y properties from[i] = { "x":Math.random()*320, "y":Math.random()*240 }; to[i] = { "x":from[i].x + Math.random()*5, "y":from[i].y+Math.random()*5 }; } // each point will be marked as good(1) or bad(0) var mask = new jsfeat.matrix_t(count, 1, jsfeat.U8_t | jsfeat.C1_t); var model_size = 3; // minimum points to estimate motion var thresh = 0; // is not used in lmeds var eps = 0.45; // hard coded internally var prob = 0.99; // probability of success var params = new jsfeat.ransac_params_t(model_size, thresh, eps, prob); var max_iters = 1000; var ok = lmeds(params, affine_kernel, from, to, count, transform, mask, max_iters);
jsfeat.imgpoc
module is like library heart! It contains all kind of methods to pre/post process and analyze image data.
Convert color array input [r0,g0,b0,a0, ...]
to grayscale using Y = 0.299*R + 0.587*G + 0.114*B
formula. You can specify the source input channel order such as BGRA, RGBA, RGB and BGR.
context2d.drawImage(video, 0, 0, width, height); var image_data = context2d.getImageData(0, 0, width, height); var gray_img = new jsfeat.matrix_t(width, height, jsfeat.U8_t | jsfeat.C1_t); var code = jsfeat.COLOR_RGBA2GRAY; jsfeat.imgproc.grayscale(image_data.data, width, height, gray_img, code);
Generic resize method. Works with single and multi channel matrix_t
. If performance is critical or you need multiple image resizings it is recommended to use canvas
built-in drawImage()
method.
jsfeat.imgproc.resample(source:matrix_t, dest:matrix_t, new_width, new_height);
Works with single channel data only. NOTE: if input is jsfeat.U8_t
and options = jsfeat.BOX_BLUR_NOSCALE
dest should be at least jsfeat.S32_t
to handle accumulated values correctly.
/* options - you can pass jsfeat.BOX_BLUR_NOSCALE to avoid result values scaling */ jsfeat.imgproc.box_blur_gray(source:matrix_t, dest:matrix_t, radius, options = 0);
Works with single channel data only. You can choose between providing kernel_size
or sigma
argument or both.
jsfeat.imgproc.gaussian_blur(source:matrix_t, dest:matrix_t, kernel_size, sigma = 0);
Downsample source
to dest
writing simple 4 pix average value. Works with single channel only.
jsfeat.imgproc.pyrdown(source:matrix_t, dest:matrix_t);
Compute gradient using Scharr kernel [3 10 3] * [-1 0 1]^T
. Works with single channel only.
// result is written as [gx0,gy0,gx1,gy1, ...] sequence in 2 channel matrix_t jsfeat.imgproc.scharr_derivatives(source:matrix_t, dest:matrix_t);
Compute gradient using Sobel kernel [1 2 1] * [-1 0 1]^T
. Works with single channel only.
// result is written as [gx0,gy0,gx1,gy1, ...] sequence in 2 channel matrix_t jsfeat.imgproc.sobel_derivatives(source:matrix_t, dest:matrix_t);
Calculates one or more integral images for the source image. Using these integral images, one may calculate sum, mean, standard deviation over arbitrary up-right or rotated rectangular region of the image in a constant time. NOTE: each destinatination should be 1 pixel larger than source width = source.cols+1, height = source.rows+1
. Single channel source only.
jsfeat.imgproc.compute_integral_image(source:matrix_t, sum:Array, sqsum:Array, tilted:Array);
Equalizes the histogram of a grayscale image. The algorithm normalizes the brightness and increases the contrast of the image.
jsfeat.imgproc.equalize_histogram(source:matrix_t, dest:matrix_t);
Canny edge detector. Result contains only 0x00
and 0xFF
values.
jsfeat.imgproc.canny(source:matrix_t, dest:matrix_t, low_threshold, high_threshold);
Applies a perspective transformation to an image using 3x3 Matrix
. Single channel source only.
To avoid sampling artifacts, the mapping is done in the reverse order, from destination to the source. That is, for each pixel of the destination image, the functions compute coordinates of the corresponding “donor” pixel in the source image and copy the pixel value.
jsfeat.imgproc.warp_perspective(source:matrix_t, dest:matrix_t, warp_mat:matrix_t, fill_value = 0);
Applies an affine transformation to an image using 2x3 or 3x3 Matrix
. Single channel source only.
To avoid sampling artifacts, the mapping is done in the reverse order, from destination to the source. That is, for each pixel of the destination image, the functions compute coordinates of the corresponding “donor” pixel in the source image and copy the pixel value.
jsfeat.imgproc.warp_affine(source:matrix_t, dest:matrix_t, warp_mat:matrix_t, fill_value = 0);
Feature Detection and Description
Detects corners using the FAST
algorithm.
// threshold on difference between intensity of the central pixel // and pixels of a circle around this pixel var threshold = 20; jsfeat.fast_corners.set_threshold(threshold); var corners = [], border = 3; // you should use preallocated keypoint_t array for(var i = 0; i < img.cols*img.rows, ++i) { corners[i] = new jsfeat.keypoint_t(0,0,0,0); } // perform detection // returns the amount of detected corners var count = jsfeat.fast_corners.detect(img:matrix_t, corners:Array, border = 3);
Laplacian and min eigen value based feature detector by CVLab (Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland).
var corners = [], laplacian_threshold = 30, min_eigen_value_threshold = 25; // choose threshold values jsfeat.yape06.laplacian_threshold = laplacian_threshold; jsfeat.yape06.min_eigen_value_threshold = min_eigen_value_threshold; // you should use preallocated keypoint_t array for(var i = 0; i < img.cols*img.rows, ++i) { corners[i] = new jsfeat.keypoint_t(0,0,0,0); } // perform detection // returns the amount of detected corners var count = jsfeat.yape06.detect(img:matrix_t, corners:Array, border = 5);
Yet Another Point Extractor by CVLab (Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland).
var corners = [], image_width = img.cols, image_height = img.rows, radius = 5, pyramid_levels = 1; // for now only single level support // YAPE needs init before running detection jsfeat.yape.init(image_width, image_height, radius, pyramid_levels); // you should use preallocated keypoint_t array for(var i = 0; i < img.cols*img.rows, ++i) { corners[i] = new jsfeat.keypoint_t(0,0,0,0); } // perform detection // returns the amount of detected corners var count = jsfeat.yape.detect(img:matrix_t, corners:Array, border = 4);
Oriented and Rotated BRIEF. [for more info see: http://en.wikipedia.org/wiki/ORB_(feature_descriptor)]
var corners = []; // jsfeat.keypoint_t Array var cols = 32; // 32 Bytes / 256 BIT descriptor var rows = num_corners; // descriptors stored per row var descriptors = new jsfeat.matrix_t(cols, rows, jsfeat.U8_t | jsfeat.C1_t); jsfeat.orb.describe(img_u8:matrix_t, corners:Array, num_corners, descriptors:matrix_t);
Calculates an optical flow for a sparse feature set using the iterative Lucas-Kanade method with pyramids.
/* prev_pyr - previous frame 8-bit pyramid_t curr_pyr - current frame 8-bit pyramid_t prev_xy - Array of 2D coordinates for which the flow needs to be found curr_xy - Array of 2D coordinates containing the calculated new positions count - number of input coordinates win_size - size of the search window at each pyramid level max_iter - stop searching after the specified maximum number of iterations status - each element is set to 1 if the flow for the corresponding features has been found overwise 0 eps - stop searching when the search window moves by less than eps min_eigen_threshold - the algorithm calculates the minimum eigen value of a 2x2 normal matrix of optical flow equations, divided by number of pixels in a window; if this value is less than min_eigen_threshold, then a corresponding feature is filtered out and its flow is not processed, it allows to remove bad points and get a performance boost */ jsfeat.optical_flow_lk.track(prev_pyr:pyramid_t, curr_pyr:pyramid_t, prev_xy:Array, curr_xy:Array, count, win_size, max_iter = 30, status:Array = null, eps = 0.01, min_eigen_threshold = 0.0001);
The object detector initially proposed by Paul Viola and improved by Rainer Lienhart.
/* Evaluates a Haar cascade classifier at a specified scale int_sum - integral of the source image int_sqsum - squared integral of the source image int_tilted - tilted integral of the source image int_canny_sum - integral of canny source image or undefined width - width of the source image height - height of the source image scale - image scale classifier - haar cascade classifier rects - rectangles representing detected object */ var rects:Array = jsfeat.haar.detect_single_scale(int_sum:Array, int_sqsum:Array, int_tilted:Array, int_canny_sum:Array, width, height, scale, classifier);
/* Evaluates a Haar cascade classifier at all scales int_sum - integral of the source image int_sqsum - squared integral of the source image int_tilted - tilted integral of the source image int_canny_sum - integral of canny source image or undefined width - width of the source image height - height of the source image classifier - haar cascade classifier scale_factor - how much the image size is reduced at each image scale scale_min - start scale rects - rectangles representing detected object */ var rects:Array = jsfeat.haar.detect_multi_scale(int_sum:Array, int_sqsum:Array, int_tilted:Array, int_canny_sum:Array, width, height, classifier, scale_factor = 1.2, scale_min = 1);
/* Groups the object candidate rectangles rects - input candidate objects sequence min_neighbors - Minimum possible number of rectangles minus 1, the threshold is used in a group of rectangles to retain it */ var rects:Array = jsfeat.haar.group_rectangles(rects:Array, min_neighbors = 1);
The original paper refers to: YEF∗ Real-Time Object Detection, Yotam Abramson and Bruno Steux.
/* This step needed only once to create local copy of features to prevent multiple Array relocation during detection */ jsfeat.bbf.prepare_cascade(cascade); /* Build image pyramid using canvas drawImage src - source grayscale matrix_t (U8_t|C1_t) min_width - minimum width to scale pyramid to min_height - minimum height to scale pyramid to interval - number of original scale levels in pyramid */ var img_pyramid:pyramid_t = jsfeat.bbf.build_pyramid(src, min_width, min_height, interval = 4); /* Run detection img_pyramid - pyramid_t from build_pyramid method cascade - cascade data */ var rects:Array = jsfeat.bbf.detect(img_pyramid:pyramid_t, cascade); /* Groups the object candidate rectangles rects - input candidate objects sequence min_neighbors - Minimum possible number of rectangles minus 1, the threshold is used in a group of rectangles to retain it */ var rects:Array = jsfeat.bbf.group_rectangles(rects:Array, min_neighbors = 1);