Segmentation Transfer v1.2 ========================== This software implements the full segmentation pipeline of [1]. It also includes the large-scale segmentation transfer of [2]. It was developed under Linux with Matlab R2011b. There is no guarantee it will run on other configurations. Introduction ------------ This software provides figure-ground segmentation for a test image as described in [1]. The implementation has also been extended for large-scale datasets as outlined in [2]. The algorithm works in a supervised setting, thus a training set with annotated figure-ground segmentations is needed. Notation -------- "%>" is used for the linux command prompt. ">>" is used for the matlab prompt. Installation ------------ Create an empty folder where you want to install this code. We call it the segtrans folder from here on. %> mkdir segtrans %> cd segtrans Download/move matlab.zip to segtrans and then extract it. %> unzip matlab.zip The matlab.zip contains all the source code for running the pipeline. Download/move the dataset zips you want to use. %> unzip voc10.zip %> unzip horses.zip %> unzip graz.zip Those zip files contain the train and test images for the respective datasets. Additionally, all the transferred masks and final segmentations on the test sets as computed for [1] are provided. Start matlab in segtrans and run compile. %> matlab >> compile This concludes the installation. Quickstart (see quickstart.m) ---------- In this quickstart we take one test image from the VOC10 test set and segment it. At the end, you can compare it to the output we compared, to confirm that your installation works. The segmentations are transferred from the VOC10 training set. Here we use the precomputed VOC10 training set. See usage.m for how to compute your own training sets. See quickstart.m to automatically execute this example. Load the prepared training data of voc10. >> source = load_value('data/xps/voc10/train_source.mat'); Segment a single test image from voc10. >> image = load_value('data/sets/voc10/quickstart_image.mat'); >> segmentation = load_value('data/sets/voc10/quickstart_segmentation.mat'); >> windows = st_windows(image,100); >> features = st_features_gist(image,windows); >> weights = ones(1,100)/100; >> mask = st_transfer(source,50,windows,features,@distance_L2,weights); >> prediction = st_segment(image,mask,0.2,20); Display the results. >> show_prediction(prediction,image,mask); If it worked correctly, the result should be similar to: >> cvpr12_prediction = load_value('data/sets/voc10/quickstart_prediction.mat'); >> cvpr12_mask = load_value('data/sets/voc10/quickstart_mask.mat'); >> show_prediction(cvpr12_prediction,image,cvpr12_mask); General usage (see usage.m) ------------- Collect the training data. >> images = ...; % cell of RGB uint8 images >> segmentations = ...; % cell of logical images (ground truth) Get the windows for transfer and compute their feature representations. >> windows = st_windows(images,100); >> features = st_featrues_gist(images,windows); Construct a "source" for segmentation transfer. >> source = st_source(50,segmentations,windows,features); This source can be used for segmentation transfer. Only the source is needed, the other data can be cleared. >> clear images segmentations windows features; To segment one test image, transfer masks from source and then segment it using the transferred masks. >> image = ...; % RGB uint8 image >> windows = st_windows(image,100); >> features = st_features_gist(image,windows); >> weights = ones(1,100)/100; >> mask = st_transfer(source,50,windows,features,@distance_L2,weights); >> prediction = st_segment(image,mask,0.2,20); Most functions work also with a set of images. Thus, to segment a set of test images: >> images = ...; % cell of RGB uint8 images >> windows = st_windows(images,100); >> features = st_features_gist(images,windows); >> weights = ones(1,100)/100; >> masks = st_transfer(source,50,windows,features,@distance_L2,weights); >> predictions = st_segment(images,masks,0.2,20); If you have ground truth for the test set, compute the scores of the prediction. >> segmentations = ...; % cell of logical images (ground truth for test set) >> ious = st_score_iou(segmentations,predictions); >> accs = st_score_acc(segmentations,predictions); >> [~,~,f1] = st_score_pr(segmentations,predictions); Large scale usage (see usage.m) ----------------- This covers the extension made for [2]. For large-scale data, the feature representation of all windows is too large to keep in memory. Instead, you should use: >> itq = load_value('data/sets/voc10/train_itq.mat'); >> features = st_features_itq(images,windows,@st_feature_gist,{},itq); This ITQ is trained on the voc10 training set. See st_itq.m to train your own ITQ configuration. Further, to speed up the segmentation transfer, we trained a weighted model for transfer. Instead of using 100 nearest neighbors, transfer is now a weighted sum with less neighbors. >> weights = load_value('data/sets/voc10/train_weights.mat'); >> masks = st_transfer(source,50,windows,features,@distance_hamming,weights); (Note also the use of @distance_hamming for ITQ instead of @distance_L2 for Gist.) (Note in [2] we use HOG features, but we didn't add a HOG implementation here.) Parallel computing (map.m) ------------------ The functions that work for multiple images use the map function (map.m). This function is used to map a function to a list of images, segments, ... By default, it just uses a for loop, on one machine. Simply adapt map.m to your local configuration for parallel computing to do segmentation transfer on your cluster. Experiments (see run.m) ----------- To run one of the predefined experiments, first get the configuration. >> c = st_config('voc10'); % or horses, or graz_bikes/cars/people The run the experiment. >> [source test scores] = run(c); See st_config.m and run.m for details. To add a new set it is usually enough to extend st_config.m and add the images of the set to the folder data/sets. Note that the predefined experiments have been precomputed by us. The data you find is the exact data as we used it for [1]. If you run these experiments, nothing will be recomputed. If you delete the precomputed data, or change the folder but keep the same settings, you should get almost the same results. However, there is some randomness in the results, because the windows are not always the same. Third party elements -------------------- Our implementation relies on several third party components. - Windows are generated using the software of [3]. - Our Gist implementation is an adapted version of [4]. - ITQ is adpated from the software of [5]. - An adapted implementation of GrabCut [6]. - To optimize the energy function of GrabCut we use maxflow from [7] Support ------- For further support, please contact one of the authors: Daniel Kuettel, dkuettel@vision.ee.ethz.ch Matthieu Guillaumin, guillaumin@vision.ee.ethz.ch Vittorio Ferrari, vferrari@staffmail.ed.ac.uk References ---------- [1] Daniel Kuettel, and Vittorio Ferrari Figure-ground segmentation by transferring window masks CVPR 2012 [2] Daniel Kuettel, Matthieu Guillaumin, and Vittorio Ferrari Segmentation Propagation in ImageNet ECCV 2012 [3] Bogdan Alexe, Thomas Deselaers and Vittorio Ferrari What is an object? CVPR 2010 http://groups.inf.ed.ac.uk/calvin/objectness/ [4] Aude Oliva, and Antonio Torralba Modeling the shape of the scene: a holistic representation of the spatial envelope IJCV 2001 http://people.csail.mit.edu/torralba/code/spatialenvelope/ [5] Yunchao Gong, and Svetlana Lazebnik Iterative Quantization: A Procrustean Approach to Learning Binary Codes CVPR 2011 http://www.unc.edu/~yunchao/itq.htm [6] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake GrabCut - Interactive Foreground Extraction using Iterated Graph Cuts SIGGRAPH 2004 [7] http://www.mathworks.com/matlabcentral/fileexchange/21310-maxflow adapted from Yuri Boykov and Vladimir Kolmogorov An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision PAMI 2004 http://vision.csd.uwo.ca/code/ Version history --------------- 1.2.1 (15.11.12) fixed st_source.m (new argument, updated all other code) 1.2 (03.08.2012) fixed quickstart.m generaly added more explanations 1.1 (23.07.2012) added large-scale code as used in [2] 1.0 (16.07.2012) original release covering only [1]