Ctrax Computer and Software Requirements

Ctrax has been installed and tested on 32-bit Windows and 32- and 64-bit Linux systems. As it is written in Python and Python is a platform-independent language, it conceivably can be run on other platforms as well. Installation instructions are at Ctrax Installation. Memory usage is not demanding for reasonably short videos -- all that is needed is enough memory to store the ellipse fit for each fly in each frame. Ctrax will run faster on faster computers, but has no minimal speed requirements.

Ctrax Input Video Formats

Ctrax versions ≥ 0.1.4 uses FFmpeg (via Pyglet's avbin wrapper) to read many types of movie files and codecs. The formats supported by FFmpeg are described at FFmpeg Supported File Formats and Codecs. Depending on the type and parameters of the video compression, Ctrax may be slow and inexact at allowing out-of-order frame access (used by the GUIs). While Ctrax may be able to read a given file format, it may not be able to successfully track the flies if compression artifacts are too prevalent. All our experiments were performed using Fly Movie Format and Static Background Fly Movie Format videos. Older versions of Ctrax only support Uncompressed AVI Format video, Fly Movie Format (FMF) video, or Static Background Fly Movie Format (SBFMF) video.

Ctrax Arena and Video Requirements

25 female and 25 male wild-type flies in a 24.5-cm diameter open arena. Video is recorded at 20 fps in near-IR, hence the flies appear as white specs on a black background. Flies are approximately 8 pixels in length. Their wings have been clipped to keep them from flying, and a heat barrier restricts them from climbing the arena walls.

A description of the tracking algorithm and the apparatuses we used it with is given in the article High-Throughput Ethomics in Large Groups of Drosophila. Next, we discuss briefly the assumptions made by the algorithm, and the constraints they put on the physical arena setup. Here are links to videos of tracking results in our two arena types:

• 25 female and 25 male wild-type flies in open arena
• 50 female wild-type flies in open arena
• 50 male wild-type flies in open arena
• 20 male fru1/fru1 flies in open arena
• 14 male fru1/fru1 flies in covered chamber
• 14 male wild-type flies in covered chamber

Please contact Kristin Branson if you have successfully applied Ctrax in your experimental set-up and have a video to share.

14 male fruitless flies in a 5-in diameter chamber with glass top. Video is recorded at 20 fps in the visible spectrum, and flies appear as black specs on a white background. Flies are approximately 17.5 pixels in length. Their wings have *not been clipped.*

Ctrax Installation: Windows Installer

For 32-bit Windows systems, we have developed a Windows Installer, available at Ctrax Win32 Installer Latest Release. This is the best option if you do not want to worry about separately installing Python and all the Python packages Ctrax builds upon. Download the file with the name Ctrax-<version>.exe, where <version> is the most recent release of Ctrax. Then, run the installer. This will install Ctrax to your Desktop and start menu. Ctrax can then be launched by double-clicking the Ctrax icon on the Desktop, or selecting Ctrax from the Start Menu.

Ctrax Installation: Ubuntu Package

For 32- and 64-bit Ubuntu systems, we have developed Debian packages containing pre-compiled versions of Ctrax. These are distributed at Andrew Straw's Apt Repository.

To install, first add http://debs.astraw.com to your list of repositories. Instructions are available at Andrew Straw's Apt Repository, and general directions for adding repositories in Ubuntu are available at Adding Repositories in Ubuntu. Next, install the package python-ctrax. General directions for how to install software in Ubuntu are available at Installing Software in Ubuntu.

Ctrax can then be run from the command-line with the command Ctrax. It will also be added to the "Applications->Sound & Video" menu.

Ctrax Installation: EasyInstall

Ctrax can also be installed using the EasyInstall tool. From scratch, the following instructions must be followed. If some of the dependencies of Ctrax are already installed, these steps of course can be skipped. Depending on the platform, binary eggs may or may not have been computed. If no binary egg is available, a C++ compiler is required to install from source.

1. Download and install Python version >= 2.5.
2. Download and install Numpy version >= 1.0.3 (no Numpy binary egg exists).
3. Download and install wxPython version >= 2.8 (no wxPython binary egg exists).
4. Download and install EasyInstall.
5. At a command line, type easy_install Ctrax.

Ctrax Installation: DistUtils

Ctrax can be installed directly from source as well. This requires installing all the Ctrax Python Dependencies and a C++ compiler. The source code for Ctrax can be downloaded from Ctrax Source Latest Release. Uncompress the archive, change into the downloaded "Ctrax" directory. Run the command

python setup.py build

to compile the code and

python setup.py install

to install Ctrax. Ctrax can then be run from the command-line with the command

Ctrax

Ctrax Settings->Background Model...

Screenshot of the Background Model Settings dialog.

The first step to start tracking is to set up the background model (see Background Subtraction Considerations). One first defines the parameters of the background model computation, then hits the "Calculate Now" button to compute. Note that this dialog has no effect if the movie loaded in is an SBFMF, as the background model is provided by that movie. The parameters to set are the following:

• Algorithm: This is the algorithm used to estimate the background image and background deviation. "Median/Median Absolute Difference" estimates the center image as the median of the sampled frames and the standard deviation from the median absolute difference from this median. "Mean/Standard Deviation" estimates the center image as the mean of the sampled frames and the deviation as the standard deviation of the sampled frames. In all our experiments, we use the "Median/Median Absolute Difference" algorithm.
• Number of frames: Background is estimated from frames sampled evenly between the first frame and the last frame of the input video. Specify the number of frames to sample here. In all our experiments, we used 200 frames.
• First Frame and Last Frame: Specify the interval from which to sample frames to estimate the background. In our experiments, we noticed that the flies toward the end of longer trials tended to sit still, which can interfere with background modeling. In our current experiments, we use the first 6000 frames = 5 minutes of video to estimate the background model.

Ctrax Settings->Background Subtraction...

Screenshot of the Background Subtraction Settings dialog

Background subtraction involves thresholding the difference between the current frame and the mean background image divided by the normalization image. If "Background Type" is "Light flies on a dark background", then we only look for pixels that are brighter than the background image, so we threshold

[distance image] = ( [current image] - [mean background image] ) / [normalization image]

If "Background Type" is "Dark flies on a light background", then we only look for pixels that are darker than the background image, so we threshold

[distance image] = - ( [current image] - [mean background image] ) / [normalization image]

If "Background Type" is "Other", then we look for any difference from the background, and threshold the absolute difference:

[distance image] = | [current image] - [mean background image] | / [normalization image]

The image that is thresholded, the "distance image" can be thought of as the distance between the background model and the current image. This distance image can be seen if "Distance from Background" is selected from the display drop-down menu.

• The mean background image can be seen by selecting "Background Image" from the display drop-down menu. It is the mean image computed during the background modeling step (Ctrax Settings->Background Model...).
• The normalization image is defined by the "Normalize by" drop-down menu. It can be seen by selecting "Normalization Image" from the display drop-down menu.

• If "Standard Deviation" is selected, then we normalize by the thresholded standard deviation, as computed in the background modeling step. The thresholds are set at "Std Range". The standard deviation image is more susceptible to errors in estimation from flies sitting still for long periods of time. If you see bright spots in the normalization image that are fly-shaped, these are probably caused by flies that sit still for too long. Having bright spots like this can cause the background subtraction to fail in these areas, as it is equivalent to setting the threshold to much higher values in these regions of the image. One can avoid this by setting the range of allowed standard deviations. In our experiments, we normalize by the standard deviation, and we set the Standard Deviation Range so that the maximum standard deviation is three times the minimum standard deviation, and the range is centered on the average standard deviation for the arena (for our open arena, this is [1,3]).
• If "Background Brightness" is selected, then we normalize by the background mean image. This is sometimes a reasonable approximation, as in general the brighter the pixel, the higher the noise is.
• If "Homomorphic Filtering" is selected, then we approximate Homomorphic Filtering on the current image. This approximation consists of finding the normalization image that would give the same results as homomorphic filtering on the background mean image, then just normalizing any given frame by this normalization image. The parameters of the homomorphic filter can be set by clicking the "Homomorphic Filter Settings..." button.

• There may be regions of the video where there will never be flies. The regions of the image that will always be classified as background are shown in white if "Background-Only Areas" is selected from the display drop-down menu. One can force the algorithm to never detect foreground in certain regions in the following three ways.

  • First, one can set the "Minimum Non-Foreground Intensity". All locations in the image such that the mean background image is above this value will always be classified as background. Similarly, one can set the "Maximum Non-Foreground Intensity" and all lcoations in the image such that the mean background image is below this value will always be classified as background.
  • Second, one can set a circular region of interest using the "Detect Circular Arena" dialog. All regions outside of the circle are assumed to be background. In this dialog, the image panel shows edges detected in the image. The circular region of interest can be manually adjusted by dragging the green and yellow circles to set the center and radius, respectively. Or, the "Radius", "X", and "Y" values can be adjusted manually. The circular region of interest can also be detected automatically by fitting a circle to edges in the image using the "Detect Arena" button. The "Refine Estimate" can be used to automatically refine the estimate. The edge threshold used can be increased or decreased using the "Edge Threshold" control. We used the automatically detected circular region of interest in our experiments, as the wall of the arena was reflective, and flies were spuriously detected on the walls when this region of interest was not set.
  • Third, one can set polygonal regions of interest using the "Regions of Interest" dialog [new in version 0.1.4]. Regions outside of all polygons set are assumed to be background. In this dialog, the image panel can show either the background model center or the currently selected regions of interest in white and the always-background regions in black. The previously selected polygons are shown by red lines. To select a region, click on the image. Continue clicking to add more points. Either double-click the start point or push the "Close" button to close and add the selected region. To cancel adding the current polygon, push the "Cancel" button. To remove the last-added polygon, use the "Undo" button. To save the currently selected regions, use the "Save" button. To close the dialog, use the "Quit" button.

Screenshot of Detect Circular Arena dialog

Screenshot of Regions of Interest dialog

Screenshot of Fix Background Model dialog

To obtain a classification of each pixel location as foreground or background, we threshold the distance image, the normalized difference between the current image and the background mean. We use a hysteresis approach to thresholding. For a pixel to be foreground, its distance from the background must be above a low threshold ("Low Thresh"), and there must be a path from it to a pixel that is above a higher threshold ("High Thresh") that only goes through pixels above the low threshold. That is, we find all pixels that are above the low threshold, then remove all connected components of pixels such that no pixel in the connected component is above the high threshold. The low and high thresholds can be set with the scrollbars on the left. The low threshold cannot be higher than the high threshold. Changing the normalization algorithm will greatly change the range of distances, so the GUI tries to compensate for these changes in magnitude. The classification of pixels into foreground and background can be seen by selecting "Foreground/Background Classification" from the display drop-down menu.

If there are flies that are not being detected, lowering the high threshold can help. If there are objects other than flies being detected, raising the high threshold can help. Lowering the low threshold will result in the area of the detected flies being bigger and raising the low threshold will result in the area of the detected flies being smaller. Setting the low threshold too low will result in noise near a fly being detected as part of the fly. The benefits of a larger area are that we can better estimate the center position and orientation of the fly. The disadvantages of a larger area are that when flies are close to each other, they will merge into a single connected component, and clustering will need to be used to separate the flies, causing the tracker to run slower. The connected components of foreground pixels can be seen by selecting "Connected Components" from the display drop-down menu. The ellipses fit to each connected component of foreground pixels can be seen by selecting "Ellipse Fits" from the display drop-down menu. Note that if two flies are part of the same connected component, they will only be fit at this step by one ellipse. When tracking is performed, this ellipse may be split into multiple flies, depending on the tracking settings. This is the best display setting for looking for errors in the background subtraction step, as it displays the raw frame below the annotated ellipses. For all display settings, the frame shown can be controlled with the slider bar below the image panel.

Camera noise and compression artifacts can potentially be compensated for by applying morphological filtering. One can apply Morphological Opening and Morphological Closing. We do not use morphological filtering in our experiments.

If a fly sits still for too large a fraction of the video, the background model may assume that this fly is part of the background. Errors in the background modeling can be fixed using the "Fix Background Model" dialog [new in version 0.1.4]. Using this dialog, one can select polygonal regions of the image, and fill these regions by interpolating from the boundaries. The interface is similar to the "Regions of Interest" dialog. The image panel can show either the background model center or deviation estimates.

Ctrax Settings->Tracking Settings...

Screenshot of the Tracking Settings dialog.

The "Settings->Tracking Settings..." dialog controls the parameters of the observation detection, identity assignment, and hindsight steps. Please see the related sections in the online methods and supplementary note of High-Throughput Ethomics in Large Groups of Drosophila.

Screenshot of some of the shape parameter-related displays in the Tracking Settings dialog.

Screenshot of some of the motion parameter-related displays in the Tracking Settings dialog.

The right side of the dialog shows the current frame annotated with estimates of the positions of the flies at different stages in the algorithm so that the user may attempt to see the effects of many of the parameters. The toolbar above the image allows one to zoom in, zoom out, and get information about the shape of the next-selected fly. The following displays are available.

• Unfiltered Observations: The current frame is annotated with the ellipses fit to each connected component of foreground pixels (the same as "Ellipse fits" in the background subtraction dialog).
• Filtered Observations: The current frame is annotated with the final results from the observation detection step -- the result of modifying the ellipse fits based on the model of allowed fly shapes.
• Small Observations: The current frame is annotated with ellipse fits for only those connected components whose area is smaller than the minimum area.
• Large Observations: The current frame is annotated with ellipse fits for the only those connected components whose area is bigger than the maximum area.
• Deleted Observations:The current frame is annotated with the initial ellipse fits for connected components that are ignored in the detection step.
• Split Observations: The current frame is annotated with observation detections that resulted from large observations being split into multiple flies.
• Merged Observations: The current frame is annotated with observation detections that resulted from small observations being merged into a single fly.
• Threshold-Lowered Observations: The current frame is annotated then observation detections whose initial area was small and were enlarged by lowering the background subtraction threshold nearby.
• Max Jump: Illustration of the "Max Jump Distance" parameter from the "Motion" tab. The current frame is annotated with a circle for each fly. The radius of this circle reflects the maximum distance between the predicted and detected positions of a fly if the change in orientation is zero. If the distance is larger than this, then the algorithm will instead create a new trajectory.
• Motion Model Prediction: Illustration of the motion model parameters. For each fly in the current frame, we show the previous two positions of the fly and the predicted position of the fly in this frame. The green bars and blue arcs illustrate the relative weight of errors in orientation matching and center position matching. The green lines are equivalent in error weight to the blue arcs.

There are four tabs in the Tracking Settings dialog, described below.

Screenshot of the four Tracking Settings dialog tabs.

Ctrax Track->Start Tracking

The "Track" menu has various ways to start/restart tracking.

• Start Tracking will delete any trajectories computed previously, and start tracking from the current frame. Be careful with this option.
• Resume Tracking will go to the last frame for which trajectories have been computed, and restart tracking from there.
• Resume Tracking from Current Frame will keep trajectories computed for frames before the current frame, remove trajectories from this frame on, and restart tracking from the current frame.

Ctrax Track->Choose Orientations...

Screenshot of "Choose Orientations..." dialog.

The tracking algorithm only computes orientation modulo pi radians -- it does not resolve the head-tail ambiguity. "Choose Orientations..." resolves this ambiguity using the velocity of the flies. It is based on the following two assumptions:

1. When the fly is walking, it is usually walking forward.
2. The orientation does not change much from one frame to the next.

"Choose Orientations..." decides whether to add pi radians to a fly's orientation in each frame to minimize the following criterion:

Here, J¹ is the velocity direction constraint, where θ_t is the orientation at frame t modulo π radians, φ_t is the velocity direction at frame t modulo 2|pi| radians, and s_t is whether or not to add π to the orientation. J² is the temporal constraint. v_t is the speed in frame t. The weight w(v_t) is the weight of the velocity term with respect to the temporal term:

λ is the "Velocity Weight Constant" and w_max is the "Maximum Velocity Weight". In all our experiments, we set "Velocity Weight Constant" to .05 and "Maximum Velocity Weight" to 0.25.

Ctrax File->Export as MAT-file...

To analyze the trajectories in Matlab using either the FixErrors GUI or the BehavioralMicroarray toolbox, one must save the trajectories to a Matlab-native format. This can be done by selecting "File->Export as MAT-file". This will prompt the user for the name of a mat file. It will save the following variables:

• ntargets: 1 x nframes vector, where ntargets(t) is the number of flies tracked in frame t.
• identity: 1 x sum(ntargets) vector, where identity(1:ntargets(1)) are the identities of the flies tracked in frame 1, identity(ntargets(1)+1:ntargets(1)+ntargets(2)) are the identies of the flies tracked in the second frame, etc.
• x_pos: 1 x sum(ntargets) vector, indexed as identity, storing the x-coordinate of the center of a fly in pixels.
• y_pos: 1 x sum(ntargets) vector, indexed as identity, storing the y-coordinate of the center of a fly in pixels.
• maj_ax: 1 x sum(ntargets) vector, indexed as identity, storing the quarter-major axis length of a fly in pixels.
• min_ax: 1 x sum(ntargets) vector, indexed as identity, storing the quarter-minor axis length of a fly in pixels.
• angle: 1 x sum(ntargets) vector, indexed as identity, storing the orientation of a fly in radians.

As is, this is not the easiest-to-use format for Matlab. The load_tracks.m function reads in this mat file and writes it out in a more usable format.

"File->Write Timestamps to MAT-file..." operates the same as "File->Export as MAT-file..." except that it also saves the 1 x nframes array timestamps which contains the timestamp from each frame.

Ctrax Display Control

The frame shown can in the main panel can be controlled with the slider bar below it. The frame number is also shown in the toolbar at the bottom, and the frame shown can also be modified by editing this shown frame number. The buttons on the toolbar, from left to right, have the following effects:

• Magnifying glass: If this is selected, when the user clicks on a fly in the main panel the zoom window (above, left) will pop up and show the selected fly in higher resolution. As the frame displayed changes, these zoomed views will update.
• Play button: This will play the video at the speed set in "Play Speed".
• Stop button: This will stop playback of the video if the video is currently playing, or stop tracking if the video is currently being tracked.
• Fast-forward button: If the video is playing, then this will increase the playback speed. If the video is being tracked, this will increase the rate with which the display is refreshed.
• Rewind button: If the video is playing, then this will decrease the playback speed. If the video is being tracked, this will decrease the rate at with the display is refreshed. For fastest tracking, one can turn off automatic refreshing altogether in the Settings->Playback Options menu.
• Refresh button: During tracking, clicking this button will result in the display refreshing after the next-tracked frame.

The "Settings->Playback Options" menu allows one to set the following playback parameters: * "Show Old Annotation*: Whether we annotate the displayed movie with the computed trajectories or not. * "Tail Length": Number of frames for which to plot the center position of the fly previous to the current frame. * "Automatically Refresh": During tracking, whether we refresh the screen with the current tracking results. * "Dim Original": Whether we should show the raw video darker -- this causes the trajectories to stand out more in the display.

Ctrax Batch Processing

Screenshot of "Batch Processing" dialog.

Ctrax can be set up to process multiple movies in a row.

1. Choose the movies to be processed using the "Add" and "Remove" buttons.
2. "Circular Arena": Set whether you want to auto-detect the circular arena independently in each new movie or whether you want to use the arena settings from the first movie. If in the currently loaded movie, it is set not to use a circular region of interest, then none will be used in all the movies. If in the currently loaded movie, it is set to use a circular region of interest, that exact region of interest will be used in all the movies.
3. "Shape": Set whether you want to auto-detect the shape parameters (Ctrax Tracking Settings Shape Parameters) in each movie, or use the shape parameters set in the first movie.
4. "Background Model": Set whether you want to recompute the background model for each new movie, or use the background model from the current movie.
5. Hit "Execute" to being tracking.

Ctrax Other Commands

• File Menu

  • Open: Open a different movie file.
  • Load Settings from File...: Load the Ctrax settings from a different annotation file.
  • Export as AVI-file: Writes the movie annotated with the computed trajectories to an uncompressed AVI.
  • Quit: Quit Ctrax.

• Track

  • Write compressed sbfmf while tracking: Writes an sbfmf (Static Background Fly Movie Format) version of the movie while tracking. This uses the background model and background subtraction parameters used by the tracker to perform the compression.
  • Compute Background: Computes the background model with the current background model parameters (see Ctrax Settings->Background Model...).
  • Compute Target Shape: Automatically estimates the shape parameters using the current shape parameters (see Ctrax Tracking Settings Shape Parameters).

• Settings

  • "Show Zoom Window": whether to show the zoom window display.

FixErrors Start-Up

FixErrors begins by prompting the user for the name of the raw movie whose trajectories are to be checked, followed by the name of the mat file containing the trajectories to be checked, and the name of the corresponding Ctrax annotation file. If the trajectories have not yet been augmented with the pixel to millimeter and frame to second conversions, the convert_units.m script is run to get these. FixErrors then checks to see if it has already been run on this movie and there is saved information available to restart running. If it finds such saved information, it prompts the user to see if he wants to restart.

FixErrors Suspiciousness Parameters

If he chooses not to restart, he is then prompted for parameters for detecting "suspicious" trajectory sequences -- these are sequences in which tracking is more likely to have failed. The suspiciousness parameters are the following:

Screenshot of FixErrors Suspiciousness Parameters dialog.

• Minimum suspicious prediction - detection error (mm): This parameter is used to identify sequences in which the constant velocity motion model poorly predicts the trajectories. All sequences in which the error between the constant velocity prediction and the measured positions is greater than the set value will be flagged. By default, this is initialized to one-fifth the "Max Jump Error" read from the Ctrax annotation file.
• Minimum suspicious orientation change (deg): All sequences in which the change in orientation is greater than the set value will be flagged. We use 45 degrees in our experiments.
• Minimum suspiciously large major axis (mm): All sequences in which the major axis length is greater than the set value will be flagged. For reference, the max and mean major axis lengths read from the Ctrax annotation file are shown.
• Minimum suspicious orientation - velocity direction mismatch (deg).
• Minimum walking speed (mm/frame): All sequences in which the fly is walking with speed greater than this minimum walking speed and the orientation and velocity direction mismatch by the above value will be flagged.
• Maximum ambiguous error (mm^2): All sequences in which the increase in error for swapping a pair of identities is less than the set value will be flagged.

Note that it is okay to set these suspiciousness parameters to be overly cautious, i.e. so that a lot of sequences are flagged. The FixErrors GUI will show the sequences in order from most suspicious to least suspicious sequence of a given type. Once the user feels that the rest of the sequences of a given type are probably correct, he can move on to another type of suspicious sequence, or quit the program altogether.

Once these parameters are set, FixErrors goes through the trajectories and identifies the suspicious sequences. Depending on the number and length of the trajectories, this may take some time. When it is done, it initializes the mat file containing the current results of FixErrors that can be used to restart the GUI, then brings up the FixErrors GUI.

FixErrors GUI Display

Screenshot of FixErrors GUI.

The left side of the GUI shows the current frame annotated with the trajectories of the fly. For each fly, we show the fit ellipse. Tail direction is indicated by a line segment. Using the "Plot Path" drop-down menu, one can select whether the paths of "All Flies", "Seq Flies" (only those flies that are part of the current suspicious sequence), or "No Flies" are plotted. The number of frames of trajectory around the current frame that are plotted can be adjusted with the "NFrames Plot" control. The "Zoom" drop-down menu controls whether we zoom in on the flies in the current sequence, or show the whole arena. The zoom level and region of the arena shown can also be set manually using the figure toolbar zoom and pan tools.

The "Sequence Info" box shows information about the current suspicious sequence. The "Error" number shows which error sequence of this type is shown (this is not indexed by suspiciousness, but rather fly and frame). It shows the numbers of the frames and the identities of the flies that are suspicious ("Frames" and "Flies"). It shows the "Type" of suspicious sequence, and finally the suspiciousness of the sequence ("Susp"). The "suspiciousness" measure varies from one type of error to another, but will always be nonnegative for the selected sequences, and zero for the least suspicious sequence selectable.

The "Frame Info" box shows information about the current frame and flies. It shows the (editable) current frame number, which frame of the current sequence this is, the suspiciousness for the current frame (the suspiciousness of the sequence is the maximum per-frame suspiciousness over all frames in the sequence). The "Selected Fly" shows the identity of the last clicked fly. This is useful if the colors of flies being examined are similar.

The "Navigation Tools" control the order in which suspicious sequences are shown. Here, the user can select the next type of error to show, and whether the sequences should be sorted by suspiciousness, fly, or frame. We recommend sorting by suspiciousness. When the user is satisfied that the trajectories are correct, he can hit the "Correct" button to go on to the next sequence. The "Back" button is not yet implemented, the "Save" button saves the current state of the GUI to the restart file, and the "Quit" button exits the GUI.

The "Seek Tools" allow one to change the current frame to the next or previous track birth or death in the currently selected axes.

FixErrors Manipulating Trajectories

The positions of the flies in the current frame can be manipulated by dragging around the plotted ellipses. The center position of the ellipse can be changed by dragging the white circle at the center of the ellipse. The white circles at the four end points of the fly's axes can be dragged independently to set these points.

For more drastic changes to the trajectories, one can use the "Edit Tools".

FixErrors Edit Tools: Auto Track

One can modify a sequence of frames for a single fly in one step using the auto-track tool. During auto-tracking, we remove all foreground pixels that can be attributed to other flies. Then, we find the closest connected component of remaining foreground pixels, and store the ellipse fit to this component as the position of the fly. There are a few parameters that can be modified in the auto-tracking. First, one can fix errors in the background modeling by selecting regions of the frame and filling them with a selected color. Second, one can set the "Track Radius" -- the size of the square around the fly's predicted position that is examined during the tracking. Third, one can set the background subtraction threshold. To automatically retrack a fly for a sequence of frames:

Screenshot of FixErrors Retrack Settings dialog.

1. Click on the fly to be tracked in the frame to be tracked from.
2. Push the "First Frame" button to select this fly and frame.
3. If desired, modify the tracking settings by pushing the "Settings..." button:

1. The image panel shows the first frame in a window around the selected fly's position in that frame. In red, it outlines the connected components of foreground pixels detected with the current settings.
2. To fix the background model, drag a rectangle in the "retrack_settings" dialog image. Then, click the "Eyedropper" button, and click on the image to select the color to fill the rectangle with. Finally, click the "Fill" button.
3. The window shown reflects the "Track Radius". The "Track Radius" can be modified with the corresponding "+" and "-" buttons.
4. The background subtraction "Threshold" can be modified with the corresponding "+" and "-" buttons.
5. One can specify that the flies are light on a dark background, dark on a light background, or other.
6. Click the "Done" button when finished modifying the tracking settings.

4. To see the tracking results as they are computed, select "Show Tracking". For faster tracking, deselect it.
5. Scroll to the frame to be tracked until, then click the "Do It" button.
6. To stop tracking at any time, hit the "Stop" button.

showtrx.m

This script allows the user to view a movie annotated with the trajectories computed by Ctrax. The user first inputs the movie and corresponding trajectory mat file. This movie is shown in the main window of the GUI that is started. The user can scroll to different frames or play the movie. The user can select any number of flies, and play the movie keeping the axes zoomed on these flies only. Per-frame properties of the last clicked fly are also shown in the "Selected Flies" panel. More per-frame statistics can be computed by clicking the "Compute Per-Frame Stats" button, which calls the compute_perframe_stats.m script.

simple_diagnostics.m

This script inputs the trajectories of flies in a video, and outputs a few simple visualizations of the data. In figure 1, it plots the (x,y) position of each fly in a separate subplot. In figure 2, it plots a histogram of the position in the arena of all flies (frequency in the left subplot, log frequency in the right subplot). In figure 3, it plots a histogram of the speed of all the flies in black as well as each fly individually in color. In figure 4, it plots a histogram of the angular speed (change in orientation) of all the flies in black as well as each fly individually in color.

make_ctrax_results_movie.m

This function makes an AVI out of the raw video of the flies in the arena and the trajectories returned by Ctrax/FixErrors like those shown in Ctrax Arena and Video Requirements. The user is prompted for the raw video to input, the name of the mat file containing the trajectories, and the name of the AVI file to output the annotated movie to. The user can then set parameters of which frames to output (first frame and number of frames to output), the video frames per second, the compressor (Windows only), and the number and size of the zoomed-in fly boxes to show in the right panel of the movie.

load_tracks.m

Usage: [trx,matname,succeeded] = load_tracks

This function loads in the trajectories output to a mat file by Ctrax into the easier-to-use structure trx. The trx structure is the representation of the trajectories used throughout this toolbox. It prompts the user for a mat file containing trajectories -- this can either be a matfile output by Ctrax or any mat file containing the trx variable: the output of load_tracks.m, convert_units.m, compute_perframe_stats.m, compute_perframe_stats_social.m, classify_by_area.m, If the input mat file is the raw output from Ctrax, the function stores trx to the auto-created file defined by the returned value matname. The returned value succeeded reflects whether trajectories were successfully loaded or not. trx is an array of structs with an element for each fly (i.e. length(trx) is the number of flies). trx(fly) has the following member variables.

• firstframe is the frame that the trajectory begins on.
• endframe is the frame that the trajectory ends on.
• nframes (nframes = endframe - startframe + 1) is the length of the trajectory in frames.
• x is a 1 x nframes array containing the x-position of the fly (in pixels) in each frame. Thus, trx(fly).x(i) is the x-position of fly in the i th frame of fly's trajectory (frame trx(fly).firstframe + i - 1).
• y is the 1 x nframes array of the y-position of fly in each frame (pixels),
• theta is the 1 x nframes array of the orientation of fly in each frame (radians),
• a is the 1 x nframes array of the quarter-major axis length of fly in each frame (pixels).
• b is the 1 x nframes array of the quarter-minor axis length of fly in each frame (pixels).

convert_units.m

This script allows a user to convert from the video units of pixels and frames to the real units of millimeters and seconds.

1. The user first selects a mat file containing the flies' trajectories to convert.
2. He then sets the number of frames per second.
3. Next, there is the option to set the conversion from pixels to millimeters manually, or to compute this conversion by entering the known distance in millimeters between two landmark points.
4. In the "Compute" option, the user first selects the movie file corresponding to the trajectories. The first frame of this movie is then read in and displayed.
5. The user then draws a line between two landmark points.
6. He then sets the length of this line in millimeters.
7. Finally, he selects a mat file to save the results to.

The resulting trx structure (see load_tracks.m) is augmented with the following member variables:

• x_mm is the 1 x nframes array of x-position in millimeters.
• y_mm is the 1 x nframes array of y-position in millimeters.
• a_mm is the 1 x nframes array of quarter-major axis length in millimeters.
• b_mm is the 1 x nframes array of quarter-minor axis length in millimeters.
• pxpermm is the number of pixels per millimeter.
• fps is the number of frames per second.

compute_perframe_stats.m

This script inputs the trajectories of flies in a video, and outputs a number of per-frame properties of interest. These include per-frame speed and acceleration properties such as velocity, angular velocity, forward and sideways velocity, velocity direction, change in velocity direction, velocity of the nose and tail of the fly, center of rotation, speed of center of rotation, acceleration. In addition, there is the option to compute arena-based properties for circular arenas, such as distance to wall, orientation relative to the wall, change in distance to the wall. Finally, there is the option to compute per-frame properties involving other flies, such as distance to the closest fly (center-center, nose-ellipse), maximum angle of the fly's vision subtended by another fly, velocity toward the closest fly, orientation and velocity direction relative to closest fly, and change in any of these properties. More specifically, the following per-frame properties are always computed. All of these fields are appended to the trx structure.

Basic parameters:

• dtheta: angular velocity (rad/s) [1 x (nframes - 1)].
• absdtheta: angular speed (rad/s) (|dtheta|) [1 x (nframes - 1)].
• signdtheta: boolean expressing whether change in orientation is positive or negative (no units) [1 x (nframes - 1)].
• d2theta: angular acceleration (rad/s^2) [1 x (nframes - 1)]. Note that d2theta(1) is always set to 0, and d2theta(t) is the acceleration computed from the angular velocities in frames t and t + 1.
• absd2theta: absolute angular acceleration (|d2theta|) (rad/s^2) [1 x (nframes - 1)].
• smooththeta: orientation, smoothed with the filter [1,4,6,4,1]/16 (rad) [1 x nframes].
• smoothdtheta: angular velocity, computed from the smoothed orientation (rad/s) [1 x (nframes - 1)].
• abssmoothdtheta: angular speed, computed from the smoothed orientation (rad/s) [1 x (nframes - 1)].
• smoothd2theta: angular acceleration, computed from the smoothed orientation (rad/s^2) [1 x (nframes - 1)]. Note that smoothd2theta(1) is always set to 0, and d2theta(t) is the acceleration computed from the angular velocities in frames t and t + 1. * abssmoothd2theta: absolute angular acceleration, computed from the smoothed orientation (|smoothd2theta|)(rad/s^2) [1 x (nframes - 1)].
• velmag_ctr: speed (mm/s) [1 x (nframes - 1)].
• du_ctr: forward velocity of the fly's center (mm/s) [1 x (nframes - 1)]. This is the projection of the fly's velocity on its orientation direction.
• dv_ctr: sideways velocity of the fly's center (mm/s) [1 x (nframes - 1)]. This is the projection of the fly's velocity on the direction orthogonal to the orientation. Right is positive, left is negative.
• corfrac_maj: projection of the center of rotation on the fly's major axis (no units) [1 x (nframes - 1)]. This is a measure of the point on the fly that translates least from one frame to the next. It is 0 at the center of the fly, 1 at the forward tip of the major axis, and -1 and the backward tip of the major axis.
• corfrac_min: projection of the center of rotation on the fly's minor axis (no units) [1 x (nframes - 1)]. This is a measure of the point on the fly that translates least from one frame to the next. It is 0 at the center of the fly, 1 at the right tip of the minor axis, and -1 and the backward tip of the minor axis.
• abscorfrac_min: absolute value of corfrac_min (no units) [1 x (nframes - 1)].
• velmag: speed of the center of rotation (defined by corfrac_maj and corfrac_min) (mm/s) [1 x (nframes - 1)].
• corisonfly: boolean expressing whether any point on the fly does not move (no units) [1 x (nframes - 1)].
• du_cor: forward velocity of the fly's center of rotation (defined by corfrac_maj and corfrac_min) (mm/s) [1 x (nframes - 1)]. This is the projection of the change in the position of the center of rotation on the fly onto the fly's orientation direction.
• dv_cor: sideways velocity of the fly's center of rotation (defined by corfrac_maj and corfrac_min) (mm/s) [1 x (nframes - 1)]. This is the projection of the change in the position of the center of rotation on the fly onto the direction orthogonal to the fly's orientation.
• absdv_cor: sideways speed of the fly's center of rotation (|dv_cor|) (mm/s) [1 x (nframes - 1)].
• flipdv_cor: sideways velocity of the fly's center of rotation, sign-normalized so that if the fly's orientation is turning right, then flipdv_cor is positive if the fly's center of rotation is also translating to the right (dv_cor * signdtheta) (mm/s) [1 x (nframes - 1)].
• accmag: acceleration of the center of rotation (defined by corfrac_maj and corfrac_min) (mm/s^2) [1 x (nframes - 1)]. Note that accmag(1) is always set to 0, and accmag(t) is the acceleration computed from the velocities in frames t and t + 1.
• phi: velocity direction (rad) [1 x nframes].
• yaw: difference between velocity direction and orientation (rad) [1 x nframes].
• absyaw: absolute difference between velocity direction and orientation (rad) [1 x nframes].
• du_tail: forward velocity of the backmost point on the fly (mm/s) [1 x (nframes - 1)]. This is the projection of the fly's velocity on its orientation direction.
• dv_tail: sideways velocity of the backmost point on the fly (mm/s) [1 x (nframes - 1)]. This is the projection of the fly's velocity on the direction orthogonal to the orientation. Right is positive, left is negative.
• absdu_tail: forward speed of the backmost point on the fly (mm/s) [1 x (nframes - 1)].
• absdv_tail: sideways speed of the backmost point on the fly (mm/s) [1 x (nframes - 1)].
• dtheta_tail: change in orientation of fly's head around mean tail location (rad/s) [1 x (nframes - 1)].
• absdtheta_tail: absolute change in orientation of fly's head around mean tail location (rad/s) [1 x (nframes - 1)].
• phisideways: difference between velocity direction and direction orthogonal to fly's orientation (range is [-pi/2,pi/2)) (rad) [1 x nframes].
• absphisideways: absolute difference between velocity direction and direction orthogonal to fly's orientation (range is [0,pi/2]) (rad) [1 x nframes].

Circular arena parameters:

• dist2wall: distance to arena wall (mm) [1 x nframes]
• theta2wall: angle to closest point on the arena wall, relative to the fly's orientation (rad) [1 x nframes].
• ddist2wall: change in distance to arena wall (mm/s) [1 x (nframes-1).
• absdangle2wall: absolute value of change in direction to closest point on the wall (rad/s) [1 x (nframes-1)].

Closest fly parameters:

• dcenter: minimum distance between this fly's center and other flies' centers (mm) [1 x nframes].
• closestfly_center: identity of closest fly, based on dcenter [1 x nframes].
• dnose2ell: minimum distance between this fly's nose and any point on another fly (mm) [1 x nframes].
• closestfly_nose2ell: identity of closest fly, based on dnose2ell [1 x nframes].
• dell2nose: minimum distance between any point on this fly and the nose of other flies (mm) [1 x nframes].
• closestfly_ell2nose: identity of closest fly, based on dell2nose [1 x nframes].
• anglesub: maximum total angle of fly's view occluded by another fly (rad) [1 x nframes]. The fly's view is restricted to [-fov/2,fov/2], where fov is the set field of view.
• closestfly_anglesub: identity of closest fly, based on anglesub [1 x nframes].
• magveldiff_center: magnitude of difference in velocity of this fly and velocity of closest fly based on dcenter (mm/s) [1 x (nframes-1)].
• magveldiff_nose2ell: magnitude of difference in velocity of this fly and velocity of closest fly based on dnose2ell (mm/s) [1 x (nframes-1)].
• magveldiff_ell2nose: magnitude of difference in velocity of this fly and velocity of closest fly based on dell2nose (mm/s) [1 x (nframes-1)].
• magveldiff_anglesub: magnitude of difference in velocity of this fly and velocity of closest fly based on anglesub (mm/s) [1 x (nframes-1)].
• veltoward_center: velocity of this fly in direction to closest fly based on dcenter (mm/s) [1 x (nframes-1)].
• veltoward_nose2ell: velocity of this fly in direction to closest fly based on dnose2ell (mm/s) [1 x (nframes-1)].
• veltoward_ell2nose: velocity of this fly in direction to closest fly based on dell2nose (mm/s) [1 x (nframes-1)].
• veltoward_anglesub: velocity of this fly in direction to closest fly based on anglesub (mm/s) [1 x (nframes-1)].
• absthetadiff_center: absolute difference in orientation between this fly and closest fly based on dcenter (rad) [1 x nframes].
• absthetadiff_nose2ell: absolute difference in orientation between this fly and closest fly based on dnose2ell (rad) [1 x nframes].
• absthetadiff_ell2nose: absolute difference in orientation between this fly and closest fly based on dell2nose (rad) [1 x nframes].
• absthetadiff_anglesub: absolute difference in orientation between this fly and closest fly based on anglesub (rad) [1 x nframes].
• absphidiff_center: absolute difference in velocity direction between this fly and closest fly based on dcenter (rad) [1 x (nframes-1)].
• absphidiff_nose2ell: absolute difference in velocity direction between this fly and closest fly based on dnose2ell (rad) [1 x (nframes-1)].
• absphidiff_ell2nose: absolute difference in velocity direction between this fly and closest fly based on dell2nose (rad) [1 x (nframes-1)].
• absphidiff_anglesub: absolute difference in velocity direction between this fly and closest fly based on anglesub (rad) [1 x (nframes-1)].
• absanglefrom1to2_center: absolute difference between direction to closest fly based on dcenter and this fly's orientation (rad) [1 x nframes].
• absanglefrom1to2_nose2ell: absolute difference between direction to closest fly based on dnose2ell and this fly's orientation (rad) [1 x nframes].
• absanglefrom1to2_ell2nose: absolute difference between direction to closest fly based on dell2nose and this fly's orientation (rad) [1 x nframes].
• absanglefrom1to2_anglesub: absolute difference between direction to closest fly based on anglesub and this fly's orientation (rad) [1 x nframes].
• ddcenter: change in minimum distance between this fly's center and other flies' centers (mm/s) [1 x (nframes-1)].
• ddnose2ell: change in minimum distance between this fly's nose and any point on another fly (mm/s) [1 x (nframes-1)].
• ddell2nose: change in minimum distance between any point on this fly and the nose of other flies (mm/s) [1 x (nframes-1)].
• danglesub: change in maximum total angle of fly's view occluded by another fly (rad/s) [1 x (nframes-1)].

compute_perframe_stats_social.m

Like compute_perframe_stats.m, this script inputs the trajectories of flies in a video and outputs a number of per-frame properties of interest. These per-frame properties are functions of the trajectories of each pair of flies. Because there will be a trajectory computed for each pair of flies, the script saves the results to multiple mat files. Each mat file corresponds to the main fly fly1 being a different fly, and the saved trajectory in pairtrx(fly2) corresponds to the pair (fly1,fly2). The names of each mat file are autogenerated, created as the original mat name appended with "_perframepairs_fly``[fly1]``_start``[t0]``_end``[t1]``", with [fly1] standing in for the index of the main fly, [t0] standing in for the first frame of the pair trajectory and [t1] standing in for the last frame of the pair trajectory. For example, if fly1 = 5, t0 = 1, and t1 = 1000, then we append the mat name with "_perframepairs_fly05_start00001_end01000". The per-frame pairwise parameters computed are:

• dcenter: distance between fly1 and fly2's centers (mm) [1 x nframes].
• distnose2ell: distance between fly1's nose and any point on fly2 (mm) [1 x nframes].
• magveldiff: magnitude of difference in velocity of fly1 and fly2 (mm/s) [1 x (nframes-1)].
• veltoward: velocity of fly1 in direction to fly2 (mm/s) [1 x (nframes-1)].
• thetadiff: difference in orientation between fly2 and fly1 (rad) [1 x nframes].
• absthetadiff: absolute difference in orientation between fly2 and fly1 (rad) [1 x nframes].
• phidiff: difference in velocity directions between fly2 and fly1 (rad) [1 x (nframes-1)].
• minvelmag: minimum velmag of fly1 and fly2 (mm/s) [1 x (nframes-1)].
• maxvelmag: maximum velmag of fly1 and fly2 (mm/s) [1 x (nframes-1)].
• anglefrom1to2: difference between direction from fly1 to fly2 and fly1's orientation (rad) [1 x nframes].
• absanglefrom1to2: absolute difference between direction from fly1 to fly2 and fly1's orientation (rad) [1 x nframes].
• danglefrom1to2: change in direction from fly1 to fly2` in ``fly1's coordinate system.
• minabsanglefrom1to2: minimum absolute angle from fly1 to some point on fly2 in fly1's coordinate system.

classify_by_area.m

This script allows one to classify the types of flies in a movie based on their areas. For example, it can be used to classify the sex of flies, as males are smaller than females.

1. The script first prompts the user for mat files containing the trajectories of the flies. Note that the user can enter multiple files to classify the types of the flies in more than one movie simultaneously.
2. If an area-based classifier has already been defined during a previous run of this script, it can be loaded in at this step by choosing the saved mat file containing the parameters of this area-based classifier.
3. The measured image area of the fly will often depend on its location in the arena, in particular because of lighting differences. There is the option to normalize for this effect by learning the quadratic regression that best predicts the area of the fly given its location in the arena. For trajectory mat files which may contain more than one type of fly, we actually predict the ratio of the area in a particular frame to the median area for a fly over the entire trajectory. Better performance may be achieved by using homogeneous trajectory files -- files where all flies are approximately the same size. Then, we predict the ratio of per-frame area to the median area over all frames and flies. One can load in more trajectory mat files to optimize these normalization parameters, including homogeneous trajectory files, at this step. The user can specify whether the extra mat files loaded in are homogeneous or not. If they are, then the normalization terms are estimated only from these homogeneous mat files, otherwise all mat files are used. Also, if the normalization parameters have already been estimated, there is an option to load these in from a saved mat file. After the normalization parameters have been estimated, a figure will pop up showing the normalization function learned -- each pixel in the image encodes the fit ratio of area to median area for that location in the image.
4. The script then brings up a figure showing the (possibly normalized) median for each area fly over its entire trajectory. The fly identities (x-axis) are sorted in order of increasing area (y-axis). The color of the plotted point reflects which mat file the fly is from, and a key is given in the legend. The for reference, the gray lines give the 5th, 25th, 75th, and 95th percentile area for the fly over its trajectory. The red cross-hairs show the current estimate of the threshold between the two types of flies. They can be dragged to move the threshold. To select the currently displayed threshold, hit ENTER.
5. The user must now define the name of the type variable, as well as the names of the two types of flies. For instance, the type variable might be defined to be "sex", and the smaller type might be defined to be "M" (for male) and the larger type might be defined to be "F" (for female). In this case, for each fly, the field sex would be added to the trx variable, and its value will be set to either 'M' or 'F', depending on the fly's median area.
6. The updated trx variable can then be saved to a mat file. The script prompts the user for the names of these files.
7. The parameters of the area-based classifier can then be saved to a mat file for future use.

learn_params.m

This script allows a user to label the starts and ends of episodes of a single behavior. It then learns the threshold-based classifier that best replicates these labels.

1. Choose whether to learn a behavior classifier from pre-labeled data (i.e. if you have already labeled data, or to label new data.
2. To label data where the fly is performing and not performing the behavior:

Screencap of labelbehaviors.m function

• Choose corresponding pairs of movies and mat files containing per-frame properties (see compute_perframe_stats.m).
• Choose the number of flies to label in each movie, and the number of frames to label for each fly labeled. Flies and intervals of frames to label are selected randomly from each movie.
• Choose the file to save the labels to. If you want to learn parameters again from this labeled data using different settings, you can load this file instead of relabeling.
• A GUI will then open, allowing you to set the start and ends of sequences in which the fly is performing the behavior.
• The frame of the video currently shown can be moved using the frame slider or by clicking on the fly's trajectory.
• To add a sequence in which the fly is performing the behavior, go to the start of the sequence and hit the "Add Start" button. This will highlight the segment starting at the current frame and going to the end of the trajectory (or until the start of the next segment).
• To set the end of this segment, go to the last frame of the behavior, and hit the "Add End" button.
• Labeled segments can be removed with the "Delete" button.
• You can zoom in and out using the figure toolbar. The small window in the top right shows what part of the video is shown in the larger main window.
• In the bottom panel below the main window, per-frame properties of the fly are shown. du_ctr is the forward velocity of the fly's center (relative to the fly's orientation, in pixels/frame), dv_ctr is the sideways velocity of the fly's center of rotation, dtheta is the change in orientation (degrees/frame). cor is the center of rotation along the fly's major axis. The center of rotation is defined as the point on the fly that translates the least from one frame to the next, and cor is this point projected onto the major axis (varies between -1 corresponding to the tail and 1 corresponding to the nose). du_cor, dv_cor are the forward and sideways velocity of the fly's center of rotation.
• The color of the plotted point is related to the fly's speed [for Matlab experts, the "Scatter Command" can be changed to any valid Matlab command to change the property encoding the trajectory's color].
• When done labeling a fly and its trajectory interval, close the window to go to the next fly/movie/finish.

3. Choose a mat file to save the learned parameters to.

Screencap of chooseproperties.m function

4. Choose properties of the behavior classifier to learn. The behavior classifier is based on four thresholds. The trajectory from from t1 to t2 can be classified as the behavior if (i) the per-frame bounds hold: in each frame, given properties are within given loose ranges, (ii) the near-frame bounds hold: there is a frame within r frames from each frame t such that given properties are within given tighter ranges, (iii) the sequence sum bounds: the total summed value of given properties over the entire sequence must be within given bounds, and (iv) the sequence mean bounds: the mean value of a given property over the entire sequence must be within given bounds. Within the "chooseproperties" dialog, you can select which properties are used for each of the four types of bounds. The exact parameter ranges will be learned automatically later, but you can also set sensible bounds on the values that these ranges can take. For instance, if the property cannot be negative, it is useful to bound the ranges at >= 0.
5. Now that all the parameters have been set, the software will automatically try to find the ranges of the bounds that result in the segmentation of the training data closest to the labels provided. Figure 1234 shows the current "Score" of the classifiers considered (higher is better). You can hit "Quit" at any time to use the best classifier found so far, or wait until the search converges. Properties of the current distribution of the classifier parameters and the types of classification errors made on the training data are printed out at fixed intervals.
6. The learned parameters have been saved to the file selected in step (3). These can be input to a behavior detector.
7. To illustrate the differences between the detected and labeled behaviors, we plot the fly's position in and around frames where either the behavior is labeled or detected. The fly's position is plotted in black. We plot labeled behaviors in magenta and detected behaviors in cyan. At the start and end of each detected behavior, we plot a green and red circle, resp. There is a subplot for each labeled fly.We plot nothing in cases where no behaviors are labeled or detected.
8. We finally print out the parameters of the classifier.

Screencap of output of learn_params.m.

learn_params_social.m

Similar to the learn_params.m script, learn_params_social.m allows a user to label the starts and ends of episodes of a single social behavior, a behavior involving a pair of flies. It then learns the threshold-based classifier that operates on a pair of flies' trjaectories and best replicates these labels. The following directions parallel the directions in learn_params.m.

1. Choose whether to learn a behavior classifier from pre-labeled data (i.e. if you have already labeled data, or to label new data.
2. To label data where the fly is performing and not performing the behavior:

Screencap of labelbehaviorssocial.m function

• Choose corresponding pairs of movies and mat files containing per-frame properties (see compute_perframe_stats.m).
• Choose the number of flies to label of each sex in each movie, and the number of frames to label for each fly labeled (the "sex" property should have previously been set using the classify_by_area.m script). Flies and intervals of frames to label are selected randomly from each movie.
• Choose the file to save the labels to. If you want to learn parameters again from this labeled data using different settings, you can load this file instead of relabeling.
• A GUI will then open, allowing you to set the start and ends of behavior sequences, and the other fly involved.
• The frame of the video currently shown can be moved using the frame slider or by clicking on the fly's trajectory.
• The currently selected other fly is highlighted, and a white line is drawn from the main fly to the other fly.
• To add a sequence in which the fly is performing the behavior, select the correct "other fly", go to the start of the sequence, and hit the "Add Start" button. This will highlight the segment starting at the current frame and going to the end of the trajectory (or until the start of the next segment). A colored line will connect the other fly's initial position to the labeled trajectory sequence.
• To set the end of this segment, go to the last frame of the behavior, and hit the "Add End" button.
• Labeled segments can be removed with the "Delete" button.
• You can zoom in and out using the figure toolbar. The small window in the top right shows what part of the video is shown in the larger main window.
• In the bottom panel below the main window, per-frame properties of the fly are shown. du_ctr is the forward velocity of the fly's center (relative to the fly's orientation, in pixels/frame), dv_ctr is the sideways velocity of the fly's center of rotation, dtheta is the change in orientation (degrees/frame). cor is the center of rotation along the fly's major axis. The center of rotation is defined as the point on the fly that translates the least from one frame to the next, and cor is this point projected onto the major axis (varies between -1 corresponding to the tail and 1 corresponding to the nose). du_cor, dv_cor are the forward and sideways velocity of the fly's center of rotation.
• The color of the plotted point is related to the distance to the closest fly [for Matlab experts, the "Scatter Command" can be changed to any valid Matlab command to change the property encoding the trajectory's color].
• When done labeling a fly and its trajectory interval, close the window to go to the next fly/movie/finish.

3. Choose a mat file to save the learned parameters to.
4. Choose properties of the behavior classifier to learn. The behavior classifier is based on four thresholds. The trajectory from from t1 to t2 can be classified as the behavior if (i) the per-frame bounds hold: in each frame, given properties are within given loose ranges, (ii) the near-frame bounds hold: there is a frame within r frames from each frame t such that given properties are within given tighter ranges, (iii) the sequence sum bounds: the total summed value of given properties over the entire sequence must be within given bounds, and (iv) the sequence mean bounds: the mean value of a given property over the entire sequence must be within given bounds. Within the "chooseproperties" dialog, you can select which properties are used for each of the four types of bounds. The exact parameter ranges will be learned automatically later, but you can also set sensible bounds on the values that these ranges can take. For instance, if the property cannot be negative, it is useful to bound the ranges at >= 0.
5. Now that all the parameters have been set, the software will automatically try to find the ranges of the bounds that result in the segmentation of the training data closest to the labels provided. Figure 1234 shows the current "Score" of the classifiers considered (higher is better). You can hit "Quit" at any time to use the best classifier found so far, or wait until the search converges. Properties of the current distribution of the classifier parameters and the types of classification errors made on the training data are printed out at fixed intervals.
6. The learned parameters have been saved to the file selected in step (3). These can be input to a behavior detector.
7. To illustrate the differences between the detected and labeled behaviors, we plot the fly's position in and around frames where either the behavior is labeled or detected. The main fly is plotted in black, the other fly is plotted in green. We connect corresponding frames in yellow.We plot labeled behaviors in magenta and detected behaviors in cyan. At the start and end of each detected behavior, we plot a green and red circle, resp. There is a subplot for each labeled fly.We plot nothing in cases where no behaviors are labeled or detected.
8. We finally print out the parameters of the classifier.

Screencap of output of learn_params_social.m.

detect_behaviors.m

This script uses a defined behavior classifier to segment trajectory of each fly (or pair of flies, for social behaviors) into sequences in which the fly is and is not performing the given behavior.

1. The user is prompted for the names of mat file(s) containing the trajectories of the flies. These mat files should contain all the per-frame properties necessary for the behavior detection (see compute_perframe_stats.m and compute_perframe_stats_social.m).
2. The user is prompted for the mat file defining the behavior to be classified. This is the output of either learn_params.m or learn_params_social.m.
3. If there are missing per-frame parameters in the read-in trajectories, the script will try to figure out which functions it must call to compute these.
4. The script then segments each trajectory, and prompts the user for the name of the mat file to save the results to. In this mat file, it saves the array of structs seg, where seg(fly) is the segmentation for fly fly. seg has the fields t1 and t2, which define the start and end frames of intervals in which the fly is performing the behavior.
5. The segmented trajectories are also plotted in a manner similar to learn_params.m and learn_params_social.m. There is a subplot for each fly. The fly's position in each frame is plotted in black. We highlight frames in which the behavior is detected in magenta. At the start and end of each detected behavior, we plot a green and red circle, resp. In the case of social behaviors, during and near detected behaviors, we plot the position of the other fly in green. We connect corresponding frames for the main and other fly in yellow.

Screencap of output of detect_behaviors.m.

histogramproperties.m

Screencap of histogramproperties.m for one property over four different data types.

The histogramproperties GUI allows a user to histogram single and pairs of per-frame properties. When the GUI is started, the user is first prompted for a mat file containing the trajectories with per-frame properties. The user can then interact with the GUI to explore statistics of the per-frame properties for the flies in the trajectories. The "No. Properties" radio buttons allow the user to specify whether he wants to histogram a single per-frame property or jointly histogram a pair of per-frame properties. In the former case, the histogram will consist of the property plotted against frequency. In the latter case, a 2-D histogram will be computed and displayed as an image, the color of the image reflecting the frequency. The "Property 1" and "Property 2" (if two properties are histogrammed) panels allow the user to specify which properties to histogram. The pop-up menu displays the name of the per-frame property (the actual name of the field of the "trx" struct). He can also specify properties of the histogram in this panel; the number of bins to histogram into and the range of the data to histogram, either in percent or in absolute units. Finally, he can specify whether a transformation should be applied to the per-frame data before histogramming. "None (Identity)" specifies that no transformation should be applied, "Absolute value" specifies that the absolute value of the data should be taken, and "Log absolute value" specifies that the log of the absolute value should be taken. If the loaded in trajectories do not have sufficient per-frame statistics to explore, clicking the "Compute Per-Frame Stats" button calls the compute_perframe_stats function on the current trajectories.

The user can compare histograms of different portions of the data. The Data panel defines different histograms to compute. One can specify a behavior segmentation file (computed using detect_behaviors.m), and look at the per-frame properties only during ("During behavior...") or only not during ("Invert behavior") the specified behavior. One can look at the histogram of the properties during all frames of the behavior, or one can condense each sequence of a behavior into a single statistic, then histogram these statistics ("Interval Averaging"). "None (Per-frame)" specifies that all frames should be used, i.e. there is no summarizing of each sequence, and instead the per-frame properties from all intervals are just concatenated together. "Interval mean" specifies that the mean per-frame property for each interval should be histogrammed. "Interval median" indicates that the median per-frame property for each interval should be histogrammed. "Interval start" indicates that the per-frame property in the first frame of the interval should be histogrammed. "Interval end" indicates that the per-frame property in the last frame of the interval should be histogrammed. If the type or sex fields of the trx struct have been set to reflect the type or sex of each fly (e.g. with the classify_by_area.m script), one can also separate the flies by type/sex ("Fly Type(s)"). One can set the name of this data type in the "Data Name" field.

Properties of the plot can be set in the "Plot Parameters" panel. First, one can set precisely what histogram statistic is plotted ("Plot Statistic"). "Total Count" is the total frequency over all flies for each bin. "Total Fraction" is the total fraction (frequency normalized to sum to 1) over all flies. "Mean Fraction per Fly" computes the fraction histogram for each fly, then averages the histograms. "Mean Count per Fly" computes the frequency histogram for each fly, then averages the histograms. Next, one can set whether the standard deviation or standard error should be plotted ("Plot Error Bars"). Standard deviations/errors are computed over flies. If one selects "Plot Individual Flies", the histograms for each individual fly will also be plotted, in addition to the population summaries. If "Plot log of counts/fraction" is checked, then the log frequencies/fractions are plotted.

Screencap of histogramproperties.m for two properties and one data type.

Clicking the "Update Plot" button will update the plotted histogram to reflect any changes to the parameters made. Clicking the "Export..." button allows the user to export the computed histograms to a selected mat file. The following variables are exported:

• countsperfly: The frequency histograms for each fly and data type. [nflies x nbins x ndata].
• fracperfly: The normalized fraction histograms for each fly and data type. [nflies x nbins x ndata].
• propnames: Name(s) of the per-frame properties histogrammed, including transformations performed on the per-frame parameters (absolute value, log absolute value, ...).
• countstotal: Frequency histograms for all flies per data type [1 x nbins x ndata].
• countsmean: Mean of frequency histograms over flies for each data type [1 x nbins x ndata].
• countsstd: Standard deviation of frequency histograms over flies for each data type [1 x nbins x ndata].
• countsstderr: Standard error of frequency histograms over flies for each data type [1 x nbins x ndata].
• fractotal: Normalized fraction histograms over all flies per data type [1 x nbins x ndata].
• fracmean: Mean of normalized fraction histograms over flies for each data type [1 x nbins x ndata].
• fracstd: Standard deviation of normalized fraction histograms over flies for each data type [1 x nbins x ndata].
• fracstderr: Standard error of normalized fraction histograms over flies for each data type [1 x nbins x ndata].
• centers: Centers of the bins for each property. This is a cell with an element for each property (so either a 1 x 1 or 1 x 2 cell). centers{propi} is 1 x nbins.
• props: Field name(s) of the per-frame properties [1 x nprops cell].
• nbins: Number of bins for each property [1 x nprops].
• lb: Lower bound(s) on per-frame properties histogrammed [1 x nprops].
• ub: Upper bound(s) on per-frame properties histogrammed [1 x nprops].
• transforms: Transformations to the per-frame properties before histogramming [1 x nprops cell].
• isbehavior: isbehavior(datai) indicates whether histogram datai is restricted to frames during/not during a behavior [1 x ndata].
• segfile: segfile{datai} is the name of the behavior segmentation file for histogram datai [1 x ndata cell].
• doinvert: doinvert(datai) indicates whether we are looking at the per-frame properties not during or during the behavior [1 x ndata].
• flytype: flytype{datai} are the type(s) of flies that are histogrammed for datai [1 x ndata cell]. If empty, then all fly types are histogrammed.
• dataname: dataname{datai} is the name of histogram datai [1 x ndata cell].

plot_behaviormicroarray.m

This script allows the user to explore differences in the behavioral statistics of different types of flies. When the GUI is started, the user is first prompted to input the name(s) of the trajectory files already augmented with the per-frame properties. The user can then interact with the GUI to plot and compare various behavioral statistics for different populations of flies.

• The Trajectories panel shows the trx mat files currently loaded in. These can be modified with the "Add..." and "Delete" buttons.
• The Behaviors panel shows the behaviors currently loaded in. To load in a behavior, one must associate a behavior segmentation file (the output of detect_behaviors.m) with each trajectory file listed in the "Trajectories" panel. A new behavior can be added with the "Add..." button. This will go through each trx file and prompt the user for the associated segmentation file. Once the segmentation files have been defined, the user can edit the name of the behavior. The currently displayed behavior can be changed using the drop-down menu with the names of all the behaviors. The user can change the segmentation file associated with a specific trx file by selecting that segmentation file and clicking the "Edit..." button. Note that the selected trx file in the "Trajectories" panel corresponds to the selected segmentation file.
• The Populations panel allows the user to define different populations of flies to be compared. A fly type consists of subsets of the trx files and sex and type fields of trx. The sex and type fields can be set, for instance, using the classify_by_area.m script. A new population can be defined by clicking the "Add New" button. The types and trx files associated with the current population can be selected using the listbox. The population currently displayed can be changed with the drop-down menu. The name of the population can be set in the edit box. The current population definition can be removed with the "Delete" button.

• The behavioral statistics to be plotted are set in the "Properties" panel. The drop-down menu allows the user to select the currently displayed property. The text edit box allows the user to set the name of this property. The current property may be associated with a behavior, selectable by behavior name using the drop-down menu. The property may be one of the following, selectable using the radio buttons:

  • Duration: The mean duration of bouts of the associated behavior for the fly.
  • Fraction of time: The fraction of time the fly performs the associated behavior.
  • Frequency: The frequency of onsets of the associated behavior for the fly.
  • Per-frame props: The average value of a per-frame property (as computed by compute_perframe_stats.m or compute_perframe_stats_social.m). The per-frame property is set in the "Per-Frame Property" panel, visible only if "Per-frame props" is selected.
• In the "Per-Frame Property" panel, the drop-down menu allows the user to select the per-frame property (as computed by compute_perframe_stats.m or compute_perframe_stats_social.m script) the current property is based on. The radio buttons allow the user to select whether the property is averaged over only frames during the associated behavior ("During behavior"), not during the associated behavior ("Not during behavior"), or over all frames ("All frames"). If the per-frame property is during or not during behaviors, then one may prefer to represent each bout of the behavior or non-behavior with a single statistic, then average the statistics over bouts. The "Interval averaging" drop-down menu allows one to specify this. If "None (per-frame)" is selected, then the average per-frame property over all relevant frames is computed. If "Interval mean"/"Interval median" is selected, then the mean/median per-frame property per bout is computed and averaged over all bouts. If "Interval start"/"Interval end" is selected, then the per-frame property at the start/end of the bout is averaged over all bouts. A new property can be defined with the "Add new" button, and the currently displayed property definition can be removed with the "Delete" button.

Screenshot of means per population output by plotbehaviormicroarray.m.

Screenshot of per-fly behavioral microarray output by plotbehaviormicroarray.m.

• The Plot panel allows the user to control properties of the plots. In the first figure, we plot the mean of each property over all flies in each population. Thus, we can compare the properties over population. Error bars may also be plotted, indicating either the population standard deviation or standard error ("Error bars" drop-down menu). If "Normalize per-prop" is selected, then the total population statistics are shown in the top subplot, and in the bottom subplot we plot the number of standard deviations each population is from the total population mean. This is often useful as the differences in values between different properties may be much larger than differences between different populations. If "Plot individuals" is selected, then the properties for each individual fly is plotted. If "Normalize per-prop" is selected, then we plot the z-scored properties (the number of standard deviations from the mean for all flies). In the second figure, we show the "behavioral microarray". We draw an image for each population of flies. Each row in the image corresponds to a different property and each column to an individual fly. The color of the pixel for a given property and fly is related to the value of that property, or the normalized value of that property if "Normalize per-prop" is selected. Clicking "Update Plot" updates the two figures to reflect the current settings.
• The properties shown can also be selected automatically selected to best differentiate the populations. If more than one population is defined, then the properties which maximize a variant of Fisher's linear discriminant are selected. More specifically, if mu_i is the mean for population i and sigma_i is the standard deviation for population i, we choose the properties that maximize

• If "Choose independently" is selected, then we choose the top properties. If it is not selected, we greedily choose the best property, remove the correlations between the chosen properties and the remaining property, then choose the property whose residual has the highest discriminative power. If there is only one population of flies, then properties are chosen so that flies cluster into two groups. For every nearly-even split of flies (at least 30% of flies must be in one group), we compute the Fisher linear discriminant. We choose the properties with maximal maximum discrimant over all allowed splits. As with multiple types of flies, if "Choose independently" is selected, we choose the top properties independently. If it is not selected, then we remove correlations with previously chosen properties before choosing again.
• There are a large number of potential properties to choose from, and it may be useful to restrict those under consideration by the auto-choose program. The allowed properties can be manipulated in the "Auto-choose" panel. The total number of available properties to choose from is shown.