Scaled Motion Dynamics

Click Points

(a)

−1 −0.5 0 0.5 1 1.5

−0.5 0 0.5 1

Original Updated

(b)

Figure 6.4: (a) Customized software for manual digitization ofDrosophilabody kinematics from [16].

Points are clicked at the head, tail, wing joint, and wing tip in multiple camera views to manually fit a geometric model to the images. Manually estimated pose is used as an initial guess for the automated algorithm. (b) At the initial frame, the profile of the body is refined, while holding the pose parameters fixed, to more closely match the actual shape of theDrosophilaby minimizing the error described in Section 6.6.

the images based on visual inspection (see Figure 6.4a). Next, the body transformation, (Q^b, T), associated with the manual initialization are refined using the registration procedure described in Section 6.6 applied to the body only segmented images, Ibody, shown in Figure 6.6e. Finally, the shape profile of the body is adjusted, while holding the pose, (Q^b, T), fixed to further match the body only segmented images. The entire width profile is modeled as the combination of two B-spline curves representing the head and body, respectively. The control points of the B-spline curves are adjusted to best match the images, similar to the width profile calculation for zebrafish in Section 4.3. Figure 6.4b illustrates the results of this shape refinement for a particular Drosophila that provides a more accurate representation of the appearance.

with body transformations relative to the fixed observing frame given by the translationTe_i, rotation (represented in quaternion form)Qe^b_i, and joint angle vectors Qe_i (details on collecting the training samples are given in Section 6.8). We denote this list of temporally ordered training samples as P =hpei. . .peNi, and the sublist inP of length m ending at time i byhpe_i−m+1. . .peii. In order to predict the statepk+1at the next time step, the training list searched to find the location in the list that best matches the sublist of previous tracked states, hpk−m+1. . . pki. For the matching to be invariant with respect to the velocity of the tracked fly, the matching is performed at different scalings s of P. The different scalings of the training data, denoted P^s, are calculated using two different techniques. The scaled body translations are obtained using linear interpolation and resampling.

However, linear interpolation in the 4D space of unit quaternions ignores the 3D spherical subspace where quaternions exist and would result in interpolated values with non-uniform velocity that are not elements of SO(3). To remain on the correct constraint surface and produce valid rotations, the technique ofspherical linear interpolation (Slerp) is employed [91]. To interpolate between two rotations given byQ1and Q2 on the intervalu∈[0, 1], calculate

Slerp(Q₁, Q₂;u) = sin(1−u)Ω

sin Ω Q₁+sinuΩ

sin Ω Q₂ (6.15)

where cos Ω = Q^T₁Q2. The resulting scaled lists are given by P^s = {pe^s_i := (Te_i^s,Qe^b,s_i ,Qe^s_i)|i = 0. . . sN}. For the Drosophila flight initiation videos that we analyzed, we scan the interval s = [0.5, 1]with stepsize 0.1 as the scaling factors because the Drosophilawere typically observed to undergo equivalent or faster wing motion than our database, which was captured at 6000 fps. The best matching sublist of the training data is then calculated by

argmin

s,j m−1

X

v=0

kQ_k−v−Qe^s_j−vk

. (6.16)

Only the quaternion representation of the wing joint angles are taken into account since their motion will be invariant with respect to the global orientation of the fly. An illustration of this technique is given in Figure 6.5. In order to calculate the predicted wing configuration, the relative rotation

0 20 40 60 80 100 120

−0.5 0 0.5 1

1.5 Left Wing Orientation

q₁ q₂ q₃ q₀

−0.52 4 0 0.5 1 1.5

0 20 40 60 80 100 120

−4

−2 0 2

4 Left Wing Orientation

A B

Figure 6.5: (A) Rotational motion ofDrosophila left wing motion during take off (120 out of 380 samples shown). Motion is parameterized by quaternions which vary smoothly with time. The query ofm= 5previously calculated poses is matched with position 106 of the prior database. The relative motion to position 107 is used to calculate the prediction. (B) The same motion as (A) but parameterized by joint angles about constant twists. Because there is no 3-dimensional, global parameterization ofSO(3), the wing motion parameters undergo discontinuities as the wing passes through a singularity, resulting in inaccurate prediction of the motion.

between the optimal location in the prior set,j, and its subsequent orientation,j+ 1is calculated.

Then, this relative motion is applied to the current state to generate the prediction of the state variables at the next time frame

∂Qe^lw,s_j+1 =Qe^lw,s_j+1 ·(Qe^lw,s_j )⁻¹ (6.17) Q^lw_k+1=∂Qe^lw,s_j+1·Q^lw_k . (6.18)

An identical calculation to (6.17) is also performed for the right wing and the body orientation. The predicted body translation is given by

Tk+1=Tk+ (Te_j+1^s −Te_j^s) (6.19)

so the entire predicted state consists ofp_k+1=

Tk+1 Q^b_k+1 Q^lw_k+1 Q^rw_k+1

. I must note, however, that this prediction assumes a correlation between the joint angle velocity and the body’s spatial velocity. In [85], Rosenhahn wisely notes that this assumption is not valid for humans because a larger person runs faster than a smaller person with the same changes in joint angles. Their use of the twist representation for the global body transformation allows them rescale the body motion using

the velocity of the previously calculated state, and not that determined from the prior database. For Drosophila, the assumption of correlation between joint velocity and body velocity is valid because there is less variation in body size across the populations used in experimental studies. Nevertheless, in order to track blowflies or other members of the orderDipterathat are larger thanDrosophila, the body motion can still be scaled conveniently using the quaternion representation. Letθe= 2 cos⁻¹q₀^b denote the body velocity of the predicted motion calculated from the scalar part ofQ^b_k+1, andθ is the average value of the lastmframes calculated fromhpk−m+1. . . p_ki. First, we calculate the twist, ξ= _ω^v

, associated with the predicted body motion

ω= 1 sin^eθ₂

q^b_k+1 (6.20)

v= (I−e^ωebθ)bω+ωω^Tθe⁻¹

Tk+1 (6.21)

where ωb denotes the skew-symmetric matrix associated with the vector ω and e^bωθ=I+ωbsinθ+ ωb²(1−cosθ)by Rodrigues’ formula. The rescaled body transformation ( ¯Tk+1,Q¯^b_k+1) is now given by

Q¯^b_k+1=







 ωsin^θ₂

cos^θ₂









(6.22)

T¯_k+1= (I−e^ωθb )ωb+ωω^Tθ

v. (6.23)

In this way, the kind of motion the body undergoes is determined by the prior data, but the velocity is determined by the previous frames.