SLAM-based Joint Calibration of Multiple Asynchronous Microphone Arrays and Sound Source Localization
Abstract
Robot audition systems with multiple microphone arrays have many applications in practice. However, accurate calibration of multiple microphone arrays remains challenging because there are many unknown parameters to be identified, including the relative transforms (i.e., orientation, translation) and asynchronous factors (i.e., initial time offset and sampling clock difference) between microphone arrays. To tackle these challenges, in this paper, we adopt batch simultaneous localization and mapping (SLAM) for joint calibration of multiple asynchronous microphone arrays and sound source localization. Using the Fisher information matrix (FIM) approach, we first conduct the observability analysis (i.e., parameter identifiability) of the above-mentioned calibration problem and establish necessary/sufficient conditions under which the FIM and the Jacobian matrix have full column rank, which implies the identifiability of the unknown parameters. We also discover several scenarios where the unknown parameters are not uniquely identifiable. Subsequently, we propose an effective framework to initialize the unknown parameters, which is used as the initial guess in batch SLAM for multiple microphone arrays calibration, aiming to further enhance optimization accuracy and convergence. Extensive numerical simulations and real experiments have been conducted to verify the performance of the proposed method. The experiment results show that the proposed pipeline achieves higher accuracy with fast convergence in comparison to methods that use the noise-corrupted ground truth of the unknown parameters as the initial guess in the optimization and other existing frameworks.
Index Terms:
Robot audition; Simultaneous localization and mapping; Multiple microphone arrays calibration; Sound source localization.I INTRODUCTION
Microphone array-based robotic auditory systems have many applications such as sound source localization and human-robot interaction [1]-[5]. As with other sensing modalities [6]-[10], precise calibration of robotic auditory system parameters is crucial for achieving satisfactory sound source localization and tracking performance [11]. Hence, the calibration of robotic auditory systems made of single or multiple microphone arrays has received significant attention recently.
Of particular interest in this paper is the parameter calibration of robotic auditory systems that are made of multiple microphone arrays. Compared to single microphone array-based audition systems, there are more parameters to be calibrated for systems with multiple microphone arrays, including the relative transforms (i.e., orientation, translation) and the asynchronous offsets among the arrays. In the following, we first give a brief overview of the relevant literature on calibration of single microphone array-based systems, and then discuss the existing calibration methods for systems with multiple arrays.
I-A Related Work
In [12], based on the time difference of arrival (TDOA) between each pair of microphones, a calibration algorithm was developed to estimate the positions of microphones within a single microphone array. In [13], a bilinear calibration method based on time of flight (TOF) between each sensor source pair was proposed to estimate the microphone and source positions in 3D under the condition that the transmitting time is known. In [14], based on time of arrival (TOA) measurements and assuming knowledge of the distances between the sources and the microphones, a method for joint calibration of the positions of multiple microphones and sound source localization was proposed. In [15], a calibration method using TOA measurements was proposed for the scenario with a planar microphone array and a sound source moving in 3D.
Note that the applicability of the above-mentioned methods is limited in that they all rely on hardware synchronization between microphone channels, which is challenging to implement for robotic platforms in practice due to spatial and cost constraints [11]. Recently, in [16]-[18], a general framework using batch simultaneous localization and mapping has been developed for joint sound source localization and calibration of a single microphone array with asynchronous effects (i.e., clock difference and initial time offset).
Compared to single microphone array-based systems, the calibration of systems with multiple arrays has gained more recent attention. For example, the proposed approach in [19] utilizes direction of arrival (DOA) measurements to determine the sound source location and inter-array TDOA measurements to obtain the microphone array location through exhaustive grid search. The work [20] employs evolutionary algorithms to improve the accuracy and real-time performance of the approach in [19]. Based on DOA and inter-array TDOA measurements, another calibration framework for multiple microphone arrays is proposed in [21] using distributed damped Newton optimization. Note that the above-mentioned methods focus on the 2D case.
For the more general 3D case, there are only a few existing works. In [22], an artificial bee colony algorithm was employed to calibrate the positions and orientation of microphone arrays in 3D. Nevertheless, this method assumes that the sound source position at different moments is partially known and the clocks of the arrays are synchronized using hardware. Simultaneous calibration of positions, orientations, and time offsets of multiple microphone arrays and sound source positions in 3D was explored in [23] and [24].
I-B Motivation
For spatially distributed microphone arrays, it is necessary to consider both the initial time offsets and the sampling clock differences between the arrays [25], especially in the case of asynchronized scenarios based on the USB protocol and wireless acoustic sensor networks. In the above situations, each microphone array captures acoustic signals through its own microprocessor-controlled analog-to-digital converter and has a unique sampling clock source. Therefore, when launching multiple microphone arrays, differences in initialization result in varying initial time offsets between arrays. Moreover, the microprocessors in these microphone arrays often have limited performance, and the oscillators/crystals used to generate clock signals typically drift around their nominal frequencies. As a result, differences in sampling clocks accumulate over time. Not properly handling the above issue will significantly degrade the performance of sound source localization/tracking algorithms embedded in the arrays [17].
To the best of our knowledge, there is no work that has addressed the simultaneous calibration of positions, orientations, time offsets and sampling clock differences of multiple microphone arrays and sound source positions in 3D. In fact, as for single microphone array, calibration of multiple microphone arrays can be considered as a SLAM problem [26]-[29], where microphone arrays and the moving sound source serve as landmarks in the environment and the robot, respectively. As illustrated in Fig. 1, the acoustic measurements from the microphone arrays and the motion measurements from the robot are utilized in the optimization process, with landmark-robot constraints and robot relative pose constraints enforced, similar to the approach used in full information estimation and batch SLAM [26]-[31]. Then, two important questions arise.
Firstly, it is critical to assess whether the information contained in the measurements is sufficient to estimate the unknown parameters of microphone arrays and sound source locations. This is the so-called observability problem in the SLAM literature [32]-[33]. Although there exist works on observability analysis of SLAM-based calibration of single microphone arrays, in-depth analysis for the case with multiple microphone arrays is lacking.
Secondly, the selection of initial values is crucial because the considered calibration is a nonlinear least squares (NLS) problem, similar to batch SLAM [26], [34]. Many existing algorithms for solving such NLS problems employ the Gauss-Newton method or its variants. These methods typically require reasonable initial guesses; otherwise, the algorithms may converge toward local minima, or in extreme cases, diverge. For some specific problems, novel algorithms with certifiable convergence properties have been proposed in [35]-[37].
I-C Contributions
Motivated by the above observations, in this paper, we adopt batch SLAM as a general framework for the simultaneous calibration of translations, orientations, time offsets and sampling clock differences of multiple microphone arrays, and sound source positions in 3D. Our contributions are two-fold.
Firstly, we concentrate on the parameter identifiability of the corresponding SLAM problem. As discussed in existing works [32]-[33], SLAM is not observable from a control theoretical perspective. Hence, in the SLAM literature, the observability problem of SLAM has been tackled from an information-theoretic perspective, where all the parameters to be identified are taken to be constant but unknown. From the information-theoretic perspective, Fisher information quantifies the amount of information contained in a set of observations about a set of unknown parameters [33]. Following the above line of argument, when the multiple microphone array calibration problem is formulated as an NLS parameter estimation problem, the full rankness of the associated FIM determines the parameter identifiability or observability of the calibration problem.
Hence, in this paper, by leveraging the FIM approach, we thoroughly investigate the identifiability of the unknown parameters, including translations, orientations, and asynchronous factors between the microphone arrays and the sound source positions. We establish necessary/sufficient conditions under which the FIM and the Jacobian matrix have full column rank, which implies the identifiability of the unknown parameters. Furthermore, we identify several scenarios where the unknown parameters are not uniquely identifiable.
Secondly, we propose an effective framework to initialize the unknown parameters from the measurements, which is used as the initial guess in batch SLAM. Specifically, the initialization procedure is composed of the following major steps: (i) estimation of the sound source position by triangulation; (ii) estimation of distance between the sound source and microphone arrays using 3D geometry; (iii) estimation of microphone array poses using the iterative closest point (ICP) method; (iv) estimation of the asynchronous factors using linear least squares (LLS). As to be explained later in the paper (see Section IV. A), the microphone array pose estimation problem addressed in step (iii) mentioned above is conceptually a point-to-point registration problem, and hence can be tackled effectively using ICP [38]. To validate the effectiveness and robustness of the proposed initialization framework, we have conducted extensive numerical simulations and real experiments. Overall, the proposed pipeline achieves higher accuracy with fast convergence, in comparison to methods that use the noise-corrupted ground truth of the unknown parameters as the initial guess in the optimization, and other state-of-the-art methods in the literature [20], [23].
Compared to existing frameworks, the proposed calibration method requires less prior information. More specifically, the knowledge of the source’s position required in [22]-[23], or the distance between the signal source and the microphones needed in [14] is not required in the proposed framework in this paper. It should also be noted that our previous works documented in [16]-[18] primarily focused on calibrating individual microphones within a single array while in this paper we address the more challenging problem of calibrating multiple microphone arrays.
Finally, we remark that the observability analysis reported in Section III has been previously reported in our conference paper [39]. However, the results of [39] are only applicable for the case where the time interval between consecutive sound source events is fixed. In the current paper, we generalize the results in [39] from the scenario of fixed-interval sound source emissions to arbitrary time intervals (i.e., the interval between every two consecutive sound events can be asynchronous and time-varying). More importantly, we have proposed an effective framework for estimating the initial values of the parameters and conducted extensive simulation studies and real experiments to validate the entire calibration pipeline. All the codes and multimodal dataset used in this paper are publicly available at https://github.com/AISLAB-sustech/Calibration_of_Multi_Mic_Arrays.
Notation: Denote , , and as scalars, vectors, and matrices, respectively. represents the transpose of matrix . stands for the identity matrix of dimensions. denotes the -dimensional Euclidean space. denotes , where are scalars/vectors/matrices with proper dimensions. denotes a block diagonal matrix with as block diagonal entries for times; denotes a block diagonal matrix with and as its block diagonal entries; and as a matrix of appropriate dimensions with its all entries as 0. means that is a positive definite matrix. We denote . Vectors/matrices, with dimensions not explicitly stated, are assumed to be algebraically compatible.
![Refer to caption](x1.png)
II PROBLEM FORMULATION
In a calibration scene containing microphone arrays, as shown in Fig. 1 (with as an example), the arrays capture consecutive acoustic signals emitted by a single sound source at several spatial positions. represents the position of the microphone array in the global reference frame and any two arrays are in different positions. We assume that there is a local reference frame attached to every microphone array; we choose as the global reference frame; is the rotation matrix of reference frame to the frame with the ZYX Euler angles vector ; is the sound source position at time instance , with respect to (w.r.t.) , where is the total number of time steps. In the calibration process, the arrays remain static while the sound source moves around.
Here we consider the most general scenario with initial time offset and sampling clock difference among microphone arrays (we assume that the configuration of each microphone array itself, including its geometry, is known). When the sound source sends the acoustic signal, the DOA information, i.e., the direction vector of the sound source in the microphone array frame is obtained as:
(1) |
Note that the Euclidean norm of is 1, i.e., is a unit vector. Denote , for , as the distance between the microphone array and the sound source at the time instant. The inter-array TDOA information between the and the first microphone arrays can be expressed as follows:
(2) |
for , where represents the sound speed in the air; the scalar (unknown) constant variables and represent the initial time offset and the sampling clock difference per second of each microphone array, respectively; is the time interval from the beginning to the sound signal. Since the first microphone array is used as the reference, then
(3) |
The positions and orientation of the microphone array (where ), i.e., and , are:
(4) |
respectively, where , and take values in the range of , and respectively. Denote the unknown parameters w.r.t. the microphone array as:
(5) |
All the unknown parameters w.r.t. microphone arrays are:
(6) |
Denote the sound source position at time as:
(7) |
Thus, all unknown parameters to be identified are:
(8) |
We denote the ideal inter-array TDOA and DOA measurements at the time instance as:
(9) |
The measurements of DOA and inter-array TDOA at time are subject to Gaussian noises as follows:
(10) |
where is defined in (9), , with , where is a positive scalar, and . Assume that the sound source relative position between two consecutive time steps can be measured with Gaussian noise, i.e.,
(11) |
where , , with . We combine the relative position measurements, the TDOA, and DOA measurements as:
(12) |
The models in (10)-(11) can be rewritten compactly as:
(13) |
where is the combined observation model, and is the noise of combined observations with
(14) |
As shown in Fig. 1, the batch SLAM framework is a feasible solution to the above problem by treating the moving sound source as a robot and the multiple microphone arrays as landmarks [26]. As in [16]-[17], the problem of joint calibration of multiple asynchronous microphone arrays and sound source localization can be treated as the following NLS using batch SLAM:
(15) |
The measurements obtained by microphone arrays and robots constitute the spatial constraints and can be included in (15) to improve estimation accuracy.
Given the problem formulation described above, our main objective is (1) to determine the identifiability of the unknown parameters (microphone arrays positions, orientations, time offsets, sampling clock differences, and sound source positions) based on the available measurements (DOAs, inter-array TDOAs, and relative position measurements), and (2) to develop an efficient algorithm pipeline for solving the corresponding NLS in (15).
III OBSERVABILITY ANALYSIS
In this section, by utilizing the FIM method, the observability analysis of the batch SLAM framework for the above calibration problem is performed. More specifically, we have established necessary/sufficient conditions under which the FIM and Jacobian matrix have full column rank (which implies the identifiability of the unknown parameters, including the microphone array positions, orientations, time offsets, sampling clock differences, and sound source positions). In addition, we also discover some scenarios where the FIM and Jacobian matrix cannot have full column rank (in this case, the unknown parameters could not be uniquely identified).
III-A The Fisher Information Matrix and the Jacobian
The covariance matrix of the estimation error corresponding to the estimated values and the true values of unknown parameters in the observation model in (13) can be calculated by
(16) |
For nonrandom vector parameter estimation, the FIM of an unbiased estimator is defined as:
(17) |
where is the gradient operator w.r.t. the vector , is the probability distribution function, and the derivatives are calculated at the true value [40, chap. 2]. It can be shown that the covariance matrix of any unbiased estimator satisfies
(18) |
i.e., when the is singular, the Cramr-Rao lower bound will not exist [40, pp. 165], one or more parameters will be unobservable. As in [32], the Fisher information matrix for the models described in (17) can be formulated as:
(19) |
where is the Jacobian of the observation model in (13), and its explicit expressions will be given in (22). When one has that
(20) |
Since the first microphone array is viewed as the reference array, its corresponding parameters are all set to zero. The remaining state vectors contain only microphone arrays parameters and the sound source position at all time steps. From the definition of the Jacobian matrix [41, pp. 569], we know that , where
From (17)-(20), a necessary and sufficient condition for to be nonsingular is that has full column rank. For to be of full column rank, it is necessary that
(21) |
where stands for the ceiling operation generating the least integer not less than the number within the operator. We then have the following results.
Proposition 1: The Jacobian can be written as
(22) |
where , with being the inter-array TDOA and DOA observation model at the time instant, (expression of can be found in (10); the detailed expressions of and can be found in (47) and (51) in Appendix A, respectively).
Proof. See Appendix A.
Given the equivalence of full rankness between the FIM and the Jacobian, in the following, we will focus on investigating conditions under which the Jacobian derived in (22) can or can not be of full column rank.
III-B Main Results of Observability
We firstly have the following results regarding the equivalence of full column rank between the Jacobian (22) and matrix in (23) which has a much simpler structure.
Theorem 1
The Jacobian matrix is of full column rank if and only if the following matrix
(23) |
is of full column rank.
Proof. The proof is similar to that of [18, Theorem 1] and is skipped here.
We next present a necessary condition (Theorem 2) and a sufficient condition (Theorem 3) under which matrix in (23) is of full column rank.
Theorem 2
The Jacobian matrix is of full column rank only if matrices and , for are of full column rank, respectively111As shown in the full proof in Appendix A, submatrices and are obtained from the matrices after applying elementary transformations to and (both defined in (23)), respectively., where
(24) |
Proof. See Appendix A.
Theorem 3
The Jacobian matrix is of full column rank if the following statements hold concurrently:
(i) Any matrix resulting from the horizontal concatenation of and is of full column rank, .
(ii) All matrices , and are of full column rank.
Proof. See Appendix A.
III-C Special Cases When Observability is Impossible
It can be seen from Proposition 1 and Theorems 1-3 that observability of the considered identification question is determined both by the configuration of microphone arrays (i.e., the relative transforms, namely, orientation and translation) and the sound source positions. This raises the question of under what conditions on the microphone array configuration and the sound source trajectory, the necessary conditions in Theorem 2 cannot hold. In this section, we will focus on this question and discover some special cases where observability is impossible. Our major result is stated in Theorems 4-5.
Theorem 4
The matrix is not of full column rank if one or more of the following conditions hold.
(i) For all microphone arrays, there exists fewer than five time steps information (i.e., the value of in (23) is less than 5).
(ii) The sound source positions at all moments are collinear with the origin of the global frame , i.e., always holds, where , and is an arbitrary non-zero scalar ( might take different values at different time steps).
(iii) The sound source lies on any Euclidean plane of , , and within the three-dimensional -- Cartesian coordinate frame , at all moments, where are arbitrary scalars.
Proof. See Appendix A.
Theorem 5
The matrices , , are not of full column rank if one or more of the following conditions hold:
(i) The sound source positions at all moments are collinear with the origin of the frame , i.e., always holds, where and is an arbitrary non-zero scalar ( might take different values at different time steps).
(ii) For the microphone array, one of the Euler angles satisfies .
Proof. See Appendix A.
III-D Discussions
The observability analysis presented in the above subsections refers to conditions concerning the ground truth value of the sound source trajectories or the configurations of microphone arrays. Hence, the observability analysis is of theoretical interest as it can serve as guidelines when designing microphone array configurations or the sound source trajectories during the calibration process. One can also rely on the results of Section III.C to avoid the unobservable scenarios from a theoretical point of view.
However, during real calibration processes, the measurements contain noises (i.e., the ground truth is not known a prior). Hence, the observability analysis results obtained above are not directly applicable. It is crucial to develop a reliable algorithmic pipeline that can achieve satisfactory convergence and accuracy. This will be discussed in the next section. One should note that the algorithmic pipeline presented in the sequel can also be applied to the nonobservable cases (but the calibration results will be unreliable). This is because, for these scenarios, the noisy measurements do not contain enough information to estimate the unknown parameters. This is also why the analysis in Sections III.A to III.C is valuable, as it suggests avoiding such unobservable situations when designing the microphone array configurations or the sound source trajectories.
Based on the above arguments, to validate the theoretical analysis, we will discuss both observable and unobservable situations in the numerical simulations in Section V. In the experimental results of Section VI, we will only design experiments that correspond to observable cases.
IV BATCH SLAM BASED CALIBRATION
In this section, we present our proposed pipeline for batch SLAM based joint calibration of multiple microphone arrays and sound source localization. As illustrated in Fig. 1, we treat the microphone arrays as landmarks and the sound source as a mobile robot in the corresponding batch SLAM problem and utilize Gauss–Newton iterations to solve the corresponding NLS problem. More specifically, we propose an effective framework to initialize the unknown parameters which are used as the initial guess in the Gauss–Newton iterative algorithm.
IV-A The Proposed Initialization Procedure
For notational simplicity, in the sequel, we use and to denote the Gaussian noise corrupted DOA and inter-array TDOA measurements, respectively. We use to represent the estimates of the unknown scalar/vector/matrix parameters. Our proposed initialization procedure is composed of the following main steps: (i) estimation of the sound source position by triangulation; (ii) estimation of the distance between the sound source and microphone arrays using 3D geometry; (iii) estimation of microphone array poses using ICP; (iv) estimation of the asynchronous factors using LLS.
![Refer to caption](x2.png)
![Refer to caption](x3.png)
![Refer to caption](x4.png)
![Refer to caption](x5.png)
(i) Estimation of the sound source position by triangulation: Without loss of generality, the initial trajectory of the moving sound source in the global frame is illustrated in Fig. 2(a). Then, from geometry, the initial position of the sound source can be obtained by triangulation and using the first two consecutive DOA measurements as follows:
(25) |
where , i.e., is the measured distance that the source moves between the first two consecutive moments, is the angle of two vectors, and is the distance between the first sound source position and the origin. Note that can be obtained from the relative position and DOA measurements, respectively. Once the initial position of the sound source is obtained as above, the sound source positions at different time steps can be estimated:
(26) |
(ii) Estimation of the distance between the sound source and microphone arrays using 3D geometry: We calculate the distance between each source node and microphone arrays to provide constraints for estimating microphone array poses. One can construct an over-constrained NLS for estimating the distance between each source node and microphone arrays by using the law of cosines constraints. To illustrate, as shown in Fig. 2(b), each microphone array and any four source positions , , , and at the corresponding time instances form a polyhedron (when the four nodes are coplanar, it is tetrahedral, and when the four nodes are on different planes, it forms a five-vertex hexahedral structure). We construct an NLS problem by enforcing the law of cosines for each face of the polyhedron (including the two inner faces). For the scenario shown in Fig. 2(b), denote the estimated squared distance between any two sound source nodes , among the four source positions , , , and as:
where and are the unit direction vectors of the corresponding sides with length and , respectively. Denote the difference between and as:
where and . Consider a system of six nonlinear equations, given by . We use to collectively denote the distances between the four sound source positions and the microphone array, which can be estimated by solving
(27) |
Note that the nonlinear optimization problem in (27) features a polynomial cost function with a fixed number of unknown parameter dimensions, namely, four edge lengths. However, the batch optimization problem in (15) has a more intricate objective function, incorporating polynomials, exponentials, and trigonometric functions, with optimization variables, where and represents the numbers of microphone arrays and time steps, respectively. Hence, in general, the optimization problem in (27) will be much easier to solve (it can be conveniently solved, for instance, using the trust region reflective method [42]) than the entire batch optimization problem in (15). To improve the estimation accuracy of at all time instances , we form combinations by selecting any four sound source positions from all time instances, where the microphone array-to- source line exists within multiple polyhedra. This implies that we can leverage multiple estimation results to achieve greater accuracy. By solving for the edge lengths of each polyhedron and employing the well-known interquartile range (IQR) method [43, pp. 236], we calculate the average value of these same edges in different polyhedra. This average serves as the estimated distance between the microphone array and the sound source position at the time step.
(iii) Estimation of microphone arrays positions and orientations using ICP: Note that the positions of the sound source in the frame can be estimated as:
(28) |
We treat the sound source positions as features in each coordinate frame. To find the transformation that optimally aligns the sound source positions with the reference frame is akin to representing the same features in the reference frame. To tackle this challenge, we formulate an NLS problem to minimize the mapping error of sound source positions between and :
(29) |
which is conceptually a point-to-point registration problem that can be tackled effectively using ICP [38]. Hence, as in [38], let and be the geometric mean of the source position in and , and they can be computed based on the estimated sound source positions and , respectively. The covariance of the sound source trajectory expressed in the two different frames becomes:
(30) |
We perform singular value decomposition on this covariance matrix:
(31) |
The optimal rotation matrix [38, 44] can be obtained as:
(32) |
Then, we can transform the rotation matrix into the corresponding ZYX Euler angles [45]. Thus, the initial guess of microphone array positions can be expressed as:
(33) |
(iv) Estimation of microphone arrays asynchronous parameters using LLS: In part (ii), the distances between the sound source and microphone arrays at different time steps have been estimated. By using the inter-array TDOA measurements, the initial guess of the microphone array asynchronous factors can be obtained by solving the following LLS problem:
(34) |
To identify outliers and improve the estimation accuracy of the inter-array asynchronous factors, we first solve the optimization problem (34). Then, we calculate the residuals by determining the differences between the value and the corresponding fitted value at each time step, and their average and standard deviation. Subsequently, we perform normalization to the residuals, i.e., dividing each residual by the standard deviation to identify and exclude the outliers. Using the data with the outliers removed as described above, we solve the optimization problem (34) again, and the final estimates of the asynchronous factors are obtained.
IV-B The Batch Optimization Procedure
As described in (15), we construct a standard NLS problem by considering the microphone arrays as landmarks and the sound source locations as robot positions. For the Gauss-Newton iterations, the increment of each iteration can be obtained by solving:
where is the approximation matrix of the Hessian matrix and is the coefficient vector [27]:
(35) |
where and are the two nodes in the graph (formed by the sound source at different positions and microphone arrays), is the full set of measurements, and is the Jacobian matrix of the error function of the corresponding nodes. For the position-position constraint, denote the error between the expected measurement and real measurement collected by the robot as:
(36) |
The Jacobian matrix w.r.t. position and position are:
(37) |
For the position-landmark constraint, denote the error between the expected measurement and the real measurement collected by microphone arrays as:
(38) |
The Jacobian matrices corresponding to landmark and position are:
(39) |
The structure of the Jacobian matrix is elaborated in Eq. (47)-Eq. (51). For corresponding nodes and , the Jacobian matrix can be succinctly represented as:
(40) |
By omitting the zero blocks, the corresponding sparse block matrix and the vector (see Eq. (35)) can be expressed as:
(41) |
(42) |
respectively. Combining the initial guess selection pipeline and the Gauss-Newton iteration procedure, we then have the entire calibration algorithm as shown in Algorithm 1.
![Refer to caption](x6.png)
![Refer to caption](x7.png)
![Refer to caption](x8.png)
![Refer to caption](x9.png)
![Refer to caption](x10.png)
![Refer to caption](x11.png)
V NUMERICAL SIMULATIONS AND RESULTS
We next present extensive numerical simulations to validate the results in Sections III and IV. Firstly, we verify the observability analysis results, along with intuitive physical interpretations of unobservable scenarios. Secondly, we compare our proposed initialization method (which does not require ground truth) with initialization schemes of adding different levels of noise to the ground truth (GT) and random initialization. Thirdly, we verify the robustness of the calibration algorithm by varying the sound source trajectories.
V-A Observable Cases
We firstly present two observable scenarios as shown in Fig. 3(a). Each scenario comprises eight stationary microphone arrays and a moving sound source. In case 1, the source follows a randomly generated 3D trajectory, while in case 2, it moves along a path on a plane that does not coincide with the global reference frame. In both scenarios, the moving sound source emits signals at ten consecutive locations, which are recorded by the microphone arrays.
The rank of the matrix in (23) changes over time steps, as illustrated in Fig. 3(b). Based on Theorem 3, since (note that is defined in (59)) and , , it is evident that with an increasing time step and the source’s movement along these two trajectories, the matrix (with dimensions ) gradually become full column rank, i.e. its Jacobian matrix (with dimensions ) in (22) is full column rank. This implies that the calibration scenarios are observable. At the time step when the Jacobian matrix becomes full column rank, it also can be verified that , , and so that Theorem 2 holds. Hence, the simulations presented so far based on the theoretical analysis worked as expected. It is worth noting that the sound source positions are not always in the same line with any array frame or on the same plane with the reference array frame. Hence, a sound source trajectory with more motion varieties often can help to ensure that the necessary conditions stated in Theorem 2 are met, thereby potentially avoiding the unobservable scenarios.
V-B Unobservable Cases
Several unobservable scenarios are presented in the following to verify the conclusions in Theorems 4-5.
(i) For the Jacobian matrix to have full column rank, it is necessary that the time steps are greater than or equal to 3 so that the number of rows of the Jacobian matrix is greater than the number of columns, according to (21). In addition, as can be seen from Fig. 3(b), when the number of time steps is greater than or equal to 3 but less than 5, the Jacobian matrix is not of full column rank. This reflects that the system is unobservable when the number of time steps is less than 5.
(ii) For the sound source trajectories shown in Fig. 3(c), the first case is that the sound source stays co-linear with the origin of the global frame during the entire process, and in Theorem 4 (ii) take on the values of , and respectively. The second case is that the sound source remains co-planar with global frame . For this scenario, the sound source positions all lie on the Euclidean plane defined by the equation within the three-dimensional Cartesian coordinate frame . From Fig. 3(d), we can see that both cases are unobservable due to the rank deficiency of matrix .
(iii) For the sound source trajectories shown in Fig. 3(e), the first case is that the sound source keeps co-linear with the origin of during the movement, and in Theorem 4 (iii) take on the values of , and respectively. In the second case, the Euler angles and of and are , and the sound source travels along the route of the observable scenario mentioned in case 1 of Fig. 3(f). The rotation angle is at the singular point of observation, rendering the system unobservable. Hence, the simulations presented above validate the conclusions in Theorems 4-5.
Parameters | Values |
Inter-array TDOA noise STD | 0.067ms |
Elevation angle (DOA) noise STD | 5 degrees |
Azimuth angle (DOA) noise STD | 5 degrees |
Relative position noise STD | |
Max. time offset | 0.1s |
Max. clock difference | 0.1ms |
Sound speed in air | 346m/s |
Max. iterations | 50 |
Threshold | 1e-5 |
V-C Calibration under Different Initialization Schemes
To validate the initialization pipeline, we employed a predefined trajectory for the sound source, as illustrated in Fig. 4. We utilized our proposed pipeline to initialize the unknown parameters. For comparison, we added varying levels of noises to the true values of the unknown parameters for the same trajectory. These noisy values were then used as initial guesses for the Gauss-Newton iterations.
In detail, we set the base noise standard deviation for the microphone array positions, orientations, asynchronous parameters, and source positions to be , (10 degrees), , , and , respectively. We selected six sets of initial values, i.e., the ground truth (GT), Random, and Lv1, Lv2, Lv3, Lv4 where Gaussian noises with a standard deviation of 1, 3, 6, 9 times of the base noise are added to the GT. For these different initialization schemes, we conducted 200 Monte Carlo simulations with randomly selected initial values and used the root mean square error (RMSE) to measure the accuracy of the estimated values (the specific formulas are provided in Appendix B).
Furthermore, we also investigated the impact of different initialization schemes on the convergence ratio of the Gauss-Newton algorithm. For each initialization scheme, we define the convergence ratio as the proportion of successful convergence instances to the total number of experiments. During the optimization process, we assessed the convergence of the Gauss-Newton algorithm based on the square norm of the optimization step size, i.e., . We classify any of the following three scenarios as divergent: (1) exceeds 1e8 in any iteration; (2) exhibits oscillations above 1e3 and does not come down below 1e3; (3) keeps growing as the iteration step increases. Otherwise, the Gauss-Newton algorithm is deemed convergent.
Note that, during the numerical experiments, we keep the multiple microphone arrays stationary while the sound source is in motion. The parameters used in the numerical experiments are summarized in Table I (note that in practice, the DOA information can be conveniently indicated by elevation and azimuth angles in 3D. Hence, we will use the latter two angles to represent DOA in the remainder of the paper). Specifically, in our Monte Carlo simulations, the true values of all unknown parameters remain fixed, and the initial values for the Gauss-Newton iterations of each simulation are obtained as described above. Additionally, each simulation utilizes measurements with the same noise level. In other words, noises are added to the theoretical measurement values with standard deviations (STD), as shown in Table I, resulting in the final measurement values used in simulations for inter-array TDOA, DOA, and sound source relative positions.
Noise Levels | Microphone Array | SRC | Convg. Ratio | |||
Pos. (m) | Orie. (deg.) | Offset (ms) | Clock (us) | Pos. (m) | ||
GT | 2.796e-02 | 1.173 | 1.078e-01 | 7.579 | 4.228e-02 | 100% |
Ours | 2.797e-02 | 2.348 | 1.078e-01 | 7.584 | 4.229e-02 | 100% |
Lv1 | 2.973e-02 | 6.299 | 0.992e-01 | 7.865 | 4.475e-02 | 100% |
Lv2 | 3.143e-02 | 19.790 | 1.348e-01 | 8.730 | 4.611e-02 | 99.0% |
Lv3 | 6.026e-02 | 42.860 | 3.010e-01 | 33.196 | 1.011e-01 | 78.5% |
Lv4 | 7.861e-01 | 64.250 | 3.239 | 68.573 | 2.754e-01 | 44.0% |
Random | 6.928e-01 | 67.636 | 2.416 | 82.810 | 2.635e-01 | 43.0% |
The results are presented in Table II. It is evident that with an increase in the noise level of initialized values for the unknown parameters, the final estimation errors gradually increase (except for the time offset, where Lv1 has a negligible advantage over GT), and the convergence ratio decreases. Furthermore, it can be observed that without relying on the GT for initial guess selection, the performance of our calibration algorithm is comparable to the case using the GT as the initial value. In terms of estimating the microphone array orientation, our method is slightly less accurate compared to using the GT as the initial guess. This demonstrates the effectiveness of our proposed framework. In contrast, the random initialization method, frequently used in many optimization problems, exhibits inferior performance. Although it outperforms Lv4 in terms of the accuracy of some parameters, it has the lowest convergence ratio, indicating the unreliability of a random strategy. The above comparisons highlight the necessity of an appropriate initialization algorithm in the calibration process and the effectiveness of our proposed pipeline.
V-D Calibration Using Random Trajectories
To verify the robustness of the proposed calibration framework, we generate ten random trajectories, each involving five microphone arrays and 80 sound-emitting events. Take a trajectory shown in Fig. 5 as an example (only the first 40 sound-emitting events for illustration purposes). Even with measurement noise interference, the parameter initialization procedure can obtain initial values that are close to the ground truth. The initialized values are used in Gauss-Newton iterations to improve calibration accuracy.
Fig. 6 shows the error distribution between the initialized values obtained by our proposed initialization method and the ground truth for ten different trajectories. In the box plot, the blue circle represents the outliers obtained from the interquartile range, while the upper and lower black horizontal lines represent the maximum and minimum values of the non-outlier errors. The upper and lower edges of each box represent the upper and lower quartiles, respectively, and the middle blue line corresponds to the median of the errors. The orange triangle represents the mean of the errors. The errors between our initial values and the ground truth are small, which promotes the convergence of the calibration algorithm.
Fig. 7 shows the error distribution between the final estimated values and the ground truth for ten different trajectories. Similar to Table II, the results indicate that while the accuracy of the microphone array orientation estimation is slightly lower than that of other parameters due to the larger DOA measurement noise, the calibration of all parameters is accurate.
Finally, we remark that the relatively poorer accuracy for microphone array orientation shown in Table II and Fig. 7 is mainly attributed to the large magnitude of DOA measurement noise used in the simulation. As indicated in Table I, for our simulations, the elevation and azimuth angle noise STD are both 5°. If we reduce the elevation and azimuth angle noise STD, the accuracy of microphone array orientation will be improved. However, due to limited space, we skip these comparisons and results here.
![Refer to caption](x12.png)
![Refer to caption](x13.png)
![Refer to caption](x14.png)
![Refer to caption](x15.png)
![Refer to caption](x16.png)
![Refer to caption](x17.png)
![Refer to caption](x18.png)
![Refer to caption](x19.png)
![Refer to caption](x20.png)
![Refer to caption](x21.png)
![Refer to caption](x22.png)
![Refer to caption](x23.png)
![Refer to caption](x24.png)
![Refer to caption](x25.png)
![Refer to caption](x26.png)
![Refer to caption](x27.png)
![Refer to caption](x28.png)
![Refer to caption](x29.png)
![Refer to caption](x30.png)
![Refer to caption](x31.png)
![Refer to caption](x32.png)
![Refer to caption](x33.png)
![Refer to caption](x34.png)
VI REAL-WORLD EXPERIMENTS
In this section, we validate our calibration method using real data. In our experiment, a Turtlebot3 mobile robot moves in an indoor environment, and multiple microphone arrays capture the sound signal emitted by the robot. More specifically, we use the iFLYTEK M160C microphone array consisting of six independent microphones arranged in a circular and evenly distributed configuration, with a diameter of 70.85mm, a sampling depth of 32 bits, a sampling rate of 16 KHz, and an effective pickup range of 3.5 meters, as shown in Fig. 8(a). The mobile robot is equipped with an Intel D435i camera with an integrated inertial measurement unit (IMU), as shown in Fig. 8(b). It also includes a four-channel trajectory detector for tracking predefined paths and a 3W 8 speaker for sound emission.
In the experimental setup, four microphone arrays are placed in an open area of an academic building. The experimental area is 15.5 meters long, 10 meters wide, and 3.3 meters high, as shown in Fig. 8(c). The microphone arrays remain stationary and receive audio signals while the mobile robot travels along the black trajectory on the ground. When the robot detects the cross-shaped sound markers on the ground, it immediately emits a chirp signal with a frequency of 1000 Hz to 2000 Hz through a speaker driven by a Class-D amplifier, lasting for 300 ms, and then moves on the trajectory. We carry out the following activities to validate the effectiveness and performance of the proposed calibration pipeline across diverse scenarios:
1) Firstly, we compare the calibration results achieved through various initialization strategies (see Section VI.B).
2) Next, we explore the influence of the absence of sound source relative position measurements on the estimation accuracy in the optimization process (see Section VI.C).
3) Moreover, we vary the spacing between the microphone arrays to cover a range of scenarios and scene scales to assess the calibration performance of our method (see Section VI.D).
4) Last but not least, in Section VI.E, we compare the performance of the proposed initialization method (IM) and its fine-tuning (FT) version (i.e., the results are obtained by feeding the initialized values to batch optimization with Gauss-Newton iterations) with those of other existing methods, including the open-source passive geometry calibration method for microphone arrays based on the differential evolution algorithm (PGM) [20] and the two-step calibration method (TSM) based on the L-BFGS algorithm [23].
VI-A Data Collection and Ground Truth
The trajectory of the robot is pre-defined to obtain the ground truth of sound source positions in the global frame. The position of the speaker during audio playback, corresponding to the sound marker’s coordinates and the mobile robot’s height, is regarded as the ground truth for the sound source positions. The microphone arrays were placed w.r.t. each other according to known preset values (i.e., these are taken to be the true values of microphone array positions) before the experiment started. The frame attached to the first microphone array is taken to be the global coordinate system. As shown in Fig. 8(a), we affix each microphone array to a pan-tilt head. Subsequently, the pan-tilt head attached to each microphone array (except the first one) is rotated by certain known pre-set angles which are used to calculate the ground truth value of the Euler angles of the microphone arrays.
We determine the GT values of time offset and sampling clock difference as follows. To compute the ground truth for time offset, we subtract the theoretical time difference (excluding time offset and sampling clock difference) between the first sound marker received by each microphone array and the reference microphone array from the actual time difference. Note that due to the robot’s quick arrival at the first sound marker, the clock difference is so small at this moment that it can be considered negligible. For clock difference, we have recorded 8 hours of audio using multiple microphone arrays placed at the same distance relative to the sound source. This recording includes start and end signals. We calculate the ground truth for sampling clock difference by comparing the number of samples recorded by each microphone array with that of the reference microphone array during this period.
The following three kinds of measurements are obtained during the experiment:
1) For inter-array TDOA measurements between any microphone array and the reference microphone array at the k-th sound marker, we employ a sliding window technique to break down the sound signal into short frames. Subsequently, we compute the power spectrum of each frame to determine the valid sound region. Each frame has a duration of 25 ms, and a Hamming window is applied to prevent spectral leakage. For the valid sound region, we apply the GCC-PHAT algorithm [46], widely used in robotic sound localization, to compute the inter-channel time differences for all combinations of 6 channels 6 channels. The average time difference is calculated as the inter-array TDOA.
2) For DOA measurements of the microphone array, we employ the Steered Response Power-Phase Transform (SRP-PHAT) algorithm [47] on the obtained signal region, with a discrete search angle resolution of 3 degrees. This technique leverages the spatial filtering capability of the microphone array to estimate the received power from a set of candidate directions. The source is then identified by selecting the location associated with the highest energy. The estimated azimuth and elevation angles are subsequently transformed into three-dimensional unit direction vectors.
3) For the sound source relative position measurements, we utilize a visual-inertial odometry (VIO) method [48] that integrates camera and IMU data. This approach fuses visual information and inertial data, providing more accurate and robust displacement measurements. This allows us to integrate more measurements related to robot motions, thereby enhancing the accuracy and reliability of the sound source relative position measurements.
Noise Levels | Microphone Array | SRC | Convg. Ratio | |||
Pos. (m) | Orie. (deg.) | Offset (ms) | Clock (us) | Pos. (m) | ||
GT | 0.233 | 7.936 | 1.514 | 12.712 | 0.156 | 100.0% |
Ours | 0.233 | 9.650 | 1.515 | 12.749 | 0.156 | 100.0% |
Lv1 | 0.233 | 8.291 | 1.521 | 12.713 | 0.156 | 99.87% |
Lv2 | 0.561 | 10.511 | 2.915 | 12.898 | 0.179 | 39.30% |
Lv3 | 1.068 | 34.799 | 4.886 | 13.819 | 0.419 | 3.53% |
Random | 0.839 | 78.303 | 20.730 | 56.709 | 0.775 | 0.10% |
VI-B Comparisons between Different Initialization Methods
For the case when four microphone arrays are placed on the corners of a square (2m 2m), we collect data for five different trajectories, each repeated three times, resulting in a total of 15 datasets. These collected datasets have been used to explore the impact of different initial values in real-world experiments.
Based on the GT and measurement models in Section II (see (10)-(11)), we then calculate the following measurement errors: the inter-array TDOA measurement error has mean value 3.15e-4 seconds with STD of 1.25e-3; the azimuth angle error has mean value of 6.02 degrees with STD of 4.69 degrees; the elevation angle error has mean value of 5.45 degrees with STD of 5.97 degrees, and the VIO measurement error has mean value [2.06e-2, 2.49e-2, 6.13e-3] meters with STD of [9.64e-3, 3.65e-2, 8.44e-3] meters. These errors were obtained by comparing the measured values from the sound signal with the theoretical values.
We set the base noise standard deviation for the microphone array positions, orientations, asynchronous parameters, and source positions to be , (10 degrees), , , and , respectively. Subsequently, for comparison, we obtain initial values with different levels of errors (GT, Lv1, Lv2, Lv3, and Random), similar to those described in Section V.C. Using each of the 15 datasets, for these different initialization schemes, we conducted 200 Monte Carlo experiments with randomly selected initial values (note that for our proposed initialization method and the initialization using GT, there is only one experiment) and the corresponding real measurements. The results using the above different initialized values are shown in Table III and the calibration results of our method for one of the datasets are depicted in Fig. 9.
From Table III, it can be observed that for real data, the overall calibration accuracy is slightly lower compared to that of simulation studies, primarily due to noise sources such as motion noise from the mobile robot, sensor measurement noise, and manual interference. However, the effectiveness of our proposed method is evident. In real-world settings, our initialization method produces calibration results almost identical to those obtained using ground truth values directly or Lv1 as initial values, with only slightly reduced orientation accuracy. This is because, unlike simulations, the measurement noises in real-world settings are, in general, not Gaussian and the accuracy of DOA and TDOA measurements is lower. Notably, the noises in DOA measurements (w.r.t. the ground truth values) almost overshadow the performance differences between the initialization methods GT, Lv1, and our initial values. Despite that, the results indicate the effectiveness of our method in scenarios with non-Gaussian and large measurement noises.
Regarding convergence ratio, both our initialization method and direct use of ground truth values achieve 100% convergence. As the initial noise level increases, the convergence ratio of initialization using noise-corrupted GT values significantly decreases, especially in cases with higher levels of noise, such as the Lv3 and random initialization methods, where all Monte Carlo experiments across all 15 datasets almost always diverge. This underscores the effectiveness and robustness of our proposed method in real-world scenarios.
Noise Levels | Microphone Array | SRC | Convg. Ratio | |||
Pos. (m) | Orie. (deg.) | Offset (ms) | Clock (us) | Pos. (m) | ||
GT | 0.425 | 11.580 | 2.015 | 12.064 | 0.226 | 100.0% |
Lv1 | 0.426 | 12.819 | 2.012 | 12.062 | 0.226 | 99.90% |
Lv2 | 0.658 | 14.270 | 3.100 | 12.185 | 0.231 | 32.07% |
Lv3 | - | - | - | - | - | 0.0% |
Random | - | - | - | - | - | 0.0% |
VI-C Calibration with Only Acoustic Measurements
To validate the influence of the sound source relative position measurements obtained from the VIO method on the calibration results, this section focuses on conducting calibration experiments using only acoustic measurements obtained from the microphone arrays (inter-array TDOA and DOA measurements). Given that our initialization method relies on relative position measurements of the sound source, we cannot use it for comparison purposes. Hence, we perform Gauss-Newton optimizations initialized by ground truth values corrupted by Gaussian noise across varying levels. Following Section VI.B, Monte Carlo experiments are carried out under varying initialization noise levels using the real measurements from the 15 datasets (excluding the sound source relative position measurements from VIO).
Comparing the results in Table III and Table IV (including and excluding the sound source relative position measurements, respectively), it is evident that without the sound source relative position measurements, the overall parameter estimation results are poorer. In particular, the estimation accuracy of the relative transforms (i.e., orientation, translation) between microphone arrays and sound source positions, and the convergence ratio are lower than the case with sound source relative position measurements. However, as it can also be seen from Table III and Table IV, the absence of sound source relative position measurements from VIO has less impact on the estimation accuracy of asynchronous parameters between the microphone arrays.
![Refer to caption](x35.png)
![Refer to caption](x36.png)
Distance | Microphone Array | SRC | |||
Positions (m) | Orientations (deg.) | Offset (ms) | Clock diff. (us) | Position (m) | |
1 m | 0.197 | 7.381 | 0.847 | 2.956 | 0.071 |
2 m | 0.173 | 5.825 | 1.272 | 4.327 | 0.116 |
3 m | 0.618 | 55.691 | 1.251 | 13.817 | 0.240 |
5 m | 2.557 | 81.966 | 3.811 | 20.712 | 0.253 |
VI-D Calibration Across Varied Scene Scales
In this section, we conducted experiments to investigate the influence of the distances between the microphone arrays and the sound source on the calibration results. This factor plays a pivotal role in the calibration process of microphone arrays, as it impacts the propagation of sound signals and the level of measurement noises. For instance, in scenarios involving long-distance sound propagation, sound signals undergo propagation loss and are subject to noise interference, resulting in signal attenuation and a decrease in the signal-to-noise ratio (SNR) [3]. We consider four scenarios with different microphone array spacings: 1 meter, 2 meters, 3 meters, and 5 meters. Under these varying distances, the Turtlebot3 robot moves in proximity to the microphone arrays, emitting chirp signals with consistent sound intensity. We record data for each setup, with each experiment repeated three times, resulting in a total of 12 datasets.
Using the datasets collected with different microphone array spacings (1 meter, 2 meters, 3 meters, and 5 meters, respectively), we calculate the SNR that the microphone arrays can capture, and the ground truth values of the microphone arrays positions and the sound markers positions in the global frame are directly measured using a rangefinder. Subsequently, the distances from the microphone arrays to the sound source at different sound marker positions in the 12 collected datasets could be easily calculated. In Fig. 10(a), it can be observed that there is a significant decrease in the maximum SNR that the microphone array can capture as the distance between the mobile robot and the microphone arrays gradually increases. The color map illustrates the ratio of having a certain distance (the x-axis of Fig. 10(a)) between the sound source and the microphone arrays, during the whole experiment process, across the 12 real datasets222For example, in our experiments, there are 4 microphone arrays; for every dataset, there were 13 sound events; so in total, there are scenarios; if there are 138 scenarios where the sound source is 1m–1.5m away from any microphone array, then its corresponding ratio is .. Meanwhile, Fig. 10(b) clearly shows a significant decrease in the accuracy of both DOA and inter-array TDOA estimations with the increasing distance between the sensor array and the signal source. Table V summarizes the calibration results for different spacing cases. It can be seen from Table V that, compared to the greater spacings of 3 meters and 5 meters, our proposed calibration pipeline achieves better performance for the spacing cases of 1 meter and 2 meters. Moreover, one can also notice from Table V that the calibration results for microphone array positions and orientations at 2m were better than those at 1m, because the SNR increases in the 1-2m range (see Fig. 10), while the elevation angle measurement error (w.r.t. the ground truth values) gradually decreases and the azimuth angle measurement error (w.r.t. the ground truth values) almost stays the same. The above results further illustrate the impact of distance on calibration performance.
VI-E Comparisons with Existing Methods
We next compare our proposed calibration pipeline with the existing algorithms using the datasets collected in Section VI.B. These algorithms include the passive geometry calibration method333The original algorithms in [20] is for the 2D case. For comparison purposes, we have revised it accordingly for the 3D case. for microphone arrays based on the differential evolution algorithm (PGM) [20] and the two-step calibration method based on the L-BFGS algorithm (TSM) [23]. It is worth noting that these two calibration methods do not incorporate relative position measurements, i.e., they overlook the constraints among the positions of the sound source. Additionally, PGM does not include the calibration of time offsets and sampling clock differences among microphone arrays, while TSM disregards sampling clock differences, and the above methods lack an effective initialization process.
To showcase the efficiency of each calibration algorithm, we measure the average time required for each of the 15 calibration datasets for different methods on a PC with 32 GB RAM and an Intel Core 3.1 GHz i5-10505 processor. Table VI provides a summary of quantitative comparisons, where the RMSE is calculated based on the metrics listed in Appendix B. The experimental results indicate that our proposed methods (both IM and FT) outperform both PGM and TSM. Besides, the proposed method takes approximately 9 seconds (the total time that both initialization and Gauss-Newton iteration take) to automatically generate a highly accurate calibration of the multiple microphone arrays in 3D, which is faster than TSM and PGM, demonstrating its desirable efficiency.
VI-F Discussions
It is evident from the previous simulation and experimental results that the proposed method demonstrates strong robustness, outperforming existing calibration methods in terms of both accuracy and speed. Moreover, one should note that calibration accuracy is influenced by the measurement noises of the sensors, which is a critical factor. It is also worth noting that the SNR decreases as distances increase between the sound source and the microphone arrays, as pointed out in the existing works [5], [49], and [50]. Consequently, calibration accuracy gradually diminishes with increasing distance. This phenomenon is also observed for our proposed calibration framework, as shown in Section VI.C.
Finally, we remark that while the proposed method can tolerate certain noises such as the robot motion noise and air conditioner noise, it might face challenges in more complex scenarios with diffraction, reflection, and multiple sound sources. In these scenarios, to achieve satisfactory calibration accuracy, one has to incorporate other advanced techniques reported in the literature [5], [50], [51, pp. 217-241].
VII CONCLUSION
This paper is concerned with the joint calibration of multiple asynchronous microphone arrays and sound source localization via batch SLAM. First of all, using the FIM approach, we have conducted a systematic observability analysis of the batch SLAM framework for the above-mentioned calibration problem. More specifically, we have established necessary/sufficient conditions guaranteeing that the FIM and the Jacobian matrix have full column rank, which further implies the identifiability of the unknown parameters. Several scenarios where the unknown parameters are not uniquely identifiable have also been discovered and discussed. Subsequently, for solving the corresponding NLS problem, an effective framework has been proposed to obtain initialized values for the unknown parameters, which are used as the initial guesses in Gauss–Newton types of iterations in batch SLAM and further improve optimization accuracy and convergence. Extensive Monte Carlo simulations and real experiments confirm that the proposed method exhibits high efficiency, accuracy, and robustness in parameter calibration in 3D cases, outperforming the state-of-the-art frameworks for multiple microphone arrays calibration.
The main focus of our current and future work is to consider the active calibration problem of single or multiple microphone arrays where the sound source can optimize its trajectory in real-time to actively collect measurements that contain richer information for improved accuracy and performance, in contrast to the scenarios where the sound source is operated by a human. The calibration problem of moving microphone arrays is also of interest in our future work.
VIII ACKNOWLEDGMENT
The authors would like to thank the reviewers and Editors for their constructive suggestions which have helped to improve the quality and presentation of this paper significantly. This work was supported by the Science, Technology, and Innovation Commission of Shenzhen Municipality, China, under Grant No. ZDSYS20220330161800001, the Shenzhen Science and Technology Program under Grant No. KQTD20221101093557010, the National Natural Science Foundation of China (NSFC) under Grant No. 62350055.
References
- [1] P. Gerstoft, Y. Hu, M. J. Bianco, C. Patil, A. Alegre, Y. Freund, and F. Grondin, Audio scene monitoring using redundant ad-hoc microphone array networks, IEEE Internet of Things Journal, Vol. 9, No. 6, pp. 4259–4268, 2022.
- [2] H. G. Okuno and K. Nakadai, Robot audition: Its rise and perspectives, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5610–5614, 2015.
- [3] C. Rascon and I. Meza, Localization of sound sources in robotics: A review, Robotics and Autonomous Systems, Vol. 96, pp. 184–210, 2017.
- [4] K. Nakadai, M. Kumon, H. Okuno, K. Hoshiba, M. Wakabayashi, K. Washizaki, T. Ishiki, Y. Bando, T. Morito, R. Kojima, and O. Sugiyama, Development of microphone-array-embedded UAV for search and rescue task, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5985–5990, 2017.
- [5] I. An, Y. Kwon, and S. Yoon, Diffraction-and reflection-aware multiple sound source localization, IEEE Transactions on Robotics, Vol. 38, No. 3, pp. 1925–1944, 2021.
- [6] J. Zhang, Q. Lyu, G. Peng, Z. Wu, Q. Yan, and D. Wang, LB-L2L-Calib: Accurate and robust extrinsic calibration for multiple 3D LiDARs with long baseline and large viewpoint difference, Proc. of the 2022 International Conference on Robotics and Automation (ICRA), Vol. 22, No. 11, pp. 926–932, 2022.
- [7] J. Lv, X. Zuo, K. Hu, J. Xu. G. Huang, and Y. Liu, Observability-aware intrinsic and extrinsic calibration of LiDAR-IMU systems, IEEE Transactions on Robotics, Vol. 38, No. 6, pp. 3734–3753, 2022.
- [8] J. Huai, Y. Lin, Y. Zhuang, C. K. Toth, and D. Chen, Observability analysis and keyframe-based filtering for visual inertial odometry with full self-calibration, IEEE Transactions on Robotics, Vol. 38, No. 5, pp. 3219–3237, 2022.
- [9] J. Wu, M. Wang, Y. Jiang, B. Yi, R. Fan, and M. Liu, Simultaneous hand–eye/robot–world/camera–IMU calibration, IEEE/ASME Transactions on Mechatronics, Vol. 27, No. 4, pp. 2278–2289, 2022.
- [10] J. Jiao, Y. Yu, Q. Liao, H. Ye, R. Fan, and M. Liu, Automatic calibration of multiple 3D lidars in urban environments, Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 15–20, 2019.
- [11] A. Plinge, F. Jacob, R. Haeb-Umbach, and G. A. Fink, Acoustic microphone geometry calibration: An overview and experimental evaluation of state-of-the-art algorithms, IEEE Signal Processing Magazine, Vol. 33, No. 4, pp. 14–29, 2016.
- [12] F. Perrodin, J. Nikolic, J. Busset and R. Siegwart, Design and calibration of large microphone arrays for robotic applications, Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4596–4601, 2012.
- [13] M. Crocco, A. Del Bue, and V. Murino, A bilinear approach to the position self-calibration of multiple sensors, IEEE Transactions on Signal Processing, Vol. 60, No. 2, pp. 660–673, 2011.
- [14] Y. Kuang, S. Burgess, A. Torstensson, and K. Åström, A complete characterization and solution to the microphone position self-calibration problem, IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3875–3879, 2013.
- [15] S. Burgess, Y. Kuang, and K. Åström, TOA sensor network self-calibration for receiver and transmitter spaces with difference in dimension, Signal Processing, Vol. 107, pp. 32–42, 2015.
- [16] D. Su, T. Vidal-Calleja, and J. V. Miro, Simultaneous asynchronous microphone array calibration and sound source localisation, Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5561–5567, 2015.
- [17] D. Su, T. Vidal-Calleja, and J. V. Miro, Asynchronous microphone arrays calibration and sound source tracking, Autonomous Robots, Vol. 44, No. 2, pp. 183–204, 2020.
- [18] D. Su, H. Kong, S. Sukkarieh, and S. Huang, Necessary and sufficient conditions for observability of SLAM-based TDOA sensor array calibration and source localization, IEEE Transactions on Robotics, Vol. 37, No. 5, pp. 1451–1468, 2021.
- [19] A. Plinge and G. A. Fink, Geometry calibration of multiple microphone arrays in highly reverberant environments, Proc. of the International Workshop on Acoustic Signal Enhancement, pp. 243–247, 2014.
- [20] A. Plinge, G. A. Fink, and S. Gannot, Passive online geometry calibration of acoustic sensor networks, IEEE Signal Processing Letters, Vol. 24, No. 3, pp. 324–328, 2017.
- [21] D. Hu, Z. Chen, and F. Yin, Geometry calibration for acoustic transceiver networks based on network Newton distributed optimization, IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 29, pp. 1023–1032, 2021.
- [22] R. Wang, Z. Chen, and F. Yin, DOA-Based three-dimensional node geometry calibration in acoustic sensor networks and its Cramér–Rao Bound and sensitivity analysis, IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 27, No. 9, pp. 1455–1468, 2019.
- [23] S. Woźniak and K. Kowalczyk, Passive joint localization and synchronization of distributed microphone arrays, IEEE Signal Processing Letters, Vol. 26, No. 2, pp. 292–296, 2019.
- [24] C. Sugiyama, K. Itoyama, K. Nishida, and K. Nakadai, Assessment of simultaneous calibration for positions, orientations, and time offsets in multiple microphone arrays systems, IEEE/SICE International Symposium on System Integration (SII), pp. 1–6, 2023.
- [25] L. Wang and S. Doclo, Correlation maximization-based sampling rate offset estimation for distributed microphone arrays, IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 24, No. 3, pp. 571–582, 2016.
- [26] S. Thrun and M. Montemerlo, The graph SLAM algorithm with applications to large-scale mapping of urban structures, The International Journal of Robotics Research, Vol. 25, No. 5–6, pp. 403–429, 2006.
- [27] G. Grisetti, R. Kümmerle, C. Stachniss, and W. Burgard, A tutorial on graph-based SLAM, IEEE Intelligent Transportation Systems Magazine, Vol. 2, No. 4, pp. 31–43, 2010.
- [28] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard, Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age, IEEE Transactions on Robotics, Vol. 32, No. 6, pp. 1309–1332, 2016.
- [29] S. M. Nasiri, R. Hosseini, and H. Moradi, Novel parameterization for Gauss–Newton methods in 3D pose graph optimization, IEEE Transactions on Robotics, Vol. 37, No. 3, pp. 780–797, 2021.
- [30] H. Kong and S. Sukkarieh, Suboptimal receding horizon estimation via noise blocking, Automatica, Vol. 98, pp. 66–75, 2018.
- [31] H. Kong and S. Sukkarieh, Metamorphic moving horizon estimation, Automatica, Vol. 97, pp. 167–171, 2018.
- [32] Z. Wang and G. Dissanayake, Observability analysis of SLAM using Fisher information matrix, Proc. of the International Conference on Control, Automation, Robotics, and Vision, pp. 1242–1247, 2008.
- [33] S. Huang and G. Dissanayake, A critique of current developments in simultaneous localization and mapping, International Journal of Advanced Robotic Systems, Vol. 13, No. 5, pp. 1–13, 2016.
- [34] S. M. Nasiri, H. Moradi and R. Hosseini, A linear least square initialization method for 3D pose graph optimization problem, IEEE International Conference on Robotics and Automation (ICRA), pp. 2474-2479, 2018.
- [35] D. M. Rosen, L. Carlone, A. S. Bandeira, and J. J. Leonard, SE-Sync: A certifiably correct algorithm for synchronization over the special Euclidean group, The International Journal of Robotics Research, Vol. 38, No. 2-3, pp. 95–125, 2019.
- [36] F. Dümbgen, C. Holmes, and T. D. Barfoot, Safe and smooth: Certified continuous-time range-only localization, IEEE Robotics and Automation Letters, Vol. 8, No. 2, pp. 1117–1124, 2023.
- [37] H. Yang and L. Carlone, Certifiably optimal outlier-robust geometric perception: Semidefinite relaxations and scalable global optimization, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, No. 3, pp. 2816-2834, 2023.
- [38] F. Pomerleau, F. Colas, and R. Siegwart. A review of point cloud registration algorithms for mobile robotics, Foundations and Trends® in Robotics, Vol. 4, No. 1, pp. 1–104, 2015.
- [39] Y. He, J. Wang, D. Su, K. Nakadai, J. Wu, S. Huang, Y. Li, and H. Kong, Observability analysis of graph SLAM-based joint calibration of multiple microphone arrays and sound source localization, IEEE/SICE International Symposium on System Integration, pp. 1–8, 2023.
- [40] Y. Bar-Shalom, X. R. Li, and T. Kirubarajan, Estimation with applications to tracking and navigation: Theory algorithms and software. New York: Wiley, 2004.
- [41] B. Siciliano, L. Sciavicco, L. Villani, and G. Oriolo, Robotics: Modeling, planning, and control, Berlin, Germany: Springer, 2009.
- [42] M. A. Branch, T. F. Coleman, and Y. Li, A subspace, interior, and conjugate gradient method for large-scale bound-constrained minimization problems, SIAM Journal on Scientific Computing, Vol. 21, No. 1, pp. 1–23, 1999.
- [43] F. M. Dekking, C. Kraaikamp, H. P. Lopuhaä, and L. E. Meester. A modern introduction to probability and statistics: Understanding why and how, London: springer, 2005.
- [44] K. S. Arun, T. S. Huang, and S. D. Blostein. Least-Squares fitting of two 3-D Point sets, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-9, No. 5, pp. 698–700, 1987.
- [45] T. Blesgen. On rotation deformation zones for finite-strain Cosserat plasticity, Acta Mechanica, Vol. 226, No. 7, pp. 2421–2434, 2015.
- [46] C. Knapp and G. Carter, The generalized correlation method for estimation of time delay, IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. 24, No. 4, pp. 320–327, 1976.
- [47] A. Badali, J. M. Valin, F. Michaud, and P. Aarabi, Evaluating real-time audio localization algorithms for artificial audition in robotics, IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), pp. 2033–2038, 2009.
- [48] T. Qin, P. Li and S. Shen, VINS-Mono: A Robust and Versatile monocular visual-inertial state estimator, IEEE Transactions on Robotics, Vol. 34, No. 4, pp. 1004–1020, 2018.
- [49] C. Evers and P. A. Naylor, Acoustic SLAM, IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 26, No. 9, pp. 1484–1498, 2018.
- [50] J. M. Valin, F. Michaud, and J. Rouat, Robust localization and tracking of simultaneous moving sound sources using beamforming and particle filtering, Robotics and Autonomous Systems, Vol. 55, No. 3, pp. 216–228, 2007.
- [51] S. Rickard, The DUET blind source separation algorithm. Dordrecht: Springer Netherlands, 2007.
Appendix A
Proof of Proposition 1. Firstly, we note that the relative position of the sound source satisfies whose corresponding Jacobian matrices are
Secondly, for , the distance between the microphone array and the sound source at time instance can be computed as
(43) |
where
(44) |
When i.e., for the first microphone array, we have
(45) |
Based on the DOA and TDOA models in (1) and (2), then
(46) |
where for and , and only entries of on its rows are nonzero. Then, can be re-expressed as:
(47) |
Denote as the partial derivative of TDOA and DOA w.r.t. microphone array positions, respectively; denote as the partial derivative of DOA w.r.t. XYZ Euler angles. We then have:
(48) |
where
(49) |
and
(50) |
where and are the rotation matrices about coordinate frame axes , and , respectively. can be expressed as with
Denote as the partial derivative of TDOA and DOA observations w.r.t. sound source position at time instance , for . We then have the expression of as follows:
(51) |
The results then follow the definition of the Jacobian matrix [41, pp. 569]. This completes the proof.
Proof of Theorem 2. By performing elementary row transformation of , we can obtain:
(52) |
where
for . Apparently, it holds that . Also, due to the structure of , their columns are independent of each other. For each microphone array, denote . We then perform the following elementary transformation on the matrix :
(i) adding the first column block of to ;
(ii) exchanging row blocks to collect all and together, respectively, thereby obtaining
(53) |
where
We further perform the following elementary operations on , :
(i) dividing the fourth column block by ;
(ii) for , deducing the row by the first row;
(iii) transforming the elements in the first row (except the third one) to zero by the third column block (the first element therein equals 1 while the other elements equal zero after the elementary operations listed above);
(iv) for , deducing the row by the second row multiplied by ;
(v) transforming the elements in the second row (except the fourth one) to zero by the fourth column block (the second element therein equals 1 while the other elements equal zero after the elementary operations listed above);
(vi) moving column blocks 3 and 4 to columns blocks 1 and 2, respectively.
After the above operations, we obtain
(54) |
where
(55) |
and
(56) | ||||
with , , and being defined in (48), represents With the above elementary row and column transformations, we have
(57) |
It holds that . From the structure of , we can see that the block columns containing , , are independent of each other. A necessary condition for to be of full column rank is that and are of full column rank, respectively, . This completes the proof.
Proof of Theorem 3. Here we take as an example. For , we could perform elementary row block changes: for , deduce row block by the first-row block and obtain:
(58) |
Denote the submatrix of this matrix as:
(59) |
From the structure in (58), we can see clearly that if:
(i) is of full column rank, and
(ii) is
of full column rank,
then will be of full column rank.
Due to the fact that ,
the Jacobian matrix is of full column rank. Similarly,
the same conditions hold when equals to . So the
Jacobian matrix is of full column rank if any matrix
consisting of the column block and the
last column block in is of full
column rank, , and
are of full column rank, and . This completes
the proof.
Proof of Theorem 4. (i) in (55) is of full column rank only if a 3 × 3 matrix formed by at least one of the three-permutation of its rows is full rank. For , the necessary condition for to be of full column rank is . If , can not be of the full column rank.
(ii) Based on (45), when , we could derive From the expression of , we can see that cannot be of full rank if is proportional to each other, . In this case, the sound source is collinear with the origin of the reference microphone array frame at all time steps.
(iii) If the sound source lies on any Euclidean plane of , , and within the three-dimensional Cartesian coordinate frame at all moments, where , , and are arbitrary scalars, the sound source position , could be expressed as , , and , respectively. will not be of full column rank. Specifically, if oder oder , the sound source position will have , , and , respectively, i.e., YOZ, XOZ, and XOY planes in global frame. This completes the proof.
Proof of Theorem 5. (i) If the sound source, at all of time steps, is collinear w.r.t. the origin of the microphone array frame , i.e., always holds true, then for , we can get the following expression:
where , and are defined in (48). For an arbitrary single time step, we have as shown in (49). It can also be seen that and the second-order sub-determinant of is not equal to 0, we know that . is a rotation matrix, , thus . Therefore, will not be of full column rank.
Appendix B
Evaluation Metrics. The errors of microphone arrays positions, orientations, time offsets, clock differences and sound source positions can be expressed as follows:
where represents the estimate of the unknown scalars/vectors/matrix parameters, represents the true value of the corresponding parameter and .
In the experiments in Sections V and VI, we utilized the root mean square error (RMSE) to evaluate the accuracy of the calibration algorithm for parameters estimations. The RMSE of the parameter was calculated as , where is equal to the total number of the corresponding parameter .
![]() |
Jiang Wang received the B.Eng. in Electrical Engineering and Automation from the Shenyang Agricultural University, Shenyang, China, in 2020. Since September 2021, he has been working towards the M.S. degree in Electronic Science and Technology, Southern University of Science and Technology, Shenzhen, China. His major research interests include sensor calibration, robot audition, SLAM, sensor fusion. |
![]() |
Yuanzheng He received the B.Eng. in Electronic and Information Engineering from the Henan University of Technology, Zhengzhou, China, in 2021. Since September 2021, he has been working towards the M.S. degree in Electronic Science and Technology, Southern University of Science and Technology, Shenzhen, China. His major research interests include robot audition, robot perception, and multi-sensor fusion. |
![]() |
Daobilige Su received his B. Eng. in Mechatronic Engineering from Zhejiang University, China in 2010, M. Eng. in Automation and Robotics from Warsaw University of Technology, Poland and M. Eng. in Automation from University of Genova, Italy through European Master on Advanced Robotics (EMARO) program in 2012, and Ph. D. in robotics at Centre for Autonomous System (CAS), University of Technology Sydney (UTS), Australia in 2017. He was a post-doctoral research associate at Australian Centre for Filed Robotics (ACFR), The University of Sydney from 2017 to 2020. He is currently an Associate Professor at College of Engineering, China Agricultural University, China. His current research areas include field robotics, SLAM, robot audition, computer vision, and machine learning. |
![]() |
Katsutoshi Itoyama received the M.S. and Ph.D. degrees in informatics from Kyoto University, Kyoto, Japan, in 2008 and 2011, respectively. He had been an Assistant Professor with the Graduate School of Informatics, Kyoto University, until 2018 and is currently a Associate Professor with the Tokyo Institute of Technology, Tokyo, Japan. His research interests include sound source separation, music listening interfaces, and music information retrieval. |
![]() |
Kazuhiro Nakadai received a B.E. in electrical engineering in 1993, an M.E. in information engineering in 1995, and a Ph.D. in electrical engineering in 2003 from the University of Tokyo. He worked with Nippon Telegraph and Telephone for four years as a system engineer from 1995 to 1999, with the Kitano Symbiotic Systems Project, ERATO, JST as a researcher from 1999 to 2003, and with Honda Research Institute Japan, Co., Ltd. as a principal scientist from 2003 to 2022. Currently he is a professor at the Department of Systems and Control Engineering, School of Engineering, Tokyo Institute of Technology. He has had a concurrent position at Tokyo Institute of Technology, as a visiting associate professor from 2006 to 2010, a visiting professor from 2011 to 2017, and a specially-appointed professor from 2017 to 2022. He also had a concurrent position as a guest professor at Waseda University from 2011 to 2018. His research interests include AI, robotics, signal processing, computational auditory scene analysis, multi-modal integration, and robot audition. He has been an executive board member for JSAI from 2015 to 2016, and for RSJ from 2017 to 2018. He is a Fellow of the IEEE and also a member of JSAI, RSJ, IPSJ, ASJ, HIS, ISCA, ACM. |
![]() |
Junfeng Wu received the B.Eng. degree from the Department of Automatic Control, Zhejiang University, Hangzhou, China, and the Ph.D. degree in electrical and computer engineering from the Hong Kong University of Science and Technology, Hong Kong, in 2009, and 2013, respectively. From 2014 to 2017, he was a Postdoctoral Researcher with the ACCESS (Autonomic Complex Communication nEtworks, Signals and Systems) Linnaeus Center, School of Electrical Engineering, KTH Royal Institute of Technology, Stockholm, Sweden. From 2017 to 2021, he was with the College of Control Science and Engineering, Zhejiang University, Hangzhou, China. He is currently an Associate Professor with the School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. His research interests include networked control systems, state estimation, and wireless sensor networks, multi-agent systems, robot perception and localization. He currently serves as an Associate Editor for IEEE Transactions on Control of Network Systems. |
![]() |
Shoudong Huang received the bachelor’s and master’s degrees in mathematics and the Ph.D. in automatic control from Northeastern University, Shenyang, China, in 1987, 1990, and 1998, respectively. He is currently a Professor with the Centre for Autonomous Systems, Faculty of Engineering and Information Technology, University of Technology, Sydney, NSW, Australia. His research interests include nonlinear control systems and mobile robots simultaneous localization and mapping (SLAM), exploration, and navigation. |
![]() |
Youfu Li received the PhD degree in robotics from the Department of Engineering Science, University of Oxford in 1993. From 1993 to 1995 he was a research staff in the Department of Computer Science at the University of Wales, Aberystwyth, UK. He joined City University of Hong Kong in 1995 and is currently professor in the Department of Mechanical Engineering. His research interests include robot sensing, robot vision, and visual tracking. In these areas, he has published over 400 papers including over 180 SCI listed journal papers. Dr Li has received many awards in robot sensing and vision including IEEE Sensors Journal Best Paper Award by IEEE Sensors Council, Second Prize of Natural Science Research Award by the Ministry of Education, 1st Prize of Natural Science Research Award of Hubei Province, 1st Prize of Natural Science Research Award of Zhejiang Province, China. He was on Top 2% of the world’s most highly cited scientists by Stanford University, 2020, 2021 and Career Long. He has served as an Associate Editor for IEEE Transactions on Automation Science and Engineering (T-ASE), Associate Editor and Guest Editor for IEEE Robotics and Automation Magazine (RAM), and Editor for CEB, IEEE International Conference on Robotics and Automation (ICRA). He is a Fellow of the IEEE. |
![]() |
He Kong received the Bachelor’s degree in Electrical Engineering from China University of Mining and Technology, Xuzhou, China, Master’s degree in Control Science and Engineering from Harbin Institute of Technology, Harbin, China, and the Ph.D. degree in Electrical Engineering from the University of Newcastle, Australia, respectively. He was a Research Fellow at the Australian Centre for Field Robotics, the University of Sydney, Australia, during 2016–2021. In early 2022, he joined the Southern University of Science and Technology, Shenzhen, China, where he is currently an Associate Professor. His research interests include active multi-modal perception, robot audition, state estimation, and control applications. He is currently serving on the editorial board of IEEE Robotics and Automation Letters, IEEE Robotics and Automation Magazine, IEEE Sensors Letters, International Journal of Adaptive Control and Signal Processing. He has also served as an Associate Editor for several international conferences in robotics and automation, including the IEEE ICRA, IEEE/RSJ IROS, IEEE CASE, etc. |