Published in Technometrics (Vol. 64, No. 4, 2022)

AbstractOnline experiments are the gold standard for evaluating impact on user experience and accelerating innovation in software. However, since experiments are typically limited in duration, observed treatment effects are not always stable, sometimes revealing increasing or decreasing patterns over time. There are multiple causes for a treatment effect to change over time. In this article, we focus on a particular cause, user-learning, which is primarily associated with novelty or primacy. Novelty describes the desire to use new technology that tends to diminish over time. Primacy describes the growing engagement with technology as a result of adoption of the innovation. Estimating user-learning is critical because it holds experimentation responsible for trustworthiness, empowers organizations to make better decisions by providing a long-term view of expected impact, and prevents user dissatisfaction. In this article, we propose an observational approach, based on difference-in-differences technique to estimate user-learning at scale. We use this approach to test and estimate user-learning in many experiments at Microsoft. We compare our approach with the existing experimental method to show its benefits in terms of ease of use and higher statistical power, and to discuss its limitation in presence of other forms of treatment interaction with time.

AbstractIn practice, optimal screening designs for arbitrary run sizes are traditionally generated using the D-criterion with factor settings fixed at ±1, even when considering continuous factors with levels in [-1,1]. This paper identifies cases of undesirable estimation variance properties for such D-optimal designs and argues that generally A-optimal designs tend to push variances closer to their minimum possible value. New insights about the behavior of the criteria are gained through a study of their respective coordinate-exchange formulas. The study confirms the existence of D-optimal designs comprised only of settings ±1 for both main effect and interaction models for blocked and unblocked experiments. Scenarios are also identified for which arbitrary manipulation of a coordinate between [-1,1] leads to infinitely many D-optimal designs each having different variance properties. For the same conditions, the A-criterion is shown to have a unique optimal coordinate value for improvement. We also compare how Bayesian versions of the A- and D-criteria balance minimization of estimation variance and bias. Multiple examples of screening designs are considered for various models under Bayesian and non-Bayesian versions of the A- and D-criteria.

AbstractIdentifying important factors from a large number of potentially important factors of a highly nonlinear and computationally expensive black box model is a difficult problem. Morris screening and Sobol’ design are two commonly used model-free methods for doing this. In this article, we establish a connection between these two seemingly different methods in terms of their underlying experimental design structure and further exploit this connection to develop an improved design for screening called Maximum One-Factor-At-A-Time (MOFAT) design. We also develop efficient methods for constructing MOFAT designs with a large number of factors. Several examples are presented to demonstrate the advantages of MOFAT designs compared to Morris screening and Sobol’ design methods.

Published in Technometrics (Vol. 64, No. 4, 2022)

AbstractThe eigenvalues and eigenvectors of the Fisher Information Matrix (FIM) can reveal the most and least sensitive directions of a system and it has wide application across science and engineering. We present a symplectic variant of the eigenvalue decomposition for the FIM and extract the sensitivity information with respect to two-parameter conjugate pairs. The symplectic approach decomposes the FIM onto an even-dimensional symplectic basis. This symplectic structure can reveal additional sensitivity information between two-parameter pairs, otherwise concealed in the orthogonal basis from the standard eigenvalue decomposition. The proposed sensitivity approach can be applied to naturally paired two-parameter distribution parameters, or a decision-oriented pairing via regrouping or re-parameterization of the FIM. It can be used in tandem with the standard eigenvalue decomposition and offer additional insights into the sensitivity analysis at negligible extra cost. Supplementary materials for this article are available online.

AbstractA novel method is proposed for detecting changes in the covariance structure of moderate dimensional time series. This non-linear test statistic has a number of useful properties. Most importantly, it is independent of the underlying structure of the covariance matrix. We discuss how results from Random Matrix Theory, can be used to study the behaviour of our test statistic in a moderate dimensional setting (i.e. the number of variables is comparable to the length of the data). In particular, we demonstrate that the test statistic converges point wise to a normal distribution under the null hypothesis. We evaluate the performance of the proposed approach on a range of simulated datasets and find that it outperforms a range of alternative recently proposed methods. Finally, we use our approach to study changes in the amount of water on the surface of a plot of soil which feeds into model development for degradation of surface piping.

Published in Technometrics (Vol. 65, No. 1, 2023)

Published in Technometrics (Vol. 65, No. 2, 2023)

Simultaneous degradation of multiple dependent performance characteristics (PCs) is a common phenomenon for industrial products. The associated degradation modeling is of practical importance yet c...

AbstractWe study optimal variance reduction solutions for count and ratio metrics in online controlled experiments. Our methods apply flexible machine learning tools to incorporate covariates that are independent from the treatment but have predictive power for the outcomes, and employ the cross-fitting technique to remove the bias in complex machine learning models. We establish CLT-type asymptotic inference based on our estimators under mild convergence conditions. Our procedures are optimal (efficient) for the corresponding targets as long as the machine learning estimators are consistent, without any requirement for their convergence rates. In complement to the general optimal procedure, we also derive a linear adjustment method for ratio metrics as a special case that is computationally efficient and can flexibly incorporate any pretreatment covariates. We evaluate the proposed variance reduction procedures with comprehensive simulation studies and provide practical suggestions regarding commonly adopted assumptions in computing ratio metrics. When tested on real online experiment data from LinkedIn, the proposed optimal procedure for ratio metrics can reduce up to 80% of variance compared to the standard difference-in-mean estimator and also further reduce up to 30% of variance compared to the CUPED approach by going beyond linearity and incorporating a large number of extra covariates.

Published in Technometrics (Vol. 65, No. 1, 2023)

AbstractMulti-output Gaussian process (MGP) regression plays an important role in the integrative analysis of different but interrelated systems/units. Existing MGP approaches assume that data from all units is collected and stored at a central location. This requires massive computing and storage power at the central location, induces significant communication traffic due to raw data exchange, and comprises privacy of units. However, recent advances in Internet of Things technologies, which have tremendously increased edge computing power, pose a significant opportunity to address such challenges. In this paper, we propose FedMGPFedMGPFedMGP, a general federated analytics (FA) framework to learn an MGP in a decentralized manner that utilizes edge computing power to distribute model learning efforts. Specifically, we propose a hierarchical modeling approach where an MGP is built upon shared global latent functions. We then develop a variational inference FA algorithm that overcomes the need to share raw data. Instead, collaborative learning is achieved by only sharing global latent function statistics. Comprehensive simulation studies and the case study on battery degradation data highlight the superior predictive performance and versatility of FedMGP, achieved while distributing computing and storage demands, reducing communication burden, fostering privacy, and personalizing analysis.

AbstractIn one-shot experiments, units are subjected to varying levels of stimulus and their binary response (go/no-go) is recorded. Experimental data is used to estimate the ‘sensitivity function’, which characterizes the probability of a ‘go’ for a given level of stimulus. We review the current GLM approaches to modeling and inference, and identify some deficiences. To address these, we propose a novel Bayesian approach using an adjustable number of cubic splines, with physically-plausible smoothness, monotonicity, and tail constraints introduced through the prior distribution on the coefficients. Our approach runs ‘out of the box’, and in roughly the same time as the GLM approaches. We illustrated with two contrasting datasets, and show that our more flexible Bayesian approach gives different inferences to the GLM approaches for both the sensitivity function and its inverse. The code and datasets are available online in the Supplementary Material.

Published in Technometrics (Vol. 65, No. 2, 2023)

AbstractThe growing prevalence of tensor data, or multiway arrays, in science and engineering applications motivates the need for tensor decompositions that are robust against outliers. In this paper, we present a robust Tucker decomposition estimator based on the L2 criterion, called the Tucker-L2E. Our numerical experiments demonstrate that Tucker-L2E has empirically stronger recovery performance in more challenging high-rank scenarios compared with existing alternatives. The appropriate Tucker-rank can be selected in a data-driven manner with cross-validation or hold-out validation. The practical effectiveness of Tucker-L2E is validated on real data applications in fMRI tensor denoising, PARAFAC analysis of fluorescence data, and feature extraction for classification of corrupted images.

Published in Technometrics (Vol. 65, No. 1, 2023)

AbstractA new class of orthogonal arrays called dual-orthogonal arrays is proposed in this paper to design order-of-addition two-level factorial experiments in which both component addition orders and component levels can be varied over treatments. Dual-orthogonal arrays can be viewed as an optimal combination of order-of-addition orthogonal arrays and two-level orthogonal arrays. Based on these two different concepts of orthogonality, when a compound model is used to fit the observed data, both pairwise order effects and component main effects can be estimated with optimal efficiency. Under the assumption of normality, these two kinds of parametric effects can also be inferred independently. A three-drug combination study is first used to show that dual-orthogonal arrays can be practical for real-world studies. Both combinatorial and computational methods are then introduced to construct dual-orthogonal arrays. Additionally, a design catalogue is generated for future work.

Published in Technometrics (Vol. 65, No. 1, 2023)

Published in Technometrics (Vol. 65, No. 1, 2023)