Metric type</th>	Examples</th>	Typical test</th></tr></thead>
Average</td>	Average revenue per user, average orders per user</td>	Welch's t-test (unequal-variance two-sample t-test)</td></tr>
Ratio of averages</td>	Average revenue per order, average orders per session</td>	Welch's t-test with variance from the delta method</td></tr>
Proportion</td>	Proportion of users with at least one order</td>	Asymptotic tests (z-test, G-test, Pearson's chi-squared) or exact tests (Boschloo, Barnard, Fisher)</td></tr> </tbody></table> Typical metric types and tests</figcaption></figure> Typical per-metric analysis output #</a></h3> For each metric, analysts usually want the same core fields:</p> Metric value in control.</li> Metric value in treatment.</li> Effect size estimate and confidence interval (absolute and/or relative).</li> P-value.</li> </ul> This output format is what makes A/B test analysis convenient for repeated use across many experiments.</p> A/B testing specifics #</a></h3> Some details matter a lot in real experimentation workflows.</p> Relative effect size is often easier to interpret than absolute effect size. A common mistake is to divide an absolute confidence interval by the control mean. That does not generally produce a valid confidence interval for a relative effect. Use the delta method or Fieller's theorem instead (both are standard approaches for ratio-style uncertainty).</li> Variance reduction (especially CUPED, which uses pre-experiment covariates) is widely used to increase power. CUPED for ratio metrics is more subtle because it combines variance reduction with ratio estimators and delta-method variance.</li> Multiple hypothesis testing correction becomes important when you track many metrics or compare multiple variants. In practice, teams usually need a clear FWER/FDR policy and adjusted p-values (or adjusted significance thresholds) in experiment reports.</li> Efficiency matters. Teams often analyze many metrics across many experiments. For many tests, only aggregated statistics are required (count, mean, variance, covariance). In those cases, it is usually more efficient to compute aggregates in the data backend and send only summary statistics to Python. A convenience layer for aggregate fetching can reduce both code and latency.</li> </ul> Example data setup #</a></h2> With that context, the examples below use the same synthetic experiment dataset generated with tea-tasting. They intentionally do not use variance reduction (no CUPED), so the package comparisons stay focused on baseline analysis.</p> import</span> tea_tasting</span> as</span> tt</span></span> </span> data</span> =</span> tt</span>.</span>make_users_data</span>(</span>return_type</span>=</span>"</span>pandas</span>"</span>,</span> rng</span>=</span>42</span>,</span> n_users</span>=</span>5_000</span>)</span></span> data</span>[</span>"</span>has_order</span>"</span>]</span> =</span> (</span>data</span>[</span>"</span>orders</span>"</span>]</span> ></span> 0</span>)</span>.</span>astype</span>(</span>int</span>)</span></span> </span> control</span> =</span> data</span>[</span>data</span>[</span>"</span>variant</span>"</span>]</span> ==</span> 0</span>]</span></span> treatment</span> =</span> data</span>[</span>data</span>[</span>"</span>variant</span>"</span>]</span> ==</span> 1</span>]</span></span> </span> print</span>(</span>data</span>.</span>head</span>(</span>3</span>)</span>.</span>to_string</span>(</span>index</span>=</span>False</span>)</span>)</span></span></code></pre> Output:</p> `user variant sessions orders revenue has_order</span></span> 0 1 1 1 5.77 1</span></span> 1 0 3 0 0.00 0</span></span> 2 1 1 0 0.00 0</span></span></code></pre> The metrics used in the examples are:</p>` `orders_per_user</code>: average orders per user.</li>` `users_with_orders</code>: proportion of users with at least one order.</li>` `revenue_per_user</code>: average revenue per user.</li>` `revenue_per_order</code>: average revenue per order (ratio of averages).</li> </ul> All package examples below reuse data</code>, control</code>, and treatment</code> from this setup.</p>` `Package-by-package comparison` #</a></h2> tea-tasting #</a></h3> tea-tasting</a> is a package specifically designed for A/B test analysis. It targets experimentation workflows directly, with metrics, relative effects, CUPED, power analysis, and concise experiment-style outputs.</p> Best for: teams that want an A/B-testing-first workflow with minimal glue code.</p> This is the most compact example among the four packages because it provides A/B-specific metric classes and a high-level Experiment</code> API.</p> experiment</span> =</span> tt</span>.</span>Experiment</span>(</span></span> orders_per_user</span>=</span>tt</span>.</span>Mean</span>(</span>"</span>orders</span>"</span>)</span>,</span></span> users_with_orders</span>=</span>tt</span>.</span>Proportion</span>(</span>"</span>has_order</span>"</span>,</span> correction</span>=</span>False</span>)</span>,</span></span> revenue_per_user</span>=</span>tt</span>.</span>Mean</span>(</span>"</span>revenue</span>"</span>)</span>,</span></span> revenue_per_order</span>=</span>tt</span>.</span>RatioOfMeans</span>(</span>"</span>revenue</span>"</span>,</span> "</span>orders</span>"</span>)</span>,</span></span> )</span></span> result</span> =</span> experiment</span>.</span>analyze</span>(</span>data</span>)</span></span> </span> print</span>(</span>result</span>)</span></span></code></pre> Output:</p> metric control treatment rel_effect_size rel_effect_size_ci pvalue</span></span> orders_per_user 0.511 0.556 8.8% [-0.74%, 19%] 0.0718</span></span> users_with_orders 0.334 0.352 5.4% [-2.4%, 14%] 0.181</span></span> revenue_per_user 5.06 5.64 11% [0.22%, 24%] 0.0455</span></span> revenue_per_order 9.91 10.2 2.5% [-3.0%, 8.3%] 0.389</span></span></code></pre> A/B testing specifics:</p> Power analysis: built-in (Experiment.solve_power</code> and metric parameters for effect size, relative effect size, and sample size).</li> Relative effect and confidence interval: first-class output (including relative CI in the printed result).</li> CUPED: built-in for averages and ratio metrics.</li> Multiple hypothesis testing correction: built-in (tt.adjust_fdr</code> and tt.adjust_fwer</code>) for experiment results, including FDR and FWER procedures.</li> Aggregated statistics workflow: built-in support via metrics that expose required aggregates and integration with data backends (e.g., through Ibis-supported engines).</li> </ul> Pingouin #</a></h3> Pingouin</a> is a user-friendly statistical package focused on convenient inferential statistics in pandas-centric workflows. It is strong for common tests and effect sizes, but it is not an A/B-specific framework.</p> Best for: quick pandas-based analyses of standard statistical tests.</p> Pingouin has a convenient t-test interface and a contingency-table chi-squared helper. It does not provide a built-in ratio-of-averages test with delta-method variance, so the revenue_per_order</code> example is omitted.</p> import</span> pingouin</span> as</span> pg</span></span> </span> orders_test</span> =</span> pg</span>.</span>ttest</span>(</span></span> treatment</span>[</span>"</span>orders</span>"</span>]</span>,</span></span> control</span>[</span>"</span>orders</span>"</span>]</span>,</span></span> correction</span>=</span>True</span>,</span></span> )</span>.</span>iloc</span>[</span>0</span>]</span></span> print</span>(</span></span> "</span>orders_per_user: </span>"</span></span> f</span>"</span>control=</span>{</span>control</span>[</span>'</span>orders</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>treatment=</span>{</span>treatment</span>[</span>'</span>orders</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>effect_size=</span>"</span></span> f</span>"</span>{</span>treatment</span>[</span>'</span>orders</span>'</span>]</span>.</span>mean</span>(</span>)</span> -</span> control</span>[</span>'</span>orders</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>effect_size_ci=</span>{</span>orders_test</span>[</span>'</span>CI95</span>'</span>]</span>}</span> "</span></span> f</span>"</span>pvalue=</span>{</span>orders_test</span>[</span>'</span>p_val</span>'</span>]</span>:.4f</span>}</span>"</span></span> )</span></span></code></pre> Output:</p> orders_per_user: control=0.511 treatment=0.556 effect_size=0.045 effect_size_ci=[-0. 0.09] pvalue=0.0718</span></span></code></pre>_</span>,</span> _</span>,</span> tests</span> =</span> pg</span>.</span>chi2_independence</span>(</span></span> data</span>,</span></span> x</span>=</span>"</span>variant</span>"</span>,</span></span> y</span>=</span>"</span>has_order</span>"</span>,</span></span> correction</span>=</span>False</span>,</span></span> )</span></span> pearson</span> =</span> tests</span>.</span>loc</span>[</span>tests</span>[</span>"</span>test</span>"</span>]</span> ==</span> "</span>pearson</span>"</span>]</span>.</span>iloc</span>[</span>0</span>]</span></span> print</span>(</span></span> "</span>users_with_orders: </span>"</span></span> f</span>"</span>control=</span>{</span>control</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>treatment=</span>{</span>treatment</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>effect_size=</span>"</span></span> f</span>"</span>{</span>treatment</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span> -</span> control</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>pvalue=</span>{</span>pearson</span>[</span>'</span>pval</span>'</span>]</span>:.4f</span>}</span>"</span></span> )</span></span></code></pre> Output:</p> users_with_orders: control=0.334 treatment=0.352 effect_size=0.018 pvalue=0.1811</span></span></code></pre> Notes:</p> revenue_per_user</code> uses the same pg.ttest(...)</code> pattern as orders_per_user</code>.</li> revenue_per_order</code> (ratio of averages) requires manual derivation if you want a statistically correct delta-method analysis.</li> The proportion example above shows a p-value, but not a built-in A/B-style effect CI.</li> </ul> A/B testing specifics:</p> Power analysis: built-in for standard cases (power_ttest</code>, power_ttest2n</code>, power_chi2</code>), but not an A/B-specific multi-metric workflow.</li> Relative effect and confidence interval: mostly manual for A/B-style relative lift CIs.</li> CUPED: no built-in CUPED abstraction. You would implement variance reduction manually (for example, with regression).</li> Multiple hypothesis testing correction: built-in p-value adjustment via pg.multicomp</code> (for example, Bonferroni, Holm, and FDR methods), but integration into an A/B reporting workflow is manual.</li> Aggregated statistics workflow: mostly expects granular arrays/DataFrames; no built-in A/B aggregate interface.</li> </ul> statsmodels #</a></h3> statsmodels</a> is a broad statistical modeling library with strong hypothesis testing, power analysis, and confidence interval utilities. It is less opinionated than an experimentation-specific package, which is a strength if you want building blocks and explicit control.</p> Best for: analysts who want mature statistical building blocks and are comfortable assembling a workflow.</p> The example below uses Welch-style t-tests for two average metrics and a risk-ratio test/CI for the proportion metric. revenue_per_order</code> is omitted because there is no built-in A/B-style ratio-of-averages delta-method helper.</p> from</span> statsmodels</span>.</span>stats</span>.</span>proportion</span> import</span> (</span></span> confint_proportions_2indep</span>,</span></span> test_proportions_2indep</span>,</span></span> )</span></span> from</span> statsmodels</span>.</span>stats</span>.</span>weightstats</span> import</span> CompareMeans</span>,</span> DescrStatsW</span></span> </span> def</span> welch_summary</span>(</span>treatment_series</span>,</span> control_series</span>)</span>:</span></span> cm</span> =</span> CompareMeans</span>(</span></span> DescrStatsW</span>(</span>treatment_series</span>)</span>,</span></span> DescrStatsW</span>(</span>control_series</span>)</span>,</span></span> )</span></span> _</span>,</span> pvalue</span>,</span> _</span> =</span> cm</span>.</span>ttest_ind</span>(</span>usevar</span>=</span>"</span>unequal</span>"</span>)</span></span> ci_low</span>,</span> ci_high</span> =</span> cm</span>.</span>tconfint_diff</span>(</span>usevar</span>=</span>"</span>unequal</span>"</span>)</span></span> return</span> (</span></span> control_series</span>.</span>mean</span>(</span>)</span>,</span></span> treatment_series</span>.</span>mean</span>(</span>)</span>,</span></span> treatment_series</span>.</span>mean</span>(</span>)</span> -</span> control_series</span>.</span>mean</span>(</span>)</span>,</span></span> ci_low</span>,</span></span> ci_high</span>,</span></span> pvalue</span>,</span></span> )</span></span> </span> for</span> metric</span> in</span> [</span>"</span>orders</span>"</span>,</span> "</span>revenue</span>"</span>]</span>:</span></span> ctrl</span>,</span> trt</span>,</span> effect</span>,</span> ci_low</span>,</span> ci_high</span>,</span> pvalue</span> =</span> welch_summary</span>(</span></span> treatment</span>[</span>metric</span>]</span>,</span></span> control</span>[</span>metric</span>]</span>,</span></span> )</span></span> print</span>(</span></span> f</span>"</span>{</span>metric</span>}</span>_per_user: </span>"</span></span> f</span>"</span>control=</span>{</span>ctrl</span>:.3f</span>}</span> treatment=</span>{</span>trt</span>:.3f</span>}</span> effect_size=</span>{</span>effect</span>:.3f</span>}</span> "</span></span> f</span>"</span>effect_size_ci=[</span>{</span>ci_low</span>:.3f</span>}</span>, </span>{</span>ci_high</span>:.3f</span>}</span>] pvalue=</span>{</span>pvalue</span>:.4f</span>}</span>"</span></span> )</span></span></code></pre> Output:</p> orders_per_user: control=0.511 treatment=0.556 effect_size=0.045 effect_size_ci=[-0.004, 0.094] pvalue=0.0718</span></span> revenue_per_user: control=5.062 treatment=5.641 effect_size=0.579 effect_size_ci=[0.012, 1.146] pvalue=0.0455</span></span></code></pre>count1</span> =</span> int</span>(</span>treatment</span>[</span>"</span>has_order</span>"</span>]</span>.</span>sum</span>(</span>)</span>)</span></span> nobs1</span> =</span> len</span>(</span>treatment</span>)</span></span> count0</span> =</span> int</span>(</span>control</span>[</span>"</span>has_order</span>"</span>]</span>.</span>sum</span>(</span>)</span>)</span></span> nobs0</span> =</span> len</span>(</span>control</span>)</span></span> prop_test</span> =</span> test_proportions_2indep</span>(</span></span> count1</span>=</span>count1</span>,</span></span> nobs1</span>=</span>nobs1</span>,</span></span> count2</span>=</span>count0</span>,</span></span> nobs2</span>=</span>nobs0</span>,</span></span> compare</span>=</span>"</span>ratio</span>"</span>,</span></span> method</span>=</span>"</span>log</span>"</span>,</span></span> )</span></span> prop_ci</span> =</span> confint_proportions_2indep</span>(</span></span> count1</span>=</span>count1</span>,</span></span> nobs1</span>=</span>nobs1</span>,</span></span> count2</span>=</span>count0</span>,</span></span> nobs2</span>=</span>nobs0</span>,</span></span> compare</span>=</span>"</span>ratio</span>"</span>,</span></span> method</span>=</span>"</span>log</span>"</span>,</span></span> )</span></span> print</span>(</span></span> "</span>users_with_orders: </span>"</span></span> f</span>"</span>control=</span>{</span>control</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>treatment=</span>{</span>treatment</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>rel_effect_size=</span>{</span>prop_test</span>.</span>ratio</span> -</span> 1</span>:.3f</span>}</span> "</span></span> f</span>"</span>rel_effect_size_ci=[</span>{</span>prop_ci</span>[</span>0</span>]</span> -</span> 1</span>:.3f</span>}</span>, </span>{</span>prop_ci</span>[</span>1</span>]</span> -</span> 1</span>:.3f</span>}</span>] </span>"</span></span> f</span>"</span>pvalue=</span>{</span>prop_test</span>.</span>pvalue</span>:.4f</span>}</span>"</span></span> )</span></span></code></pre> Output:</p> users_with_orders: control=0.334 treatment=0.352 rel_effect_size=0.054 rel_effect_size_ci=[-0.024, 0.138] pvalue=0.1811</span></span></code></pre> A/B testing specifics:</p> Power analysis: strong built-in support (TTestIndPower</code>, NormalIndPower</code>, GofChisquarePower</code>, and more).</li> Relative effect and confidence interval: partial. There are built-in options for some cases (for example, risk ratios for proportions), but no general A/B-style relative lift CI interface across metric types.</li> CUPED: no built-in one-call CUPED API, but it is practical to implement manually with regression tooling.</li> Multiple hypothesis testing correction: strong built-in support (statsmodels.stats.multitest</code>, including multipletests</code> and fdrcorrection</code>).</li> Aggregated statistics workflow: partial. Proportion tests work directly from counts/sample sizes; other workflows often still require granular arrays or more manual setup.</li> </ul> SciPy #</a></h3> SciPy</a> is a foundational scientific computing and statistics package used directly or indirectly by many higher-level libraries, including the others in this comparison. It provides robust hypothesis tests and exact tests, but it does not provide a high-level A/B testing workflow.</p> Best for: low-level building blocks and custom A/B analysis code.</p> This snippet shows a Welch t-test for orders_per_user</code> and a Pearson chi-squared test for users_with_orders</code>. Exact tests such as Fisher, Barnard, and Boschloo are also available in SciPy.</p> import</span> numpy</span> as</span> np</span></span> from</span> scipy</span> import</span> stats</span></span> </span> orders_test</span> =</span> stats</span>.</span>ttest_ind</span>(</span></span> treatment</span>[</span>"</span>orders</span>"</span>]</span>,</span></span> control</span>[</span>"</span>orders</span>"</span>]</span>,</span></span> equal_var</span>=</span>False</span>,</span></span> )</span></span> orders_ci</span> =</span> orders_test</span>.</span>confidence_interval</span>(</span>)</span></span> print</span>(</span></span> "</span>orders_per_user: </span>"</span></span> f</span>"</span>control=</span>{</span>control</span>[</span>'</span>orders</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>treatment=</span>{</span>treatment</span>[</span>'</span>orders</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>effect_size=</span>"</span></span> f</span>"</span>{</span>treatment</span>[</span>'</span>orders</span>'</span>]</span>.</span>mean</span>(</span>)</span> -</span> control</span>[</span>'</span>orders</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>effect_size_ci=[</span>{</span>orders_ci</span>.</span>low</span>:.3f</span>}</span>, </span>{</span>orders_ci</span>.</span>high</span>:.3f</span>}</span>] </span>"</span></span> f</span>"</span>pvalue=</span>{</span>orders_test</span>.</span>pvalue</span>:.4f</span>}</span>"</span></span> )</span></span></code></pre> Output:</p> orders_per_user: control=0.511 treatment=0.556 effect_size=0.045 effect_size_ci=[-0.004, 0.094] pvalue=0.0718</span></span></code></pre>contingency</span> =</span> np</span>.</span>array</span>(</span></span> [</span></span> [</span>(</span>control</span>[</span>"</span>has_order</span>"</span>]</span> ==</span> 0</span>)</span>.</span>sum</span>(</span>)</span>,</span> (</span>control</span>[</span>"</span>has_order</span>"</span>]</span> ==</span> 1</span>)</span>.</span>sum</span>(</span>)</span>]</span>,</span></span> [</span></span> (</span>treatment</span>[</span>"</span>has_order</span>"</span>]</span> ==</span> 0</span>)</span>.</span>sum</span>(</span>)</span>,</span></span> (</span>treatment</span>[</span>"</span>has_order</span>"</span>]</span> ==</span> 1</span>)</span>.</span>sum</span>(</span>)</span>,</span></span> ]</span>,</span></span> ]</span></span> )</span></span> chi2_res</span> =</span> stats</span>.</span>contingency</span>.</span>chi2_contingency</span>(</span>contingency</span>,</span> correction</span>=</span>False</span>)</span></span> print</span>(</span></span> "</span>users_with_orders: </span>"</span></span> f</span>"</span>control=</span>{</span>control</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>treatment=</span>{</span>treatment</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>effect_size=</span>"</span></span> f</span>"</span>{</span>treatment</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span> -</span> control</span>[</span>'</span>has_order</span>'</span>]</span>.</span>mean</span>(</span>)</span>:.3f</span>}</span> "</span></span> f</span>"</span>pvalue=</span>{</span>chi2_res</span>.</span>pvalue</span>:.4f</span>}</span>"</span></span> )</span></span></code></pre> Output:</p> users_with_orders: control=0.334 treatment=0.352 effect_size=0.018 pvalue=0.1811</span></span></code></pre> Notes:</p> revenue_per_user</code> uses the same stats.ttest_ind(..., equal_var=False)</code> pattern as orders_per_user</code>.</li> revenue_per_order</code> (ratio of averages) requires manual delta-method implementation.</li> Relative effect confidence intervals are also manual.</li> </ul> A/B testing specifics:</p> Power analysis: mostly manual in SciPy (or delegated to custom code / another package).</li> Relative effect and confidence interval: manual.</li> CUPED: manual.</li> Multiple hypothesis testing correction: partial. SciPy provides scipy.stats.false_discovery_control</code> for BH/BY FDR adjustment, but broader multiple-comparison correction workflows are more limited than in statsmodels.</li> Aggregated statistics workflow: partial. SciPy supports some summary-statistics and contingency-table tests (for example, ttest_ind_from_stats</code> and chi-squared/exact tests on contingency tables), but not an A/B-specific aggregate workflow.</li> </ul> Feature comparison for A/B testing #</a></h2> SciPy underpins much of the Python statistics ecosystem. In principle, all the capabilities discussed here can be implemented with NumPy + SciPy plus custom code. The practical question is convenience and code verbosity. For that reason, the table below uses three labels:</p> built-in</code>: directly supported in a way that fits common A/B analysis tasks.</li> partial</code>: some built-in support exists, but not as a complete or ergonomic A/B workflow.</li> manual</code>: possible, but requires custom implementation/glue code.</li> </ul> Feature</th> tea-tasting</th> Pingouin</th> statsmodels</th> SciPy</th></tr></thead> Power analysis to estimate required number of observations</td> built-in</td> built-in</td> built-in</td> manual</td></tr> Welch's t-test or Student's t-test for analysis of averages</td> built-in</td> built-in</td> built-in</td> built-in</td></tr> Welch's t-test with delta method for analysis of ratios of averages</td> built-in</td> manual</td> manual</td> manual</td></tr> Two-sample proportion z-test, G-test, or Pearson's chi-squared test</td> built-in</td> built-in</td> built-in</td> built-in</td></tr> Relative effect size confidence intervals</td> built-in</td> manual</td> partial</td> manual</td></tr> Variance reduction with CUPED for analysis of averages and ratios of averages</td> built-in</td> manual</td> manual</td> manual</td></tr> Multiple hypothesis testing correction (FWER/FDR p-value adjustment)</td> built-in</td> built-in</td> built-in</td> partial</td></tr> Working with aggregated statistics instead of granular data</td> built-in</td> manual</td> partial</td> partial</td></tr> </tbody></table> Feature comparison for A/B testing</figcaption></figure>Conclusions #</a></h2> The four packages sit at different levels of abstraction.</p> tea-tasting is the most A/B-specific option in this group. It is designed around metrics, experiments, relative effects, CUPED, and aggregate-based workflows.</li> Pingouin is convenient for standard statistical tests and quick analysis in pandas, but A/B-specific workflows (especially ratio metrics, relative CIs, and CUPED) are mostly manual.</li> statsmodels provides strong statistical building blocks and power analysis. It is a good choice when you want explicit control and are willing to assemble an experimentation workflow yourself.</li> SciPy is the essential foundation. It can support almost everything with enough custom code, but it is the most verbose option for repeated A/B test reporting.</li> </ul> If you run many experiments with multiple metrics and need consistent outputs, the main differentiator is not just statistical correctness. It is how much A/B-specific workflow a package gives you out of the box.</p> Inclusion criteria for the comparison #</a></h2> For transparency, here are the minimum criteria I used to decide which packages to include.</p> Maintained: a recent release (for example, within a year) and recent commits (for example, within two months) reduce the risk of stale APIs and unresolved compatibility issues.</li> Well documented: a package should have both a user guide and an API reference, because A/B testing code is often reused by analysts with different levels of statistical depth.</li> Used by a community: a practical heuristic is at least 100 GitHub stars. This is not a quality guarantee, but it often means more examples and more edge cases have already been surfaced.</li> </ul> Scope note: This comparison focuses on frequentist A/B testing workflows. Bayesian-first experimentation frameworks are out of scope.</p> Excluded notable packages (and why):</p> spotify_confidence</a>: excluded because it lacks documentation and has not added significant new features in the last couple of years.</li> ambrosia</a>: excluded because it has not added new features in the last couple of years, aside from dependency version updates.</li> </ul> Note: The maintenance and documentation notes in this section are assessed as of March 1, 2026.</p> Resources #</a></h2> The comparison above is based on the public documentation and APIs of the packages as of March 1, 2026. Current stable versions on PyPI at the time of writing:</p> tea-tasting</a>: 1.12.0, PyPI</a>.</li> Pingouin</a>: 0.6.0, PyPI</a>.</li> statsmodels</a>: 0.14.6, PyPI</a>.</li> SciPy stats</a>: 1.17.1, PyPI</a>.</li> </ul> A/B testing and statistics references:</p> Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing.</a></li> Deng, A., Knoblich, U., & Lu, J. (2018). Applying the delta method in metric analytics: A practical guide with novel ideas.</a></li> Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data.</a></li> Multiple comparisons problem (Wikipedia).</a></li> </ul> Prices as model weights: learning from dispersed data 2025-10-24T00:00:00+00:00 Market prices are similar to model weights in the sense that they are "fit" through price discovery mechanisms to optimize for equilibrium. The difference between supply and demand is analogous to the gradient in backpropagation.</p> An important property of prices is that they can be "fit" to distributed</em> and sometimes hidden</em> data. As Friedrich Hayek argued, prices convey dispersed</em> knowledge that no single person or central planner could ever fully possess. There are problems that deal with information that cannot be accessed directly. A well-designed price discovery mechanism can solve such problems more efficiently than an AI/ML model with limited information.</p> This analogy came to mind while I was considering job market congestion that occurs when too many candidates apply for a role. Most of those candidates are not a good fit for the job, but they still try their luck since it requires little effort from them. As a solution, employers automate screening with AI. Candidates also use AI to scrape job descriptions and apply automatically (like "tinda finger"</a>). This creates a vicious cycle in which nobody wins.</p> One possible solution is a market design in which candidates pay to apply, with dynamic pricing. Paying to apply may sound provocative and require thoughtful consideration and careful testing. Payments can be made with platform-issued virtual points, available in limited supply. But here, I focus on why price signals may address this problem better than AI-based screening.</p> The job market is a matching market with information asymmetry, where both sides have unique preferences resolved only through costly interaction. Any AI/ML algorithm is only as good as the data, and that data is limited and often inaccurate. Requiring payment to apply would elicit a candidate's own estimate of their fit for the job and their true level of interest. These estimates would be reflected in prices, providing a visible signal that guides other candidates considering the role. This approach could reduce congestion, lower screening costs, and help form better matches.</p> Ladder of questions in analysis 2025-05-12T00:00:00+00:00 A ladder of questions in analysis:</p> Which question</em> should we ask to address a decision or a problem?</li> Which model</em> should we choose to answer the question?</li> Which assumptions</em> must hold for the model to be valid, and do they hold?</li> What data</em> do we need to apply the model?</li> What conclusions</em> can we draw after applying the model?</li> </ol> From my personal experience, people often skip steps 1 and 3, which can lead to bad decisions or mediocre solutions.</p> You probably read this in a statistical or data-analysis context. But I believe the framework applies more broadly.</p> In statistics, answering a wrong question is sometimes called a Type III error. I've already mentioned</a> it in a blog post.</p> Curious facts about Ronald Fisher 2024-08-12T00:00:00+00:00 Fisher and smoking #</a></h2> Ronald Fisher</a>, the father of modern statistics and experimental design, refused to believe that smoking tobacco caused lung cancer. He argued that correlation does not imply causation and even published a paper in Nature titled Cancer and Smoking</a>, defending this viewpoint.</p> Ronald Fisher enjoying a pipe, 1956. Source</a>.</p> </figcaption></figure> Fisher and Guinness #</a></h2> Ronald Fisher helped William Gosset</a> develop a statistical test known as Student's t-test</a>. They exchanged over 150 letters between 1912 and 1934. Gosset published his work under the pen name Student as he was the Head Experimental Brewer at Guinness</em> Brewery and was not allowed to publish under his real name.</p> The fact about Gosset is well known. Less well known is that Fisher's wife, Ruth Eileen Fisher née Guinness, was a great-great-granddaughter of Arthur Guinness, the founder of Guinness Brewery.</p> Fisher and a lady #</a></h2> Ronald Fisher devised the famous "lady tasting tea"</a> experiment. In this experiment, he developed the null hypothesis significance testing framework to analyze a lady's claim that she could discern whether the tea or the milk was added first to the cup.</p> The lady was Muriel Bristol</a>, a phycologist at Rothamsted Research. One day at Rothamsted, Fisher offered Bristol a cup of tea, but she declined, explaining that she preferred the flavor when the milk was poured into the cup before</em> the tea. Fisher said that the order of pouring could not affect the flavor. Bristol insisted that it did and that she could tell the difference. Overhearing this debate, William Roach said, "Let's test her." They devised an experiment, and Lady Bristol proved her case.</p> Fisher and tea-tasting #</a></h2> tea-tasting</a> is a Python package that I developed for the statistical analysis of A/B tests. The name "tea-tasting" is a play on words referring to two subjects:</p> The "lady tasting tea" experiment, devised by Ronald Fisher.</li> "Tea-tasting" phonetically resembling "t-testing" or Student's t-test, developed by William Gosset with the help of Ronald Fisher.</li> </ul> The funny part is that tea-tasting doesn't include Fisher's exact test</a>, which was used in the "lady tasting tea" experiment. It's like naming a coffee shop "Espresso Corner" and not serving espresso.</p> Do you think I should implement Fisher's exact test in tea-tasting? Or Boschloo's test</a>? Or Barnard's test</a>? Let me know in the comments below.</p> tea-tasting: a Python package for the statistical analysis of A/B tests 2024-07-29T00:00:00+00:00 Note: the post has been updated on March 7, 2026. It's often referred to by AI chatbots and agents without recognizing that some information could be outdated. So I decided to update it.</em></p> I developed tea-tasting</a>, a Python package for the statistical analysis of A/B tests featuring:</p> Welch's t-test, Student's t-test, z-test, Bootstrap, variance reduction with CUPED, delta method for ratio metrics, power analysis, and other A/B-test-specific statistical methods and approaches out of the box.</li> Support for a wide range of data backends, such as BigQuery, ClickHouse, PostgreSQL, Snowflake, Trino, and other backends supported by Ibis</a>. tea-tasting also accepts dataframes supported by Narwhals</a>: cuDF, Daft, Dask, DuckDB, Modin, pandas, Polars, PyArrow, PySpark.</li> Convenient API for reducing manual work, and a framework for minimizing errors. Extensible API: define custom metrics and use statistical tests of your choice.</li> Pretty representation of analysis results: rounding to significant digits, rendering in terminals, Jupyter/IPython, and marimo notebooks, serialization to Markdown, and conversion to pandas and Polars DataFrames.</li> Detailed documentation.</li> </ul> In this blog post, I explore each of these advantages of using tea-tasting in the analysis of experiments.</p> If you are eager to try it, check the user guide</a>.</p> Statistical methods #</a></h2> tea-tasting includes statistical methods and techniques that cover most of what you might need in the analysis of experiments.</p> Analyze metric averages with t-test or z-test, proportions with asymptotic or exact tests, ranks with the Mann-Whitney U test, or use Bootstrap to analyze any other statistic of your choice. There is also a predefined method for analyzing quantiles with Bootstrap. tea-tasting detects mismatches in the sample ratios of different variants of an A/B test. You can also define a custom metric</a> with a statistical test of your choice.</p> tea-tasting applies the delta method</a> for the analysis of ratios of averages. For example, the average number of orders per average number of sessions, assuming that sessions are not randomization units.</p> Use pre-experiment data, metric forecasts, or other covariates to reduce variance and increase the sensitivity of an experiment. This approach is also known as CUPED</a> or CUPAC</a>. In tea-tasting, it can be easily combined with the delta method for analyzing ratio metrics as well.</p> The calculation of confidence intervals for percentage</em> change in t-test and z-test can be tricky. Just taking the confidence interval for absolute</em> change and dividing it by the control average will produce a biased result. tea-tasting applies the delta method to calculate the correct interval.</p> Analyze statistical power</a> for t-test and z-test. There are three possible options:</p> Calculate the effect size, given statistical power and the total number of observations.</li> Calculate the total number of observations, given statistical power and the effect size.</li> Calculate statistical power, given the effect size and the total number of observations.</li> </ul> Other A/B-test-specific features of tea-tasting:</p> Procedures for controlling the multiple hypothesis testing</a> problem: False discovery rate (FDR): Benjamini-Hochberg and Benjamini-Yekutieli procedures.</li> Family-wise error rate (FWER): Hochberg's step-up and Holm's step-down procedures.</li> </ul> </li> Simulations</a>: A/A tests are useful for identifying potential issues before conducting the actual A/B test.</li> Treatment simulations are great for power analysis—especially when you need a specific uplift distribution or when an analytical formula doesn’t exist.</li> </ul> </li> </ul> Learn more in the detailed user guide</a>.</p> Data backends #</a></h2> There are many different databases and engines for storing and processing experimental data. And in most cases it's not efficient to pull the detailed experimental data into a Python environment. Many statistical tests, such as t-test or z-test, require only aggregated data for analysis.</p> For example, if the raw experimental data are stored in ClickHouse, it's faster and more efficient to calculate counts, averages, variances, and covariances directly in ClickHouse rather than fetching granular data and performing aggregations in a Python environment.</p> Querying all the required statistics manually can be a daunting and error-prone task. For example, analysis of ratio metrics and variance reduction with CUPED require not only the number of rows and variance, but also covariances. But don't worry: tea-tasting does all this work for you.</p> tea-tasting supports Ibis Tables as input data. Ibis</a> is a Python package which serves as a DataFrame API to various data backends. It supports many backends including BigQuery, ClickHouse, PostgreSQL, Snowflake, and Trino. You can write an SQL query, wrap</a> it as an Ibis Table, and pass it to tea-tasting.</p> Keep in mind that tea-tasting assumes that:</p> Data is grouped by randomization units, such as individual users.</li> There is a column indicating the variant of the A/B test (typically labeled as A, B, etc.).</li> All necessary columns for metric calculations (like the number of orders, revenue, etc.) are included in the table.</li> </ul> Some statistical methods, like Bootstrap, require granular data for the analysis. In this case, tea-tasting fetches the detailed data as well.</p> tea-tasting also accepts dataframes supported by Narwhals</a>: cuDF, Daft, Dask, DuckDB, Modin, pandas, Polars, PyArrow, PySpark.</p> Learn more in the guide on data backends</a>.</p> Convenient API #</a></h2> You can perform most of the statistical calculations above using just SciPy. In fact, tea-tasting uses it under the hood. What tea-tasting offers on top is a convenient higher-level API.</p> It's easier to show than to describe. Here is the basic example:</p> import</span> tea_tasting</span> as</span> tt</span></span> </span> data</span> =</span> tt</span>.</span>make_users_data</span>(</span>rng</span>=</span>42</span>)</span></span> experiment</span> =</span> tt</span>.</span>Experiment</span>(</span></span> sessions_per_user</span>=</span>tt</span>.</span>Mean</span>(</span>"</span>sessions</span>"</span>)</span>,</span></span> orders_per_session</span>=</span>tt</span>.</span>RatioOfMeans</span>(</span>"</span>orders</span>"</span>,</span> "</span>sessions</span>"</span>)</span>,</span></span> orders_per_user</span>=</span>tt</span>.</span>Mean</span>(</span>"</span>orders</span>"</span>)</span>,</span></span> revenue_per_user</span>=</span>tt</span>.</span>Mean</span>(</span>"</span>revenue</span>"</span>)</span>,</span></span> )</span></span> result</span> =</span> experiment</span>.</span>analyze</span>(</span>data</span>)</span></span> result</span></span></code></pre> Output:</p> metric control treatment rel_effect_size rel_effect_size_ci pvalue</span></span> sessions_per_user 2.00 1.98 -0.66% [-3.7%, 2.5%] 0.674</span></span> orders_per_session 0.266 0.289 8.8% [-0.89%, 19%] 0.0762</span></span> orders_per_user 0.530 0.573 8.0% [-2.0%, 19%] 0.118</span></span> revenue_per_user 5.24 5.73 9.3% [-2.4%, 22%] 0.123</span></span></code></pre> The two-stage approach, with separate parametrization and inference, is common in statistical modeling. This separation helps in making the code more modular and easier to understand.</p> tea-tasting performs calculations that can be tricky and error-prone:</p> Analysis of ratio metrics with delta method.</li> Variance reduction with CUPED/CUPAC (also in combination with the delta method for ratio metrics).</li> Calculation of confidence intervals for both absolute and percentage change.</li> Analysis of statistical power.</li> </ul> It also provides a framework for representing experimental data to avoid errors. Grouping the data by randomization units and including all units in the dataset is important for correct analysis.</p> Check out the comparison post</a> to see how tea-tasting differs from general purpose statistical packages like Pingouin, statsmodels, and SciPy.</p> Pretty representation and formatting #</a></h2> tea-tasting reduces the work to format and represent the analysis results. It rounds numbers to significant digits and renders the result table automatically based on where you display it:</p> In a terminal or Python console, the result is shown as a plain-text table (as in the example above).</li> In IPython and Jupyter, the result is rendered as an HTML table.</li> In marimo notebooks, the result is rendered as a table widget.</li> </ul> </a>Experiment result rendering in marimo</p> </figcaption></figure> In addition, tea-tasting provides methods to serialize the experiment result. Notable examples:</p> to_pandas</code>: Convert the result to a Pandas DataFrame.</li> to_polars</code>: Convert the result to a Polars DataFrame.</li> to_markdown</code>: Convert the result to a Markdown table.</li> to_string</code>: Convert the result to a string (with parameters for output customization).</li> </ul> Documentation #</a></h2> Last but not least: documentation. I believe that good documentation is crucial for tool adoption. That's why I wrote several user guides and an API reference.</p> I recommend starting with the example of basic usage in the user guide</a>. Then you can explore specific topics:</p> Data backends</a>.</li> Power analysis</a>.</li> Multiple hypothesis testing</a>.</li> Custom metrics</a>.</li> Simulated experiments</a>.</li> </ul> Use the API reference</a> to explore all parameters and detailed information about the functions, classes, and methods available in tea-tasting.</p> The tea-tasting repository includes examples</a> as copies of the guides in the marimo</a> notebook format. See the corresponding README</a> to learn how to run them.</p> Conclusions #</a></h2> There are a variety of statistical methods that can be applied in the analysis of an experiment. But only a handful of them are actually used in most cases.</p> On the other hand, there are methods specific to the analysis of A/B tests that are not included in the general purpose statistical packages like SciPy.</p> tea-tasting's functionality includes the most important statistical tests, as well as methods specific to the analysis of A/B tests.</p> tea-tasting provides a convenient API that helps to reduce the time spent on analysis and minimize the probability of error.</p> In addition, tea-tasting optimizes computational efficiency by calculating the statistics in the data backend of your choice, where the data are stored.</p> With the detailed documentation</a>, you can quickly learn how to use tea-tasting for the analysis of your experiments.</p> P.S. Package name #</a></h2> The package name "tea-tasting" is a play on words that refers to two subjects:</p> Lady tasting tea</a> is a famous experiment which was devised by Ronald Fisher. In this experiment, Fisher developed the null hypothesis significance testing framework to analyze a lady's claim that she could discern whether the tea or the milk was added first to the cup.</li> "tea-tasting" phonetically resembles "t-testing" or Student's t-test, a statistical test developed by William Gosset.</li> </ul> This meeting could have been 2024-06-05T00:00:00+00:00 an email</del> a Slack message</del> a PRD</del> a one-pager</del> nothing</p> Readme Driven Development for an A/B testing analysis package 2023-12-11T00:00:00+00:00 tea-tasting: statistical analysis of A/B tests #</a></h2> tea-tasting is a Python package for statistical analysis of A/B tests that features:</p> Student's t-test, Z-test, and Bootstrap out of the box.</li> Extensible API: Define and use statistical tests of your choice.</li> Delta method</a> for ratio metrics.</li> Variance reduction with CUPED</a>/CUPAC</a> (also in combination with delta method for ratio metrics).</li> Fieller's confidence interval</a> for percent change.</li> Sample ratio mismatch check.</li> Power analysis.</li> A/A tests.</li> </ul> Currently, tea-tasting is in the planning stage, and I'm starting with a README that outlines the proposed API — an approach known as Readme Driven Development (RDD).</p> In this blog post, I'll explain the motivation for creating this package and the benefits of the RDD approach.</p> Explore the README here</a>.</p> I welcome your insights! If you have any suggestions, questions, or ideas for alternative designs, please join the conversation</a> on GitHub.</p> Why a new package? #</a></h2> One might wonder why there's a need for a new package like tea-tasting when there are already SciPy and statsmodels available. Both of these are general-purpose statistical packages offering a wide range of statistical tests. However, they lack certain methods that are specific to A/B test analysis.</p> tea-tasting aims to fill this gap. For example, it includes methods such as the delta method for ratio metrics, CUPED/CUPAC for variance reduction, and methods for calculating confidence intervals for percentage changes.</p> The intention behind tea-tasting is not only to automate the analysis process but also to reduce the probability of making a mistake. Statistical analysis of A/B tests can be challenging, with many opportunities for errors. tea-tasting is designed to mitigate these risks.</p> Many companies develop in-house platforms for A/B testing, but these platforms often don't cover every scenario, such as non-standard randomization units used in cluster-level or switchback experiments, or the analysis of new metrics not yet integrated into the platform. tea-tasting is also useful in these cases.</p> In summary, tea-tasting is designed for a more specialized, error-minimizing approach to A/B testing analysis, filling in the gaps left by existing general-purpose statistical packages.</p> Package name #</a></h2> The package name "tea-tasting" is a play on words that refers to two subjects:</p> Lady tasting tea</a> is a famous experiment which was devised by Ronald Fisher. In this experiment, Fisher developed the null hypothesis significance testing framework to analyze a lady's claim that she could discern whether the tea or the milk was added first to a cup.</li> "tea-tasting" phonetically resembles "t-testing" or Student's t-test, a statistical test developed by William Gosset.</li> </ul> Lady tasting tea by DALL-E</p> </figcaption></figure>Readme Driven Development #</a></h2> Readme Driven Development (RDD) is a software development approach where you start by writing the README file first, before writing any code. The idea is similar to Amazon's Working Backwards</a> approach, where a press release is drafted before</em> starting a new product.</p> Writing a README encourages thinking from a user's perspective, focusing on how users will interact with the API. It allows for better project planning. You don't need to revise the code every time you decide to include something in the public API.</p> Don't confuse RDD with the Waterfall approach. It's not about creating exhaustive specifications or detailed documentation upfront. In fact, with tea-tasting, I might have gone a bit deeper into details than usual, but this was a conscious choice to thoroughly plan the package's functionality.</p> To learn more about RDD, I recommend reading Tom Preston-Werner's blog post</a>.</p> Request for feedback #</a></h2> One of the key benefits of Readme Driven Development (RDD) is the ability to adapt design choices with minimal cost. After reviewing the README</a>, you might have insights that could enhance this project. It could be anything from improved naming conventions and additional features to an entirely different API approach.</p> Your perspective is valuable! If the concept of this package interests you and you have ideas for its design, I encourage you to share your thoughts</a> on GitHub. Let's collaborate to make tea-tasting as effective and user-friendly as possible.</p> P.S. Can ChatGPT write a Python package? #</a></h2> Tom Preston-Werner wrote his blog post in 2010. There were no LLMs at that time. Thinking about RDD in 2023, I couldn't resist asking ChatGPT to write the package code based on README.</p> Well, not quite yet. ChatGPT resisted performing a large task at once. After I insisted, ChatGPT started implementing it class by class. In the first class, it started hallucinating about the SciPy API. I gave up shortly.</p> For those interested in how this experiment unfolded, you can view the entire conversation here: chat</a>.</p> Debunking the Dunning–Kruger effect with random number simulation 2023-09-25T00:00:00+00:00 Intro #</a></h2> Many people associate the Dunning–Kruger effect with the following image:</p> Misinterpretation of the Dunning–Kruger effect</p> </figcaption></figure> Actually, this image is a misinterpretation of the conclusions drawn by Dunning and Kruger from their experiments. The Dunning–Kruger effect is a cognitive bias in which people with limited competence in a particular domain overestimate their abilities.1</a></sup></p> Furthermore, evidence suggests that the Dunning and Kruger experiments might not definitively prove that the effect is real. It can be shown that it's a statistical artifact and the result of the way data is manipulated.2</a></sup> 3</a></sup> 4</a></sup> I'll illustrate this using a random number simulation, which offers an intuitive and straightforward understanding.</p> In this post, I'll:</p> Provide details of the experiments and analysis conducted by Dunning and Kruger.</li> Simulate similar results using random data that do not imply that the effect is real.</li> Explain why percentiles are not a suitable measure for these experiments.</li> </ul> I also created a Streamlit app</a> as a demo for this blog post. Feel free to play with it and check out the source code</a>.</p> Dunning and Kruger experiments5</a></sup> #</a></h2> Justin Kruger and David Dunning conducted several experiments (studies), each in one of the domains: humor, logical reasoning, and English grammar. In each experiment, participants—who were Cornell University students—took a test that assessed their knowledge in the domain. After the test, they were asked to assess their ability compared to other Cornell students. To be specific, participants were required to provide percentile rankings on a scale from 0 to 99. In two of the three studies, they were also asked to assess their test performance.</p> Dunning and Kruger performed the following analyses for each experiment:</p> Assigned participants percentile ranks based on their test score, with 0 being the lowest score and 99 being the highest score.</li> Calculated the average perceived ability percentile and found out that it exceeded the average test score percentile (50, by definition).</li> Grouped participants in quartiles based on their test score.</li> Calculated the average test score percentile and the average perceived ability percentile for each test score quartile (see the plot below).</li> </ul> Perceived ability to recognize humor as a function of actual test performance (Study 1)</p> </figcaption></figure> The plot above compares the average test score percentile and the average perceived ability percentile by test score quartile for the humor study. The plots for logical reasoning and English grammar experiments look similar. They also contain the average perceived test score percentiles that are quite close to the average perceived ability percentiles.</p> The main conclusions that were drawn:</p> Participants overall overestimated their abilities.</li> "Incompetent" participants grossly overestimated their ability relative to their peers.</li> </ol> The second conclusion is commonly referred to as the Dunning–Kruger effect. In the rest of the post, I will focus mostly on this conclusion and on the calculations that led to it.</p> Dunning and Kruger also conducted other experiments that studied participants' abilities to reassess their competence. I omit the details as they are not the main interest of this post.</p> Random number simulation #</a></h2> As previously mentioned, these experiments didn't prove the Dunning–Kruger effect. To demonstrate this, I'll:</p> Simulate the experiments with random data implying that the effect is not real.</li> Perform the same calculations as Dunning and Kruger.</li> </ul> First, let's generate test scores and perceived ability as correlated variables. Each point on the plot below represents a participant with a test score on the x-axis and a perceived ability on the y-axis.</p> Test score vs. perceived ability</p> </figcaption></figure> Note that the data generation process doesn't imply that "incompetent" participants overestimate their abilities.</p> Next, we'll compute the percentiles for participants' test scores and perceived abilities. This is not exactly how perceived ability percentiles were obtained in the Dunning and Kruger experiment. But this is still valid for our purpose.</p> Test score percentile vs. perceived ability percentile</p> </figcaption></figure> Finally, let's calculate the average test score percentile and the average perceived ability percentile by test score quartiles.</p> Average percentiles by test score quartiles</p> </figcaption></figure> For the bottom quartile, the average perceived ability percentile is higher than the average test score percentile. Feel free to try different values for the random seed, correlation coefficient (below 1), or number of participants in the Streamlit app</a>.</p> You will always get this result for two variables with imperfect correlation. But Dunning and Kruger used this result as evidence that "incompetent" participants overestimated their abilities. They were wrong, as this result is a statistical artifact and the outcome of the way data is manipulated.</p> Additionally, let's use the same data to calculate the average test score percentile and the average perceived ability percentile by perceived ability</em> quartiles.</p> Average percentiles by perceived ability quartiles</p> </figcaption></figure> Participants from the bottom quartile, on average, have the lowest test scores. For them, the average perceived ability percentile is also lower than the average test score percentile. Doesn't this contradict the Dunning–Kruger effect?</p> Bias and percentile measure #</a></h2> In fact, Dunning and Kruger did discuss the regression effect due to imperfect correlation between actual and perceived ability. But they argued that the overestimation they observed was more psychological than artifactual. The main argument was that the magnitude of miscalibration among the bottom quartile was larger than that of the top quartile.5</a></sup></p> As you may have noticed, in the simulation, participants overall don't overestimate their abilities. This is different from the Dunning and Kruger experiment. I don't think it's possible to generate a dataset with average perceived ability percentile higher than the average test score percentile without implying that "incompetent" participants grossly overestimate their abilities. The reason is that percentile is a measure with bounded support, meaning it has a finite minimum and maximum.</p> Let's take a participant with a test score which corresponds to the 99th percentile. They cannot overestimate their abilities in percentiles. Participants from the top test score quartile can overestimate their abilities by 24 percentiles maximum and 12 percentiles on average. Participants from the bottom test score quartile can overestimate their abilities by 99 percentiles maximum and 87 percentiles on average.</p> If participants overall overestimate their abilities, then, with high probability, the magnitude of overestimation will be larger among participants with low test scores. And this is a pure mathematical artifact, not a psychological effect. This reasoning applies to any measure with bounded support, not just percentiles. For example, we would get the same results with test scores as a measure.</p> So, the actual reason for the Dunning and Kruger observation is a combination of two factors:</p> Participants overall overestimated their abilities.</li> Percentile is a measure with bounded support.</li> </ol> Also, I encourage you to think about the question: if a person scores the maximum number of points in a test, does this mean that they cannot overestimate their abilities?</p> Conclusion #</a></h2> Justin Kruger and David Dunning conducted several experiments and concluded that "incompetent" participants grossly overestimated their ability relative to their peers. This effect is known as the Dunning–Kruger effect.</p> I reproduced similar results using random number simulation implying that the effect is not real. If participants overall overestimate their abilities, the magnitude of the overestimation among the participants with low scores is explained by the fact that percentile is a measure with bounded support.</p> The Dunning–Kruger effect, as observed in the original experiments, is a statistical artifact rather than a psychological phenomenon.</p> I don't argue about whether the Dunning–Kruger effect is real or not. My intent is to show that the original Dunning and Kruger experiments don't prove it.</p> I created a Streamlit app</a> as a demo for this blog post. Feel free to play with it and check out the source code</a>.</p> References #</a></h2> ^1</sup>Dunning–Kruger effect – Wikipedia</a>.</p> </div> ^2</sup>Nuhfer, Edward, Christopher Cogan, Steven Fleisher, Eric Gaze, and Karl Wirth. (2016). Random Number Simulations Reveal How Random Noise Affects the Measurements and Graphical Portrayals of Self-Assessed Competency. Numeracy 9, Iss. 1: Article 4. DOI: http://dx.doi.org/10.5038/1936-4660.9.1.4</a>.</p> </div> ^3</sup>Nuhfer, Edward, Steven Fleisher, Christopher Cogan, Karl Wirth, and Eric Gaze. (2017). How Random Noise and a Graphical Convention Subverted Behavioral Scientists' Explanations of Self-Assessment Data: Numeracy Underlies Better Alternatives. Numeracy 10, Iss. 1 : Article 4. DOI: http://dx.doi.org/10.5038/1936-4660.10.1.4</a>.</p> </div> ^4</sup>Jarry, Jonathan. (2020). The Dunning-Kruger Effect Is Probably Not Real. McGill University, Office for Science and Society. Article: link</a>.</p> </div> ^5</sup>Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134. DOI: https://doi.org/10.1037/0022-3514.77.6.1121</a>. Open access: link</a>.</p> </div> Ranking in a two-sided matching platform (Part 2) 2023-08-28T00:00:00+00:00 Recap #</a></h2> This is the second post about the ranking problem in Profi, a services marketplace where customers can find tutors, beauty masters, plumbers, and other professionals, and professionals can find customers.</p> Here is the short recap of the previous post</a>:</p> In Profi, professionals search, choose orders, and send proposals to customers. And then the customers review the proposals and choose the professionals, or don't choose anyone. If a professional sends a proposal to a customer, and the customer chooses the professional then this is a deal.</li> The objective was to increase the number of deals by implementing a new order ranking algorithm in the app for professionals.</li> Ranking orders by the probability of a proposal did increase the number of proposals, but didn't increase the number of deals in the marketplace.</li> Ranking orders by the uplift in the probability of a deal significantly increased the number of deals in the marketplace.</li> </ul> Uplift in the probability of a deal</p> </figcaption></figure> In the second post, I'll explain the solution. This post is more mathematical. Please bear with me while I go through some details of the model.</p> Mathematical formulation #</a></h2> First, let's express the uplift in the probability of a deal in a mathematical form:</p> $$P(Y_i = 1 \mid S_{ij} = 1) - P(Y_i = 1 \mid S_{ij} = 0)$$</p> $Y_i$ is an outcome for the order $i$. If there is a deal, then $Y_i = 1$, otherwise $Y_i = 0$.</li> $S_{ij}$ indicates whether the professional $j$ has seen the order $i$ or not.</li> $P(Y_i = 1 \mid S_{ij} = 1)$ is the probability of a deal in the order $i$, given that the professional $j$ has seen the order.</li> $P(Y_i = 1 \mid S_{ij} = 0)$ is the probability of a deal in the order $i$, given that the professional $j$ hasn't seen the order.</li> </ul> It seems impossible to solve this problem directly. Hundreds of professionals should see the order for a deal to happen. Estimating the effect of showing an order to a professional would be very noisy.</p> Let's decompose the problem by expanding the expression using the law of total probability:</p> $$P(T_{ij} = 1 \mid S_{ij} = 1)(P(Y_i = 1 \mid T_{ij} = 1) - P(Y_i = 1 \mid T_{ij} = 0))$$</p> $T_{ij}$ indicates whether the professional $j$ has sent a proposal to the customer $i$ or not.</li> $P(T_{ij} = 1 \mid S_{ij} = 1)$ is the probability that the professional $j$ will send a proposal to the customer $i$, given that the professional has seen the order.</li> $P(Y_i = 1 \mid T_{ij} = 1)$ is the probability of a deal in the order $i$, given that the professional $j$ has sent a proposal to the customer.</li> $P(Y_i = 1 \mid T_{ij} = 0)$ is the probability of a deal in the order $i$, given that the professional $j$ hasn't sent a proposal to the customer.</li> </ul> Derivation Let's expand the left and the right part of the uplift expression:</p> $$ \small \begin{split} P(Y_i = 1 \mid S_{ij} = 1) & = P(Y_i = 1 \mid T_{ij} = 1)P(T_{ij} = 1 \mid S_{ij} = 1) \\ & \quad + P(Y_i = 1 \mid T_{ij} = 0)P(T_{ij} = 0 \mid S_{ij} = 1) \\ & = P(Y_i = 1 \mid T_{ij} = 1)P(T_{ij} = 1 \mid S_{ij} = 1) \\ & \quad + P(Y_i = 1 \mid T_{ij} = 0)(1 - P(T_{ij} = 1 \mid S_{ij} = 1)) \\ & = P(T_{ij} = 1 \mid S_{ij} = 1)(P(Y_i = 1 \mid T_{ij} = 1) - P(Y_i = 1 \mid T_{ij} = 0)) \\ & \quad + P(Y_i = 1 \mid T_{ij} = 0) \\ \\ P(Y_i = 1 \mid S_{ij} = 0) & = P(Y_i = 1 \mid T_{ij} = 1)P(T_{ij} = 1 \mid S_{ij} = 0) \\ & \quad + P(Y_i = 1 \mid T_{ij} = 0)P(T_{ij} = 0 \mid S_{ij} = 0) \\ & = P(Y_i = 1 \mid T_{ij} = 0) \\ \end{split} $$ </p> The last equation takes into account that a professional cannot send a proposal without seeing the order, i.e.:</p> $$ \small \begin{split} & P(T_{ij} = 1 \mid S_{ij} = 0) = 0 \\ & P(T_{ij} = 0 \mid S_{ij} = 0) = 1 \end{split} $$ </p> Now, we can derive that:</p> $$ \small \begin{split} & P(Y_i = 1 \mid S_{ij} = 1) - P(Y_i = 1 \mid S_{ij} = 0) \\ = P(T_{ij} = 1 \mid S_{ij} = 1)(& P(Y_i = 1 \mid T_{ij} = 1) - P(Y_i = 1 \mid T_{ij} = 0)) \end{split} $$ </p> </div> </div> </div> </div> We already have a model</a> that predicts $P(T_{ij} = 1 \mid S_{ij} = 1)$, the probability that the professional $j$ will send a proposal to the customer $i$.</p> So, now the problem reduces to predicting the uplift in the probability of a deal caused by a proposal from the professional:</p> $$P(Y_i = 1 \mid T_{ij} = 1) - P(Y_i = 1 \mid T_{ij} = 0)$$</p> To solve the problem, we'll create a model for the probability of a deal in the order. The difference between the probabilities of a deal with and without a proposal from the professional is the uplift we want to predict.</p> Discrete choice model #</a></h2> A customer can choose one of the professionals who sent a proposal, or not choose anyone.</p> Let's model customer's choice with the softmax function:</p> $$P(Y_{ik}=1) = \frac{e^{U_{ik}}}{e^{U_{i0}} + \sum_{j \in J_i} e^{U_{ij}}} = \frac{e^{U_{ik}-U_{i0}}}{1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}}}$$</p> $Y_{ik}$ indicates whether the customer $i$ chooses the professional $k$ or not.</li> $P(Y_{ik}=1)$ is the probability that the customer $i$ will choose the professional $k$.</li> $U_{ij}$ (or $U_{ik}$) is the utility the customer $i$ obtains from choosing the professional $j$ (or $k$).</li> $U_{i0}$ is the utility the customer $i$ obtains from not choosing anyone.</li> $J_i$ is a set of professionals who sent a proposal to the customer $i$.</li> </ul> This model assumes no correlation among alternatives. This might not be 100% true. But we decided to proceed with this model:</p> Given that customers and professionals have unique preferences that only they can discover, this assumption seems realistic.</li> Taking into account the sequential number of the proposal in the order can capture correlations between proposals.</li> </ul> Now, we can express the probability that the customer will choose one of the professionals who sent a proposal:</p> $$P(Y_i=1) = \sum_{j \in J_i} P(Y_{ij}=1) = \frac{\sum_{j \in J_i} e^{U_{ij}-U_{i0}}}{1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}}}$$</p> Suppose the customer $i$ has received proposals from a set of professionals $J_i$. Then the uplift in the probability of a deal caused by a proposal from the professional $k$ will be:</p> $$ \begin{split} & P(Y_i = 1 \mid T_{ik} = 1) - P(Y_i = 1 \mid T_{ik} = 0) \\ & = \frac{\sum_{j \in J_i} e^{U_{ij}-U_{i0}} + e^{U_{ik}-U_{i0}}}{1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}} + e^{U_{ik}-U_{i0}}} - \frac{\sum_{j \in J_i} e^{U_{ij}-U_{i0}}}{1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}}} \\ & = \frac{e^{U_{ik}-U_{i0}}}{(1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}})(1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}} + e^{U_{ik}-U_{i0}})} \end{split} $$ </p> If we can model the utility, we can estimate the uplift in the probability of a deal.</p> Utility function #</a></h2> The utility a customer obtains from choosing a professional depends on:</p> How much time has passed from the moment of the order to the proposal from the professional.</li> The rating and the number of reviews in the professional's profile.</li> When the customer was last online.</li> And other factors.</li> </ul> If we represent the utility as a function of these variables, then the parameters of the function can be estimated by minimizing the cross-entropy (maximum likelihood estimation):</p> $$ \begin{split} H(Y, P(Y)) & = -\sum_i (Y_i \log P(Y_i = 1) + (1 - Y_i) \log P(Y_i = 0)) \\ & = \sum_i (\log (1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}}) - Y_i \log (\sum_{j \in J_i} e^{U_{ij}-U_{i0}})) \end{split} $$ </p> We started with a basic model that accounts only for how much time has passed from the moment of the order to the proposal from the professional. And this basic model helped us significantly increase the number of deals.</p> Combined target #</a></h2> Let's review the components of the successful solution:</p> A model $f(X_1)$ that takes a feature set $X_1$ and predicts the probability that the professional will send a proposal to the customer.</li> A model $g(X_2)$ that takes a feature set $X_2$ and predicts the uplift in the probability of a deal in the order caused by a proposal from the professional.</li> Orders are sorted by the product of these models' predictions $f(X_1) g(X_2)$.</li> </ul> This solution has its disadvantages:</p> Using two models in production increases the complexity of the solution.</li> The model $f(X_1)$ should predict the probability. This limits how the model can be improved. For example, the loss function is the cross-entropy. We cannot use loss functions that are more suitable for the learning to rank problem.</li> </ul> These disadvantages can be resolved by training a model on a combined target. Instead of training a model $f(X_1)$ on an indicator of a proposal $y$, we can train a model $h(X_1, X_2)$ on a combined target $y g(X_2)$. This solution resulted in a further increase in the number of deals.</p> Conclusions #</a></h2> In two posts I explored the ranking problem in a services marketplace. It's a two-sided matching platform where both sides have to choose each other for a deal to happen. I explained the solution that significantly increased the number of deals in the marketplace.</p> The right problem formulation is more important than a sophisticated solution. When one side searches and proposes, and the other side chooses among the proposals, it's not sufficient to maximize the probability of a proposal. Ranking by the uplift in the probability of a deal increased the number of deals in the marketplace.</p> When two sides choose, it makes sense to model two processes separately:</p> Probability of a proposal from the side that chooses first.</li> Uplift in the probability of a match caused by a proposal.</li> </ul> A discrete choice model can be used to estimate the uplift in the probability of a match.</p> If one of the models should predict the probability measure, it limits the model improvements. Training a single model on a combined target can resolve this issue.</p> Ranking in a two-sided matching platform (Part 1) 2023-08-15T00:00:00+00:00 Overview #</a></h2> Profi is a two-sided matching platform where customers can find tutors, beauty masters, plumbers, and other professionals, and professionals can find customers. It's a horizontal marketplace with over 900 types of services provided by professionals.</p> The quality of search and ranking is important for the matching process there. In a two-sided matching platform, both sides have to choose each other for a deal to happen, and that makes the ranking problem harder and kind of unique.</p> In two blog posts, I'll explore the ranking problem and explain the solution that significantly increased the number of deals in the marketplace.</p> In this first post, I will cover:</p> The concept of matching markets and their distinctions from other markets.</li> The mechanics of Profi's matching process.</li> Business objectives.</li> The three versions of the problem formulation, with insights into why certain approaches succeeded while others did not.</li> </ul> In the second post</a>, I will cover the details of the successful solution that was implemented in production.</p> Matching markets #</a></h2> When a customer places an order on Amazon, the seller doesn't get to choose whether or not to sell to this customer. And this is an example of a commodity market.</p> In matching markets, you can't just choose and get what you want—you also have to be chosen. Examples of matching markets and platforms:</p> Job market.</li> Airbnb before introducing the instant booking feature.</li> Dating platforms, like Tinder and Bumble.</li> Platforms for freelancers, like Upwork and Fiverr.</li> </ul> While in commodity markets prices do most of the work and balance supply and demand, in matching markets, price is not the most important factor. Market design, search, and ranking algorithms play a crucial role there.</p> To learn more about matching markets read "Who Gets What―and Why: The New Economics of Matchmaking and Market Design"</a> by Alvin Roth, a winner of the Nobel Prize in Economics.</p> Matching process #</a></h2> The process of how customers and professionals can find each other on Profi:</p> Customers place orders and describe the details of the work.</li> Professionals search, choose orders, and send proposals to the customers.</li> The customers review the proposals and choose the professionals, or don't choose anyone.</li> If a professional sends a proposal to a customer, and the customer chooses the professional, then this is a deal.</li> </ul> Professionals choose orders and send proposals</p> </figcaption></figure>Customers review proposals and choose professionals</p> </figcaption></figure>Business objective #</a></h2> The order feed in the app for professionals is the source of 56% of proposals. Orders in the feed are selected based on a professional's preferences and previous activity. Originally, orders were sorted in reverse chronological order.</p> While order "freshness" is important for professionals, it's not the only factor of choice. The hypothesis was that we could improve the matching process by implementing a new order ranking algorithm.</p> The objective was to increase the number of deals, which is one of the two most important metrics in the company.</p> Problem formulation #</a></h2> Probability of a proposal #</a></h3> The first idea was to sort orders by the probability of a proposal. The intuition was the following:</p> Sorting by the probability of a proposal would help professionals to find more relevant orders.</li> Professionals would send more proposals.</li> Customers would receive more proposals and have more choice.</li> A wider choice of proposals would increase the probability of finding the right professional.</li> The number of deals would increase.</li> </ul> Predicting the probability of a proposal</p> </figcaption></figure> We trained a machine learning model with the following target and conducted an experiment:</p> Proposal</th> Deal</th> Target</th></tr></thead> Yes</td> Yes</td> 1</td></tr> Yes</td> No</td> 1</td></tr> No</td> No</td> 0</td></tr> </tbody></table> Predicting the probability of a proposal</figcaption></figure> The experiment resulted in the following:</p> The number of proposals had indeed increased.</li> But the number of deals had not increased.</li> </ul> The interpretation was that extra proposals were sent to customers who already had enough proposals to choose from.</p> Increasing the number of proposals without increasing the number of deals is not good for either professionals or customers:</p> Professionals make an extra effort to write and send proposals, but they don't get more customers.</li> Customers are distracted by additional proposals while they already have enough to choose from.</li> </ul> Probability of a deal #</a></h3> The next intuitive choice was to sort orders by the probability of a deal, and to train a model with the following target:</p> Proposal</th> Deal</th> Target</th></tr></thead> Yes</td> Yes</td> 1</td></tr> Yes</td> No</td> 0</td></tr> No</td> No</td> 0</td></tr> </tbody></table> Predicting the probability of a deal</figcaption></figure> To be precise, for a pair of a professional and a customer (P, C)</em>, a model should have predicted the probability that:</p> The professional P</em> would send a proposal to the customer C</em>.</li> And the customer C</em> would choose the professional P</em> among others.</li> </ul> Predicting the probability of a deal</p> </figcaption></figure> But, if you think carefully, you can see that this is not the right problem formulation in this case:</p> First, imagine that the first professional on the picture above has sent a proposal to the first customer, and that this customer has chosen this professional.</li> Then the second professional wouldn't get a deal with the first customer. Does that mean that the second professional shouldn't have sent the proposal?</li> Now, imagine that the first professional hasn't sent a proposal. Then the proposal from the second professional could be valuable.</li> </ul> A customer can choose another professional</p> </figcaption></figure> We decided not to move forward with this formulation of the problem as it doesn't correctly reflect the matching process.</p> Uplift in the probability of a deal #</a></h3> The last idea was to estimate how showing an order to a professional would change the probability of a deal in that order, no matter with which professional. The model should calculate a difference in the probability of a deal in an order between two parallel universes:</p> The one where we show the order to the professional.</li> The one where we don't show the order to the professional.</li> </ul> Models that compare parallel universes are called uplift models.</p> Uplift in the probability of a deal</p> </figcaption></figure> Ranking orders by the uplift in the probability of a deal showed the following results in the experiment (measured with proxy metrics):</p> The number of deals coming from the order feed had increased by 10%.</li> The total number of deals had increased by 5%.</li> </ul> It was a great result in itself. But it also opened a door for further improvements, not only in the ranking of orders. For example, finding more relevant orders and adding them to the order feed automatically increases the number of deals.</p> I will get into some details of the model in the next post</a>. Stay tuned.</p> P.S. Type III error #</a></h2> This case reminded me of a notion of a type III error. Before I explain what it means, let me give you examples of type I and type II errors in the context of the ranking of orders:</p> Imagine we show an order to a professional and expect the professional to send a proposal. But the professional doesn't send a proposal. That's a type I error.</li> Now imagine we don't show an order to a professional because we don't expect the professional to send a proposal. But if we showed it, the professional would send a proposal. That's a type II error.</li> </ul> Well, whether the professional would send a proposal or not is not the right question in the first place, at least not the final one. And that's a type III error</a>.</p> Also, check out Cassie Kozyrkov's excellent explanation: https://youtu.be/R51nueuAyIY</a>.</p> rico: rich content to HTML as easy as Doc(df, plot) 2023-08-10T00:00:00+00:00 What is rico? #</a></h2> rico</a> is a Python package for creating HTML documents from rich content: dataframes, plots, images, markdown, etc. It provides a high-level, easy-to-use API with reasonable defaults, as well as low-level access for better control.</p> Why rico? #</a></h2> One might wonder why they need another package if they can create HTML documents with Jupyter and nbconvert. If you are working with data in Jupyter notebooks, you probably don't need rico. But if you are working with data in Python scripts, rico can be helpful.</p> Here is my use case. I'm working with data in Python scripts and organizing them into data pipelines using Prefect. Sometimes I want to create a report as the last step of a pipeline. With Jupyter I would have the following routine:</p> Create a task to save all the dataframes I need for my report in Parquet files or in a database.</li> Create a Jupyter notebook to load the dataframes and visualize all the tables and plots I need.</li> Create a task to convert the notebook to HTML using nbconvert.</li> </ul> Creating an extra task and an extra notebook is not what bothers me the most. I have to think about how to pass data to a notebook. Sure, it's not a very hard job, but it's not a job I want to spend my time on.</p> With rico, I simply create a task to add all objects I want to visualize to a document and save it to an HTML file.</p> How to use rico? #</a></h2> Installation #</a></h3> pip</span> install</span> rico</span></span></code></pre> rico has no dependencies other than standard Python packages.</p> For Markdown support install markdown-it-py</a> or Python Markdown</a> or set your own Markdown renderer using rico.set_config</code>.</p> Basic usage #</a></h3> rico provides both declarative and imperative interfaces.</p> Declarative style:</p> import</span> pandas</span> as</span> pd</span></span> import</span> rico</span></span> </span> df</span> =</span> pd</span>.</span>DataFrame</span>(</span></span> {</span></span> "</span>x</span>"</span>:</span> [</span>2</span>,</span> 7</span>,</span> 4</span>,</span> 1</span>,</span> 2</span>,</span> 6</span>,</span> 8</span>,</span> 4</span>,</span> 7</span>]</span>,</span></span> "</span>y</span>"</span>:</span> [</span>1</span>,</span> 9</span>,</span> 2</span>,</span> 8</span>,</span> 3</span>,</span> 7</span>,</span> 4</span>,</span> 6</span>,</span> 5</span>]</span>,</span></span> }</span>,</span></span> index</span>=</span>pd</span>.</span>Index</span>(</span>list</span>(</span>"</span>AAABBBCCC</span>"</span>)</span>)</span>,</span></span> )</span></span> plot</span> =</span> df</span>.</span>plot</span>.</span>scatter</span>(</span>x</span>=</span>"</span>x</span>"</span>,</span> y</span>=</span>"</span>y</span>"</span>)</span></span> </span> doc</span> =</span> rico</span>.</span>Doc</span>(</span>"</span>Hello, World!</span>"</span>,</span> df</span>,</span> plot</span>,</span> title</span>=</span>"</span>My doc</span>"</span>)</span></span></code></pre> The result:</p> </a>Click on image to zoom in.</p> </figcaption></figure> Imperative style:</p> doc</span> =</span> rico</span>.</span>Doc</span>(</span>title</span>=</span>"</span>My doc</span>"</span>)</span></span> doc</span>.</span>append</span>(</span>"</span>Hello, World!</span>"</span>)</span></span> doc</span>.</span>append</span>(</span>df</span>,</span> plot</span>)</span></span></code></pre> The result will look the same.</p> Serialize the document to HTML using str(doc)</code>:</p> with</span> open</span>(</span>"</span>doc.html</span>"</span>,</span> "</span>w</span>"</span>)</span> as</span> f</span>:</span></span> f</span>.</span>write</span>(</span>str</span>(</span>doc</span>)</span>)</span></span></code></pre> Also, you can call doc.serialize(indent=True)</code> to indent the HTML element tree. For brevity, I will omit the serialization and saving to file in the following examples.</p> Content types #</a></h3> rico automatically recognizes the following content types:</p> rico content classes (subclasses of rico.ContentBase</code>).</li> Matplotlib Pyplot Plots.</li> Dataframes, Seaborn plots, Altair charts and other types with IPython rich representation methods</a>. Basically, rico supports the same types as Jupyter.</li> Text.</li> </ul> Use specific classes for plots and text to change the default behavior:</p> doc</span> =</span> rico</span>.</span>Doc</span>(</span></span> rico</span>.</span>Text</span>(</span>"</span>Hello, World!</span>"</span>,</span> mono</span>=</span>True</span>)</span>,</span> #</span> The default value is False.</span></span> df</span>,</span></span> rico</span>.</span>Plot</span>(</span>plot</span>,</span> format</span>=</span>"</span>png</span>"</span>,</span> bbox_inches</span>=</span>"</span>tight</span>"</span>)</span>,</span> #</span> The default value is "svg".</span></span> title</span>=</span>"</span>My doc</span>"</span>,</span></span> )</span></span></code></pre> The following code gives the same result as the code above:</p> doc</span> =</span> rico</span>.</span>Doc</span>(</span>title</span>=</span>"</span>My doc</span>"</span>)</span></span> doc</span>.</span>append_text</span>(</span>"</span>Hello, World!</span>"</span>,</span> mono</span>=</span>True</span>)</span></span> doc</span>.</span>append</span>(</span>df</span>)</span></span> doc</span>.</span>append_plot</span>(</span>plot</span>,</span> format</span>=</span>"</span>png</span>"</span>,</span> bbox_inches</span>=</span>"</span>tight</span>"</span>)</span></span></code></pre> Use specific classes and methods for other content types:</p> Images: Image</code> or Doc.append_image</code>.</li> Code: Code</code> or Doc.append_code</code>.</li> Markdown: Markdown</code> or Doc.append_markdown</code>.</li> HTML tag: Tag</code> or Doc.append_tag</code>.</li> Raw HTML: HTML</code> or Doc.append_html</code>.</li> </ul> For example:</p> doc</span> =</span> rico</span>.</span>Doc</span>(</span></span> rico</span>.</span>Markdown</span>(</span>"</span>## Dataframe</span>"</span>)</span>,</span></span> df</span>,</span></span> rico</span>.</span>Tag</span>(</span>"</span>h2</span>"</span>,</span> "</span>Plot</span>"</span>)</span>,</span> #</span> An alternative way to add a header.</span></span> plot</span>,</span></span> rico</span>.</span>HTML</span>(</span>"</span><h2>Code</h2></span>"</span>)</span>,</span> #</span> Another way to add a header.</span></span> rico</span>.</span>Code</span>(</span>"</span>print('Hello, World!')</span>"</span>)</span>,</span></span> title</span>=</span>"</span>My doc</span>"</span>,</span></span> )</span></span></code></pre> The result:</p> </a>Click on image to zoom in.</p> </figcaption></figure> The following code gives the same result as the code above:</p> doc</span> =</span> rico</span>.</span>Doc</span>(</span>title</span>=</span>"</span>My doc</span>"</span>)</span></span> doc</span>.</span>append_markdown</span>(</span>"</span>## Dataframe</span>"</span>)</span></span> doc</span>.</span>append</span>(</span>df</span>)</span></span> doc</span>.</span>append_tag</span>(</span>"</span>h2</span>"</span>,</span> "</span>Plot</span>"</span>)</span></span> doc</span>.</span>append</span>(</span>plot</span>)</span></span> doc</span>.</span>append_html</span>(</span>"</span><h2>Code</h2></span>"</span>)</span></span> doc</span>.</span>append_code</span>(</span>"</span>print('Hello, World!')</span>"</span>)</span></span></code></pre>Layout #</a></h3> rico relies on Bootstrap</a> styles. The resulting documents are responsive and mobile-friendly. Use Bootstrap classes to control document layout. For example:</p> import</span> altair</span> as</span> alt</span></span> </span> doc</span> =</span> rico</span>.</span>Doc</span>(</span></span> rico</span>.</span>Tag</span>(</span>"</span>h2</span>"</span>,</span> "</span>Dataframes</span>"</span>)</span>,</span></span> rico</span>.</span>Div</span>(</span></span> rico</span>.</span>Obj</span>(</span>rico</span>.</span>Tag</span>(</span>"</span>h3</span>"</span>,</span> "</span>A</span>"</span>)</span>,</span> df</span>.</span>loc</span>[</span>"</span>A</span>"</span>,</span> :</span>]</span>,</span> class_</span>=</span>"</span>col</span>"</span>)</span>,</span></span> rico</span>.</span>Obj</span>(</span>rico</span>.</span>Tag</span>(</span>"</span>h3</span>"</span>,</span> "</span>B</span>"</span>)</span>,</span> df</span>.</span>loc</span>[</span>"</span>B</span>"</span>,</span> :</span>]</span>,</span> class_</span>=</span>"</span>col</span>"</span>)</span>,</span></span> rico</span>.</span>Obj</span>(</span>rico</span>.</span>Tag</span>(</span>"</span>h3</span>"</span>,</span> "</span>C</span>"</span>)</span>,</span> df</span>.</span>loc</span>[</span>"</span>C</span>"</span>,</span> :</span>]</span>,</span> class_</span>=</span>"</span>col</span>"</span>)</span>,</span></span> class_</span>=</span>"</span>row row-cols-auto</span>"</span>,</span></span> )</span>,</span></span> rico</span>.</span>Tag</span>(</span>"</span>h2</span>"</span>,</span> "</span>Plots</span>"</span>)</span>,</span></span> rico</span>.</span>Div</span>(</span></span> rico</span>.</span>Obj</span>(</span></span> rico</span>.</span>Tag</span>(</span>"</span>h3</span>"</span>,</span> "</span>A</span>"</span>)</span>,</span></span> alt</span>.</span>Chart</span>(</span>df</span>.</span>loc</span>[</span>"</span>A</span>"</span>,</span> :</span>]</span>)</span>.</span>mark_point</span>(</span>)</span>.</span>encode</span>(</span>x</span>=</span>"</span>x</span>"</span>,</span> y</span>=</span>"</span>y</span>"</span>)</span>,</span></span> class_</span>=</span>"</span>col</span>"</span>,</span></span> )</span>,</span></span> rico</span>.</span>Obj</span>(</span></span> rico</span>.</span>Tag</span>(</span>"</span>h3</span>"</span>,</span> "</span>B</span>"</span>)</span>,</span></span> alt</span>.</span>Chart</span>(</span>df</span>.</span>loc</span>[</span>"</span>B</span>"</span>,</span> :</span>]</span>)</span>.</span>mark_point</span>(</span>)</span>.</span>encode</span>(</span>x</span>=</span>"</span>x</span>"</span>,</span> y</span>=</span>"</span>y</span>"</span>)</span>,</span></span> class_</span>=</span>"</span>col</span>"</span>,</span></span> )</span>,</span></span> rico</span>.</span>Obj</span>(</span></span> rico</span>.</span>Tag</span>(</span>"</span>h3</span>"</span>,</span> "</span>C</span>"</span>)</span>,</span></span> alt</span>.</span>Chart</span>(</span>df</span>.</span>loc</span>[</span>"</span>C</span>"</span>,</span> :</span>]</span>)</span>.</span>mark_point</span>(</span>)</span>.</span>encode</span>(</span>x</span>=</span>"</span>x</span>"</span>,</span> y</span>=</span>"</span>y</span>"</span>)</span>,</span></span> class_</span>=</span>"</span>col</span>"</span>,</span></span> )</span>,</span></span> class_</span>=</span>"</span>row row-cols-auto</span>"</span>,</span></span> )</span>,</span></span> title</span>=</span>"</span>Grid system</span>"</span>,</span></span> )</span></span></code></pre> The result:</p> </a>Click on image to zoom in.</p> </figcaption></figure> How it works:</p> rico.Div</code> represents an HTML container (<div></code>). You can add objects to it, just like with rico.Doc</code>.</li> Each content element is also wrapped in a <div></code> container.</li> rico.Obj</code> is a magic class that automatically determines the content type, just like rico.Doc</code> does.</li> You can specify an HTML class</code> attribute of a <div></code> container for rico.Div</code> and for any content element such as rico.Obj</code>.</li> The "row"</code>, "row-cols-auto"</code>, "col"</code> classes are part of the Bootstrap Grid System. Read more in the documentation</a>.</li> </ul> The following code gives the same result as the code above:</p> doc</span> =</span> rico</span>.</span>Doc</span>(</span>title</span>=</span>"</span>Grid system</span>"</span>)</span></span> </span> doc</span>.</span>append_tag</span>(</span>"</span>h2</span>"</span>,</span> "</span>Dataframes</span>"</span>)</span></span> div1</span> =</span> rico</span>.</span>Div</span>(</span>class_</span>=</span>"</span>row row-cols-auto</span>"</span>)</span></span> doc</span>.</span>append</span>(</span>div1</span>)</span></span> for</span> name</span>,</span> data</span> in</span> df</span>.</span>groupby</span>(</span>df</span>.</span>index</span>)</span>:</span></span> div1</span>.</span>append</span>(</span>rico</span>.</span>Tag</span>(</span>"</span>h3</span>"</span>,</span> name</span>)</span>,</span> data</span>,</span> class_</span>=</span>"</span>col</span>"</span>)</span></span> </span> doc</span>.</span>append_tag</span>(</span>"</span>h2</span>"</span>,</span> "</span>Plots</span>"</span>)</span></span> div2</span> =</span> rico</span>.</span>Div</span>(</span>class_</span>=</span>"</span>row row-cols-auto</span>"</span>)</span></span> doc</span>.</span>append</span>(</span>div2</span>)</span></span> for</span> name</span>,</span> data</span> in</span> df</span>.</span>groupby</span>(</span>df</span>.</span>index</span>)</span>:</span></span> div2</span>.</span>append</span>(</span></span> rico</span>.</span>Tag</span>(</span>"</span>h3</span>"</span>,</span> name</span>)</span>,</span></span> alt</span>.</span>Chart</span>(</span>data</span>)</span>.</span>mark_point</span>(</span>)</span>.</span>encode</span>(</span>x</span>=</span>"</span>x</span>"</span>,</span> y</span>=</span>"</span>y</span>"</span>)</span>,</span></span> class_</span>=</span>"</span>col</span>"</span>,</span></span> )</span></span></code></pre>More information #</a></h2> To learn more about how to use rico:</p> Read the more detailed user guide</a>.</li> Take a look at the self-explanatory examples</a> with resulting HTML documents.</li> Install</a> rico, create your own documents.</li> Check the docstrings for details.</li> </ul> Feel free to create an issue</a> to report bugs or submit a suggestion.</p>

Comparing Python packages for A/B test analysis: tea-tasting, Pingouin, statsmodels, and SciPy

2026-03-01T00:00:00+00:00

Disclosure: I am also the author of tea-tasting.

This article compares four Python packages that are relevant to A/B test analysis: tea-tasting, Pingouin, statsmodels, and SciPy. It does not try to pick a universal winner. Instead, it clarifies what each package does well for common experimentation tasks and how much manual work is needed to produce production-style A/B test outputs.

It assumes familiarity with A/B testing basics, including randomization, p-values, and confidence intervals.

A/B test setting and analysis requirements #</a></h2>
A/B tests in a nutshell #</a></h3>
An A/B test compares two (or more) variants of a product change by randomly assigning experimental units to variants and measuring outcomes. In online experiments, the randomization unit is usually the user, and the standard assumption is that units are independent.
A typical workflow is:

Design the experiment: choose the randomization unit (usually users), define the target population, and estimate sample size and duration with power analysis.</li>
Run the experiment: ship the treatment, randomize traffic, and collect data.</li>
Analyze and interpret results: compute control and treatment metric values, estimate effects with confidence intervals, and report p-values.</li> </ol>
Good references for this mindset include books and papers by Ron Kohavi and Alex Deng, especially on trustworthy experimentation, delta-method metrics, and CUPED.
Typical metric types and tests #</a></h3>
The table below summarizes common A/B test metric families and the tests usually applied.

Metric type</th> Examples</th> Typical test</th></tr></thead>

Average</td> Average revenue per user, average orders per user</td> Welch's t-test (unequal-variance two-sample t-test)</td></tr>
Ratio of averages</td> Average revenue per order, average orders per session</td> Welch's t-test with variance from the delta method</td></tr>
Proportion</td> Proportion of users with at least one order</td> Asymptotic tests (z-test, G-test, Pearson's chi-squared) or exact tests (Boschloo, Barnard, Fisher)</td></tr> </tbody></table>
Typical metric types and tests</figcaption></figure>
Typical per-metric analysis output #</a></h3>
For each metric, analysts usually want the same core fields:

Metric value in control.</li>
Metric value in treatment.</li>
Effect size estimate and confidence interval (absolute and/or relative).</li>
P-value.</li> </ul>
This output format is what makes A/B test analysis convenient for repeated use across many experiments.
A/B testing specifics #</a></h3>
Some details matter a lot in real experimentation workflows.

Relative effect size is often easier to interpret than absolute effect size. A common mistake is to divide an absolute confidence interval by the control mean. That does not generally produce a valid confidence interval for a relative effect. Use the delta method or Fieller's theorem instead (both are standard approaches for ratio-style uncertainty).</li>
Variance reduction (especially CUPED, which uses pre-experiment covariates) is widely used to increase power. CUPED for ratio metrics is more subtle because it combines variance reduction with ratio estimators and delta-method variance.</li>
Multiple hypothesis testing correction becomes important when you track many metrics or compare multiple variants. In practice, teams usually need a clear FWER/FDR policy and adjusted p-values (or adjusted significance thresholds) in experiment reports.</li>
Efficiency matters. Teams often analyze many metrics across many experiments. For many tests, only aggregated statistics are required (count, mean, variance, covariance). In those cases, it is usually more efficient to compute aggregates in the data backend and send only summary statistics to Python. A convenience layer for aggregate fetching can reduce both code and latency.</li> </ul>
Example data setup #</a></h2>
With that context, the examples below use the same synthetic experiment dataset generated with tea-tasting. They intentionally do not use variance reduction (no CUPED), so the package comparisons stay focused on baseline analysis.
import tea_tasting as tt data = tt.make_users_data(return_type="pandas", rng=42, n_users=5_000) data["has_order"] = (data["orders"] > 0).astype(int) control = data[data["variant"] == 0] treatment = data[data["variant"] == 1] print(data.head(3).to_string(index=False))</code></pre> Output:
`user variant sessions orders revenue has_order 0 1 1 1 5.77 1 1 0 3 0 0.00 0 2 1 1 0 0.00 0</code></pre> The metrics used in the examples are:`

`orders_per_user</code>: average orders per user.</li>`
`users_with_orders</code>: proportion of users with at least one order.</li>`
`revenue_per_user</code>: average revenue per user.</li>`
`revenue_per_order</code>: average revenue per order (ratio of averages).</li> </ul> All package examples below reuse data</code>, control</code>, and treatment</code> from this setup.`
`Package-by-package comparison` #</a></h2> tea-tasting #</a></h3> tea-tasting</a> is a package specifically designed for A/B test analysis. It targets experimentation workflows directly, with metrics, relative effects, CUPED, power analysis, and concise experiment-style outputs. Best for: teams that want an A/B-testing-first workflow with minimal glue code. This is the most compact example among the four packages because it provides A/B-specific metric classes and a high-level Experiment</code> API. experiment = tt.Experiment( orders_per_user=tt.Mean("orders"), users_with_orders=tt.Proportion("has_order", correction=False), revenue_per_user=tt.Mean("revenue"), revenue_per_order=tt.RatioOfMeans("revenue", "orders"), ) result = experiment.analyze(data) print(result)</code></pre> Output: metric control treatment rel_effect_size rel_effect_size_ci pvalue orders_per_user 0.511 0.556 8.8% [-0.74%, 19%] 0.0718 users_with_orders 0.334 0.352 5.4% [-2.4%, 14%] 0.181 revenue_per_user 5.06 5.64 11% [0.22%, 24%] 0.0455 revenue_per_order 9.91 10.2 2.5% [-3.0%, 8.3%] 0.389</code></pre> A/B testing specifics: Power analysis: built-in (Experiment.solve_power</code> and metric parameters for effect size, relative effect size, and sample size).</li> Relative effect and confidence interval: first-class output (including relative CI in the printed result).</li> CUPED: built-in for averages and ratio metrics.</li> Multiple hypothesis testing correction: built-in (tt.adjust_fdr</code> and tt.adjust_fwer</code>) for experiment results, including FDR and FWER procedures.</li> Aggregated statistics workflow: built-in support via metrics that expose required aggregates and integration with data backends (e.g., through Ibis-supported engines).</li> </ul> Pingouin #</a></h3> Pingouin</a> is a user-friendly statistical package focused on convenient inferential statistics in pandas-centric workflows. It is strong for common tests and effect sizes, but it is not an A/B-specific framework. Best for: quick pandas-based analyses of standard statistical tests. Pingouin has a convenient t-test interface and a contingency-table chi-squared helper. It does not provide a built-in ratio-of-averages test with delta-method variance, so the revenue_per_order</code> example is omitted. import pingouin as pg orders_test = pg.ttest( treatment["orders"], control["orders"], correction=True, ).iloc[0] print( "orders_per_user: " f"control={control['orders'].mean():.3f} " f"treatment={treatment['orders'].mean():.3f} " f"effect_size=" f"{treatment['orders'].mean() - control['orders'].mean():.3f} " f"effect_size_ci={orders_test['CI95']} " f"pvalue={orders_test['p_val']:.4f}" )</code></pre> Output: orders_per_user: control=0.511 treatment=0.556 effect_size=0.045 effect_size_ci=[-0. 0.09] pvalue=0.0718</code></pre>_, _, tests = pg.chi2_independence( data, x="variant", y="has_order", correction=False, ) pearson = tests.loc[tests["test"] == "pearson"].iloc[0] print( "users_with_orders: " f"control={control['has_order'].mean():.3f} " f"treatment={treatment['has_order'].mean():.3f} " f"effect_size=" f"{treatment['has_order'].mean() - control['has_order'].mean():.3f} " f"pvalue={pearson['pval']:.4f}" )</code></pre> Output: users_with_orders: control=0.334 treatment=0.352 effect_size=0.018 pvalue=0.1811</code></pre> Notes: revenue_per_user</code> uses the same pg.ttest(...)</code> pattern as orders_per_user</code>.</li> revenue_per_order</code> (ratio of averages) requires manual derivation if you want a statistically correct delta-method analysis.</li> The proportion example above shows a p-value, but not a built-in A/B-style effect CI.</li> </ul> A/B testing specifics: Power analysis: built-in for standard cases (power_ttest</code>, power_ttest2n</code>, power_chi2</code>), but not an A/B-specific multi-metric workflow.</li> Relative effect and confidence interval: mostly manual for A/B-style relative lift CIs.</li> CUPED: no built-in CUPED abstraction. You would implement variance reduction manually (for example, with regression).</li> Multiple hypothesis testing correction: built-in p-value adjustment via pg.multicomp</code> (for example, Bonferroni, Holm, and FDR methods), but integration into an A/B reporting workflow is manual.</li> Aggregated statistics workflow: mostly expects granular arrays/DataFrames; no built-in A/B aggregate interface.</li> </ul> statsmodels #</a></h3> statsmodels</a> is a broad statistical modeling library with strong hypothesis testing, power analysis, and confidence interval utilities. It is less opinionated than an experimentation-specific package, which is a strength if you want building blocks and explicit control. Best for: analysts who want mature statistical building blocks and are comfortable assembling a workflow. The example below uses Welch-style t-tests for two average metrics and a risk-ratio test/CI for the proportion metric. revenue_per_order</code> is omitted because there is no built-in A/B-style ratio-of-averages delta-method helper. from statsmodels.stats.proportion import ( confint_proportions_2indep, test_proportions_2indep, ) from statsmodels.stats.weightstats import CompareMeans, DescrStatsW def welch_summary(treatment_series, control_series): cm = CompareMeans( DescrStatsW(treatment_series), DescrStatsW(control_series), ) _, pvalue, _ = cm.ttest_ind(usevar="unequal") ci_low, ci_high = cm.tconfint_diff(usevar="unequal") return ( control_series.mean(), treatment_series.mean(), treatment_series.mean() - control_series.mean(), ci_low, ci_high, pvalue, ) for metric in ["orders", "revenue"]: ctrl, trt, effect, ci_low, ci_high, pvalue = welch_summary( treatment[metric], control[metric], ) print( f"{metric}_per_user: " f"control={ctrl:.3f} treatment={trt:.3f} effect_size={effect:.3f} " f"effect_size_ci=[{ci_low:.3f}, {ci_high:.3f}] pvalue={pvalue:.4f}" )</code></pre> Output: orders_per_user: control=0.511 treatment=0.556 effect_size=0.045 effect_size_ci=[-0.004, 0.094] pvalue=0.0718 revenue_per_user: control=5.062 treatment=5.641 effect_size=0.579 effect_size_ci=[0.012, 1.146] pvalue=0.0455</code></pre>count1 = int(treatment["has_order"].sum()) nobs1 = len(treatment) count0 = int(control["has_order"].sum()) nobs0 = len(control) prop_test = test_proportions_2indep( count1=count1, nobs1=nobs1, count2=count0, nobs2=nobs0, compare="ratio", method="log", ) prop_ci = confint_proportions_2indep( count1=count1, nobs1=nobs1, count2=count0, nobs2=nobs0, compare="ratio", method="log", ) print( "users_with_orders: " f"control={control['has_order'].mean():.3f} " f"treatment={treatment['has_order'].mean():.3f} " f"rel_effect_size={prop_test.ratio - 1:.3f} " f"rel_effect_size_ci=[{prop_ci[0] - 1:.3f}, {prop_ci[1] - 1:.3f}] " f"pvalue={prop_test.pvalue:.4f}" )</code></pre> Output: users_with_orders: control=0.334 treatment=0.352 rel_effect_size=0.054 rel_effect_size_ci=[-0.024, 0.138] pvalue=0.1811</code></pre> A/B testing specifics: Power analysis: strong built-in support (TTestIndPower</code>, NormalIndPower</code>, GofChisquarePower</code>, and more).</li> Relative effect and confidence interval: partial. There are built-in options for some cases (for example, risk ratios for proportions), but no general A/B-style relative lift CI interface across metric types.</li> CUPED: no built-in one-call CUPED API, but it is practical to implement manually with regression tooling.</li> Multiple hypothesis testing correction: strong built-in support (statsmodels.stats.multitest</code>, including multipletests</code> and fdrcorrection</code>).</li> Aggregated statistics workflow: partial. Proportion tests work directly from counts/sample sizes; other workflows often still require granular arrays or more manual setup.</li> </ul> SciPy #</a></h3> SciPy</a> is a foundational scientific computing and statistics package used directly or indirectly by many higher-level libraries, including the others in this comparison. It provides robust hypothesis tests and exact tests, but it does not provide a high-level A/B testing workflow. Best for: low-level building blocks and custom A/B analysis code. This snippet shows a Welch t-test for orders_per_user</code> and a Pearson chi-squared test for users_with_orders</code>. Exact tests such as Fisher, Barnard, and Boschloo are also available in SciPy. import numpy as np from scipy import stats orders_test = stats.ttest_ind( treatment["orders"], control["orders"], equal_var=False, ) orders_ci = orders_test.confidence_interval() print( "orders_per_user: " f"control={control['orders'].mean():.3f} " f"treatment={treatment['orders'].mean():.3f} " f"effect_size=" f"{treatment['orders'].mean() - control['orders'].mean():.3f} " f"effect_size_ci=[{orders_ci.low:.3f}, {orders_ci.high:.3f}] " f"pvalue={orders_test.pvalue:.4f}" )</code></pre> Output: orders_per_user: control=0.511 treatment=0.556 effect_size=0.045 effect_size_ci=[-0.004, 0.094] pvalue=0.0718</code></pre>contingency = np.array( [ [(control["has_order"] == 0).sum(), (control["has_order"] == 1).sum()], [ (treatment["has_order"] == 0).sum(), (treatment["has_order"] == 1).sum(), ], ] ) chi2_res = stats.contingency.chi2_contingency(contingency, correction=False) print( "users_with_orders: " f"control={control['has_order'].mean():.3f} " f"treatment={treatment['has_order'].mean():.3f} " f"effect_size=" f"{treatment['has_order'].mean() - control['has_order'].mean():.3f} " f"pvalue={chi2_res.pvalue:.4f}" )</code></pre> Output: users_with_orders: control=0.334 treatment=0.352 effect_size=0.018 pvalue=0.1811</code></pre> Notes: revenue_per_user</code> uses the same stats.ttest_ind(..., equal_var=False)</code> pattern as orders_per_user</code>.</li> revenue_per_order</code> (ratio of averages) requires manual delta-method implementation.</li> Relative effect confidence intervals are also manual.</li> </ul> A/B testing specifics: Power analysis: mostly manual in SciPy (or delegated to custom code / another package).</li> Relative effect and confidence interval: manual.</li> CUPED: manual.</li> Multiple hypothesis testing correction: partial. SciPy provides scipy.stats.false_discovery_control</code> for BH/BY FDR adjustment, but broader multiple-comparison correction workflows are more limited than in statsmodels.</li> Aggregated statistics workflow: partial. SciPy supports some summary-statistics and contingency-table tests (for example, ttest_ind_from_stats</code> and chi-squared/exact tests on contingency tables), but not an A/B-specific aggregate workflow.</li> </ul> Feature comparison for A/B testing #</a></h2> SciPy underpins much of the Python statistics ecosystem. In principle, all the capabilities discussed here can be implemented with NumPy + SciPy plus custom code. The practical question is convenience and code verbosity. For that reason, the table below uses three labels: built-in</code>: directly supported in a way that fits common A/B analysis tasks.</li> partial</code>: some built-in support exists, but not as a complete or ergonomic A/B workflow.</li> manual</code>: possible, but requires custom implementation/glue code.</li> </ul> Feature</th> tea-tasting</th> Pingouin</th> statsmodels</th> SciPy</th></tr></thead> Power analysis to estimate required number of observations</td> built-in</td> built-in</td> built-in</td> manual</td></tr> Welch's t-test or Student's t-test for analysis of averages</td> built-in</td> built-in</td> built-in</td> built-in</td></tr> Welch's t-test with delta method for analysis of ratios of averages</td> built-in</td> manual</td> manual</td> manual</td></tr> Two-sample proportion z-test, G-test, or Pearson's chi-squared test</td> built-in</td> built-in</td> built-in</td> built-in</td></tr> Relative effect size confidence intervals</td> built-in</td> manual</td> partial</td> manual</td></tr> Variance reduction with CUPED for analysis of averages and ratios of averages</td> built-in</td> manual</td> manual</td> manual</td></tr> Multiple hypothesis testing correction (FWER/FDR p-value adjustment)</td> built-in</td> built-in</td> built-in</td> partial</td></tr> Working with aggregated statistics instead of granular data</td> built-in</td> manual</td> partial</td> partial</td></tr> </tbody></table> Feature comparison for A/B testing</figcaption></figure>Conclusions #</a></h2> The four packages sit at different levels of abstraction. tea-tasting is the most A/B-specific option in this group. It is designed around metrics, experiments, relative effects, CUPED, and aggregate-based workflows.</li> Pingouin is convenient for standard statistical tests and quick analysis in pandas, but A/B-specific workflows (especially ratio metrics, relative CIs, and CUPED) are mostly manual.</li> statsmodels provides strong statistical building blocks and power analysis. It is a good choice when you want explicit control and are willing to assemble an experimentation workflow yourself.</li> SciPy is the essential foundation. It can support almost everything with enough custom code, but it is the most verbose option for repeated A/B test reporting.</li> </ul> If you run many experiments with multiple metrics and need consistent outputs, the main differentiator is not just statistical correctness. It is how much A/B-specific workflow a package gives you out of the box. Inclusion criteria for the comparison #</a></h2> For transparency, here are the minimum criteria I used to decide which packages to include. Maintained: a recent release (for example, within a year) and recent commits (for example, within two months) reduce the risk of stale APIs and unresolved compatibility issues.</li> Well documented: a package should have both a user guide and an API reference, because A/B testing code is often reused by analysts with different levels of statistical depth.</li> Used by a community: a practical heuristic is at least 100 GitHub stars. This is not a quality guarantee, but it often means more examples and more edge cases have already been surfaced.</li> </ul> Scope note: This comparison focuses on frequentist A/B testing workflows. Bayesian-first experimentation frameworks are out of scope. Excluded notable packages (and why): spotify_confidence</a>: excluded because it lacks documentation and has not added significant new features in the last couple of years.</li> ambrosia</a>: excluded because it has not added new features in the last couple of years, aside from dependency version updates.</li> </ul> Note: The maintenance and documentation notes in this section are assessed as of March 1, 2026. Resources #</a></h2> The comparison above is based on the public documentation and APIs of the packages as of March 1, 2026. Current stable versions on PyPI at the time of writing: tea-tasting</a>: 1.12.0, PyPI</a>.</li> Pingouin</a>: 0.6.0, PyPI</a>.</li> statsmodels</a>: 0.14.6, PyPI</a>.</li> SciPy stats</a>: 1.17.1, PyPI</a>.</li> </ul> A/B testing and statistics references: Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing.</a></li> Deng, A., Knoblich, U., & Lu, J. (2018). Applying the delta method in metric analytics: A practical guide with novel ideas.</a></li> Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data.</a></li> Multiple comparisons problem (Wikipedia).</a></li> </ul> Prices as model weights: learning from dispersed data 2025-10-24T00:00:00+00:00 Market prices are similar to model weights in the sense that they are "fit" through price discovery mechanisms to optimize for equilibrium. The difference between supply and demand is analogous to the gradient in backpropagation. An important property of prices is that they can be "fit" to distributed and sometimes hidden data. As Friedrich Hayek argued, prices convey dispersed knowledge that no single person or central planner could ever fully possess. There are problems that deal with information that cannot be accessed directly. A well-designed price discovery mechanism can solve such problems more efficiently than an AI/ML model with limited information. This analogy came to mind while I was considering job market congestion that occurs when too many candidates apply for a role. Most of those candidates are not a good fit for the job, but they still try their luck since it requires little effort from them. As a solution, employers automate screening with AI. Candidates also use AI to scrape job descriptions and apply automatically (like "tinda finger"</a>). This creates a vicious cycle in which nobody wins. One possible solution is a market design in which candidates pay to apply, with dynamic pricing. Paying to apply may sound provocative and require thoughtful consideration and careful testing. Payments can be made with platform-issued virtual points, available in limited supply. But here, I focus on why price signals may address this problem better than AI-based screening. The job market is a matching market with information asymmetry, where both sides have unique preferences resolved only through costly interaction. Any AI/ML algorithm is only as good as the data, and that data is limited and often inaccurate. Requiring payment to apply would elicit a candidate's own estimate of their fit for the job and their true level of interest. These estimates would be reflected in prices, providing a visible signal that guides other candidates considering the role. This approach could reduce congestion, lower screening costs, and help form better matches. Ladder of questions in analysis 2025-05-12T00:00:00+00:00 A ladder of questions in analysis: Which question should we ask to address a decision or a problem?</li> Which model should we choose to answer the question?</li> Which assumptions must hold for the model to be valid, and do they hold?</li> What data do we need to apply the model?</li> What conclusions can we draw after applying the model?</li> </ol> From my personal experience, people often skip steps 1 and 3, which can lead to bad decisions or mediocre solutions. You probably read this in a statistical or data-analysis context. But I believe the framework applies more broadly. In statistics, answering a wrong question is sometimes called a Type III error. I've already mentioned</a> it in a blog post. Curious facts about Ronald Fisher 2024-08-12T00:00:00+00:00 Fisher and smoking #</a></h2> Ronald Fisher</a>, the father of modern statistics and experimental design, refused to believe that smoking tobacco caused lung cancer. He argued that correlation does not imply causation and even published a paper in Nature titled Cancer and Smoking</a>, defending this viewpoint. Ronald Fisher enjoying a pipe, 1956. Source</a>. </figcaption></figure> Fisher and Guinness #</a></h2> Ronald Fisher helped William Gosset</a> develop a statistical test known as Student's t-test</a>. They exchanged over 150 letters between 1912 and 1934. Gosset published his work under the pen name Student as he was the Head Experimental Brewer at Guinness Brewery and was not allowed to publish under his real name. The fact about Gosset is well known. Less well known is that Fisher's wife, Ruth Eileen Fisher née Guinness, was a great-great-granddaughter of Arthur Guinness, the founder of Guinness Brewery. Fisher and a lady #</a></h2> Ronald Fisher devised the famous "lady tasting tea"</a> experiment. In this experiment, he developed the null hypothesis significance testing framework to analyze a lady's claim that she could discern whether the tea or the milk was added first to the cup. The lady was Muriel Bristol</a>, a phycologist at Rothamsted Research. One day at Rothamsted, Fisher offered Bristol a cup of tea, but she declined, explaining that she preferred the flavor when the milk was poured into the cup before the tea. Fisher said that the order of pouring could not affect the flavor. Bristol insisted that it did and that she could tell the difference. Overhearing this debate, William Roach said, "Let's test her." They devised an experiment, and Lady Bristol proved her case. Fisher and tea-tasting #</a></h2> tea-tasting</a> is a Python package that I developed for the statistical analysis of A/B tests. The name "tea-tasting" is a play on words referring to two subjects: The "lady tasting tea" experiment, devised by Ronald Fisher.</li> "Tea-tasting" phonetically resembling "t-testing" or Student's t-test, developed by William Gosset with the help of Ronald Fisher.</li> </ul> The funny part is that tea-tasting doesn't include Fisher's exact test</a>, which was used in the "lady tasting tea" experiment. It's like naming a coffee shop "Espresso Corner" and not serving espresso. Do you think I should implement Fisher's exact test in tea-tasting? Or Boschloo's test</a>? Or Barnard's test</a>? Let me know in the comments below. tea-tasting: a Python package for the statistical analysis of A/B tests 2024-07-29T00:00:00+00:00 Note: the post has been updated on March 7, 2026. It's often referred to by AI chatbots and agents without recognizing that some information could be outdated. So I decided to update it. I developed tea-tasting</a>, a Python package for the statistical analysis of A/B tests featuring: Welch's t-test, Student's t-test, z-test, Bootstrap, variance reduction with CUPED, delta method for ratio metrics, power analysis, and other A/B-test-specific statistical methods and approaches out of the box.</li> Support for a wide range of data backends, such as BigQuery, ClickHouse, PostgreSQL, Snowflake, Trino, and other backends supported by Ibis</a>. tea-tasting also accepts dataframes supported by Narwhals</a>: cuDF, Daft, Dask, DuckDB, Modin, pandas, Polars, PyArrow, PySpark.</li> Convenient API for reducing manual work, and a framework for minimizing errors. Extensible API: define custom metrics and use statistical tests of your choice.</li> Pretty representation of analysis results: rounding to significant digits, rendering in terminals, Jupyter/IPython, and marimo notebooks, serialization to Markdown, and conversion to pandas and Polars DataFrames.</li> Detailed documentation.</li> </ul> In this blog post, I explore each of these advantages of using tea-tasting in the analysis of experiments. If you are eager to try it, check the user guide</a>. Statistical methods #</a></h2> tea-tasting includes statistical methods and techniques that cover most of what you might need in the analysis of experiments. Analyze metric averages with t-test or z-test, proportions with asymptotic or exact tests, ranks with the Mann-Whitney U test, or use Bootstrap to analyze any other statistic of your choice. There is also a predefined method for analyzing quantiles with Bootstrap. tea-tasting detects mismatches in the sample ratios of different variants of an A/B test. You can also define a custom metric</a> with a statistical test of your choice. tea-tasting applies the delta method</a> for the analysis of ratios of averages. For example, the average number of orders per average number of sessions, assuming that sessions are not randomization units. Use pre-experiment data, metric forecasts, or other covariates to reduce variance and increase the sensitivity of an experiment. This approach is also known as CUPED</a> or CUPAC</a>. In tea-tasting, it can be easily combined with the delta method for analyzing ratio metrics as well. The calculation of confidence intervals for percentage change in t-test and z-test can be tricky. Just taking the confidence interval for absolute change and dividing it by the control average will produce a biased result. tea-tasting applies the delta method to calculate the correct interval. Analyze statistical power</a> for t-test and z-test. There are three possible options: Calculate the effect size, given statistical power and the total number of observations.</li> Calculate the total number of observations, given statistical power and the effect size.</li> Calculate statistical power, given the effect size and the total number of observations.</li> </ul> Other A/B-test-specific features of tea-tasting: Procedures for controlling the multiple hypothesis testing</a> problem: False discovery rate (FDR): Benjamini-Hochberg and Benjamini-Yekutieli procedures.</li> Family-wise error rate (FWER): Hochberg's step-up and Holm's step-down procedures.</li> </ul> </li> Simulations</a>: A/A tests are useful for identifying potential issues before conducting the actual A/B test.</li> Treatment simulations are great for power analysis—especially when you need a specific uplift distribution or when an analytical formula doesn’t exist.</li> </ul> </li> </ul> Learn more in the detailed user guide</a>. Data backends #</a></h2> There are many different databases and engines for storing and processing experimental data. And in most cases it's not efficient to pull the detailed experimental data into a Python environment. Many statistical tests, such as t-test or z-test, require only aggregated data for analysis. For example, if the raw experimental data are stored in ClickHouse, it's faster and more efficient to calculate counts, averages, variances, and covariances directly in ClickHouse rather than fetching granular data and performing aggregations in a Python environment. Querying all the required statistics manually can be a daunting and error-prone task. For example, analysis of ratio metrics and variance reduction with CUPED require not only the number of rows and variance, but also covariances. But don't worry: tea-tasting does all this work for you. tea-tasting supports Ibis Tables as input data. Ibis</a> is a Python package which serves as a DataFrame API to various data backends. It supports many backends including BigQuery, ClickHouse, PostgreSQL, Snowflake, and Trino. You can write an SQL query, wrap</a> it as an Ibis Table, and pass it to tea-tasting. Keep in mind that tea-tasting assumes that: Data is grouped by randomization units, such as individual users.</li> There is a column indicating the variant of the A/B test (typically labeled as A, B, etc.).</li> All necessary columns for metric calculations (like the number of orders, revenue, etc.) are included in the table.</li> </ul> Some statistical methods, like Bootstrap, require granular data for the analysis. In this case, tea-tasting fetches the detailed data as well. tea-tasting also accepts dataframes supported by Narwhals</a>: cuDF, Daft, Dask, DuckDB, Modin, pandas, Polars, PyArrow, PySpark. Learn more in the guide on data backends</a>. Convenient API #</a></h2> You can perform most of the statistical calculations above using just SciPy. In fact, tea-tasting uses it under the hood. What tea-tasting offers on top is a convenient higher-level API. It's easier to show than to describe. Here is the basic example: import tea_tasting as tt data = tt.make_users_data(rng=42) experiment = tt.Experiment( sessions_per_user=tt.Mean("sessions"), orders_per_session=tt.RatioOfMeans("orders", "sessions"), orders_per_user=tt.Mean("orders"), revenue_per_user=tt.Mean("revenue"), ) result = experiment.analyze(data) result</code></pre> Output: metric control treatment rel_effect_size rel_effect_size_ci pvalue sessions_per_user 2.00 1.98 -0.66% [-3.7%, 2.5%] 0.674 orders_per_session 0.266 0.289 8.8% [-0.89%, 19%] 0.0762 orders_per_user 0.530 0.573 8.0% [-2.0%, 19%] 0.118 revenue_per_user 5.24 5.73 9.3% [-2.4%, 22%] 0.123</code></pre> The two-stage approach, with separate parametrization and inference, is common in statistical modeling. This separation helps in making the code more modular and easier to understand. tea-tasting performs calculations that can be tricky and error-prone: Analysis of ratio metrics with delta method.</li> Variance reduction with CUPED/CUPAC (also in combination with the delta method for ratio metrics).</li> Calculation of confidence intervals for both absolute and percentage change.</li> Analysis of statistical power.</li> </ul> It also provides a framework for representing experimental data to avoid errors. Grouping the data by randomization units and including all units in the dataset is important for correct analysis. Check out the comparison post</a> to see how tea-tasting differs from general purpose statistical packages like Pingouin, statsmodels, and SciPy. Pretty representation and formatting #</a></h2> tea-tasting reduces the work to format and represent the analysis results. It rounds numbers to significant digits and renders the result table automatically based on where you display it: In a terminal or Python console, the result is shown as a plain-text table (as in the example above).</li> In IPython and Jupyter, the result is rendered as an HTML table.</li> In marimo notebooks, the result is rendered as a table widget.</li> </ul> </a>Experiment result rendering in marimo </figcaption></figure> In addition, tea-tasting provides methods to serialize the experiment result. Notable examples: to_pandas</code>: Convert the result to a Pandas DataFrame.</li> to_polars</code>: Convert the result to a Polars DataFrame.</li> to_markdown</code>: Convert the result to a Markdown table.</li> to_string</code>: Convert the result to a string (with parameters for output customization).</li> </ul> Documentation #</a></h2> Last but not least: documentation. I believe that good documentation is crucial for tool adoption. That's why I wrote several user guides and an API reference. I recommend starting with the example of basic usage in the user guide</a>. Then you can explore specific topics: Data backends</a>.</li> Power analysis</a>.</li> Multiple hypothesis testing</a>.</li> Custom metrics</a>.</li> Simulated experiments</a>.</li> </ul> Use the API reference</a> to explore all parameters and detailed information about the functions, classes, and methods available in tea-tasting. The tea-tasting repository includes examples</a> as copies of the guides in the marimo</a> notebook format. See the corresponding README</a> to learn how to run them. Conclusions #</a></h2> There are a variety of statistical methods that can be applied in the analysis of an experiment. But only a handful of them are actually used in most cases. On the other hand, there are methods specific to the analysis of A/B tests that are not included in the general purpose statistical packages like SciPy. tea-tasting's functionality includes the most important statistical tests, as well as methods specific to the analysis of A/B tests. tea-tasting provides a convenient API that helps to reduce the time spent on analysis and minimize the probability of error. In addition, tea-tasting optimizes computational efficiency by calculating the statistics in the data backend of your choice, where the data are stored. With the detailed documentation</a>, you can quickly learn how to use tea-tasting for the analysis of your experiments. P.S. Package name #</a></h2> The package name "tea-tasting" is a play on words that refers to two subjects: Lady tasting tea</a> is a famous experiment which was devised by Ronald Fisher. In this experiment, Fisher developed the null hypothesis significance testing framework to analyze a lady's claim that she could discern whether the tea or the milk was added first to the cup.</li> "tea-tasting" phonetically resembles "t-testing" or Student's t-test, a statistical test developed by William Gosset.</li> </ul> This meeting could have been 2024-06-05T00:00:00+00:00 an email</del> a Slack message</del> a PRD</del> a one-pager</del> nothing Readme Driven Development for an A/B testing analysis package 2023-12-11T00:00:00+00:00 tea-tasting: statistical analysis of A/B tests #</a></h2> tea-tasting is a Python package for statistical analysis of A/B tests that features: Student's t-test, Z-test, and Bootstrap out of the box.</li> Extensible API: Define and use statistical tests of your choice.</li> Delta method</a> for ratio metrics.</li> Variance reduction with CUPED</a>/CUPAC</a> (also in combination with delta method for ratio metrics).</li> Fieller's confidence interval</a> for percent change.</li> Sample ratio mismatch check.</li> Power analysis.</li> A/A tests.</li> </ul> Currently, tea-tasting is in the planning stage, and I'm starting with a README that outlines the proposed API — an approach known as Readme Driven Development (RDD). In this blog post, I'll explain the motivation for creating this package and the benefits of the RDD approach. Explore the README here</a>. I welcome your insights! If you have any suggestions, questions, or ideas for alternative designs, please join the conversation</a> on GitHub. Why a new package? #</a></h2> One might wonder why there's a need for a new package like tea-tasting when there are already SciPy and statsmodels available. Both of these are general-purpose statistical packages offering a wide range of statistical tests. However, they lack certain methods that are specific to A/B test analysis. tea-tasting aims to fill this gap. For example, it includes methods such as the delta method for ratio metrics, CUPED/CUPAC for variance reduction, and methods for calculating confidence intervals for percentage changes. The intention behind tea-tasting is not only to automate the analysis process but also to reduce the probability of making a mistake. Statistical analysis of A/B tests can be challenging, with many opportunities for errors. tea-tasting is designed to mitigate these risks. Many companies develop in-house platforms for A/B testing, but these platforms often don't cover every scenario, such as non-standard randomization units used in cluster-level or switchback experiments, or the analysis of new metrics not yet integrated into the platform. tea-tasting is also useful in these cases. In summary, tea-tasting is designed for a more specialized, error-minimizing approach to A/B testing analysis, filling in the gaps left by existing general-purpose statistical packages. Package name #</a></h2> The package name "tea-tasting" is a play on words that refers to two subjects: Lady tasting tea</a> is a famous experiment which was devised by Ronald Fisher. In this experiment, Fisher developed the null hypothesis significance testing framework to analyze a lady's claim that she could discern whether the tea or the milk was added first to a cup.</li> "tea-tasting" phonetically resembles "t-testing" or Student's t-test, a statistical test developed by William Gosset.</li> </ul> Lady tasting tea by DALL-E </figcaption></figure>Readme Driven Development #</a></h2> Readme Driven Development (RDD) is a software development approach where you start by writing the README file first, before writing any code. The idea is similar to Amazon's Working Backwards</a> approach, where a press release is drafted before starting a new product. Writing a README encourages thinking from a user's perspective, focusing on how users will interact with the API. It allows for better project planning. You don't need to revise the code every time you decide to include something in the public API. Don't confuse RDD with the Waterfall approach. It's not about creating exhaustive specifications or detailed documentation upfront. In fact, with tea-tasting, I might have gone a bit deeper into details than usual, but this was a conscious choice to thoroughly plan the package's functionality. To learn more about RDD, I recommend reading Tom Preston-Werner's blog post</a>. Request for feedback #</a></h2> One of the key benefits of Readme Driven Development (RDD) is the ability to adapt design choices with minimal cost. After reviewing the README</a>, you might have insights that could enhance this project. It could be anything from improved naming conventions and additional features to an entirely different API approach. Your perspective is valuable! If the concept of this package interests you and you have ideas for its design, I encourage you to share your thoughts</a> on GitHub. Let's collaborate to make tea-tasting as effective and user-friendly as possible. P.S. Can ChatGPT write a Python package? #</a></h2> Tom Preston-Werner wrote his blog post in 2010. There were no LLMs at that time. Thinking about RDD in 2023, I couldn't resist asking ChatGPT to write the package code based on README. Well, not quite yet. ChatGPT resisted performing a large task at once. After I insisted, ChatGPT started implementing it class by class. In the first class, it started hallucinating about the SciPy API. I gave up shortly. For those interested in how this experiment unfolded, you can view the entire conversation here: chat</a>. Debunking the Dunning–Kruger effect with random number simulation 2023-09-25T00:00:00+00:00 Intro #</a></h2> Many people associate the Dunning–Kruger effect with the following image: Misinterpretation of the Dunning–Kruger effect </figcaption></figure> Actually, this image is a misinterpretation of the conclusions drawn by Dunning and Kruger from their experiments. The Dunning–Kruger effect is a cognitive bias in which people with limited competence in a particular domain overestimate their abilities.1</a> Furthermore, evidence suggests that the Dunning and Kruger experiments might not definitively prove that the effect is real. It can be shown that it's a statistical artifact and the result of the way data is manipulated.2</a> 3</a> 4</a> I'll illustrate this using a random number simulation, which offers an intuitive and straightforward understanding. In this post, I'll: Provide details of the experiments and analysis conducted by Dunning and Kruger.</li> Simulate similar results using random data that do not imply that the effect is real.</li> Explain why percentiles are not a suitable measure for these experiments.</li> </ul> I also created a Streamlit app</a> as a demo for this blog post. Feel free to play with it and check out the source code</a>. Dunning and Kruger experiments5</a> #</a></h2> Justin Kruger and David Dunning conducted several experiments (studies), each in one of the domains: humor, logical reasoning, and English grammar. In each experiment, participants—who were Cornell University students—took a test that assessed their knowledge in the domain. After the test, they were asked to assess their ability compared to other Cornell students. To be specific, participants were required to provide percentile rankings on a scale from 0 to 99. In two of the three studies, they were also asked to assess their test performance. Dunning and Kruger performed the following analyses for each experiment: Assigned participants percentile ranks based on their test score, with 0 being the lowest score and 99 being the highest score.</li> Calculated the average perceived ability percentile and found out that it exceeded the average test score percentile (50, by definition).</li> Grouped participants in quartiles based on their test score.</li> Calculated the average test score percentile and the average perceived ability percentile for each test score quartile (see the plot below).</li> </ul> Perceived ability to recognize humor as a function of actual test performance (Study 1) </figcaption></figure> The plot above compares the average test score percentile and the average perceived ability percentile by test score quartile for the humor study. The plots for logical reasoning and English grammar experiments look similar. They also contain the average perceived test score percentiles that are quite close to the average perceived ability percentiles. The main conclusions that were drawn: Participants overall overestimated their abilities.</li> "Incompetent" participants grossly overestimated their ability relative to their peers.</li> </ol> The second conclusion is commonly referred to as the Dunning–Kruger effect. In the rest of the post, I will focus mostly on this conclusion and on the calculations that led to it. Dunning and Kruger also conducted other experiments that studied participants' abilities to reassess their competence. I omit the details as they are not the main interest of this post. Random number simulation #</a></h2> As previously mentioned, these experiments didn't prove the Dunning–Kruger effect. To demonstrate this, I'll: Simulate the experiments with random data implying that the effect is not real.</li> Perform the same calculations as Dunning and Kruger.</li> </ul> First, let's generate test scores and perceived ability as correlated variables. Each point on the plot below represents a participant with a test score on the x-axis and a perceived ability on the y-axis. Test score vs. perceived ability </figcaption></figure> Note that the data generation process doesn't imply that "incompetent" participants overestimate their abilities. Next, we'll compute the percentiles for participants' test scores and perceived abilities. This is not exactly how perceived ability percentiles were obtained in the Dunning and Kruger experiment. But this is still valid for our purpose. Test score percentile vs. perceived ability percentile </figcaption></figure> Finally, let's calculate the average test score percentile and the average perceived ability percentile by test score quartiles. Average percentiles by test score quartiles </figcaption></figure> For the bottom quartile, the average perceived ability percentile is higher than the average test score percentile. Feel free to try different values for the random seed, correlation coefficient (below 1), or number of participants in the Streamlit app</a>. You will always get this result for two variables with imperfect correlation. But Dunning and Kruger used this result as evidence that "incompetent" participants overestimated their abilities. They were wrong, as this result is a statistical artifact and the outcome of the way data is manipulated. Additionally, let's use the same data to calculate the average test score percentile and the average perceived ability percentile by perceived ability quartiles. Average percentiles by perceived ability quartiles </figcaption></figure> Participants from the bottom quartile, on average, have the lowest test scores. For them, the average perceived ability percentile is also lower than the average test score percentile. Doesn't this contradict the Dunning–Kruger effect? Bias and percentile measure #</a></h2> In fact, Dunning and Kruger did discuss the regression effect due to imperfect correlation between actual and perceived ability. But they argued that the overestimation they observed was more psychological than artifactual. The main argument was that the magnitude of miscalibration among the bottom quartile was larger than that of the top quartile.5</a> As you may have noticed, in the simulation, participants overall don't overestimate their abilities. This is different from the Dunning and Kruger experiment. I don't think it's possible to generate a dataset with average perceived ability percentile higher than the average test score percentile without implying that "incompetent" participants grossly overestimate their abilities. The reason is that percentile is a measure with bounded support, meaning it has a finite minimum and maximum. Let's take a participant with a test score which corresponds to the 99th percentile. They cannot overestimate their abilities in percentiles. Participants from the top test score quartile can overestimate their abilities by 24 percentiles maximum and 12 percentiles on average. Participants from the bottom test score quartile can overestimate their abilities by 99 percentiles maximum and 87 percentiles on average. If participants overall overestimate their abilities, then, with high probability, the magnitude of overestimation will be larger among participants with low test scores. And this is a pure mathematical artifact, not a psychological effect. This reasoning applies to any measure with bounded support, not just percentiles. For example, we would get the same results with test scores as a measure. So, the actual reason for the Dunning and Kruger observation is a combination of two factors: Participants overall overestimated their abilities.</li> Percentile is a measure with bounded support.</li> </ol> Also, I encourage you to think about the question: if a person scores the maximum number of points in a test, does this mean that they cannot overestimate their abilities? Conclusion #</a></h2> Justin Kruger and David Dunning conducted several experiments and concluded that "incompetent" participants grossly overestimated their ability relative to their peers. This effect is known as the Dunning–Kruger effect. I reproduced similar results using random number simulation implying that the effect is not real. If participants overall overestimate their abilities, the magnitude of the overestimation among the participants with low scores is explained by the fact that percentile is a measure with bounded support. The Dunning–Kruger effect, as observed in the original experiments, is a statistical artifact rather than a psychological phenomenon. I don't argue about whether the Dunning–Kruger effect is real or not. My intent is to show that the original Dunning and Kruger experiments don't prove it. I created a Streamlit app</a> as a demo for this blog post. Feel free to play with it and check out the source code</a>. References #</a></h2> ^1Dunning–Kruger effect – Wikipedia</a>. </div> ^2Nuhfer, Edward, Christopher Cogan, Steven Fleisher, Eric Gaze, and Karl Wirth. (2016). Random Number Simulations Reveal How Random Noise Affects the Measurements and Graphical Portrayals of Self-Assessed Competency. Numeracy 9, Iss. 1: Article 4. DOI: http://dx.doi.org/10.5038/1936-4660.9.1.4</a>. </div> ^3Nuhfer, Edward, Steven Fleisher, Christopher Cogan, Karl Wirth, and Eric Gaze. (2017). How Random Noise and a Graphical Convention Subverted Behavioral Scientists' Explanations of Self-Assessment Data: Numeracy Underlies Better Alternatives. Numeracy 10, Iss. 1 : Article 4. DOI: http://dx.doi.org/10.5038/1936-4660.10.1.4</a>. </div> ^4Jarry, Jonathan. (2020). The Dunning-Kruger Effect Is Probably Not Real. McGill University, Office for Science and Society. Article: link</a>. </div> ^5Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134. DOI: https://doi.org/10.1037/0022-3514.77.6.1121</a>. Open access: link</a>. </div> Ranking in a two-sided matching platform (Part 2) 2023-08-28T00:00:00+00:00 Recap #</a></h2> This is the second post about the ranking problem in Profi, a services marketplace where customers can find tutors, beauty masters, plumbers, and other professionals, and professionals can find customers. Here is the short recap of the previous post</a>: In Profi, professionals search, choose orders, and send proposals to customers. And then the customers review the proposals and choose the professionals, or don't choose anyone. If a professional sends a proposal to a customer, and the customer chooses the professional then this is a deal.</li> The objective was to increase the number of deals by implementing a new order ranking algorithm in the app for professionals.</li> Ranking orders by the probability of a proposal did increase the number of proposals, but didn't increase the number of deals in the marketplace.</li> Ranking orders by the uplift in the probability of a deal significantly increased the number of deals in the marketplace.</li> </ul> Uplift in the probability of a deal </figcaption></figure> In the second post, I'll explain the solution. This post is more mathematical. Please bear with me while I go through some details of the model. Mathematical formulation #</a></h2> First, let's express the uplift in the probability of a deal in a mathematical form: $$P(Y_i = 1 \mid S_{ij} = 1) - P(Y_i = 1 \mid S_{ij} = 0)$$ $Y_i$ is an outcome for the order $i$. If there is a deal, then $Y_i = 1$, otherwise $Y_i = 0$.</li> $S_{ij}$ indicates whether the professional $j$ has seen the order $i$ or not.</li> $P(Y_i = 1 \mid S_{ij} = 1)$ is the probability of a deal in the order $i$, given that the professional $j$ has seen the order.</li> $P(Y_i = 1 \mid S_{ij} = 0)$ is the probability of a deal in the order $i$, given that the professional $j$ hasn't seen the order.</li> </ul> It seems impossible to solve this problem directly. Hundreds of professionals should see the order for a deal to happen. Estimating the effect of showing an order to a professional would be very noisy. Let's decompose the problem by expanding the expression using the law of total probability: $$P(T_{ij} = 1 \mid S_{ij} = 1)(P(Y_i = 1 \mid T_{ij} = 1) - P(Y_i = 1 \mid T_{ij} = 0))$$ $T_{ij}$ indicates whether the professional $j$ has sent a proposal to the customer $i$ or not.</li> $P(T_{ij} = 1 \mid S_{ij} = 1)$ is the probability that the professional $j$ will send a proposal to the customer $i$, given that the professional has seen the order.</li> $P(Y_i = 1 \mid T_{ij} = 1)$ is the probability of a deal in the order $i$, given that the professional $j$ has sent a proposal to the customer.</li> $P(Y_i = 1 \mid T_{ij} = 0)$ is the probability of a deal in the order $i$, given that the professional $j$ hasn't sent a proposal to the customer.</li> </ul> Derivation Let's expand the left and the right part of the uplift expression: $$ \small \begin{split} P(Y_i = 1 \mid S_{ij} = 1) & = P(Y_i = 1 \mid T_{ij} = 1)P(T_{ij} = 1 \mid S_{ij} = 1) \\ & \quad + P(Y_i = 1 \mid T_{ij} = 0)P(T_{ij} = 0 \mid S_{ij} = 1) \\ & = P(Y_i = 1 \mid T_{ij} = 1)P(T_{ij} = 1 \mid S_{ij} = 1) \\ & \quad + P(Y_i = 1 \mid T_{ij} = 0)(1 - P(T_{ij} = 1 \mid S_{ij} = 1)) \\ & = P(T_{ij} = 1 \mid S_{ij} = 1)(P(Y_i = 1 \mid T_{ij} = 1) - P(Y_i = 1 \mid T_{ij} = 0)) \\ & \quad + P(Y_i = 1 \mid T_{ij} = 0) \\ \\ P(Y_i = 1 \mid S_{ij} = 0) & = P(Y_i = 1 \mid T_{ij} = 1)P(T_{ij} = 1 \mid S_{ij} = 0) \\ & \quad + P(Y_i = 1 \mid T_{ij} = 0)P(T_{ij} = 0 \mid S_{ij} = 0) \\ & = P(Y_i = 1 \mid T_{ij} = 0) \\ \end{split} $$ The last equation takes into account that a professional cannot send a proposal without seeing the order, i.e.: $$ \small \begin{split} & P(T_{ij} = 1 \mid S_{ij} = 0) = 0 \\ & P(T_{ij} = 0 \mid S_{ij} = 0) = 1 \end{split} $$ Now, we can derive that: $$ \small \begin{split} & P(Y_i = 1 \mid S_{ij} = 1) - P(Y_i = 1 \mid S_{ij} = 0) \\ = P(T_{ij} = 1 \mid S_{ij} = 1)(& P(Y_i = 1 \mid T_{ij} = 1) - P(Y_i = 1 \mid T_{ij} = 0)) \end{split} $$ </div> </div> </div> </div> We already have a model</a> that predicts $P(T_{ij} = 1 \mid S_{ij} = 1)$, the probability that the professional $j$ will send a proposal to the customer $i$. So, now the problem reduces to predicting the uplift in the probability of a deal caused by a proposal from the professional: $$P(Y_i = 1 \mid T_{ij} = 1) - P(Y_i = 1 \mid T_{ij} = 0)$$ To solve the problem, we'll create a model for the probability of a deal in the order. The difference between the probabilities of a deal with and without a proposal from the professional is the uplift we want to predict. Discrete choice model #</a></h2> A customer can choose one of the professionals who sent a proposal, or not choose anyone. Let's model customer's choice with the softmax function: $$P(Y_{ik}=1) = \frac{e^{U_{ik}}}{e^{U_{i0}} + \sum_{j \in J_i} e^{U_{ij}}} = \frac{e^{U_{ik}-U_{i0}}}{1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}}}$$ $Y_{ik}$ indicates whether the customer $i$ chooses the professional $k$ or not.</li> $P(Y_{ik}=1)$ is the probability that the customer $i$ will choose the professional $k$.</li> $U_{ij}$ (or $U_{ik}$) is the utility the customer $i$ obtains from choosing the professional $j$ (or $k$).</li> $U_{i0}$ is the utility the customer $i$ obtains from not choosing anyone.</li> $J_i$ is a set of professionals who sent a proposal to the customer $i$.</li> </ul> This model assumes no correlation among alternatives. This might not be 100% true. But we decided to proceed with this model: Given that customers and professionals have unique preferences that only they can discover, this assumption seems realistic.</li> Taking into account the sequential number of the proposal in the order can capture correlations between proposals.</li> </ul> Now, we can express the probability that the customer will choose one of the professionals who sent a proposal: $$P(Y_i=1) = \sum_{j \in J_i} P(Y_{ij}=1) = \frac{\sum_{j \in J_i} e^{U_{ij}-U_{i0}}}{1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}}}$$ Suppose the customer $i$ has received proposals from a set of professionals $J_i$. Then the uplift in the probability of a deal caused by a proposal from the professional $k$ will be: $$ \begin{split} & P(Y_i = 1 \mid T_{ik} = 1) - P(Y_i = 1 \mid T_{ik} = 0) \\ & = \frac{\sum_{j \in J_i} e^{U_{ij}-U_{i0}} + e^{U_{ik}-U_{i0}}}{1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}} + e^{U_{ik}-U_{i0}}} - \frac{\sum_{j \in J_i} e^{U_{ij}-U_{i0}}}{1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}}} \\ & = \frac{e^{U_{ik}-U_{i0}}}{(1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}})(1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}} + e^{U_{ik}-U_{i0}})} \end{split} $$ If we can model the utility, we can estimate the uplift in the probability of a deal. Utility function #</a></h2> The utility a customer obtains from choosing a professional depends on: How much time has passed from the moment of the order to the proposal from the professional.</li> The rating and the number of reviews in the professional's profile.</li> When the customer was last online.</li> And other factors.</li> </ul> If we represent the utility as a function of these variables, then the parameters of the function can be estimated by minimizing the cross-entropy (maximum likelihood estimation): $$ \begin{split} H(Y, P(Y)) & = -\sum_i (Y_i \log P(Y_i = 1) + (1 - Y_i) \log P(Y_i = 0)) \\ & = \sum_i (\log (1 + \sum_{j \in J_i} e^{U_{ij}-U_{i0}}) - Y_i \log (\sum_{j \in J_i} e^{U_{ij}-U_{i0}})) \end{split} $$ We started with a basic model that accounts only for how much time has passed from the moment of the order to the proposal from the professional. And this basic model helped us significantly increase the number of deals. Combined target #</a></h2> Let's review the components of the successful solution: A model $f(X_1)$ that takes a feature set $X_1$ and predicts the probability that the professional will send a proposal to the customer.</li> A model $g(X_2)$ that takes a feature set $X_2$ and predicts the uplift in the probability of a deal in the order caused by a proposal from the professional.</li> Orders are sorted by the product of these models' predictions $f(X_1) g(X_2)$.</li> </ul> This solution has its disadvantages: Using two models in production increases the complexity of the solution.</li> The model $f(X_1)$ should predict the probability. This limits how the model can be improved. For example, the loss function is the cross-entropy. We cannot use loss functions that are more suitable for the learning to rank problem.</li> </ul> These disadvantages can be resolved by training a model on a combined target. Instead of training a model $f(X_1)$ on an indicator of a proposal $y$, we can train a model $h(X_1, X_2)$ on a combined target $y g(X_2)$. This solution resulted in a further increase in the number of deals. Conclusions #</a></h2> In two posts I explored the ranking problem in a services marketplace. It's a two-sided matching platform where both sides have to choose each other for a deal to happen. I explained the solution that significantly increased the number of deals in the marketplace. The right problem formulation is more important than a sophisticated solution. When one side searches and proposes, and the other side chooses among the proposals, it's not sufficient to maximize the probability of a proposal. Ranking by the uplift in the probability of a deal increased the number of deals in the marketplace. When two sides choose, it makes sense to model two processes separately: Probability of a proposal from the side that chooses first.</li> Uplift in the probability of a match caused by a proposal.</li> </ul> A discrete choice model can be used to estimate the uplift in the probability of a match. If one of the models should predict the probability measure, it limits the model improvements. Training a single model on a combined target can resolve this issue. Ranking in a two-sided matching platform (Part 1) 2023-08-15T00:00:00+00:00 Overview #</a></h2> Profi is a two-sided matching platform where customers can find tutors, beauty masters, plumbers, and other professionals, and professionals can find customers. It's a horizontal marketplace with over 900 types of services provided by professionals. The quality of search and ranking is important for the matching process there. In a two-sided matching platform, both sides have to choose each other for a deal to happen, and that makes the ranking problem harder and kind of unique. In two blog posts, I'll explore the ranking problem and explain the solution that significantly increased the number of deals in the marketplace. In this first post, I will cover: The concept of matching markets and their distinctions from other markets.</li> The mechanics of Profi's matching process.</li> Business objectives.</li> The three versions of the problem formulation, with insights into why certain approaches succeeded while others did not.</li> </ul> In the second post</a>, I will cover the details of the successful solution that was implemented in production. Matching markets #</a></h2> When a customer places an order on Amazon, the seller doesn't get to choose whether or not to sell to this customer. And this is an example of a commodity market. In matching markets, you can't just choose and get what you want—you also have to be chosen. Examples of matching markets and platforms: Job market.</li> Airbnb before introducing the instant booking feature.</li> Dating platforms, like Tinder and Bumble.</li> Platforms for freelancers, like Upwork and Fiverr.</li> </ul> While in commodity markets prices do most of the work and balance supply and demand, in matching markets, price is not the most important factor. Market design, search, and ranking algorithms play a crucial role there. To learn more about matching markets read "Who Gets What―and Why: The New Economics of Matchmaking and Market Design"</a> by Alvin Roth, a winner of the Nobel Prize in Economics. Matching process #</a></h2> The process of how customers and professionals can find each other on Profi: Customers place orders and describe the details of the work.</li> Professionals search, choose orders, and send proposals to the customers.</li> The customers review the proposals and choose the professionals, or don't choose anyone.</li> If a professional sends a proposal to a customer, and the customer chooses the professional, then this is a deal.</li> </ul> Professionals choose orders and send proposals </figcaption></figure>Customers review proposals and choose professionals </figcaption></figure>Business objective #</a></h2> The order feed in the app for professionals is the source of 56% of proposals. Orders in the feed are selected based on a professional's preferences and previous activity. Originally, orders were sorted in reverse chronological order. While order "freshness" is important for professionals, it's not the only factor of choice. The hypothesis was that we could improve the matching process by implementing a new order ranking algorithm. The objective was to increase the number of deals, which is one of the two most important metrics in the company. Problem formulation #</a></h2> Probability of a proposal #</a></h3> The first idea was to sort orders by the probability of a proposal. The intuition was the following: Sorting by the probability of a proposal would help professionals to find more relevant orders.</li> Professionals would send more proposals.</li> Customers would receive more proposals and have more choice.</li> A wider choice of proposals would increase the probability of finding the right professional.</li> The number of deals would increase.</li> </ul> Predicting the probability of a proposal </figcaption></figure> We trained a machine learning model with the following target and conducted an experiment: Proposal</th> Deal</th> Target</th></tr></thead> Yes</td> Yes</td> 1</td></tr> Yes</td> No</td> 1</td></tr> No</td> No</td> 0</td></tr> </tbody></table> Predicting the probability of a proposal</figcaption></figure> The experiment resulted in the following: The number of proposals had indeed increased.</li> But the number of deals had not increased.</li> </ul> The interpretation was that extra proposals were sent to customers who already had enough proposals to choose from. Increasing the number of proposals without increasing the number of deals is not good for either professionals or customers: Professionals make an extra effort to write and send proposals, but they don't get more customers.</li> Customers are distracted by additional proposals while they already have enough to choose from.</li> </ul> Probability of a deal #</a></h3> The next intuitive choice was to sort orders by the probability of a deal, and to train a model with the following target: Proposal</th> Deal</th> Target</th></tr></thead> Yes</td> Yes</td> 1</td></tr> Yes</td> No</td> 0</td></tr> No</td> No</td> 0</td></tr> </tbody></table> Predicting the probability of a deal</figcaption></figure> To be precise, for a pair of a professional and a customer (P, C), a model should have predicted the probability that: The professional P would send a proposal to the customer C.</li> And the customer C would choose the professional P among others.</li> </ul> Predicting the probability of a deal </figcaption></figure> But, if you think carefully, you can see that this is not the right problem formulation in this case: First, imagine that the first professional on the picture above has sent a proposal to the first customer, and that this customer has chosen this professional.</li> Then the second professional wouldn't get a deal with the first customer. Does that mean that the second professional shouldn't have sent the proposal?</li> Now, imagine that the first professional hasn't sent a proposal. Then the proposal from the second professional could be valuable.</li> </ul> A customer can choose another professional </figcaption></figure> We decided not to move forward with this formulation of the problem as it doesn't correctly reflect the matching process. Uplift in the probability of a deal #</a></h3> The last idea was to estimate how showing an order to a professional would change the probability of a deal in that order, no matter with which professional. The model should calculate a difference in the probability of a deal in an order between two parallel universes: The one where we show the order to the professional.</li> The one where we don't show the order to the professional.</li> </ul> Models that compare parallel universes are called uplift models. Uplift in the probability of a deal </figcaption></figure> Ranking orders by the uplift in the probability of a deal showed the following results in the experiment (measured with proxy metrics): The number of deals coming from the order feed had increased by 10%.</li> The total number of deals had increased by 5%.</li> </ul> It was a great result in itself. But it also opened a door for further improvements, not only in the ranking of orders. For example, finding more relevant orders and adding them to the order feed automatically increases the number of deals. I will get into some details of the model in the next post</a>. Stay tuned. P.S. Type III error #</a></h2> This case reminded me of a notion of a type III error. Before I explain what it means, let me give you examples of type I and type II errors in the context of the ranking of orders: Imagine we show an order to a professional and expect the professional to send a proposal. But the professional doesn't send a proposal. That's a type I error.</li> Now imagine we don't show an order to a professional because we don't expect the professional to send a proposal. But if we showed it, the professional would send a proposal. That's a type II error.</li> </ul> Well, whether the professional would send a proposal or not is not the right question in the first place, at least not the final one. And that's a type III error</a>. Also, check out Cassie Kozyrkov's excellent explanation: https://youtu.be/R51nueuAyIY</a>. rico: rich content to HTML as easy as Doc(df, plot) 2023-08-10T00:00:00+00:00 What is rico? #</a></h2> rico</a> is a Python package for creating HTML documents from rich content: dataframes, plots, images, markdown, etc. It provides a high-level, easy-to-use API with reasonable defaults, as well as low-level access for better control. Why rico? #</a></h2> One might wonder why they need another package if they can create HTML documents with Jupyter and nbconvert. If you are working with data in Jupyter notebooks, you probably don't need rico. But if you are working with data in Python scripts, rico can be helpful. Here is my use case. I'm working with data in Python scripts and organizing them into data pipelines using Prefect. Sometimes I want to create a report as the last step of a pipeline. With Jupyter I would have the following routine: Create a task to save all the dataframes I need for my report in Parquet files or in a database.</li> Create a Jupyter notebook to load the dataframes and visualize all the tables and plots I need.</li> Create a task to convert the notebook to HTML using nbconvert.</li> </ul> Creating an extra task and an extra notebook is not what bothers me the most. I have to think about how to pass data to a notebook. Sure, it's not a very hard job, but it's not a job I want to spend my time on. With rico, I simply create a task to add all objects I want to visualize to a document and save it to an HTML file. How to use rico? #</a></h2> Installation #</a></h3> pip install rico</code></pre> rico has no dependencies other than standard Python packages. For Markdown support install markdown-it-py</a> or Python Markdown</a> or set your own Markdown renderer using rico.set_config</code>. Basic usage #</a></h3> rico provides both declarative and imperative interfaces. Declarative style: import pandas as pd import rico df = pd.DataFrame( { "x": [2, 7, 4, 1, 2, 6, 8, 4, 7], "y": [1, 9, 2, 8, 3, 7, 4, 6, 5], }, index=pd.Index(list("AAABBBCCC")), ) plot = df.plot.scatter(x="x", y="y") doc = rico.Doc("Hello, World!", df, plot, title="My doc")</code></pre> The result: </a>Click on image to zoom in. </figcaption></figure> Imperative style: doc = rico.Doc(title="My doc") doc.append("Hello, World!") doc.append(df, plot)</code></pre> The result will look the same. Serialize the document to HTML using str(doc)</code>: with open("doc.html", "w") as f: f.write(str(doc))</code></pre> Also, you can call doc.serialize(indent=True)</code> to indent the HTML element tree. For brevity, I will omit the serialization and saving to file in the following examples. Content types #</a></h3> rico automatically recognizes the following content types: rico content classes (subclasses of rico.ContentBase</code>).</li> Matplotlib Pyplot Plots.</li> Dataframes, Seaborn plots, Altair charts and other types with IPython rich representation methods</a>. Basically, rico supports the same types as Jupyter.</li> Text.</li> </ul> Use specific classes for plots and text to change the default behavior: doc = rico.Doc( rico.Text("Hello, World!", mono=True), # The default value is False. df, rico.Plot(plot, format="png", bbox_inches="tight"), # The default value is "svg". title="My doc", )</code></pre> The following code gives the same result as the code above: doc = rico.Doc(title="My doc") doc.append_text("Hello, World!", mono=True) doc.append(df) doc.append_plot(plot, format="png", bbox_inches="tight")</code></pre> Use specific classes and methods for other content types: Images: Image</code> or Doc.append_image</code>.</li> Code: Code</code> or Doc.append_code</code>.</li> Markdown: Markdown</code> or Doc.append_markdown</code>.</li> HTML tag: Tag</code> or Doc.append_tag</code>.</li> Raw HTML: HTML</code> or Doc.append_html</code>.</li> </ul> For example: doc = rico.Doc( rico.Markdown("## Dataframe"), df, rico.Tag("h2", "Plot"), # An alternative way to add a header. plot, rico.HTML("<h2>Code</h2>"), # Another way to add a header. rico.Code("print('Hello, World!')"), title="My doc", )</code></pre> The result: </a>Click on image to zoom in. </figcaption></figure> The following code gives the same result as the code above: doc = rico.Doc(title="My doc") doc.append_markdown("## Dataframe") doc.append(df) doc.append_tag("h2", "Plot") doc.append(plot) doc.append_html("<h2>Code</h2>") doc.append_code("print('Hello, World!')")</code></pre>Layout #</a></h3> rico relies on Bootstrap</a> styles. The resulting documents are responsive and mobile-friendly. Use Bootstrap classes to control document layout. For example: import altair as alt doc = rico.Doc( rico.Tag("h2", "Dataframes"), rico.Div( rico.Obj(rico.Tag("h3", "A"), df.loc["A", :], class_="col"), rico.Obj(rico.Tag("h3", "B"), df.loc["B", :], class_="col"), rico.Obj(rico.Tag("h3", "C"), df.loc["C", :], class_="col"), class_="row row-cols-auto", ), rico.Tag("h2", "Plots"), rico.Div( rico.Obj( rico.Tag("h3", "A"), alt.Chart(df.loc["A", :]).mark_point().encode(x="x", y="y"), class_="col", ), rico.Obj( rico.Tag("h3", "B"), alt.Chart(df.loc["B", :]).mark_point().encode(x="x", y="y"), class_="col", ), rico.Obj( rico.Tag("h3", "C"), alt.Chart(df.loc["C", :]).mark_point().encode(x="x", y="y"), class_="col", ), class_="row row-cols-auto", ), title="Grid system", )</code></pre> The result: </a>Click on image to zoom in. </figcaption></figure> How it works: rico.Div</code> represents an HTML container (<div></code>). You can add objects to it, just like with rico.Doc</code>.</li> Each content element is also wrapped in a <div></code> container.</li> rico.Obj</code> is a magic class that automatically determines the content type, just like rico.Doc</code> does.</li> You can specify an HTML class</code> attribute of a <div></code> container for rico.Div</code> and for any content element such as rico.Obj</code>.</li> The "row"</code>, "row-cols-auto"</code>, "col"</code> classes are part of the Bootstrap Grid System. Read more in the documentation</a>.</li> </ul> The following code gives the same result as the code above: doc = rico.Doc(title="Grid system") doc.append_tag("h2", "Dataframes") div1 = rico.Div(class_="row row-cols-auto") doc.append(div1) for name, data in df.groupby(df.index): div1.append(rico.Tag("h3", name), data, class_="col") doc.append_tag("h2", "Plots") div2 = rico.Div(class_="row row-cols-auto") doc.append(div2) for name, data in df.groupby(df.index): div2.append( rico.Tag("h3", name), alt.Chart(data).mark_point().encode(x="x", y="y"), class_="col", )</code></pre>More information #</a></h2> To learn more about how to use rico: Read the more detailed user guide</a>.</li> Take a look at the self-explanatory examples</a> with resulting HTML documents.</li> Install</a> rico, create your own documents.</li> Check the docstrings for details.</li> </ul> Feel free to create an issue</a> to report bugs or submit a suggestion.

Prices as model weights: learning from dispersed data

Ladder of questions in analysis

Curious facts about Ronald Fisher

tea-tasting: a Python package for the statistical analysis of A/B tests

This meeting could have been

Readme Driven Development for an A/B testing analysis package

Debunking the Dunning–Kruger effect with random number simulation

Ranking in a two-sided matching platform (Part 2)

Ranking in a two-sided matching platform (Part 1)

rico: rich content to HTML as easy as Doc(df, plot)

What is rico? #</a></h2>
rico</a> is a Python package for creating HTML documents from rich content: dataframes, plots, images, markdown, etc. It provides a high-level, easy-to-use API with reasonable defaults, as well as low-level access for better control.</p>

Evgeny Ivanov

Open source contribution and job search as matching markets under AI-induced congestion

Comparing Python packages for A/B test analysis: tea-tasting, Pingouin, statsmodels, and SciPy

Evgeny Ivanov

Open source contribution and job search as matching markets under AI-induced congestion

Dating platform #</a></h3> Dating platforms are another example of matching markets. Let's consider Tinder:</p> People create a profile with photos and a short bio, then browse nearby users.</li> You swipe right if you’re interested and left if you’re not.</li>

Comparing Python packages for A/B test analysis: tea-tasting, Pingouin, statsmodels, and SciPy

Typical per-metric analysis output #</a></h3> For each metric, analysts usually want the same core fields:</p> Metric value in control.</li> Metric value in treatment.</li> Effect size estimate and confidence interval (absolute and/or relative).</li>

Prices as model weights: learning from dispersed data

Ladder of questions in analysis

Curious facts about Ronald Fisher

tea-tasting: a Python package for the statistical analysis of A/B tests

This meeting could have been

Readme Driven Development for an A/B testing analysis package

tea-tasting: statistical analysis of A/B tests #</a></h2> tea-tasting is a Python package for statistical analysis of A/B tests that features:</p> Student's t-test, Z-test, and Bootstrap out of the box.</li> Extensible API: Define and use statistical tests of your choice.</li>

Debunking the Dunning–Kruger effect with random number simulation

Random number simulation #</a></h2> As previously mentioned, these experiments didn't prove the Dunning–Kruger effect. To demonstrate this, I'll:</p> Simulate the experiments with random data implying that the effect is not real.</li>

Ranking in a two-sided matching platform (Part 2)

Ranking in a two-sided matching platform (Part 1)

rico: rich content to HTML as easy as Doc(df, plot)