Documentation revamp #232
Documentation revamp #232
Conversation
|
Probably is worth to add Experiment analysis scorecards |
There was a problem hiding this comment.
Hey @luizhsuperti , thanks for the PR! I've had a quick read, let me know what you think
| power_normal = npw.power_analysis(df, average_effect=0.1) | ||
| power_line_normal = npw.power_line(df, average_effects=[0.1, 0.2, 0.3]) | ||
| ### 📌 **Experiment Design & Planning** | ||
| - **Power analysis** and **Minimal Detectable Effect (MDE)** estimation |
There was a problem hiding this comment.
I'd add something on simulation-based power analysis
| - **Power analysis** and **Minimal Detectable Effect (MDE)** estimation | ||
| - Support for **complex experimental designs**, including: | ||
| - 🏢 **Cluster randomization** | ||
| - 🔄 **Switchback experiments** |
| - 🏢 **Cluster randomization** | ||
| - 🔄 **Switchback experiments** | ||
|
|
||
| ### 🛠 **Data Preprocessing** |
There was a problem hiding this comment.
I'd remove this section, I don't think the pandas integration is very relevant nor there are tools for data preparation in the lib
|
|
||
| ### 📊 **Comprehensive Experiment Analysis** | ||
| ##### **✅ Metrics** |
There was a problem hiding this comment.
I'd drop the metrics one for now since it looks like we have a bug (see last issue)
| - 📌 **Generalized Estimating Equations (GEE)** | ||
| - 📌 **Mixed Linear Models** for robust inference | ||
| - 📌 **Ordinary Least Squares (OLS)** and **Clustered OLS** with covariates | ||
| - 📌 **T-tests** with variance reduction techniques (**CUPED, CUPAC**) |
There was a problem hiding this comment.
I'd merge this and the one above, and not mention t-tests since mostly its OLS with covariates, cuped, cupac
| 📦 **Installation:** | ||
| ```sh | ||
| pip install cluster-experiments | ||
| ======= | ||
| # MDE calculation | ||
| mde = npw.mde(df, power=0.8) |
There was a problem hiding this comment.
for the MDE example, I have to asks: needs to be reproducible (so dataframe needs to be created), and show the methods power_analysis, mde, power_line and mde_line. wdyt?
| ``` | ||
|
|
||
| !!! info "Python Version Support" | ||
| **Cluster Experiments** requires **Python 3.9 or higher**. Make sure your environment meets this requirement before proceeding with the installation. |
| ## Quick Start | ||
|
|
||
| ### Power Analysis Example | ||
| **`cluster experiments`** is a comprehensive Python library for end-to-end A/B testing workflows, designed for seamless integration with Pandas in production environments. |
There was a problem hiding this comment.
designed for seamless integration with Pandas in production environments.
I'd remove any production mention, I don't think it's fair to call this production. "seamless integration" sounds generated by an LLM, do you have a more natural equivalent?
| Designing and analyzing experiments can feel overwhelming at times. After formulating a testable hypothesis, | ||
| you're faced with a series of routine tasks. From collecting and transforming raw data to measuring the statistical significance of your experiment results and constructing confidence intervals, | ||
| it can quickly become a repetitive and error-prone process. | ||
| *Cluster Experiments* is here to change that. Built on top of well-known packages like `pandas`, `numpy`, `scipy` and `statsmodels`, it automates the core steps of an experiment, streamlining your workflow, saving you time and effort, while maintaining statistical rigor. |
There was a problem hiding this comment.
I'd make the paragraph shorter and stress what it automates: being MDE/power calculation and inference scorecards
There was a problem hiding this comment.
given the next examples, I think it's worth mentioning that you're describing the simulaiton-based power analysis, and there are other pipelines like power analysis based on normal approximation and scorecard generation
There was a problem hiding this comment.
I like the explanation style, maybe you could write a similar thing for NormalPowerAnalysis and AnalysisPlan
| ```python | ||
| from cluster_experiments import TTestClusteredAnalysis | ||
|
|
||
| analysis = TTestClusteredAnalysis( |
There was a problem hiding this comment.
let's use ClusteredOLS, I think this analysis method is a bit weird
|
Hey @luizhsuperti, I was playing with this and found an issue, all examples are under switchback, whenever you can have a look please :) |
Proposal for Documentation Reorganization
linked issue
I was thinking about reorganizing the documentation to improve clarity and usability. Here’s the proposed structure:
Home (README)
Quickstart (Installation + Package Overview: Splitter, Perturbator, and Analysis classes)
Documentation (APIs, details on classes and methods)
Usage Examples (Notebooks with practical examples)
Contribution (Guidelines for contributing)
In my fork,
👉 [I revamped the README to be more appealing to a broader audience
I also modified the Quickstart section to align with this idea
I’d love your feedback on whether the Quickstart section information is accurate—I’m still new to the package, so there might be some errors in definitions or concepts. (The Python notebooks currently in the docs are safe and should still be included in some form.)
For inspiration, I looked at the documentation structures of:
Ambrosia
pysurvival
Why This Change?
Makes it easier for new users to navigate and understand the package.
Provides a clearer structure for future contributors.
The examples section can be refined over time for better clarity and accuracy.
In the future, we could also add a "Stats 101" section for foundational concepts.
Let me know what you think!