Random Forest calculator

Fit a random forest online for classification or regression, with out-of-bag error, permutation variable importance, and a cross-validated comparison against a logistic or linear benchmark.

What it does

Runs several model families against each other on identical cross-validation folds, scores them on the same metric, and reports a leaderboard with a recommendation. Which families compete is decided by the task: classification when the target is binary, regression when it is continuous.

When to use it

You want to know which kind of model suits this data before committing to one, and prediction — not explanation — is the goal.

Alternatives

• the Prediction module — you have already decided on a logistic model or a forest and want a deployable risk model with calibration and external validation
• Regression — you want the effect of a particular predictor with a confidence interval
• Ridge or Lasso in the Regression tab — you know you want a penalized linear model and want the cross-validated λ curve

How to read the output

The recommendation — read this before the leaderboard

The recommendation is boring-benchmark-first: the simple baseline is recommended unless a more complex model beats it by more than one standard deviation of the cross-validated metric. A model that wins by less than that has not been shown to be better on this data. The top row of the leaderboard is not automatically the answer. Fold-to-fold variation is real, and a 0.004 AUROC lead across five folds is noise wearing a rosette. When the recommendation disagrees with the ranking, the recommendation is the one that accounts for that.

The leaderboard and its metric

Every model scored on the same folds with the same primary metric — AUROC, Brier or accuracy for classification; R², RMSE or MAE for regression. Same folds is what makes the comparison legitimate: differences are between models, not between lucky splits. It also means the scores are correlated, so their differences are more reliable than each score's own precision suggests.

The spread across folds

How much each model's score moved from fold to fold — the noise band the recommendation is measured against. A model with a high mean and a wide spread is not better than a steadier one just below it. With few rows, or few events, the spread can exceed every gap on the board, and the honest conclusion is that this data cannot separate the models.

k-fold cross-validation

The data are split into k parts; each part is scored by a model trained on the others. k is capped automatically so no fold is empty and, for classification, so every fold contains members of the smaller class. The cross-validated score corrects for fitting the model. It does NOT correct for anything you decide after seeing the leaderboard — re-running with a different predictor set and keeping the best result is fitting on the cross-validation, and the reported score stops being honest at that point.

AUROC and Brier (classification)

AUROC measures ranking — whether cases with the event score higher. Brier measures the squared error of the probabilities, so it penalises being confidently wrong and rewards being well calibrated. They can disagree, and the disagreement is informative: a model can rank well (high AUROC) while its probabilities are badly wrong (poor Brier). If the numbers will be shown to anyone or used at a fixed threshold, the Brier score is the one that matters.

R², RMSE and MAE (regression)

R² is the share of variance explained; RMSE is the typical error in the outcome's own units, weighted towards large misses; MAE is the typical error weighted equally. RMSE and MAE are in your units, which makes them the ones a reader can judge. A cross-validated R² can be NEGATIVE — that means the model predicts worse than the outcome's own mean, which is a real and reportable result rather than a bug.

What is not here

No calibration plot, no coefficients, no confidence intervals, and no external validation. This tab picks a family. Take the winner to the Prediction tab to develop it properly — a leaderboard position is not a validated model, and nothing here has been checked on a cohort the folds did not touch.

How this calculator is validated · Which statistical test should I use?