I've been working on a late delivery prediction project (will an order arrive late, predicted at the time it's placed) using order data from a supply chain company. The model works okay, but I've hit a wall and I'm not sure if it's normal or if I'm doing something wrong.
What I did, quickly: I removed canceled and suspected fraud orders (7,754 rows) because their labels didn't follow the delivery rule. The rule "real days > scheduled days" matched the label on 97.55% of rows before, and 100% after removing them. I dropped anything that leaks the answer (delivery status, actual shipping days, shipping date) plus IDs and personal info. The data has one row per item, but the label is the same for every item in an order, so I collapsed it to one row per order (172,765 rows down to 62,897 orders). Then I split by date instead of randomly: train is Jan 2015 to Mar 2017, validation to Sep 2017, test to Jan 2018. Anything I learn from the data (encodings, thresholds) comes from train only.
I compared 5 models on 7 features, then tuned LightGBM and XGBoost with Optuna (25 trials, time-ordered folds). Test period results at the 0.50 cutoff:
| Model |
Accuracy |
AUC |
Precision |
Recall |
| Logistic regression |
0.724 |
0.775 |
0.897 |
0.587 |
| Random forest |
0.723 |
0.788 |
0.882 |
0.598 |
| LightGBM |
0.724 |
0.800 |
0.883 |
0.601 |
| XGBoost |
0.725 |
0.789 |
0.887 |
0.598 |
| CatBoost |
0.724 |
0.782 |
0.895 |
0.591 |
| LightGBM tuned |
0.724 |
0.777 |
0.897 |
0.587 |
| XGBoost tuned |
0.724 |
0.775 |
0.897 |
0.587 |
Always guessing "late" gets 57.5% on the test set. So the models all land in the same spot, plain logistic regression matches everything, and tuning did nothing.
The reason, as far as I can tell, is that almost all the signal is shipping mode (77% of the importance in the LightGBM model). First Class is late 100% of the time, Second Class about 80%, Standard Class about 40%. Same Day orders placed before noon were never late in train and the ones placed from noon were always late, and that held on validation too. So the model is perfect on First Class and Same Day. The problem is Standard Class. It's 60% of my test orders and I only get 60.4% accuracy on it, which is basically the same as always saying "on time".
I tried a lot of other stuff. 26 features in groups: location, customer segment, payment type, order size, market and region, and shipping mode combined with hour, weekday and region. None of them improved validation enough to keep, so the final model only uses 7 simple ones. Sales, discount, profit and quantity all have 0.00 correlation with the label. Market is flat (56.8% to 57.7% late).
One more weird thing: in the last two quarters orders go from about 3 items to about 1, and almost every order comes from a customer with no earlier order in the data. The late rate stays around 57% though.
So I have a few questions:
Is this normal for this kind of data? The labels feel rule driven, maybe synthetic, and I'm wondering if anyone has found real signal beyond shipping mode.
Would you change the framing, like modeling only the Standard Class orders, or predicting how many days late instead of yes/no?
I haven't built customer history features (past late rate, number of orders) yet. Is it worth it, and how do you build those without leaking future info?
And honestly, is it okay to present this as a portfolio project when the main finding is "one feature does almost everything"? Or should I change direction?
Happy to share more details if it helps.
Thanks!