Reference: Flank and Rear Charges, Page 22.

If a unit is charged in the flank or rear by an enemy unit of 5 models or more then it loses its rank bonus in combat. This represents the unit being thrown into confusion as a result of the attack, preventing troops from lending their weight to the fighting ranks.

Note that the penalty applies for the rest of the combat and continues to apply even if the unit charging to the flank or rear is subsequently broken and flees. The point is that the unit has become disordered, and won't get a chance to reorganise itself until the combat is over.

We decided that 5 models were needed to prevent single flyers or character models gaining an unfair advantage by zipping round onto an enemy's flanks. However, you could make a case where especially large monsters count. In the end we found it impossible to frame a general rule to this effect, but suggest that large monsters with 5 or more wounds on their profile count as 5 models for this purpose.