Clarification on tuning "frozen layers" hyperparameter with RayTune
#570
by
ro2h5
- opened
Hi,
in your supplementary regarding disease modeling it says "Hyperparameters (max learning rate, learning scheduler, warmup steps, weight decay, seed, frozen layers) were tuned using RayTune."
I was wondering how to tune the number of frozen layers in practice. Which parameter should be set in the RayTune search space (ray_config) to optimize this?
From what I can see, the number of frozen layers appears to be set inside the Classifier class before the HPO starts.
Thanks!
Thank you for your question! You can run different instances of RayTune for each selection of frozen layers, or you can use Optuna, which is implemented in the multitask method (can be run as single task), to set the range to be included in the tuning process.
ctheodoris
changed discussion status to
closed