- Even thoughthere are several low-rank adaptation techniques
in the literature, such as Adapter (Houlsby et al.,2019b), Compacter (Karimi Mahabadi et al., 2021), and LoRA (Hu et al., 2021b); they all suffer from
two major common problems: first, it is not clear how to select the size of their rank (while their per-formance is very sensitive to this rank selection); second, their training is static which means that if a low-rank model is trained based on a particular rank size, it will not work well in other rank values (i.e. for any other rank value we need to train a separate model)
