An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning

Fig 1. Conceptual illustrations of the model adaptation on the blue, red and yellow tasks. (a) MAML is the classical inductive method that meta-learns a network initialization θ that is used to learn a single base-learner on each task. (b) SIB is a transductive method that formulates a variational posterior as a function of both labeled training data T(tr) and unlabeled test data x(te). It also uses a single base-learner and optimizes the learner by running several synthetic gradient steps on x(te). (c) Our E3BM is a generic method that learns to combine the epoch-wise base-learners, and to generate task-specific learningcrates α and combination weights v that encourage robust adaptation.

Abstract

Few-shot learning aims to train efficient predictive models with a few examples. The lack of training data leads to poor models that perform high-variance or low-confidence predictions. In this paper, we propose to meta-learn the ensemble of epoch-wise empirical Bayes models (E3BM) to achieve robust predictions. "Epoch-wise" means that each training epoch has a Bayes model whose parameters are specifically learned and deployed. "Empirical" means that the hyperparameters, e.g., used for learning and ensembling the epoch-wise models, are generated by hyperprior learners conditional on task-specific data. We introduce four kinds of hyperprior learners by considering inductive vs. transductive, and epoch-dependent vs. epoch-independent, in the paradigm of meta-learning. We conduct extensive experiments for five-class few-shot tasks on three challenging benchmarks: miniImageNet, tieredImageNet, and FC100, and achieve top performance using the epoch-dependent transductive hyperprior learner, which captures the richest information. Our ablation study shows that both "epoch-wise ensemble" and "empirical" encourage high efficiency and robustness in the model performance.

Pipeline

Fig 2. The computing flow of the proposed E3BM approach on one meta-training episode.

Performance

Table 1.  Few-shot classification accuracy (%) on miniImageNet, tieredImageNet, and FC100 (5-class).

Citation

Please cite our paper if it is helpful to your work:
@article{Liu2020E3BM,
  author    = {Yaoyao Liu and
               Bernt Schiele and
               Qianru Sun},
  title     = {An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning},
  journal   = {arXiv},
  year      = {2020}
}