The objective of metric learning is to search a suitable metric for measuring distance or similarity between samples. Usually, it aims to minimize the distance between samples of same class and maximizes the distance between samples of different classes. However, most metric learning methods do not consider the sizes of classes, which may cause negative impact on the performance in classification since the size of a cluster is usually ignored in the distance comparison. In this work, we propose a triplet loss with variance constraint. Our method focuses not only on the distances between samples but also on the sizes of classes. The size difference between classes is also minimized in our objective function. The experimental results confirm that our method outperforms the one without the class size variance.