Really easy, if you use sass, just ad:

@include media-breakpoint-down(sm) {
  @include pagination-size($pagination-padding-y-sm, $pagination-padding-x-sm, $font-size-sm, $pagination-border-radius-sm);
}