Motivated by a study of inter-surgeon variability in assessments of surgical priority scores, we develop a hierarchical Bayesian modeling approach to the analysis of studies of observer variation for bounded integer-valued scoring systems. The hierarchical model structure accounts for the correlated nature of the study design which involves multiple raters and multiple scoring tools. A latent priority score serves to link assessments of the same vignette. We account for the tendency for variability to be less at the extremes than in the middle of the scoring range by modeling within surgeon variation using a beta-binomial distribution and between surgeon variation with a logit-normal distribution. Interpretable inferences can be obtained by focussing on descriptors of the distributions of scores and expected scores implied by the fitted model. The combination of hierarchical modeling and an inferential focus on distributions of responses, or expected responses, provides a comprehensive and flexible approach to the analysis and reporting of observer variability studies. An appealing feature of the approach adopted is that inferences can be obtained for scoring tools not directly evaluated but deterministically linked to studied tools. This provides a framework for post-study exploration of the impact on variability of alternative strategies for constraining scores.