Some additional references on Bayesian statistics:

Leonard and Hsu. Bayesian Methods. Cambridge University Press.
Gelman, Carlin, Stern, and Rubin. Bayesian Data Analysis. Second edition 2003 or first edition 1995. Chapman and Hall/CRC press.
Gilks, Richardson, and Spiegelhalter. Markov Chain Monte Carlo in Practice. 1995. Chapman and Hall/CRC press.
Gamerman, D. Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference. 1997. Chapman and Hall/CRC press.
Draper, D. Bayesian Modeling, Inference and Prediction. Draft version.

For more details on Markov chains and Markov chain Monte Carlo:

Haggstrom. Finite Markov Chains and Algorithmic Applications. Cambridge University Press

For a review of calculus-based probability and inference, I suggest

DeGroot, M. and Schervish, M. Probability and Statistics. 2002. Addison-Wesley, third edition.

Other alternatives include (in decreasing order of difficulty):

Casella, G. and Berger, R. Statistical Inference. 2001. Wadsworth, second edition.
Hogg, McKean, and Craig. Introduction to Mathematical Statistics. 2005. Prentice Hall, sixth edition.
Ross, S. A First Course in Probability. 2001. Prentice Hall, sixth edition.
Wackerly, Mendenhall, and Scheaffer. Mathematical Statistics with Applications. 2002. Duxbury, sixth edition.