We consider a class of discrete-time Markov control processes with Borel state and action spaces, and $\Re^{k}$-valued i.i.d. disturbances with unknown density $\rho.$ Supposing possibly unbounded costs, we combine suitable density estimation methods of $\rho$ with approximation procedures of the optimal cost function, to show the existence of a sequence $\{\hat{f}_{t}\}$ of minimizers converging to an optimal stationary policy $f_{\infty}.$