In numerical interpretation of potential field data, inverse methods are utilized aiming at more accurate mapping the subsurface Earth. In order to solve nonlinear inverse problems of potential field data, conventional methods based on Jacobean matrix and hessian matrix are used. Such methods usually lead to local minimums and consequently representation of inaccurate geophysical models. In this study, to constrain inverse problem and reach better convergence of the optimization process, a regularization algorithm called trust region sub-problem algorithm is employed. The proposed method possesses high convergence rate and leaves behinds local minimums with much less numerical computations than evolutionary algorithms. This algorithm is first tested on the standard mathematical benchmark and the results is compared with the solution of Newton line search-based method. An algorithm that adjoin it to least squares problem is presented. Then, the algorithm is applied to potential field data along with comparing the results with Levenberg-Marquardt algorithm and the drilling information.