-
Local Approximation of Discounted Markov Decision Problems by Mathematical Programming Methods
(2011)
- We develop a method to approximate the value vector of discounted Markov decision problems (MDP) with guaranteed error bounds. It is based on the linear programming characterization of the optimal expected cost. The new idea is to use column generation to dynamically generate only such states that are most relevant for the bounds by incorporating the reduced cost information. The number of states that is sufficient in general and necessary in the worst case to prove such bounds is independent of the cardinality of the state space. Still, in many instances, the column generation algorithm can prove bounds using much fewer states. In this paper, we explain the foundations of the method. Moreover, the method is used to improve the well-known nearest-neighbor policy for the elevator control problem.
-
Stability with uniform bounds for online dial-a-ride problems under reasonable load
(2011)
- In continuously running logistic systems (like in-house pallet transportation systems), finite buffer capacities usually require controls achieving uniformly bounded waiting queues (strong stability). Standard stochastic traffic assumptions (arrival rates below service rates) can, in general, not guarantee these strong stability requirements, no matter which control. Therefore, the worst-case traffic notion of reasonable load was introduced, originally for the analysis of the Online-Dial-a-Ride Problem. A set of requests is reasonable if the requests that are presented in a sufficiently large time period can be served in a time period of at most the same length. The rationale behind this concept is that the occurrence of non-reasonable request sets renders the system overloaded, and capacity should be extended. For reasonable load, there are control policies that can guarantee uniformly bounded flow times, leading to strong stability in many cases. Control policies based on naive eoptimization, however, can in general achieve neither bounded flow times nor strong ability. In this chapter, we review the concept and examples for reasonable load. Moreover, we present new control policies achieving strong stability as well as new elementary examples of request sets where naive reoptimization fails.
