Abstract: A detection algorithm of moving objects based on background subtraction method was proposed for a video surveillance system with static camera. Some multi-background models dealing with RGB colors were built during the leaning process without any restraint. The modes can be updated in good time with the change of background during moving object segmentation. Chromatic difference and cross-entropy were employed to eliminate shadows and made the moving object areas more accurate. This algorithm demonstrates better performance in the experiments with video sequences captured by a general USB camera.