Crowd Anomaly Detection Using Motion Based Spatio-Temporal Feature Analysis

Recently, the demand for surveillance system is increasing in real time application to enhance the security system. These surveillance systems are mainly used in crowded places such as shopping malls, sports stadium etc. In order to support enhance the security system, crowd behavior analysis has been proven a significant technique which is used for crowd monitoring, visual surveillance etc. For crowd behavior analysis, motion analysis is a crucial task which can be achieved with the help of trajectories and tracking of objects. Various approaches have been proposed for crowd behavior analysis which has limitation for densely crowded scenarios, a new object entering the scene etc. In this work, we propose a new approach for abnormal crowd behavior detection. Proposed approach is a motion based spatio-temporal feature analysis technique which is capable of obtaining trajectories of each detected object. We also present a technique to carry out the evaluation of individual object and group of objects by considering relational descriptors based on their environmental context. Finally, a classification is carried out for detection of abnormal or normal crowd behavior by following patch based process. In the results, we have reported that proposed model is able to achieve better performance when compared to existing techniques in terms of classification accuracy, true positive rate, and false positive rate.


Introduction
Recently, video surveillance technology has grown due to its importance for security requirement and detection of events in public places such as streets, shopping malls, subway station etc. Video surveillance system is widely used to monitor the crowd activities during any public events.Automated analysis and detection of anomalies in crowd activities is a challenging issue in video surveillance system.According to the study presented in [1], abnormal activities are determined based on deviation from the normal or abnormal standard.By considering this interpretation, abnormal activities are defined based on the deviation from normal activities.In real-world scenarios such as pedestrian walking, subjects follow neighboring subjects aiming for the same destination.Abnormal crowd activities affect public safety such as an explosion, fire, disasters etc.This technique of crowd behavior analysis is widely used in real-time applications such as: (a) Visual surveillance Crowd behaviors analysis is applicable for various visual surveillance scenarios such as shopping malls, railway stations etc. Conventional methods are not capable of providing better efficiency due to a huge density of the crowd.(b) Crowd management This approach is utilized for a mass gathering of a crowd such as sports event, music festival etc.Using this approach, the gathering areas can be analyzed and the crowd can be assisted for movement.(c) Public space design Crowd behavior analysis can be used designing public spaces during movement of the crowd in a railway station, shopping malls to increase the safety and efficiency.During last decade, various methods have been developed to address the issue of abnormal crowd behavior.In order to analyze abnormal crowd behavior, crowd modeling is a challenging task for researchers.With the help of crowd modeling, behavior prediction and group formation of people can be analyzed easily which utilizes information related to the geographical or logical state of affected region.In [2], an efficient model is present for crowd behavior modeling by proposing bottom-up methodology.Similarly, decision making for crowd abnormality is a crucial task.In [3], Braun et al. addressed this issue with the help of Helbing method.Thida et al [4] addressed the issue of crowd localization and detection of abnormal crowd activities.Recently, Rao et al [5] developed a new model for abnormal crowd behavior detection based on the motion pattern extraction.This work mainly aims at optical flow framework for motion analysis.Furthermore, this approach is extended to obtain a classification of merging and splitting of the crowd.Another similar approach for classification of moving object is carried out with the help of optical flow measurement.According to this methodology, object segmentation and tracking are performed by applying background subtraction, Kalman filter, silhouette computation and correlation computation between appearance models.Segmented objects are classified by applying a combination of spatiotemporal and static features by utilizing appearance and motion pattern model where spatio-temporal features are computed using adaptive block gradient intensities and HoG features.But these studies suffer from the issue of object boundaries and aperture problem which degrades the performance of analysis.Zhou et al [7] presented a new model for crowd behavior analysis by utilizing pedestrian trajectories fragment.This model suffers from the issue of efficient tracking caused by incomplete information about fragment trajectories, missing object trajectories, entering of a new object in the scene etc.Still, there are various challenges present in this field of crowd behavior detection, tracking, and activity classification.Crowded scenes are extremely cluttered and induced by various occlusions where conventional methods cannot provide the significant performance of detection and classification.According to literature study presented by Ali in [8], it is concluded that nature of human crowd is complex which consists of psychological and dynamic characteristics.This nature of human crowd makes is more complex to estimate the granularity level for the dynamic crowd.

Related Work
In this work, our main aim is to address the issue of analyzing crowd behavior in various surveillance scenarios.In order to perform this, we have developed a robust approach based on sematic concept which is able to provide a relationship between people present in the crowded scene depending upon their environmental context.In the proposed model, the descriptor based model is also developed which results in information extraction between individual object and scene.The novelty of proposed approach is that it is capable of extracting the relational feature in an automatic procedure without performing any manual annotations.A classification model is developed based on the patches of estimated trajectory and motion parameters with the help of spatiotemporal feature extraction which provides better performance for classification of crowd behavior analysis.
Rest of the articles is organized as follows: a brief literature review is presented in section II, proposed approach is depicted in section III and section IV provides an experimental study of proposed model and finally concluding remarks are presented in section V.
This section provides a brief study of recent techniques which are developed recently in the field of crowd behavior analysis.S. Yi et al [9] developed a new model for crowd modeling in video surveillance systems for pedestrian walking scenario.For stationary crowd scenarios, walking objects are ignored which is an affecting parameter for crowd behavior analysis.This work proposes a novel model for pedestrian behavior analysis.In [10] a new method explored for crowd behavior analysis by incorporating virtual environments within information space of computation.This work represents behavior mining approach for crowd behavior analysis and deals with crowd merging and splitting scenario.
In [11] Chen et al. proposed a novel algorithm for crowd behavior detection based on acceleration features of the crowd.Unlike state-of-art techniques, this work uses local features and relationship is extracted between the current and previous state of behavior for activity analysis.Since this method uses optical flow based computation approach which causes IJEECS ISSN: 2502-4752  instability in feature analysis.For detection, foreground extraction based model is developed by using acceleration computation method.For anomaly detection, object segmentation is also applied in this work which makes it more robust for real time scenarios and finally classification of activities is carried out with the help of threshold analysis method.C. Y. Chen et al [12] discussed the abnormal crowd behavior analysis method.In this work behavior, detection and localization carried out by using divergent centers of any video surveillance system.For motion analysis, a well-known technique is used known as optical flow estimation.The motion of given sequence is modeled by obtaining magnitude, direction and position of detected objects and velocity are computed by incorporating motion velocity computation.
H. S. Wong et al [13] developed a new framework for anomaly detection based on the Bayesian model.This work mainly aims of escape detection in the crowd by performing crowd modeling.This method uses a similar concept of divergent center as discussed before which is used for motion characterization and a probability density function is formulated based on optical flow computation.
As we have discussed before that spatiotemporal feature has been proven a significant technique for crowd behavior analysis by estimation motion flow.By taking this into consideration, Zheng et al [14] developed a new method to model the crowd motion patterns with the help of spatio-temporal viscous fluid field feature computation.This helps to compute and estimate the motion for densely crowded scenarios.According to this model, the spatiotemporal matrix is computed first which provides the measurement of local fluctuations in the video data by considering spatial and temporal domain.After computing the spatio-temporal matrix, fluid field scheme is applied to extract the information based on eigenvalue analysis method and finally, a codebook is constructed by utilizing clustering approach in spatio-temporal feature similarity.The classification model is developed by applying Dirichlet model.
Still, the various challenging task is present in this research field.In this literature, we have presented most recent works.From this study is concluded that most of these approaches use optical flow based motion estimation method which is not capable of extracting the boundary of given object.Since crowds are always unstructured during mass gathering which causes various occlusions and ambiguities which cannot be addressed by using state-of-art techniques.State-of-art technique suffers from the issue of performance accuracy.These issues motivate us to develop an efficient model for crowd behavior analysis.

Proposed Model
This section describes proposed an approach for abnormal crowd behavior detection.In order to model the motion flow dynamics is used.Streakline is computed by considering spatial and temporal gradients of computed flow map.In the case of abnormal activities, position and time are crucial parameters which affect flow map computations.To overcome this issue, we develop an adaptive representation of streakline flow with the help of streamline representation of the motion.

Representation of Motion Vector Field
For any given motion sequence particles are computed which are present in the flow field.Motion is analyzed using the movement of particles which are computed using dense optical flow.According to this model, a video sequence is considered as an input which is denoted by considering 3-dimensional array denoted as  × ℋ ×  where width of frame is denoted by , height of frame is denoted as ℋ and  denotes total number of frames which has optical flow such as �  (),  ℎ ()� where , ℎ and  are expressed as  ∈ [1, ], ℎ ∈ [1, ℋ] and  ∈ [1. − 1].First of all particle position are computed at computation grid point for a time  which can be denoted as: Particle positions are denoted by �  (),  ℎ ()� where , ℎ denotes grid point of flow.By computing this process in an iterative approach, curves are obtained which represent particle set which contains trajectory information.As discussed before that for unstable flow we use streakline, pathline and streamline for flow information extraction.

Computation of Streakline
In the previous section, we have discussed that streakline flow computation is time and motion dependent.During computation of streakline flow, shape inconsistency and motion are the main factors which affect the computation of streakline flow.In previous works, optical flow [15] is used for motion information extraction but fails to provide better results in terms of motion information whereas streakline flow presents better results which can be obtained by integrating time of field of velocity which results in better analysis of faster dynamic flow of motion and helps to represent particles efficiently over the computation grid.
Another paradigm considered here is known as streamline which is used for motion estimation.A brief description is presented in next section.

Computation of Streamline
Streamline of any motion vector can be computed by performing the bi-directional integration.As discussed before, streamline can be defined as curves which are tangent in nature to vector field at a given point of motion flow.
This integration computation initialized from a pre-defined seed point and ends when it achieves boundary of the closed path.This process is mainly categorized into three subsections as mentioned below [16]: a) Initial placement of seeds b) Data diffusion c) Final stopping criteria By considering these two techniques, in next section, we develop an efficient model for motion analysis which is used for crowd activity detection.

System Model
In this section we describe proposed system model for motion analysis.According to literature study presented in section II, it can be concluded that existing approaches for motion analysis feature such as moving object detection, object tracking and track analysis etc. but when there is unstable motion present, in that case existing models fail to provide the efficient results.In order to address this issue, here we present a new method which is capable of extracting motion information from unstable movement and encodes spatial and temporal In order to extract particle information, the temporal domain of motion is evaluated under Euler view.In below figure 1 complete flow of proposed model is described.According to the Figure 1, the video sequence is considered as input which is converted into frames before processing.In next stage, we apply sampling along with motion estimation.In order to estimate the motion of given sequence spatio-temporal integration and spatiotemporal integration are computed.This stage provides a complete motion structure of input sequence.After motion estimation, filtering and cell distribution of data into various cells are applied.With the help of motion distribution, complete information of motion is computed.Using this dynamics statistics, streakline and streamline flow is computed which results in segmentation and finally, the motion is classified into normal and abnormal which aims at activity classification.Step 11: Flow interpolation and integration computation Step 12: Trajectory estimation First video sample is converted into frames.For each frame sampling and motion estimation process has implemented.Sampling process provides a number of key points and motion estimation defines the instant flow map.Further filtering is applied with the help of key points and instant flow map.After filtering process, distribution of flow vectors is processed through the spatially enclosed cell.For each cell quantization and clustering has performed to get representation of flow vector.Motion advection is computed through average flow map of previous frame and current frame.Now average streak map of all mini batches has processed.With the help of motion flow vector representation, we extract interpolated flow map.Afterwards we compute the streak flow line which is combination between average streak flow and fine tocoarse streak flow.Applying diffusion technique, we evaluate stream lines which forms video trajectory.

Data Sampling and Estimation of Motion
Here in this section, we perform data sampling where key points are extracted from the input sequence by applying sparse or dense distributions.During processing of video, sequence noise is induced which is estimated by applying data sampling and complexity is reduced.Later motion flow is estimated by computing frame to frame spatio temporal displacement analysis.

Filtering
Once data is sampled then vector flow is expressed is   = (  ,   ,   ,   ), sampling points are denoted by (  ,   ) and motion vectors are denoted as (  ,   ) in  and  direction respecively.In order to build filtering model, each vector flow considers key point location by conisdering initial position vector which is expressed as   () = �  (),   ()�.Here it is assumed that key points and flow vectors are equal and finally median filtering is applied here to perform filtering of motion vector.

Cell Distribution
This subsection provides cell distribution methodology discretion.As mentioned before that data is distributed spatio temporally where each frame is divided into a grid.The resolution of the image depends on frame size and video duration is divided temporally.In this work, each spatiotemporal vector is stored into cell   and consist motion flow vectors.Encoding of flow vector is given as   = (  ,   ,   ,   ,   ) where sampling points are denoted as (  ,   ), flow magnitude length is denoted by   ,   denotes flow angle which is represented with respect to  −  for   frame.These steps are applied for each frame of input sequence.

Computation of Motion Flow and Spatio Temporal Interpolation
For motion estimation and computation, the dense grid is considered which has particles in the grid where each particle contains the information about fluid and their position in the grid.An average flow map is computed by integrating all flow vectors based on the time of sequence.For an, each time-step particles are positioned at  and old particles are imitated from same position as flow field.This process can be expressed as equation 1.Here it is assumed that streak line is a collection of various particles which are achieved from motion estimation resulting in representation of data  in the direction   .
Here we compute streamline of vector flow in terms of probability is defined as: And similarity is defined as: During motion estimation velocity variations are considered where point-to-point velocity difference is computed, this can be expressed as: Using this velocity information of each frame and each particle motion is classified according to the similarity measurement of vector flow.

Experimental results and discussion
In this section, we present comparative experimental results and analysis of abnormal crowd behavior detection using proposed model.

Dataset
In order to evaluate the performance, proposed model is tested on publicly available abnormal crowd dataset.These datasets acquired from University of Minnesota (UMN).This dataset consists of 11 videos from different scenes considering the indoor and outdoor environment.Motion field vector is extracted by partitioning the image into grid size of 8 × 8 blocks.In the UMN dataset, we have conducted two experiments for abnormal crowd behavior analysis.Classification of activity is obtained by the method which is discussed in section III.

Experimental Analysis
In this section, we show experimental study for crowd behavior analysis using proposed approach.Initially, we consider sequence 1 (corridor) as the first input for processing.According to proposed model, the video sequence is converted into the frame.First frame and corresponding steps according to proposed model are analyzed here.In Figure 3, initial frame is shown.This frame is processed further and motion flow map is computed as depicted in Figure 4.For this sequence input video frame is given in Figure 3, motion map computation corresponding to this frame motion flow map is computed by considering x and y-direction motion vector which is presented in Figure 6 and Figure 7. Based on the motion map, gradients are computed in x and y-direction gradients are computed as shown in Figure 8 and 9. Motion dynamics and motion information is extracted and detected in Figure 10 and figure 11.Finally, the segmented motion is presented in Figure 12 and Figure 13 shows detection of abnormal activity using this process.
The performance of proposed model is computed in terms of false positive rate, true positive rate and classification accuracy for varied scenarios.Performance of proposed model is compared with state of art techniques which are present in [17] and [18]  In Figure 14, a comparative study is presented for considered scenario 1 as mentioned before.This analysis is carried out by computing false positive rate and true positive rate and compared with viscous fluid field method and dense trajectory-based method.Proposed model is able to achieve better performance when compared with these techniques.

Conclusion
In this work, we have addressed the issue of abnormal crowd behavior analysis for surveillance scenarios.This is achieved by using semantic approach by global dense flow and local motion information of video data.spatio-temporal feature analysis technique has used local motion information.In this work, the relational descriptor is presented for classification purpose which utilizes the relationship between each scene and individual objects.An automated process of feature extraction is developed here for behavior analysis of crowd.For event classification, the patch based process is performed which shows promising results for dynamic motion feature and can be used for real-time application scenarios.Experimental study shows that proposed model is able to provide better performance when compared to state of art techniques.


ISSN: 2502-4752 IJEECS Vol. 7, No. 3, September 2017 : 737 -747 738 Flow descriptors are used to describe the motions which are formed by considering various reference frames and temporal structure of dynamics which represents the trajectory.Lagrangian modeling is used to estimate the movement and tracking of particles which are present in motion flow which helps us to obtain flow deformation and complete movement of motion.The Eulerian technique provides flow coverage.Particles of flow are computed for fixed positions which result in overall flow extraction.Since, the motion vector is time dependent fields which contain distinctive curves such as streamline curve, streakline, pathline and timeline curves.Pathline and streamline curves are used to define the tangent curve of a vector field.After computing flow map,  ISSN: 2502-4752 IJEECS Vol. 7, No. 3, September 2017 : 737 -747 740 Using Motion Based Spatio … (Basavaraj G M) 741 variation of motion.

Figure 2
and 3 show sample frame of input video sequence.

Figure 15 .
Figure 15.ROC Performance comparison for case 2