Sequence Processing

Our examples of neural network applications have focused on tasks where an output (label) should be predicted from one input vector. Another common application domain that we’ll discuss now is that of processing sequences.

There are many types of processes that generate sequences. Some examples are given below as follows:

• Forecasting the stock market from past observations
• Forecasting the weather from past observations
• The modeling of the progress of patients’ improvements due to medical interventions

While it is common to have naturally sequential data, there are reasons to process static data in a sequential form, such as searching large images in patches to look for specific objects. Doing this is more memory efficient. Even the human visual system uses sequence processing, because scenes are commonly explored by a series of eye movements called saccades. We will start discussing sequence processing with the basic tasks of temporal predictions, where a value of a sequence at position $t$ is to be predicted from previous data points. This is shown below: