:orphan:

******************
A ``@phase`` block
******************

The environment that the subject interacts with is specified using one or more ``@phase`` blocks.
A ``@phase`` block defines the sequence of stimuli that the subject is exposed to, and how the next stimulus
in the sequence depends on the response to the previous stimulus.

If the environment is an experiment setup, say a maze, one may use one ``@phase`` block per phase in the experiment,
for example one ``@phase`` for the training phase and one for the test phase.

The structure of a phase block is outlined in the below example::

  @phase training stop: reward==100
  TRIAL_START  lever      | pull_lever: REWARD | NO_REWARD
  REWARD       reward     | TRIAL_START
  NO_REWARD    background | TRIAL_START

The simulation will start with the first phase line below the title line (except for inherited phase blocks, see :ref:`inheriting-a-phase`).

The ``@phase`` title line
*************************

In general, a ``@phase`` block starts with a *title line* of the format::

  @phase phase_name stop:stop_condition

where 

- ``phase_name`` is the name of the phase (see :ref:`the-phase-name`), and
- ``stop_condition`` is the
  stop condition (see :ref:`the-stop-condition`) specifying when to exit the phase, usually after a prededermined number of exposures
  to a certain stimulus element, or after a predetermined number of responses with a certain behavior.

.. note::
  ``@phase`` and ``stop:`` are keywords, and cannot be changed. The name of the phase (``phase_name``) and the stop condition (``stop_condition``) are user defined.

For example, in a script with the following parameter definitions::

  stimulus_elements = lever, reward
  behaviors = pull_lever, ignore

the title line of a @phase block may look like this::

  @phase training stop:reward==100

or::

  @phase training stop:pull_lever==100

.. _the-phase-name:

The phase name
--------------

The phase name must be a :ref:`valid name<valid-names>`.

.. _the-stop-condition:

The phase stop condition
------------------------

A phase stop condition is a
`boolean expression <Boolean expression_>`_
that may include the following variables:

- Any stimulus element (evaluates to the number of exposures to this stimulus element)
- Any behavior (evaluates to the number of responses with this behavior)
- Any phase line label (evaluates to the number of times this phase line has been visited)
- Any local phase variable (evaluates to the value of this variable)

For example, the first line of a @phase block may look like this::

  @phase training stop: pull_lever>=10**2 or reward==10

which means that the phase stops when either

- the number of responses with the behavior ``pull_lever`` is greater than or equal to :math:`10^2=100`.
- the number of exposures to the stimulus element ``reward`` is 100, or

See also :ref:`using-local-phase-variables-in-phase-stop-condition`.

Boolean expression
------------------

A boolean expression is a mathematical expression 
that evaluates to *true* or *false*. It may
include number literals, variables, parentheses, and the below operators.

The supported operators in *Learning Simulator* are the arithmetic operators

- ``+`` (addition)
- ``-`` (subtraction)
- ``*`` (multiplication)
- ``/`` (division)
- ``**`` (power)

and the comparison operators

- ``==`` (equals)
- ``>`` (greater than)
- ``>=`` (greater than or equal to)
- ``<`` (less than)
- ``<=`` (less than or equal to)

and the logical operators

- ``and``
- ``or``

and the functions

- ``choice`` (See :ref:`choice-function`)
- ``rand`` (See :ref:`rand-function`)

For example, ``x>2 and y<=5`` is a boolean expression of the
variables ``x`` and ``y``, the constants 2 and 5, and
the operators ``>`` and ``<=``. It evaluates to *true*
whenever ``x`` is greater than two and ``y`` is smaller
than or equal to 5, and *false* otherwise.

If ``x`` is a number, the expression ``2x-5`` is not a boolean expression since it
does not evaluate to *true* or *false*, but to a number.

.. _phase-line:

A phase line
************

Below the phase title line follows a number of *phase lines*. Each phase line 

- presents a :ref:`stimulus <stimulus-specification>` to the subject (optional),
- performs a number of (optional) *actions* (see `Actions <Actions_>`_), and
- informs which of the phase lines to go to next

A phase line has the general format::

  LINE_LABEL s | d11,d12,...,COND1:a11,a12,... | d21,d22,...,COND2: a21,a22,... | an1,an2,...,ann

where s is a :ref:`stimulus <stimulus-specification>`, ``d11``, ``d12``, ``a11``, ``a12`` etc. are `actions <Actions_>`_, and ``COND1``, ``COND2`` etc. are *phase line conditions* (see :ref:`phase-line-conditions`).
The actions are performed sequentially, in the order they appear. For example, the line::

  LINE_LABEL s | d11,d12, COND1:a11,a12 | d21,d22, COND2:a21,a22 | d31,d32, COND3:a31,a32 | a41,a42

is interpreted by the simulator as follows::

  Present the stimulus s
  Perform actions d11 and d12
  If COND1 is fulfilled:
      Perform action a11
      Perform action a12
  Otherwise:
      Perform action d21
      Perform action d22
      If COND2 is fulfilled:
          Perform action a21
          Perform action a22
      Otherwise:
          Perform action d31
          Perform action d32
          If COND3 is fulfilled:
              Perform action a31
              Perform action a32
          Otherwise:
              Perform action a41
              Perform action a42

Note that one of the action types is "go to phase line with label X".
During the simulation, if the above logic does not determine which phase line to go to next, an error message will appear.
For example, if ``COND1`` is fulfilled, ``a12`` cannot be any other action than a phase line label::

  LINE_LABEL1 stimulus | d11,d12, COND1:a11,LINE_LABEL2
  LINE_LABEL2 ...

Since the actions are performed sequentially, an error will appear if the action ``LINE_LABEL2`` is followed by additional actions
as these additional actions will never be performed.

.. _phase-line-conditions:

Phase line conditions
*********************

A phase line condition is a
`boolean expression <Boolean expression_>`_
that may depend on the following variables:

- Any behavior (evaluates to *true* for the behavior that is the response to the stimulus on the line, and *false* for all other behaviors)
- Any phase line label (evaluates to the number of times this phase line has been visited)
- Any local phase variable (evaluates to the value of this variable)

and the following functions:

- ``count`` (See :ref:`the-function-count`)
- ``count_line`` (See :ref:`the-function-count_line`)


.. _the-function-count:

The function ``count``
----------------------

The function ``count`` counts occurrences of each behavior, each stimulus element, and 
number of visits to each phase line (using the phase line label as identifier). For example::

  count(lever)

evaluates to the number of times the stimulus element ``lever`` has been exposed
to the subject (since the beginning of the phase, or since it was last reset
using ``count_reset``, see below). Similarly,::

  count(pull_lever)

evaluates to the number of times the agent has responded with the behavior ``pull_lever`` (to any stimulus) since the beginning of the phase, or since it was last reset.

To return the number of visits to a phase line with label ``LBL``, use::

  count(LBL)

The counter for a specific stimulus element, behavior, or phase line may be
reset to zero using the action ``count_reset`` (see :ref:`the-action-count_reset`).


.. _the-function-count_line:

The function ``count_line``
---------------------------

The function ``count_line`` counts consecutive occurrences of each
behavior, each stimulus element, and number of consecutive visits 
to the phase line where this function is used. It is perhaps most useful for behaviors, to count
the number of consecutive responses to the stimulus on a phase line. For example::

  stimulus_element = s1, s2
  behaviors = b1, b2, b3
  
  @phase ...
  LBL1 s1 | count_line(b1)==3: LBL2 | LBL1
  LBL2 s2 | LBL1

will expose the subject to the stimulus ``s1`` repeatedly until the response
to ``s1`` is the behavior ``b`` three times in a row. Then ``s2`` will be
presented. A sequence of stimulus-response pairs may be::

  s1 -> b1 -> s1 -> b2 -> s1 -> b1 -> b1 -> b1 -> s2 -> b3

If instead the phase lines looked like this::

  LBL1 s1 | count_line(s1)==3: LBL2 | LBL1
  LBL2 s2 | LBL1

the subject will be presented with ``s1`` three times, and regardless of 
the responses, will then be presented with ``s2``::

  s1 -> * -> s1 -> * -> s1 -> * -> s2

Note that the above example is equivalent to::

  LBL1 s1 | count_line(LBL1)==3: LBL2 | LBL1
  LBL2 s2 | LBL1

and that ``count_line(LBL1)`` may be abbreviated ``count_line()`` as the
line label is redundant (since ``count_line`` count consecutive occurrences
on the very line where it is used)::

  LBL1 s1 | count_line()==3: LBL2 | LBL1
  LBL2 s2 | LBL1


.. _local-phase-variables:

Local phase variables
*********************

It is possible to use custom local variables within a phase, whose values may change during the phase.
They may be assigned a value in an action, and may be referred to in a condition.
Local phase variables must be initialized (have an assigned value) before they are used.
A phase variable must have a :ref:`valid name<valid-names>`.

An action that assigns the value 0.1 to the local variable ``x`` looks as follows::

  x=0.1

A phase with an action that sets ``x`` and with a
condition that uses ``x`` may look like this::

  @phase ...
  L0 x=0.1 | L1                        # Set x to 0.1 and go to L1
  L1 s1    | x=x+0.1, x>=0.5: L2 | L1  # Present s1, increase x with 0.1. If x>=0.5 go to L2, otherwise L1
  L2 s2    | L1                        # Present s2 and go to L1


.. _using-local-phase-variables-in-phase-stop-condition:

Using local variables in a phase stop condition
-----------------------------------------------

It is possible to use a local phase variable in a phase stop condition.
For example::

  @phase phase_name stop:x>2
  L0 x=0.1 | L1
  L1 s1    | x=x+0.1, x>=0.5: L2 | L1
  L2 s2    | L1

The stop condition is not checked until the first stimulus is presented.
If the phase stop condition depends on local variables, these local
variables must therefore be initialized before the first stimulus is presented
(like in the example above), otherwise an "Unknown variable" error will occur.


Actions
*******

An action is one of the following:

- Set a local phase variable (for example, ``x=2``)
- Go to a phase line (by simply stating the corresponding phase line label)
- Go to one of several phase lines, with a probability for each (for example, ``L1(0.25),L2(0.75)``). See :ref:`probabilistic-go-tos`. 
- Tell the simulation to omit the state variable update in the next step with ``@omit_learn``. (See :ref:`omit_learn`.)
- Reset the counting of an event with ``count_reset`` (See :ref:`the-action-count_reset`.)

Note that global variables (defined using ``@variables``) cannot be set in an action.

.. _probabilistic-go-tos:

Probabilistic go-tos
--------------------

A probabilistic go-to is an action of the type "go to line L1 with probability p1,
go to line L2 with probability p2, etc." and has the syntax::

  L1(p1),L2(p2),...

where ``p1``, ``p2`` etc. are expressions of global variables, local phase variables and numbers.

For example, to go to line ``L1`` with probability 0.4 and to ``L2`` with probability 0.6, use::

  L1(0.4),L2(0.6)

Note that the probabilities may be global variables or local phase variables. For example::

  START | p1=0.1, p2=0.7, p3=0.2 | L0
  L0 s  | L1(p1),L2(p2),L3(p3)
  L1 ...
  L2 ...
  L3 ...

The probabilities within parentheses do not need to add up to 1::

  L0 s | L1(0.1),L2(0.7) | L3

means that the probability to go to ``L3`` is 0.2, so this is the same as::

  L0 s | L1(0.1),L2(0.7),L3(0.2)


.. _omit_learn:

The action ``@omit_learn``
--------------------------

The action ``@omit_learn`` tells the simulator to omit the updating of the subject's state
variables in the next step. Typically this is done at the end of a trial, when the
phase represents a number of trials. In that case, it is often not desired to carry
over the reinforcement value of, say, a reward at the and of a trial to the
first stimulus in the next.

For example, in mechanisms with conditioned reinforcement values (w-values),
and we have the following stimulus-response history::

  s -> response -> reward -> eat -> s

one typically wants to avoid that the w-value for the first stimulus in the
trial (``s``) affects

- the reinforcement value of ``reward``, and
- the v-value (associative strength) between ``reward`` and ``eat``


.. _the-action-count_reset:

The action ``count_reset``
--------------------------

The counter for a specific stimulus element, behavior, or phase line may be
reset to zero using the action ``count_reset``.

For example::

  count_reset(pull)

resets the counter of ``pull`` to zero, so that ``count(pull)``
will return zero. This action may be used like this::

  stimulus_elements = lever, reward
  behaviors = pull, ignore

  @phase training stop:reward=100
  START                 | count_reset(pull), LEVER_OFF
  LEVER_OFF lever       | count(pull)=3: LEVER_ON | LEVER_OFF  
  LEVER_ON  lever       | pull: REWARD | LEVER_ON
  BACKGROUND background | START
  REWARD reward         | START

After three ``pull`` of the ``lever``, the lever will "activate" and give reward when pulled.
Then it is inactivated and three more ``pull`` are required to activate it.

To reset the counter for the stimulus element ``e1``, use::

  count_reset(e1)

To reset the counter for the phase line with label ``LBL``, use::

  count_reset(LBL)

Note that resetting one stimulus element (or behavior or phase line) will not
affect any other.

.. _inheriting-a-phase:

Inheriting a phase
******************

If you have several @phase-blocks that are very similar to each other, there is a way to avoid
duplicating each phase line, and instead only state the lines that differ between the phases.
This is done through *phase inheritance*. Do create a phase with name ``phase2`` that inherits
from from a phase with label ``phase1``, use::

  @phase phase2(phase1) ...

For example, suppose we have the following phase block in a script::

  @PHASE training stop: stimulus=50
  new_trial   stimulus   | response1: REWARD1 | response2: REWARD2 | response3: REWARD3 | NO_REWARD
  REWARD1     reward1    | new_trial
  REWARD2     reward2    | new_trial
  REWARD3     reward3    | new_trial
  NO_REWARD   background | new_trial

and we want to create a new phase ``training2`` that is the same as the above but where the line with label ``REWARD3`` should read::

  REWARD3     reward4    | new_trial

This can be accomplished by simply re-stating the entire phase block ``training``, and make the change. However, this requires
a lot of lines and a change in a part of the phase block that should be the same must be performed in each
phase block. Therefore, using phase inheritance is better::

  @PHASE training2(training) stop: stimulus=50
  REWARD3     reward4    | new_trial

The above phase block represents the same phase as::

  @PHASE training2 stop: stimulus=50
  new_trial   stimulus   | response1: REWARD1 | response2: REWARD2 | response3: REWARD3 | NO_REWARD
  REWARD1     reward1    | new_trial
  REWARD2     reward2    | new_trial
  REWARD3     reward4    | new_trial
  NO_REWARD   background | new_trial
