> For the complete documentation index, see [llms.txt](https://asail.gitbook.io/hogwarts/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://asail.gitbook.io/hogwarts/graph/rdrl.md).

# Relational RL

TL;DR: RRL combine Reinforcement Learning with Relation Learning or Inductive Logic Programming by representing states, actions and policies using a first order(relational) language.

## Structure

### Feature Extraction

We apply a CNN on the raw image and get a feature map: where we have one $k$-dimension vector for each pixel($$k$$ is the number of output channels of CNN), the vector is then concatenated with $$x$$ and $$y$$ coordinates to indicate its position in the map.

We treat the resulting $$n^2$$ pixel-feature vectors as the set of entities.

### Relational Module

The output is then passed to the relational module, where we iteratively apply "attention block" on entity representations:

The attention block is the same as a Multi-Head Attention module in Transformer.

![img](https://i.imgur.com/11U62Cz.png)

### Policy and Value Network

The output of relational module is aggregated via a feature-wise max-pooling across space($$n\times n \times k$$ tensor to a $$k$$-dimensional vector), the feature is then used to produce value and policy for Actor-Critic Algorithm.

## Task

### Box-World

* Action: left, right, up, down.
* pick up keys and open boxes(two adjacent colored pixels)
  * Agent could pick up loose keys(isolated colored pixel) and open boxes with corresponding locks.
  * Most boxes contain keys in then, and one box contain a gem(colored with white), the target of the game is to reach the gem.

![img](https://i.imgur.com/xe0017w.png)

### Starcraft