# Relational RL

TL;DR: RRL combine Reinforcement Learning with Relation Learning or Inductive Logic Programming by representing states, actions and policies using a first order(relational) language.

## Structure

### Feature Extraction

We apply a CNN on the raw image and get a feature map: where we have one $k$-dimension vector for each pixel($$k$$ is the number of output channels of CNN), the vector is then concatenated with $$x$$ and $$y$$ coordinates to indicate its position in the map.

We treat the resulting $$n^2$$ pixel-feature vectors as the set of entities.

### Relational Module

The output is then passed to the relational module, where we iteratively apply "attention block" on entity representations:

The attention block is the same as a Multi-Head Attention module in Transformer.

![img](https://i.imgur.com/11U62Cz.png)

### Policy and Value Network

The output of relational module is aggregated via a feature-wise max-pooling across space($$n\times n \times k$$ tensor to a $$k$$-dimensional vector), the feature is then used to produce value and policy for Actor-Critic Algorithm.

## Task

### Box-World

* Action: left, right, up, down.
* pick up keys and open boxes(two adjacent colored pixels)
  * Agent could pick up loose keys(isolated colored pixel) and open boxes with corresponding locks.
  * Most boxes contain keys in then, and one box contain a gem(colored with white), the target of the game is to reach the gem.

![img](https://i.imgur.com/xe0017w.png)

### Starcraft


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://asail.gitbook.io/hogwarts/graph/rdrl.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.