DReyeVR 开发

对于想要深入了解 DReyeVR 内部运作原理以及如何开始开发和编写代码的用户来说，无需再四处寻找了！

（本指南假设您已阅读 Usage.md 文档并已安装 DReyeVR ）。

入门

我们建议您采用一种开发环境，以便能够快速识别您对 DReyeVR 的更改与上游更改之间的差异。为此，我们提供了一个预装（并已提交）DReyeVR 的 CARLA 分支，这样您就可以使用一个干净的初始代码库： 

# 克隆我们的分支并替换你原有的 CARLA 仓库。
git clone https://github.com/harplab/carla -b DReyeVR-0.9.13 --depth 1
cd carla
# ./Update.sh # 在 Linux/Mac 系统中
Update.bat # 在 Windows 系统中

cd ../DReyeVR/ # 假设 DReyeVR 代码库与 carla 代码库相邻。

# (在 DReyeVR 仓库)
make install CARLA=../carla # 安装未被 Git 跟踪的内容，例如蓝图/二进制文件

cd ../carla # 切换回 carla 目录
git status
# 现在，git 应该显示相对于我们上游 DReyeVR 分支的更改，而不是相对于 CARLA 0.9.13 分支的更改。

反向安装

一旦您对 Carla 代码库中与 DReyeVR 相关的部分进行了更改，手动将所有这些更改复制回 DReyeVR 代码库（如果您想将其提交到上游）将非常繁琐。作为我们构建系统的一部分，我们提供了一个“反向安装”（r-install）程序，用于镜像安装 install 功能，并将 DReyeVR（通过 make install）安装的所有相应文件复制回 DReyeVR：

点击打开示例以生成输出 
make r-install CARLA=../carla # 相当于 "make rev"
make rev CARLA=../carla       # r-install 的别名

Proceeding on /PATH/TO/CARLA (git branch)
/PATH/TO/CARLA/Unreal/CarlaUE4/Source/CarlaUE4/DReyeVR/ -- found
/PATH/TO/CARLA/Unreal/CarlaUE4/Source/CarlaUE4/DReyeVR/EgoVehicle.h -- found
/PATH/TO/CARLA/Unreal/CarlaUE4/Source/CarlaUE4/DReyeVR/EgoVehicle.h -> /Users/gustavo/carla/DReyeVR-Dev/DReyeVR/EgoVehicle.h
/PATH/TO/CARLA/Unreal/CarlaUE4/Source/CarlaUE4/DReyeVR/FlatHUD.cpp -- found
/PATH/TO/CARLA/Unreal/CarlaUE4/Source/CarlaUE4/DReyeVR/FlatHUD.cpp -> /Users/gustavo/carla/DReyeVR-Dev/DReyeVR/FlatHUD.cpp
...etc.
...
Done Reverse Install!


请注意，复制回 DReyeVR 的文件遵循 Paths/*.csv 中定义的 DReyeVR <--> Carla 文件对应关系，因此，如果您修改了一个全新的文件（DReyeVR 未跟踪的文件），则需要手动将该文件添加到 DReyeVR 存储库并更新对应关系文件 (.csv)。

典型工作流程

The workflow we have designed for our development process on DReyeVR includes using our fork of carla (DReyeVR-0.9.13 branch) alongside a cloned DReyeVR repo that we can use to both push and pull from upstream.

Click to open terminal example 
> ls
carla.harp/    # our HarpLab fork for primary development
DReyeVR        # our DReyeVR installation

cd carla.harp
... # make some changes in carla.harp
make launch && make package # ensure carla still works with these changes

cd ../DReyeVR
make rev CARLA=../carla.harp # "reverse-install" changes from carla.harp to DReyeVR
git stuff # do all sorts of upstreaming and whatnot.

----------------- # if changes have been made upstream for you to install
cd DReyeVR/
git pull # upstream changes
make clean CARLA=../carla.harp   # optional to reset carla.harp to a clean git state
make install CARLA=../carla.harp # install new DReyeVR changes over it
cd ../carla.harp && make launch && make package && etc.

# optionally, you can keep a carla.vanilla around to test that a fresh install of your updated DReyeVR repo works on carla
make install CARLA=../carla.vanilla


Directories

了解 Carla + DReyeVR 代码库的位置

在 Carla 上进行开发时，您需要重点关注以下几个方面：

  1. Unreal/CarlaUE4/Source/CarlaUE4/DReyeVR/
    • 这包含了我们所有的自定义 DReyeVR C++ 代码，这些代码通常是在现有的 Carla 代码基础上构建的。
  2. Unreal/CarlaUE4/Plugins/Carla/Source/Carla/
    • 这里定义了 UE4 C++ Carla 的主要逻辑，涵盖了从传感器到车辆，再到记录器/回放器和天气等所有内容。
    • 这里有一些代码，例如用于自定义传感器和小功能补丁的代码。
  3. LibCarla/source/carla/
    • 这里存放了几乎所有与 Python 交互的 Carla C++ 代码。其中大部分是对 CarlaUE4/Plugins 代码的重新实现，但没有使用 Unreal C++ API，并且非常注重向 Python API 传输数据流逻辑。
    • 这里有一小段代码，用于确保传感器数据能够正确地传输到 Python。
  4. PythonAPI/examples/
    • 在这里您可以找到与 Carla 交互的大部分重要 Python 脚本。
    • 这里有一些文件，用于改善 DReyeVR 和 Carla 的 PythonAPI 的使用体验。

内部运作

本节将讨论 DReyeVR 的内部运作，包括设计范式以及与 Carla 的相应握手。

EgoVehicle

EgoVehicle 是我们的“英雄车”，也是我们的主要载体。为了提升 Carla 中人类驾驶员的沉浸感，EgoVehicle 包含许多普通 Carla 车辆所不具备的以人为中心的功能。例如，EgoVehicle 定义了车内后视镜、动态方向盘、仪表盘、人为输入、音频等的逻辑。这些都是人工智能车辆无需关注的功能，因此 Carla 在其他所有车辆中都省略了这些功能。

不过，EgoVehicle 只是标准 ACarlaWheeledVehicle 的一个封装（子类），因此它会自动继承所有 Carla 车辆操作，并兼容所有 CarlaVehicle 功能。值得注意的是，EgoVehicle 并非由玩家拥有，而是由默认的 AWheeledVehicleAIController 拥有。这样做是为了允许玩家和内置的 Carla 自动驾驶系统同时进行输入。我们将在下文的 DReyeVRPawn 部分对此进行更详细的讨论。

重要的是，我们所有的节拍同步逻辑都基于 EgoVehicle，其 Tick 方法会调用 DReyeVR 中许多其他关键组件的 Tick 方法。这确保了模拟器中更新的顺序是确定且一致的，并且将来可以依赖这种顺序。

在这里，我们也手动管理 EgoVehicle 的回放行为，它会遵循从 EgoSensor 捕获的值，而不是 Carla 的默认行为，这样我们就可以更精确地重现从 EgoSensor 收集的确切数据，例如眼睛凝视、相机方向、车辆输入和姿态等。

我们还定义了车辆中三个后视镜的生成和管理逻辑，因为它们默认情况下并未包含在蓝图网格中。将它们分开处理是明智之举，因为在模拟引擎中，使用平面反射实现的后视镜会严重影响性能，因此应谨慎使用。我们还可以为每个后视镜定义画质设置，以动态调整其分辨率及其相应的性能影响。

EgoVehicle 包含指向几乎所有其他主要 DReyeVR 对象的指针，以便它们能够无缝通信。这些指针在构造时设置，并在这些对象的整个生命周期内保持不变。重要的是，EgoVehicle 会生成并附加 EgoSensor，因此它们本质上是相互关联的，彼此不可或缺。

此外，EgoVehicle 的所有输入逻辑都保存在 EgoInputs.cpp 源文件中，这样做纯粹是为了将该逻辑与 EgoVehicle 源代码的其余部分分离。

最后，我们在 Carla 世界中生成 EgoVehicle 的方法是：复制一个现有的 Carla 载具蓝图，并将蓝图中的基类 重新父级化 reparenting 到我们的 EgoVehicle。该蓝图位于 EgoVehicle 的 内容(Content) 文件夹中，所有相关的蓝图都整理在这里。

EgoSensor

EgoSensor 是我们用于追踪各种我们可能感兴趣的以人为中心的数据的虚拟 Carla 传感器。它可以被视为一个在 Carla 世界中运行的隐形数据采集器。与其他大多数具有物理描述并安装在 Actor 上的 Carla 传感器不同，EgoSensor 会随 EgoVehicle 自动生成和销毁，并在其整个生命周期内绑定到 EgoVehicle 实例。

EgoSensor 是 DReyeVRSensor 的子类，而 DReyeVRSensor 又是通用 CarlaSensor 的子类，后者源自 Carla 的“"添加传感器教程(add a sensor tutorial)"”。DReyeVRSensor 父类位于代码库的 CarlaUE4/Plugin/Source 区域，因为它遵循了 Carla 的规范实现。重要的是，该类是虚类virtual（抽象类），这意味着它应该被另一个提供实现的类（即 EgoSensor）继承。

由于 DReyeVR 引入了一些 Carla 本身并不依赖的组件（例如用于眼动追踪的 SRanipal 和用于方向盘硬件的 LogitechWheelPlugin），因此我们在 EgoSensor 中实现了它们的接口，而不是在 CarlaUE4/Plugin/Source 区域（该区域仅供 Carla 使用）。EgoSensor 随后实现了从 SRanipal 和 Logitech 获取数据并将其格式化为适用于当前模拟器时间步的 DReyeVRData 数据包所需的方法。

DReyeVRData 类是 CarlaUE4/Plugin/Source/Carla/DReyeVRData.h 中定义的一系列结构体，它定义了 DReyeVR 跟踪的各种数据类型的结构。例如，它包含了眼睛凝视数据、车辆输入、其他自车变量等结构体。这样的设计旨在鼓励未来的数据类型遵循类似的结构体设计，并与 DReyeVR 进行接口集成，以便将所有数据收集到 AggregateData 中，然后作为一个完整的数据包发送到 Python API 进行流式传输，或发送到记录器进行序列化。

EgoSensor 还实现了其他一些不错的功能，例如相机屏幕截图和启用后带有注视点渲染的可变速率着色。

继承关系图：

为了阐明这里涉及的继承结构（从老一代到年轻一代）：

  1. AActor (UE4): 用于在世界中生成任何对象的底层虚幻类
  2. ASensor (Carla): Carla 参与者为Carla 世界中的传感器表现提供了结构模板
  3. ADReyeVRSensor (DReyeVR): 我们的传感器实例包含所有与 Carla 相关的任务逻辑
    • 流式传输到 PythonAPI
    • 从回放器接收数据以进行进行重放
    • 包含 DReyeVR::AggregateData 实例，其中包含所有数据
  4. AEgoSensor (DReyeVR): 我们的主要参与者包含了所有与 DReyeVR 相关的自定义数据变量/函数逻辑。
    • 眼动跟踪逻辑（SRanipal）、自主车辆跟踪等。

DReyeVRPawn

回到我们之前关于玩家与 AI 同时输入 EgoVehicle 的讨论，DReyeVRPawn 是玩家在关卡期间实际控制的实体。与 EgoVehicle/EgoSensor 不同，DReyeVRPawn 不绑定任何特定物体，可以将其视为 一个定义玩家视口的隐形浮动摄像机 。

因此，DReyeVRPawn 负责管理游戏内的 UCameraComponent 组件以及玩家所需的视觉和输入逻辑。SteamVR 集成和 LogiWheel 控制方案映射也由它管理，因为这是玩家拥有的对象，因此具有最高的输入优先级。我们还添加了一些视觉效果逻辑，例如用于显示注视轨迹的可视化指示器，该指示器以准星的形式绘制在 观众屏幕(SpectatorScreen) （VR 玩家不可见）或平面屏幕 HUD 上。

EgoVehicle AI 和玩家 的双输入逻辑源于 DReyeVRPawn 的实现方式：它只是将指令转发给 EgoVehicle，而不直接控制它，因此 Carla AI 仍然可以控制 EgoVehicle。这使得玩家和 Carla AI 控制器能够同时“控制”EgoVehicle，因为所有玩家的输入仍然会到达 EgoVehicle，并且优先级高于 AI 的输入。

DReyeVRGameMode

UE4 中的 GameMode 用于定义跨关卡的游戏逻辑，这可以通过代码轻松完成（与 LevelScripts 不同，LevelScripts 与单个关卡蓝图绑定）。

游戏模式(gamemode)类很有用，因为 我们可以依靠它在任何关卡实例中始终存在，因此我们可以定义超出单个 EgoVehicle/EgoSensor 生命周期的逻辑，并在更全局的层面上进行操作。

例如，我们有代码可以改变游戏内音效的音量，在特定位置生成 EgoVehicle，将控制权转移给默认的浮动观察者（从 EgoVehicle 分离），以及管理记录播放的媒体控制（播放/暂停/单步/倒带等）。

DReyeVR 游戏模式最重要的功能是生成 DReyeVRPawn，这样玩家就可以控制某个角色并与游戏世界进行互动。

DReyeVRFactory

Carla uses Factories to spawn all their relevent actors (See CarlaActorFactory, CarlaActorFactoryBlueprint, SensorFactory, etc.) which spawn everything from vehicles to pedestrians to sensors and props. This design allows Carla to handle all the dirty work of registering the actors with LibCarla so that LibCarla knows about each actor and they can be interacted with in Python.

We follow suit with a similar design in our DReyeVRFactory which defines the important characteristics for our DReyeVR actors and provides the logic necessary to spawn them. For instance, here we define our actors are labeled uniquely such as "harplab.dreyevr_vehicle.model3" to avoid conflict with existing Carla "vehicle.*" queries.

This is class you'll want to modify if you're looking to create new DReyeVR actors (such as new vehicle models), walkers, sensors, etc.

向自我传感器(ego-sensor)添加自定义数据

While we provide a fairly comprehensive suite of data in our DReyeVRSensor, you may be interested in also tracking other data that we don't currently enable.

The first file you'll want to look at is Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/DReyeVRData.h which contains the data structures that compose the contents of the ego sensor. Here you'll define the variable and its serialization methods (read/write/print). 

/// DReyeVRData.h
class AggregateData // all DReyeVR sensor data is held here
{
public:
    ... // existing code
    float GetNewVariable() const;
    ////////////////////:SETTERS://////////////////////

    ...
    void SetNewVariable(const float NewVariableIn);

    ////////////////////:SERIALIZATION://////////////////////
    void Read(std::ifstream &InFile);

    void Write(std::ofstream &OutFile) const;

    FString ToString() const; // this printing is used when showing recorder info

private:
... // existing code
float NewVariable; // <-- Your new variable
};

Then, you'll want to write the implementation in Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/DReyeVRData.cpp as inline funcitons. 

/// DReyeVRData.cpp
...
float AggregateData::GetNewVariable() const
{
    return NewVariable;
}

...
void AggregateData::SetNewVariable(const float NewVariableIn)
{
NewVariable = NewVariableIn;
}

void AggregateData::Read(std::ifstream &InFile)
{
    /// CAUTION: make sure the order of writes/reads is the same
    ... // existing code
    ReadValue<float>(InFile, NewVariable);
}

void AggregateData::Write(std::ofstream &OutFile) const
{
    /// CAUTION: make sure the order of writes/reads is the same
    ... // existing code
    WriteValue<int64_t>(OutFile, GetNewVariable());
}

FString AggregateData::ToString() const // this printing is used when showing recorder info
{
    FString print;
    ... // existing code
    print += FString::Printf(TEXT("[DReyeVR]NewVariable:%.3f,\n"), GetNewVariable());
    return print;
}
...
Notes: - It is nice to contain collections of relevant variables together in structures so they can be better organized. To facilitate this we designed our DReyeVRData to contain various DReyeVR::DataSerializer objects, which each implement their own serialization methods. Our AggregateData instance contains all our structs and a lightweight API to access member variables. - The above is an example of modifying/adding a new variable directly to a DReyeVR::AggregateData. But it would be better to either modify an existing DReyeVR::DReyeVRSerializer object or create a new one (inheriting from the virtual class) and define all the abstract methods yourself. This enables a more granular sub-class/struct abstraction like most of our variables.

With this step complete, you are free to read/write to this variable by getting the single global (static) instance of the DReyeVR::AggregateData class using the GetData() function of the EgoSensor as follows: 

// In some other file, for example EgoVehicle.cpp:
float NewVariable = EgoSensor->GetData()->GetNewVariable();
... // your code
EgoSensor->GetData()->SetNewVariable(NewVariable + 5.f); // update the new variable

[可选] 向 PythonAPI 客户端传输数据：

In order to see the new data from a PythonAPI client, you'll need to duplicate the code to the LibCarla serializer. This requires looking at LibCarla/Sensor/s11n/DReyeVRSerializer.h and following the same template as all the other variables: 

class DReyeVRSerializer
{
    public:
    struct Data
    {
        ... // existing code
        float NewVariable;

        MSGPACK_DEFINE_ARRAY(
        ... // existing code
        NewVariable, // <-- New variable
        )
    };
};
///NOTE: you'll also need to interface with this updated struct:
Then, to actually interface with the DReyeVR sensor, you'll need to modify the call to the LibCarla stream to include your NewVariable. 
// in Carla/Sensor/DReyeVRSensor.cpp
void ADReyeVRSensor::PostPhysTick(UWorld *W, ELevelTick TickType, float DeltaSeconds)
{
    ... // existing code
    Stream.Send(*this,
                carla::sensor::s11n::DReyeVRSerializer::Data{
                    ... // existing code
                    Data->GetNewVariable(), // <-- New variable
                });
}
And finally, to actually get the data from a PythonAPI call, you'll need to modify the list of available attributes to the DReyeVR sensor object as follows: 
// in LibCarla/source/carla/sensor/data/DReyeVREvent.h
class DReyeVREvent : public SensorData
{
    ...

    public:
    ... // existing code
    float GetNewVariable() const // <-- new code
    {
        return InternalData.NewVariable;
    }

    private:
    carla::sensor::s11n::DReyeVRSerializer::Data InternalData;
};
Then finally here you'll define what function to call (the variable getter) to get that data from a PythonAPI client. 
// in PythonAPI/carla/source/libcarla/SensorData.cpp
class_<csd::DReyeVREvent, bases<cs::SensorData>, boost::noncopyable, boost::shared_ptr<csd::DReyeVREvent>>("DReyeVREvent", no_init)
    ... // existing code
    .add_property("new_variable", CALL_RETURNING_COPY(csd::DReyeVREvent, GetNewVariable))
    .def(self_ns::str(self_ns::self))
;
After you modify files in PythonAPI or LibCarla the PythonAPI will need to be rebuilt in order for your changes to take effect: 
conda activate carla13 # if using conda
(carla13) make PythonAPI
# make sure to fix any build errors that may occur!

TODO: 添加更多开发笔记

技巧与诀窍

1. 启用交付模式下的日志记录

It is super useful to see the CarlaUE4.log file in shipping mode and this is not the default in Carla (or Unreal) perhaps for performance reasons?

If you want to enable these features then you'll need to add the flag for bUseLoggingInShipping in the Carla/Unreal/CarlaUE4/Source/CarlaUE4.Target.cs file.

public class CarlaUE4Target : TargetRules
{
    public CarlaUE4Target(TargetInfo Target) : base(Target)
    {
        Type = TargetType.Game;
        ExtraModuleNames.Add("CarlaUE4");
        bUseLoggingInShipping = true;//  <--- added here
    }
}

Then you should be able to find the CarlaUE4.log files (timestamped to avoid overwrite) at C:\Users\%YOUR_USER_NAME%\AppData\Local\CarlaUE4\Saved\Logs\CarlaUE4.log (on Windows). Also works for Mac/Linux. See this for more information.

2. 如何执行 LOG

Logging is useful to track code logic and debug (especially since debugging UE4 code can be a bit rough). By default in Unreal C++ you can always use UE_LOG(LogTemp, Log, TEXT("some text and %d here"), 55); but we streamlined this for DReyeVR specific logging. You can use our LOG macros (defined in CarlaUE4.h) when you are editing CarlaUE4/DReyeVR/*.[cpp|h] files.

The main benefits include: - Less boilerplate code for the programmer (thats you!) - All logs have the prefix DReyeVRLog so they are easy to filter in the overall CarlaUE4.log file - We also attached neat compile-time prefixes to include "[{INVOKED_FILE}::{INVOKED_FUNCTION}:{LINE_NUMBER}] {message}" so you can quickly find where this log was invoked from and differentiate from others - A typical DReyeVR log using our macros looks like this: 

LogDReyeVR: [DReyeVRUtils.h::ReadConfigValue:141] "your message here"


... // in CarlaUE4/DReyeVR files
void example() {
    LOG("some text and %d (this text is grey)", 55);
    FString warning_str("warning");
    LOG_WARN("some %s here (this text is yellow!)", *warning_str);
    LOG_ERROR("this text is red!");
}
...

If you are working instead in the CarlaUE4/Plugins/Source/Carla codebase then you'll be able to use similar macros but prefixed with DReyeVR_: DReyeVR_LOG("blah blah"), DReyeVR_LOG_WARN("blah blah warning"), etc.

3. 管理多个 Carla/DReyeVR 版本

  • Having separate python environments (such as conda) is extremely useful to have different carla Python packages on the same machine with different versions (such as LibCarla). To do this, you can simply create an individual conda environment for each Carla version as described in Install.md. Remember to activate your new conda environment on a per-shell basis!
  • If you plan on having multiple CARLA's installed, you'll need to keep them updated with the appropriate Content. Rather than calling the Update script every time to update this, you can save the Content.tar.gz file and copy it into new Unreal/CarlaUE4/Content/Carla directories whenever you have a new repo.

4. TODO 添加更多技巧和诀窍