Sim-to-Real Reinforcement Learning for F1TENTH Racing: An Engineering Evaluation

Abstract

This article examines an F1TENTH sim-to-real reinforcement-learning implementation at its May 1, 2025 code snapshot. The system combines Soft Actor-Critic (SAC), imitation initialization, and randomization of dynamics, sensing, and command latency. Retrospective project notes committed after that snapshot report a simulated Levine lap of approximately seven seconds for an imitation-initialized policy, followed by poor transfer to hardware. The same notes describe a slower randomized policy whose behavior was more similar across simulation and hardware, although reduced speed was a confounding factor. Context-conditioned control and two-car racing remained under development. These observations are engineering records, not results from a controlled, repeated benchmark.

Problem Definition

The objective was to train a racing policy in simulation and deploy it on a physical F1TENTH vehicle. Simulation and hardware differ in tire friction, vehicle dynamics, LiDAR measurements, and command timing, with larger consequences at higher velocity. The study therefore treated lap time and transfer robustness as distinct objectives.

The implementation description is bounded by the May 1, 2025 code snapshot. The performance narrative is drawn from later retrospective project notes and is labeled accordingly. Components not integrated in the snapshot are identified as unfinished.

System Design

Reinforcement-Learning Formulation

The environment used F1TENTH Gym (O’Kelly et al., 2020) and Soft Actor-Critic (SAC; Haarnoja et al., 2018) for continuous control. Observations combined Frenet progress, lateral offset, longitudinal velocity, yaw, and a 1,080-ray LiDAR scan. Actions specified steering angle and target speed. The reward favored progress and velocity and penalized collision. The default configuration supported 24 parallel environments.

SAC was initialized from classical-control demonstrations. Wall following and Pure Pursuit were examined as experts; Pure Pursuit produced the more useful demonstrations. SAC optimization continued after this imitation stage.

Domain Randomization and Latency Model

Following the dynamics-randomization approach of Peng et al. (2018), the simulator varied friction, front and rear cornering stiffness, mass, yaw inertia, LiDAR noise, state noise, and command timing. Command latency was modeled explicitly because the standard simulator applies one action for every observation. At each control step, a queue received zero, one, or two copies of the new command, after which the oldest queued command was executed. The sampling probabilities maintained an expected insertion count of one while allowing individual commands to be dropped, repeated, or delayed.

This timing variation is physically meaningful at racing speed. At 5 m/s, a single 0.04-second control interval corresponds to approximately 20 cm of travel before a revised steering command takes effect. On a narrow track, such displacement can substantially alter the state at which the command is executed.

Context-Conditioned Control

To condition the policy on randomized dynamics, the simulator parameters were normalized and appended to the observation. This context contained 12 values, including friction, cornering stiffness, mass, inertia, sensor-noise levels, and command-delay probabilities.

An LSTM was also trained to infer environment parameters from short state-action trajectories. This intended combination of a context-conditioned policy and trajectory-based parameter inference is conceptually related to universal policies with online system identification (Yu et al., 2017). Its recurrent state was intended to represent latent dynamics, but the estimator had not been integrated into the reinforcement-learning pipeline by May 1; no result is therefore attributed to this method.

Experimental Setup

The retrospective evaluation notes compare an early SAC policy, a policy initialized from Pure Pursuit demonstrations, and a domain-randomized policy. Selected policies were evaluated in simulation and on the physical vehicle without real-world fine-tuning. These were development runs rather than repeated trials under a standardized protocol.

Results

Retrospective project notes report that an early SAC policy completed the Levine track in approximately 16 seconds in simulation and approximately 30 seconds on the physical car without real-world fine-tuning. These times were not obtained through a repeated protocol with uncertainty estimates.

The fastest documented simulated policy used Pure-Pursuit initialization and completed a Levine lap in approximately seven seconds. Hardware deployment revealed a substantial sim-to-real discrepancy: the policy requested aggressive acceleration and began drifting almost immediately. Thus, the shorter simulated lap did not correspond to improved physical performance in this development run.

The retrospective notes describe the domain-randomized policy as substantially more conservative, with speeds near 2 m/s at best and below 1 m/s in some turns. Its physical behavior more closely resembled its simulated behavior, and deployment did not produce the immediate drifting failure observed for the faster policy. However, the available record cannot determine whether this difference resulted from domain randomization or from the lower operating speed. It is therefore reported as an observation requiring further controlled evaluation.

Providing the true simulator parameters as policy context did not immediately improve reward. The trajectory-based LSTM context estimator remained unintegrated, so no end-to-end control result is attributed to that method.

Competitive Racing

The implementation also included a two-car mode in which the learned car raced a Pure Pursuit opponent from randomized initial configurations. Each vehicle acted from its own observation without inter-vehicle communication, establishing a decentralized control interface. Racing rewards and overtaking behavior were still under development. In particular, an overtaking reward term was present but disabled in the May 1 code. The corresponding video is consequently a progress demonstration, not evidence of a completed multi-agent racing policy.

Limitations and Discussion

The evaluation lacked repeated trials, standardized starts, aggregate lap-time statistics, and ablations of individual randomized parameters. The logged times therefore compare development runs rather than estimate benchmark performance. Physical testing also confounds robustness with speed because the randomized policy was both slower and more consistent across domains.

The imitation-initialized policy recorded the fastest simulated lap but transferred poorly to hardware. The randomized policy exhibited a smaller observed cross-domain discrepancy under a lower-speed regime; because speed was confounded with training method, the available runs do not isolate an effect of randomization. Contextual methods did not show an improvement in the available development record. Future evaluation should control target speed, repeat hardware runs, and measure failures as well as lap time.

Conclusion

The May 1, 2025 code snapshot establishes an SAC training and hardware-deployment pipeline. Later retrospective notes indicate that simulated lap time alone was an inadequate measure of transfer performance: imitation initialization produced a fast simulated lap but an unstable physical deployment, while the randomized policy exhibited a smaller observed discrepancy under a lower-speed regime. Reduced speed confounds attribution. Context inference and competitive racing remained incomplete in the snapshot. Further evaluation requires repeated trials that control speed and starting state and report lap-time distributions and deployment failures.

References

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.” International Conference on Machine Learning, 2018.
M. O’Kelly et al. “F1TENTH: An Open-Source Evaluation Environment for Continuous Control and Reinforcement Learning.” Proceedings of Machine Learning Research, 2020.
X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization.” IEEE International Conference on Robotics and Automation, 2018.
W. Yu, J. Tan, C. K. Liu, and G. Turk. “Preparing for the Unknown: Learning a Universal Policy with Online System Identification.” Robotics: Science and Systems, 2017.

Project Materials

Demonstrations

Imitation-initialized SAC completing a Levine lap in approximately seven seconds in simulation.

The learned racing policy competing against a Pure Pursuit controller. This is a progress demonstration rather than a controlled evaluation.