On this page
Research in progress · August 2026 snapshot
SmartFly asks: how can an agent use vision, odor, and wind to stay stable and find a target? Built around a MuJoCo fly simulation, the project combines a learned high-level navigation policy with a frozen low-level flight controller to explore the relationship between perception, memory, and action.
Research questions
- Can one policy support two navigation tasks? The visual task provides relative target information. The vision–odor task hides the source location and uses odor, apparent wind, and vision for source seeking and obstacle avoidance. The goal is to support both tasks with the same checkpoint.
- How do visual priors relate to representations and behavior? The comparison uses a trainable CNN and FlyVis with a frozen pretrained backbone, under matched tasks and training budgets.
- What separates a demonstration from reliable evidence? Stable-flight replays help inspect closed-loop behavior and actual policy inputs. Full navigation claims require independent testing and statistics across training seeds.
Current implementation
The project uses Python, PyTorch, Ray RLlib / PPO, MuJoCo, and dm_control, with four main components:
- Multimodal observations. Binocular vision, odor, and apparent wind provide sensory information. A recurrent odor branch preserves temporal context; the hidden source coordinates are not given to the policy in the multimodal task.
- Hierarchical control. The high-level policy outputs bounded forward-speed residuals, yaw rates, and vertical velocities. A reference generator converts these commands for the frozen low-level controller.
- Curriculum training. Training starts with stabilization and progressively increases navigation distance and forest density. Repeated stage evaluations gate progression.
- Visual representation analysis. The code includes a shared stimulus bank, probes grouped by complete trajectories, and closed-loop metrics, so a compelling trajectory alone is not treated as evidence of encoder superiority.
The low-level flight and FlyVis components use upstream pretrained assets. These existing models are not presented as models trained from scratch for this project.
Available stable-flight replays
Local artifacts contain real replays of the same CNN–RNN stabilization checkpoint in both observation modes. With the fixed seed 83001, both trajectories complete approximately 3 seconds of simulation without a recorded fatal termination. The recordings include external cameras, 3D position, actual binocular policy inputs, and height and speed diagnostics.
Original project output; select the image to inspect it at full resolution. Coordinates are in centimeters. The displayed y/z axis proportions are enlarged; the underlying coordinates are unchanged.
Visual observation mode
Vision–odor observation mode
These two fixed-seed replays are neither population success rates nor results for full navigation. Height drift and speed oscillations remain visible in the original recordings. This page summarizes existing experiment artifacts; no training or simulation was rerun to produce the website.
Evaluation still to complete
- Complete training and evaluation for visual navigation and vision–odor source seeking in the full forest setting.
- Compare CNN–RNN and FlyVis–RNN across multiple training seeds under matched budgets.
- Evaluate on an independent odor field excluded from model selection, with ablations to investigate whether the policy actually uses odor information.
The current odor field is a two-dimensional approximation without tree-induced wind/odor wakes or a three-dimensional concentration structure. The CNN/FlyVis comparison also includes differences in visual pretraining, so effects cannot be attributed directly to connectome topology. Conclusions must remain within the evaluated simulation conditions and should not be extrapolated directly to real insects or robots.