In the last decades, nanostructures have unlocked myriads of functionalities in nanophotonics by engineering light–matter interaction beyond what is possible with conventional bulk optics. The space of parameters available for design is practically unlimited due to the large variety of optical materials and nanofabrication techniques. Thus, computational approaches are necessary to efficiently search for the optimal solutions. In this paper, we enable the free-form inverse design in 3D of linear optical materials with arbitrary dispersion and anisotropy. This is achieved by (1) deriving an analytical adjoint scheme based on the complex-conjugate pole-residue pair model in the time domain and (2) its implementation in a parallel finite-difference time-domain framework with a topology optimization routine, efficiently running on high-performance computing systems. Our method is tested on the design problem of field confinement using dispersive nanostructures. The obtained designs satisfy the fundamental curiosity of how free-form metallic and dielectric nanostructures perform when optimized in 3D, also in comparison to fabrication-constrained designs. Unconventional free-form designs revealed by computational methods, although may be challenging or unfeasible to realize with current technology, bring new insights into how light can more efficiently interact with nanostructures and provide new ideas for forward design.