3D Polarized Light Imaging (3D-PLI) is a neuroimaging technique that has opened up new avenues to study the complex architecture of nerve fibers in postmortem brains. The spatial orientations of the fibers are derived from birefringence measurements of unstained histological brain sections that are interpreted by a voxel-based analysis. This, however, implies that a single fiber orientation vector is obtained for each voxel and reflects the net effect of all comprised fibers. The mixture of various fiber orientations within an individual voxel is a priori not accessible by a standard 3D-PLI measurement. In order to better understand the effects of fiber mixture on the measured 3D-PLI signal and to improve the interpretation of real data, we have developed a simulation method referred to as SimPLI. By means of SimPLI, it is possible to reproduce the entire 3D-PLI analysis starting from synthetic fiber models in user-defined arrangements and ending with measurement-like tissue images. For the simulation, each synthetic fiber is considered as an optical retarder, i.e., multiple fibers within one voxel are described by multiple retarder elements. The investigation of different synthetic crossing fiber arrangements generated with SimPLI demonstrated that the derived fiber orientations are strongly influenced by the relative mixture of crossing fibers. In case of perpendicularly crossing fibers, for example, the derived fiber direction corresponds to the predominant fiber direction. The derived fiber inclination turned out to be not only influenced by myelin density but also systematically overestimated due to signal attenuation. Similar observations were made for synthetic models of optic chiasms of a human and a hooded seal which were opposed to experimental 3D-PLI data sets obtained from the chiasms of both species. Our study showed that SimPLI is a powerful method able to test hypotheses on the underlying fiber structure of brain tissue and, therefore, to improve the reliability of the extraction of nerve fiber orientations with 3D-PLI.