Resonant soft x-ray scattering (RSoXS) leverages chemical specificity to characterize thin films but is limited near the nitrogen edge. The challenge is that commercially available x-ray transparent substrates are composed of Si3N4 and thereby absorb incident x-rays and generate incoherent fluorescence. To overcome this challenge, we designed and fabricated Al2O3 free-standing films for use as RSoXS windows. Al2O3 films offer higher x-ray transmittance and minimal fluorescence near the nitrogen edge. As an example, Al2O3 windows allow for nitrogen RSoXS of conjugated block copolymer thin films that reveal domain spacings, which are not apparent with commercially available Si3N4 substrates.