The underlying reason is probably that hashicorp/terraform:light image is an alpine derivative and this variable assignment might not be supported by its default shell ash... Still I'm not 100% convinced having different token brings any extra security here.
After further investigation it turned out this is not limited by the image being used but more generally due to how variable expansion (carried out by Ruby itself and not by the shell) works in variable: constructs. Only "simple" expansion seems to be supported a.t.m. Even referencing elsewhere defined variables from gitlab-ci.yml itself is not yet supported: effort to improve this behaviour is tracked here. See its related issues for a more exhaustive list of the limitations of the current implementation.