When the COVID-19 pandemic forced higher education institutions to implement their programs in an online setting, different groups of students were influenced to different extents. In many cases, the main locus of learning moved to students’ homes, and their learning experiences were suddenly contextualized in their residential situation and immediate physical learning environment. The present study consequently examines the role of physical learning environments on different factors influencing students’ learning when pursuing their study from at home. It contrasts the situation of traditional students in a higher education institution and non-traditional students in an academic continuing education institution, which address target groups with different living conditions and needs in learning support. Data were collected via an online survey sent to students enrolled in these two institutions, with a total of 353 students participating during a timeframe impacted by COVID-related lockdowns. We found that stress and well-being is strongly linked to the quality of the surrounding environment of the learning place, whereas perceived motivation is more strongly related to the quality of the learning place itself. How strongly students are affected by these factors is moderated by their overall socio-spatial context. Academic continuing education students are more resilient to sub-optimal physical learning environment than traditional students. Altering the design of the immediate learning environment consequently can help to mitigate factors that negatively impact students’ well-being and learning motivation, which is particularly important for traditional students, who primarily dedicate their time to pursuing their studies.