I think you can make it work. Your main concern will be storage space for all your gear. An empty 5 Gal beverage cooler sitting in the corner is a tough sell to the mr/missus/roommate(s). I use a 2 gal cooler that takes up 1/4 the space. At a ratio of 1qt of brewing liquor to 1lb of grain, you can mash a max of ~5.5lbs, and fill out the rest of your gravity with extract. You can get more than enough base malt going to convert any lower diastatic specialty grains or adjuncts. Most times, I'm mashing around 4lbs at 1.2 qts/lb, and adding in 2-4lbs of extract.
The rest of my rig consists of a 6 gal bottling bucket (doubles as my primary fermenter), a 5 gal better bottle for secondary fermentation, and a SOFC box chiller for keeping the temp constant. You can get away without the chiller in the winter, but you need some form of cooling/temp control in an apartment in the summer unless you want to run the AC all day.
My boil pot is a re-purposed aluminum tamale steamer with a ~4 gal capacity. This means I have to do partial boils for any batch larger than about 3.75 gal. To be honest, I don't think my little gas stove could hold a full 5 gallons at a rolling boil anyway. Sometimes, I will even collect a bit more wort into a 6qt stew pot, boil that down alongside the main wort and add at the end to bring my volume back up. After a little practice and trial/error, I'm able to hit my OG within about .005 most times.
For wort cooling, I bent an immersion chiller out of 20' of soft copper pipe and run ice water through it with the help of a small aquarium pump. Aside from the ice water, all my other liquid transfers are powered by gravity, which translates to a lot of lifting and setting down and ladling, etc.